Encyclopedia > Tokenize

  Article Content

Tokenize

Tokenizing is the operation of replacing one set of symbols with another, typically to make the resulting set of symbols smaller.

The term is most commonly used in computers, where a programming language source code, a set of symbols in an english-like format, is converted into another format that is much smaller. Most BASIC interpreters used this to save room, a command such as print would be replaced by a single number which uses much less room in memory. In fact most lossless compression systems use a form of tokenizing, although it's typically not referred to as such.



All Wikipedia text is available under the terms of the GNU Free Documentation License

 
  Search Encyclopedia

Search over one million articles, find something about almost anything!
 
 
  
  Featured Article
East Farmingdale, New York

... average family size is 3.52. In the town the population is spread out with 26.0% under the age of 18, 9.4% from 18 to 24, 34.6% from 25 to 44, 19.4% from 45 to 64, and ...

 
 
 
This page was created in 28.4 ms