Tokenization: What the hell is a token?

In my first post in this series, I talked about collecting data from the internet and other sources, cleaning it, and preparing it for the next stage of AI training.  Now we move on to Tokenization.

After a massive amount of information has been collected and intended for use as training material for AI, it can't simply be fed in and read or processed in what you may consider a normal way.  It needs to be broken down into chunks or tokens.

What is a token?

A token is a small unit of text that an AI model can work with.  Importantly, a token is not necessarily a word.  Let's take a simple sentence.

The cat sat on the mat.

Yup, that's simple, no idea where that came from, but that's what my head came up with first.

Here is a very important first point about AI models: we understand sentences; AI does not read and understand them the way we do.  When it comes to training AI Models, sentences are not processed as sentences; instead, text is broken down into tokens.  You could imagine the tokens this way:

The | cat | sat | on | the | mat

So, six tokens, not a sentence.  But it gets more interesting: tokens are not necessarily words either.  So something like:

Unbelievable

could be broken down into tokens that look like this.

Un | believ | able

I am talking conceptually here; how a word is split into tokens varies.

Ok, so why the need for these tokens?

This is where it gets interesting.  Computers don't understand the word "cat" just because the letters c-a-t inherently mean that small, furry animal.

Tokenization gives the end model a finite vocabulary.  Pieces of text it knows how to represent.

Each token can be assigned an ID, essentially a number, to identify it.

Again, as a concept, our token and ID may look like this:

The = 123
cat = 456
sat = 789

The numbers do not mean anything, though.  For example, 456 does not mean cat.  It's just the ID used to identify that particular token.

Why not use words?

A fair question, right?  Why not just use words? That would surely make more sense.  Yes, for people, not computers.

The core issue here is that language is enormous.

If you think about the whole English language, then every other language.  Now add people's names, place names, technical terms, slang, different spellings, and all the other odd terms ever written on the internet from which all the training data comes, in the main.  Consider very old books from a time long ago when language was also different.

Now consider a word that isn't in the tokenizer's vocabulary. If the vocabulary consisted only of complete words, what would it do with a word it didn't know?

Breaking text into smaller and more reusable pieces helps solve that.  Go back to the last word I used, unbelievable.

We broke that down into

un | believ | able

Each of those three tokens could be useful in other words.  Instead of needing a unique entry for every word, the tokenizer can construct text from smaller pieces it already knows.

Ok, why not just use individual letters?

Right?  Let's go the other way, instead of:

un | believ | able

We could just use:

u | n | b | e | l | i | e | v | a | b | l | e

Ok, that could work, but no, we have now turned a single word into 12 tokens.  Let's say an entire book is being scanned or a website with a lot of data; now scale that up.  The number of tokens is now massive.

What I am really trying to say here, if you're following me so far, is that tokenization is essentially a compromise.  Common words or text can be represented much more efficiently; less common words can still be constructed from the smaller pieces.

Who decides what becomes a token?

It's not a person sitting there going through them one at a time.  Nobody in OpenAI or Anthropic is sitting going through a list or spreadsheet deciding what the tokens should be.  That would be unrealistic, given how much data is involved.

The Tokenizer

A tokenizer is a piece of software. Before it's fed the massive volume of data AI extracts from the internet and other sources, it needs training first.  Now, pause there.  No model is being built; this is not AI as such, it's just the next stage in the process of building a model.

Before the tokenizer can be used, it needs a vocabulary. To build that vocabulary, a tokenization algorithm is run over a large body of text to identify useful patterns and frequently occurring sequences.  The algorithm analyzes which characters and sequences occur frequently and uses that information to create an initial vocabulary of tokens.

With that starting position, the large amount of cleaned data we discussed in the previous article is now fed into the tokenizer, which converts the new text into tokens and IDs.

Simply, the tokenizer process is:

1. Text goes in
2. Tokens and their IDs come back out

All numbers

Ok, so OpenAI went to the internet, collected a massive volume of data, cleaned it, and fed it through a tokenizer. All that data we, as humans, would instantly recognize has now been converted into a massive sequence of tokens and numerical IDs.

We started with:

The cat sat on the mat

We now have (for illustration):

148 175 203 917 521 024

Right, but now we have another issue.

When I look at the text and tokens above, I know that 175 corresponds to the token "cat". But 175 itself doesn't mean cat. It's just an arbitrary number identifying that token.

There is nothing about the number 175 that tells the model that cats are animals, that they often have four legs and fur, or that words such as kitten, pet, dog, and mouse might be related to it.

So how do we get from a meaningless token ID to a model that can learn all of those relationships?

This is where we move beyond tokenization and into the actual training of the model. It needs to start learning patterns and relationships from the tokens it has been given.

And this is where things get considerably more interesting. And this is what we will look at in the next article. Model training.