# Understanding Tokenization

## Introduction

When you talk to AI like ChatGPT, it feels like it understands full sentences.

It feels like it reads words the same way humans do.

But that’s not true.

AI does **not** read words.  
AI does **not** read sentences.

AI reads **tokens**.

To understand how AI works, tokenization is one of the most important concepts to learn.

## Computers Don’t Understand Language

Humans understand language naturally.

When you read:

> “My name is Jay.”

You understand the meaning instantly.

A computer cannot do this.

Computers only understand **numbers**.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1768150570670/d4a5129f-5d2e-46ee-898f-b26d320590b5.png align="center")

So before AI can work with language, text must be converted into a form computers can process.

That process is called **tokenization**.

## What Is Tokenization?

**Tokenization** means breaking text into small parts called **tokens**.

AI models like GPT do not predict letters or full sentences.

They predict **tokens**, one at a time.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1768150608594/c24ed538-83a0-42a2-b037-63fb360d4d3e.png align="center")

A token can be:

* a single letter
    
* a full word
    
* or part of a word
    

There is no fixed rule.

How text is split depends on the **tokenizer** used by the model.

## What Is a Token?

A **token** is the smallest unit of text that AI understands.

A token can be:

* `"My"`
    
* `"name"`
    
* `"is"`
    
* `"Jay"`
    

Or it can be smaller:

* `"M"`, `"y"`
    
* `"n"`, `"a"`, `"m"`, `"e"`
    

Even spaces and punctuation can become tokens.

## Same Sentence, Different Tokens

Let’s take a simple sentence:

```plaintext
My name is Jay.
```

Different models can break this sentence differently.

### Model A (Word-based tokenization)

```plaintext
My
name
is
Jay
```

### Model B (Character / sub-word tokenization)

```plaintext
M
y
(space)
n
a
m
e
(space)
i
s
(space)
J
a
y
```

Both models see the **same sentence**.

But the tokens are different.

That is why:

* Token count changes
    
* Cost changes
    
* Behavior changes slightly
    

## Why Tokenization Is Important

AI does not think in words.

It thinks in **tokens**.

When you type a prompt, the model predicts the **next token**, not the next sentence.

It keeps doing this again and again until a full response is created.

That is how AI “writes”.

## How GPT Predicts Text

Let’s say you type:

```plaintext
Once upon a
```

GPT might predict:

```plaintext
time
```

Then it predicts:

```plaintext
there
```

Then:

```plaintext
was
```

One token at a time.

This continues until the response is complete.

That’s it.

No thinking.  
No understanding.  
Just prediction.

## Why Small Prompt Changes Matter

Because AI works with tokens, even small changes affect the output.

**For example:**

```plaintext
Explain JavaScript.
```

**vs**

```plaintext
Explain JavaScript simply.
```

These two prompts produce different tokens.

Different tokens → different predictions → different output.

That’s why prompt wording matters so much.

## What are Token Limits?

Every AI model has a **token limit**.

This limit includes:

* Your input
    
* Previous messages
    
* The model’s reply
    

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1768150647490/ae70c316-cb60-4dd0-85e7-c626a9236774.png align="center")

If the total tokens exceed the limit, older messages are removed.

That’s why AI sometimes “forgets” things.

It did not forget.

The tokens just did not fit.

## Tokenization Depends on the Model

Each AI model uses its own tokenizer.

**For example:**

* GPT has its own tokenizer
    
* Gemini has a different one
    
* Claude uses another
    

So the same text can produce **different token counts** in different models.

There is no universal token system.

## Tokenization Happens First

Before AI can do anything useful, tokenization happens.

The basic flow looks like this:

```plaintext
Text
→ Tokens
→ Numbers
→ AI Model
→ Predicted Tokens
→ Text Output
```

Everything starts with tokens.

## Why Freshers Should Learn Tokenization

Understanding tokenization helps you understand:

* **Why prompt engineering works**
    
* **Why AI pricing is token-based**
    
* **Why context is limited**
    
* **Why AI answers change with wording**
    

Once you understand tokens, AI stops feeling **magical**.

It starts feeling **logical**.

## Conclusion

Tokenization is the first step in how AI understands language.

Before a model can answer, explain, or generate anything, it must break text into tokens.  
Those tokens decide **what the model sees**, **how much it remembers**, and **what it predicts next**.

Once you understand tokenization, many AI behaviors start making sense:  
why prompts matter, why limits exist, and why wording changes results.

AI may feel intelligent, but at its core, it’s simply working with tokens and probabilities.

And that’s exactly where everything begins.

### Want More…?

I write articles on [blog.devwithjay.com](https://blog.devwithjay.com) and also post development-related content on the following platforms:

* [**Twitter/X**](https://x.com/devwithjay)
    
* [**LinkedIn**](https://www.linkedin.com/in/devwithjay)