AI Tokens: See How They Are Derived From Scratch

General Discussion
1

Setting Up a Safe, Clean Sandbox

Note: Checked with moderator and received permission to post.

I’m starting a series of posts showing the creation of a local, character-level Transformer model to see how tokens are derived from scratch.

Full disclosure: This isn’t going to be my code! I do not know how to code in python. I asked Gemini to put together a hands-on learning project to break down and show how tokens work. To keep from having a posting that is too long, I will post the different steps that I went through as I did them.

The first step was about setting up the sandbox.

AI suggested this environment setup, and it sounded good to me—plus, Linux Mint needed it to keep things stable. It gave me an isolated environment to work in and keeps me from loading libraries into root that I might not need in the future. To avoid that, the setup does two things:

I set up a separate partition to keep the main drive clean.

It used a Python Virtual Environment (venv), which acts like a self-contained bubble.

The Setup Commands

First, I opened the terminal and navigated to the the directory:

cd /media/easyt/ALVM

Next, I started the virtual environment bubble (I called it ai_env) by entering:

python3 -m venv ai_env

Finally, I activated it. I knew I was in the bubble when the terminal prompt changed to show (ai_env) at the front:

source ai_env/bin/activate

The initial sandbox setup completed the first step for me. Next some code.

1 Like
2

Tokens - continued.

Next I needed a text file for the tokenization. I choose the ‘Fables of Aesop’ and places it in a regular text (.txt) file (fables.txt). The text file is about 65k in size. The first 10 lines of the file looks like this;

Title: The Fables of Aesop
Author: Aesop

Contents

The Cock and the Pearl
The Wolf and the Lamb
The Dog and the Shadow
The Lion’s Share
The Wolf and the Crane

This post is taking the first real code step toward turning that raw text into math: building the vocabulary.

Because AI is making a character-level model, our vocabulary isn’t a massive dictionary of full words. It is simply a list of every single unique character that appears anywhere in our text file—letters (uppercase and lowercase), punctuation, numbers, and spaces.

The Python Code was places in a file called build_vocab.py. Code generated by AI.

Python

C Open and read the raw text file
with open(‘fables.txt’, ‘r’, encoding=‘utf-8’) as f:
text = f.read()

C Get all unique characters and sort them
chars = sorted(list(set(text)))
vocab_size = len(chars)

C Print out what we found
print(“Unique characters found:”, “”.join(chars))
print(f"Vocabulary Size: {vocab_size}")

I ran the script. I also remove the ‘#’ form the comments in the code above to remove bold printing. Replace with ‘C’.

python3 build_vocab.py

The Python code scanned all the text in fables.txt, threw out all duplicates using set(), sorted what was left, and printed out the total count.

For my dataset, it found 65 unique characters (including spaces, periods, quotes, and newlines). The entire “vocabulary” for this model is only about 65 items.

Copy of terminal

easyt@E14:~$ cd /media/easyt/ALVM
easyt@E14:/media/easyt/ALVM$ python3 -m venv ai_env
easyt@E14:/media/easyt/ALVM$ source ai_env/bin/activate
(ai_env) easyt@E14:/media/easyt/ALVM$ python3 build_vocab.py
Unique characters found:
!,-.:;?ABCDEFGHIJKLMNOPQRSTUVWY_abcdefghijklmnopqrstuvwxyzÆ—’“”
Vocabulary Size: 65

Note, no cap Z was found.