The next step was to show how many unique charterers there were in the fables.txt and build a vocabulary.
python s2-find-uq.py
Total characters in book: 63497
Unique characters (vocabulary size): 65
First 10 characters in vocabulary: ['\n', ' ', '!', ',', '-', '.', ':', ';', '?', 'A']
Computers can not process letters; they process numbers. To give our neural network something it can calculate, every single character in the vocabulary gets assigned its own number (unique integer ID).
Next I needed to apply an encoder to the entire dataset and turning it into a PyTorch Tensor.
A PyTorch Tensor is basically a multi-dimensional array designed specifically for high-speed mathematical operations on neural networks.
Installing PyTorch
Before I could run the script, I needed to make sure PyTorch was installed inside my virtual environment (ai_env). In my terminal, I ran:
pip install torch
The AI then created a new python script called s4-train1.py inside my project folder to tokenize the entire text file and store it as a Tensor:
The Python Code
import torch
# Step 1: Open and read your fables book
with open("fables.txt", "r", encoding="utf-8") as f:
text = f.read()
# Step 2: Find all unique characters in the book
chars = sorted(list(set(text)))
vocab_size = len(chars)
# Step 3: Create mappings (character to number, and number to character)
char_to_int = { ch:i for i,ch in enumerate(chars) }
int_to_char = { i:ch for i,ch in enumerate(chars) }
# Step 4: Print our results to see what the AI's "alphabet" looks like
print(f"Total characters in book: {len(text)}")
print(f"Unique characters (vocabulary size): {vocab_size}")
print(f"First 10 characters in vocabulary: {chars[:10]}")
# Step 5: Encode the entire book into numbers
data = torch.tensor([char_to_int[c] for c in text], dtype=torch.long)
# Step 6: Let's see what the first 100 characters look like as numbers
print("\nFirst 100 characters as text:")
print(text[:100])
print("\nFirst 100 characters as numbers:")
print(data[:100])
# Step 7: Split data into training and validation sets
n = int(0.9 * len(data)) # 90% of the total characters
train_data = data[:n]
val_data = data[n:]
print(f"\nTraining set size: {len(train_data)} characters")
print(f"Validation set size: {len(val_data)} characters")
Inside my activated ai_env terminal, I ran the python script:
python s4-train1.py
What Happened
Python took the entire fables file, translated every single character into its integer ID, and stored it inside a single 1D Tensor array.
When printed, the terminal showed something like:
(output of all the steps)
python s4-train1.py
Total characters in book: 63497
Unique characters (vocabulary size): 65
First 10 characters in vocabulary: ['\n', ' ', '!', ',', '-', '.', ':', ';', '?', 'A']
First 100 characters as text:
Title: The Fables of Aesop
Author: Aesop
Contents
The Cock and the Pearl
The Wolf and the Lamb
First 100 characters as numbers:
tensor([28, 42, 53, 45, 38, 6, 1, 28, 41, 38, 1, 14, 34, 35, 45, 38, 52, 1,
48, 39, 1, 9, 38, 52, 48, 49, 0, 9, 54, 53, 41, 48, 51, 6, 1, 9, 38, 52,
48, 49, 0, 0, 11, 48, 47, 53, 38, 47, 53, 52, 0, 0, 1, 28, 41, 38, 1, 11,
48, 36, 44, 1, 34, 47, 37, 1, 53, 41, 38, 1, 24, 38, 34, 51, 45, 0, 1, 28,
41, 38, 1, 31, 48, 45, 39, 1, 34, 47, 37, 1, 53, 41, 38, 1, 20, 34, 46, 35,
0, 1])
Training set size: 57147 characters
Validation set size: 6350 characters
The entire text file is now officially converted into a clean stream of numbers that PyTorch can feed straight into an AI model! All 57,147 numbers.
Next posting will show training chunks and set up the train/validation.