Smol LLM Lab · Lesson 1 — Start with a baseline, not a transformer

🧪 Build-in-public: we're building small, production-grade text models — under 1M parameters — that train and serve on a plain 4-core VPS with no GPU. Every lesson documents a real run: the commands, the numbers, and the surprises.

Where this is going (the real goal)

The end product is not a generic "bad words" filter. It's a data-quality guard for B2B contact records — the kind of thing ZoomInfo or Dun & Bradstreet needs: when someone signs a form as "Mickey Mouse, Chief Butthead, asdf@qwerty.com", the pipeline should catch that the name is fake, the job title is profane or garbage, and score the record's trustworthiness.

That problem needs the exact skills we build here:

1. a model that recognizes profanity and abuse even when obfuscated (this lesson + lesson 2),
2. a model that knows what real names and real job titles look like (the companion title-guard project, which validates titles against golden data like the O*NET taxonomy and SEC filings).

We start with general toxicity data because it's large, labeled, and free — the perfect gym. Then we transfer the technique to short strings like "VP of F***ing Nothing".

What "small" buys us

For contact-data cleaning you might score 50 million records in a batch job. At 1 ms per record, that's ~14 CPU-hours — fine. At 100 ms (a big transformer), it's 58 days. Small is a feature.

Step 0 — What we're learning on

wiki_toxic: 159,549 comments from Wikipedia talk pages, hand-labeled by the Jigsaw project as toxic (insults, threats, profanity, harassment) or not.

Two things to notice before training — this habit will save you weeks over your career:

• Class imbalance. Only ~10% of comments are toxic. A model that answers "not toxic" every single time scores 90% accuracy while catching zero toxic comments. So accuracy is meaningless here; we track precision (of the things we flagged, how many were really toxic?), recall (of all toxic things, how many did we catch?), and their harmonic mean F1 — computed on the toxic class only.
• Three splits, three jobs. We train on train, tune choices on validation, and touch test only at the end, once, as an honest exam.

Step 1 — Turn text into numbers (TF-IDF, demystified)

Machine-learning models eat numbers, not words. The 60-year-old trick that still works:

• Bag of words: count how often each word appears in a comment. "you are an idiot" → {you:1, are:1, an:1, idiot:1}.
• TF-IDF ("term frequency × inverse document frequency"): the same counts, but words that appear in every comment ("you", "the") get scaled down, and rare, distinctive words ("idiot") get scaled up. Each comment becomes a long, mostly-zero vector of scores.
• n-grams: instead of single words, also count pairs ("kill yourself" carries more signal than "kill" and "yourself" separately).

And the trick that makes this actually production-worthy — character n-grams. Slide a 3-to-5-letter window over the raw characters: "sh1t" → sh1, h1t, sh1t… A word-level filter is blind to sh1t, f__k, st00pid; a char-level model sees that sh1 and h1t look suspiciously like their uncensored cousins, because it learned that from data. Nobody wrote a rule for it.

Step 2 — The simplest model that could work

Logistic regression: one weight per feature, multiply-and-add, squash to a 0–1 probability. It is literally a single neuron. Why start here?

• It trains in seconds on a CPU.
• It's interpretable: you can print the weights and read what it learned.
• It sets the number to beat. If a fancy model can't beat it, the fancy model loses. This discipline is Rule #1 of applied ML.

The entire model, for real:

word_vec = TfidfVectorizer(ngram_range=(1, 2), max_features=100_000,
                           sublinear_tf=True, strip_accents="unicode")
char_vec = TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5),
                           max_features=150_000, sublinear_tf=True)
X = hstack([word_vec.fit_transform(texts), char_vec.fit_transform(texts)])
clf = LogisticRegression(C=4.0, max_iter=2000, class_weight="balanced")
clf.fit(X, labels)

Notes on the three knobs that matter:

• max_features caps the vocabulary (100k word + 150k char features = 250k weights — a quarter of our 1M budget).
• class_weight="balanced" tells the model that missing a toxic comment costs more than a false alarm, compensating the 10:1 imbalance.
• C=4.0 is regularization strength (higher = trust the training data more). Tuned on the validation split, never on test.

Step 3 — The numbers

Training took 25 seconds. Featurizing the text took 2 minutes. On our 4-core box.

Read the validation column first: we catch 86% of toxic comments, and 77% of our flags are correct. For 25 seconds of training, that's genuinely strong.

Step 4 — The plot twist (this is the real lesson)

Test F1 cratered to 0.63. A beginner's instinct is "the model is broken." Debug like an engineer instead:

1. Suspect #1 — different class balance? Checked: all three splits are ~10% toxic. Not it.
2. Look at the shape of the failure. Recall stayed high (0.92) but precision collapsed (0.48). The model finds toxic comments fine — but half its flags are "wrong" according to the test labels.
3. Question the labels. The Jigsaw test set is documented to come from a different annotation batch with different labeling standards. Many comments our model flags were probably let through by the second batch of human labelers. The disagreement between annotators is baked into the benchmark.

Takeaway you'll reuse forever: validation is a promise the test set doesn't have to keep. In production there's always a second distribution — new users, new slang, a different labeling vendor. When a metric drops, decompose it (precision vs recall), check base rates, then question the labels before questioning the model.

Step 5 — Read the model's mind

Because it's linear, we can print the highest-weighted features: ass, uck, fuck, shit, crap, bitch, idiot, stupid, cunt…

Look at uck and fuc — those are character n-grams, and they're exactly the anti-obfuscation machinery: any creative respelling that keeps those letter patterns still fires. That's the property we'll want when someone types a "clever" profane job title into a form.

Step 6 — Production reality check

Already inside every budget we set. This is the bar lesson 2's transformer has to clear: beat 0.81 validation F1 under 1M parameters — and if it can't, the boring baseline wins. That would also be a lesson.

Replicate it yourself (10 minutes)

mkdir smol-lab && cd smol-lab
python3 -m venv .venv
.venv/bin/pip install scikit-learn pandas scipy joblib

mkdir data && for f in train validation test; do
  curl -L "https://huggingface.co/datasets/OxAISH-AL-LLM/wiki_toxic/resolve/main/$f.csv" -o data/$f.csv
done

# grab the training script from the lab repo (lab/01_baseline/train_baseline.py)
.venv/bin/python lab/01_baseline/train_baseline.py

The script prints every number in this lesson and saves baseline.joblib (the deployable model) plus metrics.json. Change one thing, rerun, watch the metrics move — try ngram_range=(1,1) with no char features and watch obfuscated profanity slip through. That's the whole game of applied ML: change, measure, understand.

Next up

Lesson 2: a transformer built from scratch in readable PyTorch — byte-pair tokenizer, embeddings, attention blocks, ~700k parameters — trained on this same box, judged on this same table. And in parallel, title-guard: validating job titles against real golden data (the O*NET occupation taxonomy and officer titles pulled from SEC filings). Same lab, real B2B problem.

📌 All code: smol-lab repo on the VPS — lab/01_baseline/train_baseline.py.