The Bake-Off: Do You Need a Neural Net to Catch 'Chief Happiness Wizard'?

Title-guard C2.5 — 2026-07-16. Every number in this post comes from a measured run on the same 4-core/8GB VPS that runs everything else in this series (`bench_semantic.py`, results in `data/eval/semantic_bakeoff.json`).

The itch we needed to scratch

Title-guard's lexical pipeline (C2.7) validates job titles at 909/s on this box with 97.7% accuracy on the frozen eval. But it has a documented tail it can't reach, and we've been honest about it since C2:

1. Joke titles built from real words. "Chief Happiness Wizard", "Senior Vibes Coordinator", "Dragon Wrangler Executive" — every word is real, the grammar is fine, only the meaning is absurd. Lexical matching caught ~56% of these.
2. Wrong-occupation mapping. The char+word TF-IDF nearest-neighbor maps "Head of Growth Marketing" to Soil and Plant Scientists (character overlap with "growth"), and "Pharmacovigilance Specialist" to Landscaping and Groundskeeping Workers. The titles pass validation — but the canonical occupation is nonsense, and that mapping is half the product.

The obvious answer is "use AI embeddings." But C2.7 taught us a rule we keep re-learning: lists decay, statistics hold — and also that plausible-sounding signals fail in embarrassing ways when you actually measure them (the word-pair novelty signal looked great until it flagged a third of legitimate international titles). So instead of assuming, we ran a bake-off.

The contenders

All three are built from the same corpus — 45,302 unique real titles (O*NET alternate + reported titles + 600 SEC officer titles), nothing derived from the eval sets:

Plus the C2 baseline (the existing lexical score) for context.

The decision rule, written down before running: if a counting method catches the same tail as embeddings, the system stays neural-net-free. Protocol per C2.7: thresholds tuned on the frozen set only (at 80% joke recall); the holdout is scored exactly once with those frozen thresholds, no fixes afterward.

Results

Frozen set (417 titles, 9 categories) — AUC separating legit titles from garbage_joke, plus false-positive rates at the 80%-joke-recall threshold:

Holdout (423 titles, scored once, frozen thresholds):

\PMI's "98% recall" is it flagging literally everything — see the autopsy.*

Three headlines:

1. The counting method won. A word-bigram LM — 0.48s to build, no dependencies, 74,000 titles/s — beats an 8M-parameter embedding model on the joke tail, on both frozen and holdout. At the frozen threshold it catches 90% of holdout jokes at 6% legit false-positive rate; the C2 baseline managed 82% at 20%.
2. Both new scorers held up on the holdout (AUC dropped only ~0.015–0.02). This is the "statistics hold" pattern again: neither scorer memorizes joke titles; they measure properties of title-ness. Compare C2.7, where the fictional-titles list decayed 98%→12%.
3. PMI didn't just lose — it inverted. AUC 0.185 means jokes scored higher than legit titles. This deserves an autopsy, because the mechanism is a general lesson.

Autopsy: why PMI inverts (and why "just count word pairs" keeps failing)

Real examples from the run, with (pair count, word counts):

Two compounding failures:

• PMI penalizes popular words. "Sales" appears 373 times and "manager" 1,293 times, so even 16 real co-occurrences can't beat the expected-by-chance denominator. The most common legitimate pairings score worst.
• Smoothing rewards unknown words. An out-of-vocabulary word gets a tiny smoothed unigram probability, so any pair containing it has an enormous PMI ratio — the formula reads "these two words co-occur infinitely more than chance" when the truth is "we know nothing about them."

Fix the second problem by treating OOV pairs as maximally suspicious and you've reinvented the word-pair novelty signal that C2.7 already measured and rejected for flagging ⅓ of legitimate international/executive titles. Composition-counting on a 45k corpus is trapped either way: score novelty → false-flag rare-but-real; smooth it → reward gibberish. The C2.7 rejection now has a mechanism, confirmed from a second direction.

The word LM escapes the trap because interpolated backoff makes it measure familiarity, not association: an OOV word like "vibes" gets a low unigram probability directly, and frequent words raise (never lower) sequence probability. Familiarity is the right question for validity.

What embeddings are actually for: the mapping tail

Embeddings lost the joke-detection contest but decisively fixed the other tail — the wrong-occupation mappings, straight from the run:

Two clean fixes; "Managing Director" improves from absurd to merely imperfect (it should reach general/chief executives — a c-suite prior on the reference index would finish the job). So the bake-off's answer isn't "embeddings lose" — it's each tool has exactly one job: the word LM guards validity, embeddings repair the canonical mapping.

Speed and cost, measured

The semantic layer is free relative to the existing pipeline: the word LM runs 250× faster than the current bottleneck, and even brute-force embedding of all 10M titles costs half an hour on one core. Peak RSS for the whole bench — all four scorers plus the C2 TF-IDF index — was 3.9GB, inside this box's 8GB budget; the three new scorers alone need well under 100MB (the embedding reference matrix is 45,302 × 256 float32 ≈ 44MB).

For the "do I need OpenAI/Claude/Gemini?" question the numbers are now concrete: a local 30MB static model scored 20,000 titles in 4.2 seconds on one core of a budget VPS. API embeddings would cost pennies (~4M tokens ≈ $0.08 for a deduped 10M-record corpus) but add rate limits and a dependency for quality this task doesn't need. LLM-as-judge stays where the cascade design put it: the final 1–2% uncertain band only (~$50–150 per 10M-record run with a Haiku-class model), never bulk.

Pros and cons

Word-bigram LM — pros: best joke-tail accuracy, survives the holdout, 74k/s, half-second build, zero dependencies, fully interpretable (you can print the improbable transition that fired). Cons: no semantic understanding — it validates but cannot map; a fluent-sounding fake ("Regional Development Manager" for a company with no such role) sails through, because that's a truth problem, not a language problem — that's what the C3 evidence layer is for.

Static embeddings — pros: fixes the mapping failures the product has carried since C2; 30MB; fast enough to run on everything; also a competent joke detector (holdout AUC 0.955). Cons: lost to a bigram counter on the headline task; adds a model artifact + HF fetch to the build; 256-dim vectors explain themselves less well than a bigram probability.

PMI composition — pros: the autopsy. Cons: everything else. Rejected, documented, and the rejection generalizes: this is the second independent failure of word-pair statistics on this corpus size.

The C2 baseline stays: it's still the best all-garbage detector (frozen AUC 0.987) because profanity/gibberish/placeholder flags are its home turf. Nothing here replaces it; the new layers stack on top.

What production looks like now

raw title
  → L1  C2 lexical (291/s/core): profanity, gibberish, placeholder, seniority
  → L1½ word-LM gate (74k/s): joke/fluency score            ← NEW, ~free
  → L2  embedding mapping (4.7k/s): canonical occupation     ← NEW, replaces
        when lexical similarity is low or occupations disagree   worst mappings
  → L3  evidence layer (C3 design): ATS / registries / filings — truth, not language

Limitations, honestly

• The holdout has now been seen (scored once, as the protocol allows). Per the flywheel rule, the next iteration — integrating the word-LM gate into the C2 score and re-tuning the blend — must generate a fresh holdout before claiming victory.
• Joke titles are grok-generated; real-world fake titles skew different (inflation and stale seats more than wizards). The C3 evidence layer, not this one, addresses those.
• "Managing Director" shows the embedding reference index needs a c-suite/occupation-level prior; corpus-title cosine alone slightly favors surface-similar strings.
• Throughputs are single-process on a shared VPS; multiply by cores with independent processes (never multiprocessing.Pool with this stack — see C2.6 notes).

The takeaway

We asked whether the joke-title tail needs a neural net. Measured answer: no for detection, yes (a tiny one) for mapping. A 0.48-second bigram model beat an 8M-parameter embedding model at catching absurd titles — while the embedding model quietly fixed the product's documented class of mapping bugs for 30MB and three seconds of build time. The general lesson is the same one this series keeps paying for and cashing in: don't argue about approaches, bench them — the plausible-sounding one (PMI) finished dead last with an inverted score, and the boring one won.

Next: fold the word-LM gate into the production score, generate a fresh holdout, then start C3.1 (the offline evidence layer — H-1B bulk index + industry×occupation matrix) per the [C3 design](https://vault.binary.ovh/title-guard-c3-design).