Catching "Chief Butthead": how we built and stress-tested a fake-contact detector on a $5 server

Smol LLM Lab, July 2026. Every number in this post comes from a measured run on one 4-core, 8GB VPS — no GPU, no paid APIs. Code: `~/claude_/title-guard`; raw results: `bench_results.json`, `docs/extended_eval_results.json`.*

Part 1 — The problem, in plain English

Every company that collects sign-ups has met these people:

💡 Mickey Mouse, Chief Magic Officer Anita Bath, Supreme Overlord of Snacks asdf qwerty, test

B2B data vendors like ZoomInfo and Dun & Bradstreet make real money cleaning this up: deciding which contact records are real, and translating the real job titles into something a machine can use ("EVP & CFO" → Chief Executives, c-suite). We built a miniature open version of that pipeline — TitleGuard for job titles, NameGuard for person names — and then did what this post is really about: attacked it with machine-generated test data until it broke, fixed it, and measured honestly what generalized and what didn't.

The punchline for the impatient:

• Real contacts pass 99.9% of the time (97%+ even on deliberately exotic synthetic names and titles); classic garbage is caught at 86–100%, freshly-invented adversarial garbage at ~60% — and that gap is the story.
• The whole thing scores 909 titles/second on a $5/month server — 10 million records in ~3 hours for roughly $0.
• Word lists win battles; statistics win wars. Our lookup-table fixes went from 98% → 12% effectiveness on fresh adversarial data. Our statistical detectors went from 100% → 100%.

Part 2 — How it works

Both guards follow the same philosophy: cheap, transparent, explainable. No neural network, no black box. Every rejection comes with named flags you can audit.

TitleGuard takes a raw title through six steps:

1. Clean — fix HTML mess (EVP & CFO), fold accents (Ásgeir → asgeir), strip noise.
2. Expand — 50+ abbreviations: CFO → chief financial officer, Sr. → senior, mgmt → management.
3. Seniority — a regex ladder assigns c-suite / vp / director / manager / senior / entry.
4. Match — find the nearest real job title among 63,934 variants of 1,016 occupations from O*NET, the US Department of Labor's taxonomy (details in Part 3).
5. Flags — profanity, gibberish, placeholder ("n/a", "TBD"), joke buzzwords, words that never appear in real job titles.
6. Score — similarity, discounted by flags, into one 0–1 confidence.

NameGuard checks a person name against 162,253 US Census surnames and 104,819 SSA first names (372 million births, 1880–2024). A name found in the data is real; frequency shades the score. An unseen name isn't rejected — "Lakshmi Venkatasubramanian" isn't in the SSA data either — instead a character-level language model asks: does this look like human language, or like a cat on a keyboard?

Why you need both checks

The best demo in the project: "Mickey Mouse, Chief Magic Officer". The title check passes it — "Chief Magic Officer" scores 0.82; it's a perfectly plausible title shape. The name check kills it at 0.05 (fake identity). Either filter alone leaks; the product of both is the verdict.

Part 3 — The math, for those who want it

TF-IDF nearest neighbor (the title matcher)

Every known title variant becomes a sparse vector of character 2–4-grams and word 1–2-grams, weighted by TF-IDF:

weight(term, doc) = (1 + log tf) × log(N / df)

Rare, discriminative fragments ("perfusion", "actuar") get high weight; ubiquitous ones ("manager") get low weight. A new title is vectorized the same way and matched by cosine similarity — the angle between vectors:

sim(a, b) = (a · b) / (|a| |b|)

Character n-grams make it typo-tolerant ("Sofware Enginer" still lands on Software Engineer); word n-grams keep phrases intact. One trick matters more than everything else: ONET organizes jobs by function*, not corporate rank, so "VP of Engineering" also gets queried as just "engineering" after stripping rank words, and the better match wins.

Character-bigram language models (the gibberish detector)

Train nothing fancy: count every adjacent character pair in the 63,934 known titles (or 267k known names). "ma", "er", "an" are common; "qx", "zk" essentially never occur. A candidate string is scored by its mean negative log-probability:

gibberish(s) = −(1/n) Σ log P(cᵢ₊₁ | window)

Real language sits around 4–6; "aslkdjhqwlekjh" scores far above the 7.0 threshold. This is a 1950s-era idea (Shannon) that remains devastatingly effective — and, as you'll see, it generalizes, unlike everything list-based.

Score composition

A name's score is the geometric mean of its parts' scores — both parts must be plausible; one great surname can't rescue a keyboard-mash first name (arithmetic mean would let it). Each flag multiplies the score by 0.7; hard evidence (profanity, known fake identity) caps it at 0.05. The pass threshold is 0.4, and a sweep from 0.10 to 0.70 confirmed 0.4 maximizes F1 — the sensible default was also the measured optimum.

Part 4 — How we tested it

Round 1: golden data

577 real executive titles from SEC insider filings (legally attested, with authentic mess like EVP & CFO): 100% pass. Hand-picked garbage: caught. Looked great. Hand-picked evals always look great — they test what you thought of.

Round 2: an adversary generates the test set

We had a second LLM (Grok, via a local bridge) generate 16 categories of labeled test data — 417 titles and 290 names: industry-specific titles, UK/Indian business English, messy web-form typing, plus profane, joke, gibberish, and placeholder garbage; diverse real names (Icelandic, Thai, hyphenated, apostrophes) plus fictional characters, pun names, and troll profanity. The set was frozen and committed before any fixes — it's a test set, not a moving target.

Two lessons before a single row was scored:

• LLMs degenerate on long lists. Asked for 50 international titles, Grok produced 1,273 lines, ending in an infinite loop of "Liquidator under the Crown Lands Act 1927/1936/1949…". Cap and validate everything.
• LLM labels need human review. Grok labeled "John Smith" a fake placeholder. It is the most common real name in America. A validator that rejects every John Smith is broken by definition; we relabeled it (documented in the generator) — at the name level, a fake John Smith is only catchable by cross-field signals like a mismatched email.

Round 2 results: the ambush worked

The failures had texture. "Chief Moron Officer" sailed through at 0.84 — "moron" wasn't in the profanity list, and TF-IDF cheerfully matched Police Officers. "VP of Shitposting" passed because the filter did exact word lookup and "shitposting" ≠ "shit". Eric Cartman scored 0.84 — "Eric" and "Cartman" are both real name tokens. And "n/a" survived because the cleaning step itself rewrote the slash into "n and a" before the placeholder regex ever ran — the bug was in the plumbing, not the logic.

Round 3: fix, and measure the fix

Five fixes, each aimed at a failure class, not a failure case: profanity root matching ("fuck·ery", "shit·posting" — morphology beats enumeration), placeholder checks on the raw string, a 396-entry fake-identity catalogue, unicode folding (Ásgeir Jónsson went from 0.10 to passing), and an out-of-vocabulary flag — joke titles use words ("vibes", "napper", "meme") that appear zero times in 63,934 real titles.

We also rejected a tempting signal, and this is the part worth stealing: word-pair novelty ("cat video" never occurs in real titles) catches 90% of jokes — but falsely flags a third of legitimate international and executive titles ("Chartered Accountant", "Chief Strategy Officer" are also novel pairs to ONET). Our headline property is that real contacts almost never get rejected* (1 in 994). No garbage-recall gain was worth breaking that.

Frozen set, after fixes: titles 94.4% → 97.7% (garbage F1 0.776 → 0.919), names 75.2% → 100%. SEC golden went to 577/577.

Round 4: the honest part — a fresh holdout

Here's the trap in "test → fix → re-test": the re-test flatters you, because the fixes were tuned on those exact failures. So after freezing the fixes we had Grok generate a brand-new 706-row holdout — same categories, explicitly instructed to avoid the classic examples — and evaluated it exactly once, with no fixes allowed afterward.

That last line is the finding of the project. Our 396-name catalogue caught 98% of famous fakes on the frozen set and 12% of less-famous ones on the holdout — the world contains infinitely many minor cartoon characters, and a list can only ever memorize the past. Meanwhile the bigram model and the frequency dictionaries — pure statistics — didn't drop a single point. (The holdout's "profane" titles also mutated: Grok, told to be fresher, produced insults without profanity — "Executive Numpty of Product", "Chief Bonehead Operations". The adversary moves.)

What the baselines prove

Against three cheap alternatives on the same data, the ensemble's value is the combination:

Part 5 — Speed, cost, and what we'd do next

The hardening wasn't free, and we measured the bill: single-core throughput went from 315 to 273–291 titles/s (~10%), the whole 4-core box from 1,030 to 909 titles/s, and the 10-million-record projection from 2.7 to 3.1 hours — still 8× inside a 24-hour requirement, at ~$0 marginal cost on hardware costing $5/month. NameGuard runs 36,300 names/s on one core. (For scale: doing the same 10M records through an LLM API costs ~$2,000 even with batch discounts; the smart play is a hybrid — this filter on everything, an LLM only on the ~5% low-confidence tail, ≈ $110.)

The production recipe this project earned:

1. Statistical layers (frequency data, character models, TF-IDF) are the trustworthy backbone — they generalized perfectly to adversarial data they'd never seen.
2. Lists and regexes are cheap precision boosters that decay fast. Treat them like antivirus signatures: useful, perishable, needing a refresh pipeline — not "done".
3. Never trust an eval you can iterate on. Freeze test sets; measure fixes on data generated after the fixes.
4. Route the ambiguous middle (scores 0.3–0.6) to something smarter — an embedding layer or an LLM — and let the cheap filter handle the confident 95%.

The known open weakness is semantic, not lexical: "Head of Growth Marketing" still maps to Soil and Plant Scientists, because character overlap is not meaning. That's the next chapter (an embedding re-ranker), and unlike everything in this post, it might finally be a job for a GPU.

Companion reading: the [research write-up](/title-guard-paper) (method + full benchmark tables), the [case study](/title-guard-case-study) (the OOM-killer and multiprocessing war stories), and the [exec summary](/title-guard-exec-summary). All data sources are free and authoritative: ONET 29.1 (CC-BY-4.0), SEC EDGAR, US Census 2010, SSA baby names.*