Job-Title Validation — Master Dashboard
Everything tested, everything designed, everything still open — one page. Updated 2026-07-16. Sources: measured runs on this 4-core/8GB VPS (`bench_results.json`, `data/eval/semantic_bakeoff.json`), the [C3 design doc](https://vault.binary.ovh/title-guard-c3-design), the [C2.7 blog](https://vault.binary.ovh/title-guard-blog), the [bake-off blog](https://vault.binary.ovh/title-guard-c25-bakeoff).
1. Every option on the table — status board
| # | Method | Status | Measured result | Speed (1 core) |
|---|---|---|---|---|
| 1 | Cleaning + normalization (html-unescape, unicode fold, abbrev expansion CEO→chief executive officer) | ✅ shipped (C2) | prerequisite for everything below | ~free |
| 2 | Rule flags: profanity set+roots, placeholder ("n/a", "test"), joke patterns, camelCase corruption, OOV ratio | ✅ shipped (C2–C2.7) | gibberish/profane/placeholder ≈100% on frozen AND holdout — statistics hold | ~free |
| 3 | Char-bigram gibberish score (improbability > 7.0) | ✅ shipped | 100% gibberish catch, both eval sets | ~free |
| 4 | Seniority grammar ladder (c-suite/vp/director/…, (?<!vice )president) |
✅ shipped | enables c-suite leniency floor 0.45 | ~free |
| 5 | Char+word TF-IDF nearest-neighbor → canonical O*NET occupation | ✅ shipped (C2) | frozen 97.7%, F1 .919; SEC 577/577; holdout 83.5% | 273–291/s (pipeline bottleneck) |
| 6 | Word-pair novelty signal (flag unseen pairs) | ❌ tested & REJECTED (C2.7) | flags ⅓ of legit intl/exec titles | — |
| 7 | PMI composition scoring | ❌ tested & REJECTED (C2.5) | AUC 0.185 — inverted: smoothing rewards OOV pairs ("happiness wizard" +.51), punishes popular real pairs ("sales manager" −.23) | 79k/s |
| 8 | Word-bigram LM (familiarity, not association) | 🏆 tested & WON (C2.5) | holdout AUC .974, joke recall 90% @ 6% FPR (baseline: 82% @ 20%) | 74k/s, 0.48s build |
| 9 | Static embeddings Model2Vec potion-base-8M (30MB) | 🏆 tested & WON mapping (C2.5) | fixes mapping bugs: Growth Mktg→marketing manager, Pharmacovigilance→pharmacologist; joke AUC .955 (lost to #8) | 4.7k/s, 3s build |
| 10 | Transformer embeddings (MiniLM 22M / BGE 33M, ONNX int8) | 🔜 designed, untested | expected ~1–2k/s; only if #9 quality gaps appear | est. 1–2k/s |
| 11 | TF-IDF → SVD → XGBoost supervised classifier | 💡 untested | viable as the meta-combiner: features = wordlm score + embed sim + flags + evidence; needs labeled outcomes | fast at inference |
| 12 | Linear head on frozen embeddings → SOC group | 🔜 designed (Part 4 ladder v2) | O*NET gives 63k free labels; sklearn, minutes to train | fast |
| 13 | Contrastive fine-tune of embedding model | 🔜 designed (ladder v3) | only if v2 provably falls short | — |
| 14 | From-scratch tiny model (\<1M params) | ❌ rejected on analysis | pretraining is where "Growth Marketing ≠ Soil" lives; learning exercise only | — |
| 15 | LLM-as-judge (Claude/GPT/Gemini) | 🔜 designed, costed | cascade tail ONLY (1–2% uncertain band): ~$50–150 per 10M records; never bulk | API-bound |
| 16 | Evidence layer L3: ATS APIs, EDGAR officers, H-1B LCA index, registries | 🔜 designed + sources probed live, not built | see C3 design Parts 1–3; contradiction-class = only auto-reject | offline index: k/s |
| 17 | Batch forensics: Zipf/entropy test, org-pyramid shape, Highlander rule | 💡 designed, untested | pure statistics, no network | ~free |
| 18 | Conformal calibration (precision guarantees) | 🔜 designed for C3.5 | turns scores into contracts ("≥99% precision band") | ~free |
2. How it was built, developed, tested
Build path: C1 rules → C2 TF-IDF NN over 63,934 O*NET variants + 600 SEC officer titles → C2.6 batch/name guard + benchmarks → C2.7 adversarial hardening → C2.5 semantic bake-off. All on one 4-core/8GB VPS, no GPU; index cached (2.3GB build once, 350MB/0.5s load).
Test discipline (the part that matters): a 16-category frozen eval set (417 titles + 290 names, grok-generated, committed) for iteration, plus a one-shot holdout (423+283) that is scored once and never used for fixes. This protocol is why we caught our own overfitting: fictional-title detection read 98% on frozen and collapsed to 12% on holdout — memorized lists decay. Signals that measure properties (gibberish stats, word familiarity) held at ~100% and AUC −0.02 respectively. Two plausible signals (#6, #7) were killed by measurement, not opinion — #7 finished inverted.
3. What it can do today, and to what extent
- Garbage (profanity/gibberish/placeholder): ~100%, decay-proof. Solved.
- Joke titles from real words: was ~56% lexically → 90% recall @ 6% FPR with the wordlm gate (bake-off, holdout, frozen threshold). Integration into the shipped score still pending (needs fresh holdout — old one is spent).
- Normalization to canonical occupation: good on common titles; the bad-mapping tail (surface-similarity traps) now fixable by #9 as a repair tier.
- Names: 100% frozen / 79.5% holdout (NameGuard, 36.3k/s/core).
- Cannot do — by design, not failure: verify that a person actually holds a title at a company (truth, not language). That is the C3 evidence layer: ATS job boards, EDGAR officers, H-1B employer-title index, EU/UK/FR registries — all probed live, none built yet.
4. The rules catalog
Shipped: profanity roots (substring, with Dickinson/Hitchcock exclusions) · placeholder-on-raw-string · char-bigram improbability >7.0 · joke/fake patterns · camelCase corruption · OOV-ratio flag · buzzword scoring · seniority ladder + c-suite floor 0.45 · flags multiply score 0.7^n, profanity caps at 0.05.
Identified, awaiting build (from the C3 design):
- Highlander: singleton titles must be singletons — two CEOs at one company = one bad record; Poisson prior on expected count per level given headcount.
- Industry-conditional calibration: "VP" at a bank is mid-level, at a startup an executive — reinterpret seniority per industry before any seniority rule fires (learnable from H-1B data).
- Org-pyramid shape: a batch that's 60% "VP" is synthetic.
- Negative-space: titles never observed in millions of real employer-title rows despite lexical plausibility = empirical "doesn't exist" list that can't decay.
- Company-vocabulary consensus: title far outside what a company demonstrably hires for (ATS/LCA/own-corpus).
- Record-level: freemail × C-suite, name↔email-local-part mismatch × high title.
How to find new rules (the method behind all of the above): take a measured failure category from the eval, hypothesize a property that separates it, build the cheapest version, test frozen → confirm on a fresh holdout, and reject without mercy (2 of our 9 candidate signals died this way).
5. The 600M-record question
Will it work at 600M global titles? The compute is a non-issue; the languages are the issue.
Compute (measured throughputs, extrapolated): the pipeline never sees 600M unique strings — titles repeat massively. Expect maybe 20–30M unique after normalization. On this single 8GB box: dedupe (streaming, hours) → wordlm on 30M unique ≈ 7 min/core → embeddings ≈ 1.8h/core → L1 lexical on uniques ≈ 29h/core (÷4 cores ≈ 7h) → join back. Days on a $10 VPS, hours on one real 32-core machine. Memory doesn't grow with input size (streaming + fixed indexes ≈ 4GB peak). Nothing fails or degrades — throughput is linear, per-record cost is constant.
What actually breaks: language. Every current component is English/US-trained (ONET, SEC). Feed it "Geschäftsführer", "Directeur Commercial", "営業部長" and the wordlm sees OOV → flags legit titles as garbage; the TF-IDF NN maps to nonsense. On global data today the system would be confidently wrong at scale* — worse than slow.
US + Europe: solvable with known data. Route by language (fasttext langid, ~1MB model, µs/title) → per-language corpora. ESCO (probed live ✅) is Europe's ONET: ~3,000 occupations with preferred+alternate labels in 28 languages — one download replaces ONET for the EU; national taxonomies (UK SOC 2020, France ROME, Germany KldB) refine it. Retraining is rebuilding counts — the wordlm builds in 0.48s; per-language builds are trivial. The embedding model swaps to a multilingual one (potion-multilingual-128M exists). No architecture change, just data.
Global (beyond US/EU): same recipe, weaker data. ISCO-08 (the ILO spine ESCO maps to) + Wikidata occupation labels (all languages) + WDC JobPosting extractions give coverage; expect accuracy to drop where taxonomies are thin (much of Asia/Africa/LATAM) and to need per-market eval sets before trusting any number. Honest answer: US ✅ today, EU ≈ a week of data work, global ≈ a real project with per-language validation.
6. Dataset catalog for training/rules
In use: O*NET 29.1 (63,934 variants, CC-BY) · SEC EDGAR officer titles (600, refetchable) · grok-generated adversarial evals.
Probed live, ready to use: ESCO (EU, 28 languages — the Europe unlock) · DOL H-1B LCA bulk (millions of real employer+title+SOC+wage rows — incumbents, negative-space, industry calibration, wage sanity) · ATS public APIs (Greenhouse/Lever/Ashby — live title universes) · EDGAR filings · NPPES (8M US healthcare providers) · ProPublica 990 (nonprofit officers+comp) · Wikidata (multilingual occupation labels + notable-person positions) · French registry (officers+industry+headcount) · Companies House UK · Senate lobbying disclosures · WARN notices.
Known, not yet probed: WDC/Common Crawl schema.org JobPosting extractions (job postings at web scale, multilingual — the global training corpus candidate) · ISCO-08 · UK SOC 2020 / ROME (FR) / KldB (DE) / NOC (CA) / ANZSCO (AU-NZ) · national census occupation-coding indexes (e.g., US Census alphabetical index — tens of thousands of raw title strings mapped to codes) · EURES EU job-mobility postings · USAJobs API (US federal postings, free key) · BLS OES wage tables · UK visa sponsor register.
Ruled out: LinkedIn (ToS) · Kaggle LinkedIn scrapes (licensing) · anything requiring name-culture priors (ethics — measured cousin of this idea false-flagged ⅓ of international names).
7. Still needs testing — priority order
- Integration of wordlm gate + embed repair into the shipped score, verified on a fresh holdout (Part 7 Phase A — the old holdout is spent).
- Evidence layer offline core: LCA index + industry×occupation matrix (Phase C) — unlocks Highlander/calibration/negative-space rules.
-
11 meta-combiner (the TF-IDF→SVD→XGBoost idea, generalized): once ≥3 signals exist, a small supervised blender on labeled outcomes should beat hand-tuned weights — needs the labels the evidence layer and production feedback would generate.
- ESCO ingestion + langid routing (the Europe milestone).
- Multilingual embedding swap + per-language wordlm rebuilds.
- Conformal calibration for guarantee-grade thresholds.