Title-Guard C3: From String Validator to Claims Verifier
Design & brainstorm document — 2026-07-15. Status: ideation approved for write-up; no C3 code written yet. All API probes in this document were run live on 2026-07-15 from this VPS.
1. Where this came from
Title-guard C1–C2.7 answers one question well: is this string shaped like a real job title (or name)? It does so lexically and statistically — TF-IDF nearest-neighbour over 63,934 O*NET title variants, seniority grammar, profanity/gibberish/placeholder flags — at ~900 titles/s on this 4-core box, with 97.7% accuracy on the frozen eval and 100% on SEC officer titles.
The C2.7 holdout taught us the lesson this whole design is built on: memorized lists decay, statistical signals hold. Fictional-title detection fell from 98% (frozen set) to 12% (one-shot holdout), while gibberish and legit-title detection stayed at 100%. Any signal that amounts to "a list of things we've seen" will rot. Signals grounded in structure or in external reality won't.
The user's challenge: given that records usually carry (name, title, company), why not use the company? Check the company's careers page, about page, public filings — build production rules from public data. This document is the worked-out answer.
2. The reframe: three layers of validation
| Layer | Question | Status |
|---|---|---|
| L1 — Intrinsic | Is this string shaped like a real title? | ✅ built (C1–C2.7) |
| L2 — Contextual | Is this title plausible at this company? | this design |
| L3 — Extrinsic | Does public evidence support this exact person+title+company claim? | this design |
That progression turns title-guard from a string validator into a claims verifier — which is what KYC, lead-list hygiene (ZoomInfo/Apollo-style data vendors), and fraud teams actually pay for.
3. The core design principle: asymmetric evidence
Every external source is evidence with three possible polarities, and they are not symmetric:
- Positive match (title found in the company's real vocabulary; name found in filings) → strong boost.
- Absence (not found) → weak or zero penalty. Every source has coverage holes; "not on the careers page" does not mean fake. Naive designs die here.
- Contradiction (record claims "CEO of $PUBLIC_CO" but SEC filings name someone else) → very strong penalty — and the only class safe enough to auto-reject on in production.
A second structural insight: different sources see different slices of a company's true title universe.
- ATS job boards show openings (roles being hired now).
- H-1B LCA filings show incumbents (roles actually held by employees).
- SEC/registry filings show officers.
They complement rather than duplicate; a good combiner treats them as independent witnesses.
4. Live probe results (2026-07-15)
Every source below was tested from this VPS before being written into the plan.
| Source | Probe | Result |
|---|---|---|
Greenhouse boards-api.greenhouse.io/v1/boards/{slug}/jobs |
stripe, airbnb, databricks | ✅ structured JSON, no auth |
Lever api.lever.co/v0/postings/{slug} |
spotify ✅, netflix ∅ (not a Lever shop) | ✅ works; empty ≠ error |
Ashby api.ashbyhq.com/posting-api/job-board/{slug} |
openai | ✅ titles + departments + teams |
SEC EDGAR data.sec.gov/submissions/CIK…json |
Apple | ✅ (UA header required, ~8 req/s) |
GLEIF api.gleif.org/api/v1/lei-records |
"Stripe" | ✅ 55 legal entities, no key |
NPPES npiregistry.cms.hhs.gov/api |
John Smith | ✅ full provider records, no key |
| ProPublica Nonprofit Explorer v2 | "red cross" | ✅ 190 orgs, no key |
| Wikidata SPARQL (P169 chief executive officer) | Apple | ✅ returned "Tim Cook" |
OpenAlex api.openalex.org/authors |
Yoshua Bengio | ✅ 89 author records, no key |
| DOL H-1B LCA disclosure page | HTTP check | ✅ 200, quarterly bulk files |
| PatentsView | search | ❌ now requires (free) API key |
5. Architecture
claim = (name?, title, company?, country?) # any subset works
titleguard/
evidence/
base.py # EvidenceAdapter: lookup(claim) -> Evidence(source, polarity, strength, raw, fetched_at)
slugs.py # company name -> domain -> ATS detection (the one genuinely hard component)
cache.py # sqlite, per-source TTL (ATS 7d, EDGAR 90d, GLEIF 180d)
adapters/ # one small file per source, all behind the same protocol
offline/ # L2: built on disk, zero network at query time
lca_index/ # H-1B employer->title index (sqlite FTS)
sic_soc.py # industry x occupation compatibility matrix
combine.py # log-odds evidence combiner; per-source weights calibrated on labeled eval
cascade.py # L1 lexical (900/s) -> L2 offline -> L3 network; escalate only the uncertain band
The cascade preserves the throughput story. L1 clears ~95% of rows at current speed. L2 is local sqlite lookups (thousands/s, no network). Only the uncertain residue pays network latency — behind an aggressive cache, a rate limiter, and a per-batch budget. Each tier is 10–100× more expensive and exponentially rarer.
The combiner is naive-Bayes-style log-odds aggregation — the same statistical ethos as C2, calibratable with the existing eval machinery. It natively expresses the asymmetry: per-source weights for match/absence/contradiction are learned, and absence weights will calibrate near zero for low-coverage sources automatically.
8GB-VPS discipline (per project rules): offline indexes are streamed builds into sqlite/FTS, no pandas, adapters are plain requests with generators; nothing holds a dataset in RAM.
6. Sources — development plan and honest success forecast
Tier 1 — build these
DOL H-1B LCA bulk files — the gem. Quarterly XLSX disclosures with EMPLOYER_NAME, JOB_TITLE, SOC_CODE, WAGE: millions of real employer→title→occupation rows, bulk download, no rate limits, fully offline — a perfect match for this box. Answers both "does this employer hire" and "is this title in their real vocabulary" locally, for incumbents rather than openings. Plan: downloader + streaming XLSX→sqlite ETL (~1 day); the real work is employer-name normalization (legal suffixes, DBA names — reuse existing fold/clean code). Forecast: strong coverage of mid-to-large US employers, skewed toward skilled roles; high precision; ~85% success for the US L2 layer. Blind spot: small non-sponsoring businesses.
SEC EDGAR officers. fetch_sec_titles.py already speaks EDGAR; extend to the submissions API + Form 3/4 XML for name + exact title + company on every US public-company officer, plus DEF 14A for named executives. Plan: CIK resolver from company_tickers.json + officer extractor, ~2 days. Forecast: narrow coverage (~7k companies) but near-100% precision — the only source producing high-confidence contradictions, e.g. "CFO of $PUBLIC_CO" when filings say otherwise. ~90% success within its niche; the flagship demo feature.
SIC×SOC compatibility matrix (offline, universal). Company industry code (from EDGAR/GLEIF/Companies House) × the title's ONET/SOC family → plausibility prior. "Head Chef at a fintech" gets flagged with zero per-company data. Elegant twist: the matrix is estimated from the LCA data itself (each row has employer industry and SOC) — one dataset feeds two components. Plan: ~2 days on top of the LCA ETL. Forecast: coarse but universal; catches cross-domain absurdity L1 fundamentally can't. ~75% success*, fuzzy at conglomerates.
ATS adapters (the user's original idea). Verified live and clean. Five adapters (Greenhouse/Lever/Ashby/Workable/SmartRecruiters, ~½ day each). The real component is slug discovery: company name → website → careers link → recognize the ATS URL pattern — a mini-crawler with heuristics, ~3 days, which will fail on roughly half of companies. Forecast: coverage ~30–50% of tech/mid-market, near-zero for local SMBs and much of Workday-land; positive-evidence only (openings ≠ all employees, so absence must score ~0). ~60% success overall, ~85% for tech-sector claims. Build after LCA.
GLEIF + Companies House (UK). GLEIF: "does this legal entity exist at all", global, no key — a 2-hour adapter, cheap sanity gate. Companies House: every UK company's directors by name, free API key — high precision for UK director-level claims, ~1 day. ~95% success within their narrow questions.
Tier 2 — vertical plugins (each is a market segment)
- NPPES (probed ✅): all ~8M US healthcare providers — name, credential, org affiliation; bulk file available. Fake-doctor detection is a real product. ~1 day, ~90% in-domain.
- ProPublica 990 (probed ✅): US nonprofit officers with titles and compensation; 1–2 year filing lag. ~1 day, ~80% in-domain.
- Wikidata SPARQL (probed ✅): exec verification for notable orgs worldwide; watch staleness. ~½ day.
- OpenAlex (probed ✅): researcher→institution affiliations for academic-title claims. ~½ day.
- FINRA BrokerCheck: finance person→firm, high precision, US-only.
Tier 3 — demoted, with reasons
- RDAP / domain age / MX — user called this, correctly. Old domains prove nothing (aged domains are purchasable; legit startups have young ones); MX proves deliverability only; and it verifies company infrastructure, not the person's claim. At most a zero-cost tie-breaker inside existence checks. Not a roadmap item.
- Temporal / era consistency ("Webmaster in 2026", "Prompt Engineer in 2015") — also correctly called. Needs record dates most datasets don't carry, fuzzy priors, catches only lazy fraud. Dropped.
- GitHub company field — self-reported (the fox guarding the henhouse); only org membership is trustworthy and that's a sliver. Dropped.
- PatentsView — now key-gated; inventor→assignee is high-precision but tiny coverage. Parking lot.
- Name-culture × company-geography priors — rejected on ethics. Same trap as the word-pair novelty signal C2.7 measured and rejected (flagged ⅓ of legitimate international/exec names): it's bias with extra steps. Documented so it stays rejected.
- LinkedIn — off the table entirely (ToS).
7. Non-source ideas from the same brainstorm
- Batch-level fraud signals. Fraud rarely arrives one row at a time. Score the batch: title-frequency entropy vs the SEC/O*NET baseline, org-pyramid shape (a real org has more ICs than directors — a lead list that's 60% "VP" is synthetic or inflated), duplicate/near-duplicate people. Pure statistics, no network, plays to the project's strengths.
- Org-chart grammar. Titles encode a partial order (Intern \< Senior \< Lead \< Director \< VP \< C-suite). Within one company's records, check consistency: two CEOs, a "Senior Intern", an impossible reporting pyramid. Cheap, deterministic, and nobody does it.
- Cost-aware cascade as *the* production architecture — with measured cost/latency/accuracy per tier, this is a publishable systems result and the natural sequel blog.
- Title-inflation index (spinoff): not "is this fake" but "how inflated is this title vs what this company/industry actually files for" — lead-scoring teams want this; LCA wage data enables it.
- Reverse mode for KYC: given a company, generate its plausible title universe and flag outliers in bulk — the shape of what data vendors and AML teams buy.
8. Evaluation plan
Reuse the C2.7 frozen/holdout discipline. Positives: held-out LCA rows and EDGAR officers (real person/title/company triples). Negatives, synthesized as three distinct fraud modes the combiner must separate: real name × wrong company, inflated title at a real company, real title at a nonexistent company. Production gate: contradiction-class flags need ≳99% precision before anything auto-rejects; everything else surfaces as a score. Per the C2.7 lesson: the holdout is evaluated once; fixes get a freshly generated holdout.
9. Phased roadmap
| Phase | Deliverable | ~Effort |
|---|---|---|
| C3.0 | Claim schema, evidence protocol, log-odds combiner, sqlite cache (no network) | 2 days |
| C3.1 | LCA bulk index + SIC×SOC matrix — the offline L2, biggest single jump | 3 days |
| C3.2 | EDGAR officer verification — contradiction detection, the demo-able wow | 2 days |
| C3.3 | ATS slug resolver + 5 adapters | 4 days |
| C3.4 | Vertical plugins (NPPES, 990, Companies House, Wikidata) as opt-in | 2 days |
| C3.5 | Eval + calibration + blog sequel | 2 days |
The pre-existing "C3 FastAPI demo" backlog item becomes the serving layer on top of this, not a separate feature. (Deployment per project rules: bind 127.0.0.1; no DNS changes.)
10. Honest caveats
- US-heavy. LCA, EDGAR, NPPES, 990 are all US; Companies House/GLEIF are the international fig leaf. Expected lift drops noticeably outside the US/UK — say so in any blog or pitch.
- Slug discovery is the fiddly 20% that takes 80% of the ATS effort.
- Rate limits everywhere except the bulk files — the cache and per-batch budget are load-bearing, not nice-to-haves.
- Absence is not fabrication. If one sentence of this document survives, make it that one.
Part 2 — Round 2: more probed sources and new components (2026-07-15)
2.1 New probes
| Source | Probe | Result |
|---|---|---|
French registry recherche-entreprises.api.gouv.fr |
"airbus" | ✅ no key; returned actual officers (Guillaume Faury, "Président de SAS", 2 Directeurs Généraux) plus NAF industry code plus headcount bracket |
| Wayback Machine CDX API | stripe.com/about |
✅ snapshots back to 2011, free, works from this datacenter IP |
| GDELT v2 news API | exec-appointment query | ❌ rate-limited/blocked from this VPS even after backoff — parked |
French registry is the find: one free call yields officer verification (Companies-House-grade for France), the industry input for the SIC×SOC matrix, and the size-vs-seniority prior. ~90% success within France, ~1 day adapter. With Companies House UK + GLEIF + Denmark's CVR (same model, unprobed), the "US-heavy" caveat becomes a real EU story.
Wayback CDX enables the one thing no other source can: past-tenure verification — fetch the archived team/about page for a date range and search for the name. Limits: arbitrary-HTML parsing, one claim at a time, only companies whose team pages were archived. So it is not a bulk signal; it slots in as the last cascade tier before human review, for high-value single claims (executive KYC, due diligence). ~50% coverage on tech companies, high precision on hit.
2.2 New components
- CompanyGuard — the gap in the triad. We validate names and titles intrinsically but take the company string on faith. Reuse existing machinery (gibberish char-bigrams, placeholder/profanity flags) plus two cheap new checks: legal-suffix grammar ("Inc"/"GmbH"/"SAS" vs claimed country) and typosquat detection — edit distance against a top-10k brand list catches "Gooogle LLC" and homoglyph spoofs. ~2 days, L1 speed, and every L2/L3 lookup benefits because a garbage company string short-circuits before wasting a network call.
- Cross-record entity resolution. Same email with two names, one person with contradictory titles across a batch — probabilistic record linkage via
splinkon DuckDB (memory-frugal, fits the 8GB rule). Big enough to be its own track (C4), not a bolt-on. - Zipf test for synthetic batches. Real title distributions are Zipfian (many "Software Engineer", one "Chief of Staff"); generated lists come out too uniform. A KL-divergence check against the O*NET/SEC frequency baseline — whole-batch forensics, essentially free.
- Conformal calibration. Calibrate the combiner with conformal prediction so the production rule ships as a guarantee ("auto-reject band has ≥99% precision"), not a hand-picked threshold. The difference between "a score" and "a contract".
- Adversarial data flywheel. Automate the C2.7 lesson: every release, generate a fresh holdout (never reused), run evals, plot decay across versions. "Never iterate on a holdout" becomes a pipeline, not a memory.
Part 3 — Validating whole contact records, not strings (2026-07-16)
The wider question: a record is (name, email, phone, company, title). String validation is only the cheapest tier. For any holder of a large multi-source contact dataset, three approaches beat adding more string rules:
3.1 The corpus validates itself
- Per-company email-format consensus. Most companies use one canonical pattern (
first.last@,flast@). Learn it from a few known-good emails per domain; a record claimingjsmith@at afirst.last@company is flagged with zero external calls. (Hunter.io is an entire business built on this one signal.) - Per-company title vocabulary from the corpus itself. Where the dataset holds many contacts at one company, their consensus is the title universe — an incoming record claiming a title far off that company's observed distribution is the anomaly. No careers page needed for covered companies.
- Name ↔ email-local-part consistency.
jsmith↔ "John Smith" checks are cheap and surprisingly discriminative; a mismatched local-part plus a high-seniority title is a strong suspicion pattern. Corollary rule: freemail domain (gmail/yahoo) × C-suite title at a large company → flag. - Phone plausibility. libphonenumber for syntactic/region validity; area code vs known company office locations; and DID-block proximity — employee direct-dials usually share the company switchboard's first 6–7 digits. A direct-dial inside the block is near-proof; a distant mobile is merely neutral (asymmetric evidence again).
- Truth discovery over sources. When records arrive from multiple vendors/streams, treat each as a noisy voter and learn per-source reliability (EM-style truth-discovery, the math behind Knowledge Vault). Corroboration count × source reliability is a validity score that needs no external data at all. Provenance-per-field must be in the schema from day one — it is the input to this.
3.2 Event-driven decay (business logic, not lookup)
Contact data rots ~25–30%/year and not uniformly. Rules:
- Freshness half-life per record: P(still valid | age, role family, industry, company size), fit from re-verification history. Validation budget goes where value × staleness is highest.
- WARN Act notices (probed ✅, free): employer + location + date + headcount of layoffs. When a company files a WARN for 500 people in Austin, bulk-decay every matching record that day.
- ATS boards inverted into churn detectors: monitor postings over time — a new opening for "VP Marketing, Austin" often means the seat just emptied, i.e. the existing "VP Marketing at Acme, Austin" record is now suspect. Openings are invalidation signals for incumbents.
- Press-wire RSS (BusinessWire/GlobeNewswire exec-moves feeds): "Acme appoints Jane Doe as CFO" validates Jane and invalidates the previous CFO record. Primary feeds, unlike GDELT which aggregates (and blocked us).
3.3 New external sources (probed 2026-07-16)
| Source | Probe | What it gives |
|---|---|---|
| Web Data Commons (schema.org extractions from Common Crawl) | ✅ site live | Millions of schema.org/Person JSON-LD blocks from company team pages — name+jobTitle+worksFor, first-party assertions, free bulk download, offline-processable. The sleeper hit. |
| Gravatar | ✅ per-email response | Email hash → does a human-registered profile exist? Weak-positive, one HTTPS call. |
| Have I Been Pwned | not probed (keyed) | Email present in a 2016 breach = aged & real, not minted last week. Odd but genuinely useful. |
| Senate lobbying disclosures | ✅ 108,817 filings for 2025, free API | Registered lobbyists: name+title+organization. |
| State professional licensing boards | not probed (fragmented) | CPAs, engineers, nurses, real-estate — with NPPES + FINRA this becomes a "licensed-professional verification pack" per vertical. |
| SMTP RCPT-TO | not probed (IP-reputation-sensitive) | Mailbox existence without sending; catch-all domains cap precision ~70–80%. |
Part 4 — Embeddings: full feasibility engineering (2026-07-16)
4.1 What embeddings fix — and what they can't
They fix the C2.5 known failure class: surface-lexical NN maps "Head of Growth Marketing" → Soil Scientists because of character overlap; an embedding maps it into semantic space where it lands near marketing occupations. Joke titles built from real words ("Chief Happiness Wizard") land in low-density regions far from every real occupation cluster — detectable via distance-to-nearest-centroid + local density, which plain lexical matching fundamentally cannot do.
They do not fix truth: an embedding cannot tell you whether this person holds this title at this company. Embeddings upgrade L1/L2 (the semantic mapping layer); L3 evidence stays the only truth layer.
4.2 The deduplication insight that changes all the math
10M records ≠ 10M strings. After normalization, a 10M-record dataset typically contains a few hundred thousand unique titles. Embed unique strings once, cache, join back. Every cost below shrinks ~20×.
4.3 Model menu (all CPU-only, fit this 4-core/8GB box)
| Option | Params | Disk | Est. CPU throughput (short strings) | Notes |
|---|---|---|---|---|
| Model2Vec (potion-base-8M) | 8M static | ~30MB | ~30k+/s | Distilled static embeddings, no transformer at inference — the speed king; slightly weaker semantics |
| all-MiniLM-L6-v2, ONNX int8 | 22M | ~23MB | ~1–2k/s | The default choice; well-studied |
| bge-small-en-v1.5 / gte-small | 33M | ~35MB int8 | ~0.8–1.5k/s | Slightly better quality |
| 7B-class / API embeddings | — | — | — | Unnecessary for 5-word strings |
Reference index: 63,934 ONET variants × 384-dim float32 ≈ 98MB* — brute-force numpy matmul is fine (500k queries ≈ ~20 min); hnswlib if we want ms-level. All within the 8GB rule (and spawn-not-fork applies as ever).
10M-record wall-clock estimate (MiniLM int8, ~500k unique titles): dedupe ≈ minutes → embed ≈ 6–10 min → index lookup ≈ ~20 min brute-force → join back ≈ minutes. Under an hour end-to-end, vs 3.1h for the current lexical pipeline. Even with zero dedup it's ~2h. These are estimates to bench in C2.5, not measured numbers.
4.4 Train your own? Decision ladder
- v1 — zero-shot pretrained (MiniLM/BGE-small): no training at all; likely fixes most of the C2.5 tail. Start here, eval against the frozen set.
- v2 — linear head on frozen embeddings: logistic regression 384-dim → SOC major group, trained on O*NET's 63,934 labeled variants (free labels, sklearn, minutes on this box). Classifier confidence becomes a validity feature. Cheap, recommended.
- v3 — contrastive fine-tune (sentence-transformers, title→SOC pairs): hours-to-a-day on CPU; only if eval proves v2 insufficient.
- From-scratch tiny model (\<1M params, smol-lab style): will not match pretrained semantics — pretraining is where the "Growth Marketing ≠ Soil" knowledge lives. Fine as a learning exercise, wrong for production.
4.5 External APIs (OpenAI/Gemini/Anthropic) — when and cost
- Embedding APIs: ~500k unique titles × ~8 tokens ≈ 4M tokens → e.g. text-embedding-3-small ≈ $0.08 for the whole corpus. Cost is a non-issue; the reasons to stay local are latency, rate limits on bulk, privacy, and zero dependency. (Note: Anthropic has no embedding API — they point to Voyage.) Verdict: local model preferred; API acceptable.
- LLM-as-judge (Claude/GPT/Gemini): never for bulk — 10M records through an LLM is thousands of dollars and days. Correct use: the uncertain tail of the cascade (~1–2% after L1+embeddings+L2), ≈100–200k calls with a Haiku-class model ≈ 60M tokens ≈ $50–150 per 10M-record run, hours with concurrency. The LLM is the most expensive, rarest tier — exactly where the cascade puts it.
4.6 Non-neural rivals worth benching against embeddings
- KenLM n-gram perplexity over ONET+SEC+LCA titles: joke titles built from real words still have weird word-order* statistics. ~100k strings/s, tiny memory. If it catches most of the tail, embeddings become optional.
- PMI composition check: parse titles as [seniority][function][scope]; score modifier-head pairs by corpus co-occurrence PMI. "Growth Marketing" scores high; "Soil Marketing" scores near zero. Pure counting, no neural net, interpretable — very much this project's ethos.
Part 5 — Fresh validation-rule ideas (round 3, 2026-07-16)
- The Highlander rule. Singleton titles must be singletons: two CEOs or two CFOs at one company means at least one record is stale or wrong. Generalized: expected count per title-level given company headcount (Poisson prior) — five "Chief of Staff" at a 40-person company fails it.
- Industry-conditional title calibration. "VP" at an investment bank is a mid-level IC; "VP" at a 50-person startup is an executive. Title-seniority must be reinterpreted per industry before any seniority-based rule fires — a lookup table learnable from bulk employer-title data (LCA). Ignoring this poisons every seniority rule with systematic false flags on finance.
- PMI composition scoring (see 4.6) — the interpretable answer to the joke-title tail.
- Trust propagation across coworkers. If 29 of 30 records at one company form a coherent org (pyramid shape, consistent vocabulary), the 30th inherits trust — label propagation on the per-company record graph. Anomalies concentrate; trust flows.
- Department-taxonomy check. ATS APIs return departments/teams (Ashby's probe literally returned
departmentandteamfields): validate a title's function against departments the company actually has, not just its title strings. - Negative-space mining. From millions of LCA rows, derive titles that never occur despite being lexically plausible — an empirical "does not exist" prior per industry, the data-driven replacement for the fictional-titles list that decayed 98→12%.
Where everything slots
CompanyGuard + name↔email rules → C3.0; Zipf/batch + Highlander + PMI → C3.1; embeddings (4.3–4.4) = C2.5 executed as the cascade's middle tier; French registry + WDC + Wayback → C3.4; conformal + flywheel → C3.5; entity resolution → C4.
Part 6 — Decision: the C2.5 bake-off (2026-07-16)
The bet for the next sprint: bench the two counting methods against zero-shot embeddings on the frozen eval before committing to a neural middle tier.
| Contender | Type | Hypothesis |
|---|---|---|
| PMI composition | counting | modifier↔head co-occurrence catches "Soil Marketing"-class jokes, interpretably |
| Word n-gram LM perplexity | counting | joke titles from real words still have weird word-order statistics |
| Static/small embeddings (Model2Vec / MiniLM) | neural | semantic space fixes the lexical-NN failure class outright |
Decision rule: if the counting methods catch the same tail as embeddings, the system stays neural-net-free (simpler, faster, interpretable). If embeddings win decisively, they slot in as the cascade's middle tier — at under an hour per 10M records either way. Results go in the C2.5 blog.
Outcome (2026-07-16, measured — full detail in the [bake-off blog](https://vault.binary.ovh/title-guard-c25-bakeoff)): WordLM won the joke tail (holdout AUC .974, 90% recall @ 6% FPR, 74k titles/s, 0.48s build) — the system stays neural-net-free for detection. Embeddings (Model2Vec potion-base-8M) lost detection but fixed the C2 mapping bugs (Growth Marketing→marketing manager; Pharmacovigilance→pharmacologist) — they get the mapping job only. PMI inverted (AUC .185): add-alpha smoothing makes OOV pairs score high ("happiness wizard" +.51) while popular real pairs score low ("sales manager" −.23) — the C2.7 word-pair rejection, now with a mechanism. Code: titleguard/semantic.py, bench_semantic.py; numbers: data/eval/semantic_bakeoff.json.
Part 7 — Execution plan for the next session (written 2026-07-16, pre-compact)
This section exists so a fresh session can execute without re-deriving context. Constraint reminders: 8GB box (streaming, no pandas, spawn-not-fork, sklearn working_memory=256 already set in core), bind 127.0.0.1, never touch DNS, every published number from a measured run, `publish_md.py` needs `cd ~/claude_/atlas-docs` (cd back before git ops).
Phase A — Fold the bake-off winners into the product (C2.5 completion, ~½ day)
- FIRST, before any code: generate a fresh holdout —
gen_eval_data.py holdoutvia grok-bridge (raw cached underdata/eval/raw/). The old holdout has been seen twice (C2.7 + bake-off) and is dead for verdicts. Do not look at the new one until step 5. - WordLM gate into `core.py`: build
WordLMScorercounts at index time, persist insidedata/index.joblib(bumpCACHE_VERSIONto 3). New flagjoke_lmwhen score ≤ the frozen-set threshold stored insemantic_bakeoff.json(frozen.wordlm.threshold); flags multiply 0.7^n as usual. Throughput cost ~nil (74k/s vs 291/s pipeline). - Embed mapping as a repair tier, not a rewrite: only when lexical similarity \< 0.5 (the Pharmacovigilance case) or the top match comes from a different SOC major group than the embedding's nearest — replace canonical occupation with the embedding nearest-neighbour's SOC. Reference index must carry SOC codes (extend
EmbedScorerto keep(title, soc)from Alternate Titles). Keepmodel2vecan optional import — core must degrade gracefully without it. Fix to attempt: a c-suite prior on the reference so "Managing Director" reaches chief executives. - Re-tune on frozen only. Gate: frozen accuracy ≥ 97.7% (no regression), joke recall strictly up. Re-run
bench.pyfor the throughput delta and measure how often the embed repair tier fires (expected: low single-digit % of titles). - One-shot fresh-holdout eval. Report whatever it says, no fixes after. Publish an addendum on the bake-off blog page (same slug) with the integration numbers.
- Commit; update CLAUDE.md status block.
Phase B — C3.0 serving layer (~½ day)
FastAPI app api.py: POST /check/title, POST /check/name, POST /check/batch (JSON list, cap ~1k/call), GET /health. Bind 127.0.0.1:8710 (8700 is grok-bridge). Load TitleGuard once at startup (350MB index). Run as a systemd user service modeled on ~/grok-bridge/ (uvicorn, single worker). Check configs/binary.ovh.zone for an existing unused subdomain before any add-site.sh; if none fits, stay local-only and demo via curl in the blog — never modify DNS. Measure request overhead vs library-call throughput.
Phase C — C3.1 offline evidence layer (~2 days, the big one)
- LCA ETL: download 1–2 recent quarterly LCA disclosure XLSX files from the DOL OFLC performance page (~300–700MB each; disk has ~56GB free). Stream-parse with
openpyxlread-only mode → sqlite with FTS5:employer_titles(employer_norm, title, soc, naics, wage, year). Employer normalization =fold_ascii/clean+ legal-suffix stripping. Verify NAICS column exists in the file layout before building the matrix (it should — confirm against the record layout PDF on the same page). - NAICS×SOC compatibility matrix estimated from that same table (counts → smoothed log-odds, npz on disk).
- Evidence skeleton:
titleguard/evidence/base.py(Evidence dataclass: source, polarity match/absent/contradiction, strength, raw, fetched_at) + the offline LCA adapter first; log-odds combiner stub. Network adapters (EDGAR officers, ATS, French registry) are Phase D — do not start them in the same session as the ETL. - Eval per Part 1 §8: positives = held-out LCA rows; negatives = real title × wrong employer, inflated title at real employer, real title at fake employer. Gate: contradiction-class precision ≳99% before anything auto-rejects.
- Blog part 3: the evidence layer, with measured coverage/precision numbers.
Sequencing and risks
A → B → C strictly (A is small and unblocks the blog addendum; B makes everything demoable; C is the differentiator). Risks: grok-bridge flakiness for the fresh holdout (retry once, then cap-at-n like C2.7); LCA XLSX parse time on this box (openpyxl read-only streams ~1–2k rows/s — a 500k-row file is ~5 min, acceptable; if slower, convert once to CSV via libreoffice --headless or ssconvert and stream that); HF fetch for potion-8M is already cached in the venv's HF cache.