Knowledge Base

How to Validate Job Titles — A Beginner's Guide

2026-07-16llmtitle-guardvps

Plain-language version, 2026-07-16. No prior machine-learning knowledge assumed. Each method below tells you: what it is, how it works, how to build it yourself on your own data, and what to realistically expect. Methods we tested and rejected are left out — this page only contains things that work. The technical deep-dives live in the [dashboard](https://vault.binary.ovh/title-guard-dashboard) and [bake-off](https://vault.binary.ovh/title-guard-c25-bakeoff) pages.

The problem in one paragraph

You have a spreadsheet of contacts: names, job titles, companies, emails. Some titles are real ("Marketing Manager"), some are garbage ("asdfgh"), some are jokes ("Chief Happiness Wizard"), some are placeholders ("N/A", "test"), and some are real-looking but wrong for that person or company. You want a system that reads each title and outputs a confidence score: is this a real job title? The methods below are ordered from simplest to most advanced — and importantly, each one catches a different kind of bad title, so a real system stacks them like filters.

Method 1: Clean the text first (mandatory, everything depends on it)

What it is. Before judging a title, fix its formatting. Real-world data is full of HTML junk (&), weird unicode (Ｍanager in full-width characters), abbreviations (CFO), and noise words.

How it works. A chain of small text fixes, applied in order:

  1. Un-escape HTML twice (data often gets encoded twice: & → & → &).
  2. Convert accented/exotic unicode to plain letters (é → e) so lookalike characters can't sneak past later checks.
  3. Lowercase everything.
  4. Expand abbreviations from a lookup table: CEO → chief executive officer, EVP → executive vice president, Sr. → senior. We keep ~30 of these.
  5. Strip meaningless prefixes like "Co-" tracking junk.

How to reproduce. Pure Python, no libraries needed beyond html and unicodedata from the standard library. Write one function clean(raw_text) that does the five steps, and call it before every other method on this page. Build the abbreviation table by scanning your own data for the 30 most common short forms.

What to expect. This isn't a validator by itself — but skipping it silently breaks everything else. Half our early bugs traced back to unclean input (example: the placeholder check must run on the raw string, because cleaning turns "n/a" into "na" and hides it).

Method 2: Rule flags — the cheap filters that catch obvious garbage

What it is. A handful of hand-written checks, each catching one specific kind of junk. Simple, fast, and in our tests these stayed ~100% accurate even on data designed to fool the system.

How it works — the five checks:

How to reproduce. For gibberish: take any large list of real job titles (see Method 3 for where to get one), count all adjacent-letter pairs, turn counts into probabilities, then score new strings by average improbability. ~50 lines of Python. Tune the threshold by eyeballing scores of real vs. mashed-keyboard strings — there's a wide gap.

What to expect. Near-perfect on the junk it targets (gibberish, profanity, placeholders), zero ability to catch anything subtle. That's fine — that's what the next methods are for.

Method 3: Match against a list of known-real titles (the workhorse)

What it is. Governments publish huge free catalogs of real job titles. Match every incoming title against the catalog: close match to a known title → probably real, and you also learn which occupation it is (useful for normalizing "VP Eng", "Head of Engineering", "Engineering Director" into one category).

Where to get the catalog (free):

How it works. You can't just do exact matching — "Sr. Software Eng" won't exactly equal "Software Engineer". The standard trick is TF-IDF similarity: represent each title by which words and which 3-to-5-letter fragments it contains, weighted so that rare fragments count more than common ones, then find the catalog title with the most overlap. Fragments (not just words) are what make "Enginer" (typo) still match "Engineer". This is 10 lines with scikit-learn (TfidfVectorizer + NearestNeighbors).

How to reproduce.

  1. Download O*NET's "Alternate Titles" file (it's a plain text table).
  2. Build a TF-IDF index over all 64k titles (one-time; cache it to disk — ours takes 2.3GB of RAM to build once, then loads in half a second).
  3. For each incoming title (cleaned via Method 1): find the nearest catalog title and its similarity 0–1.
  4. Score: high similarity → valid; medium → uncertain; low → suspicious.
  5. One tuning tip from our runs: executive titles ("Chief Revenue Officer") are phrased more freely than regular jobs, so give anything with a detected seniority word (chief/VP/director — a 20-line regex ladder) a more lenient floor.

What to expect (our measured numbers). 97.7% accuracy on our test set, 577/577 real SEC executive titles accepted, ~290 titles/second per CPU core. Two known weaknesses: (a) surface similarity sometimes picks a wrong occupation — "Head of Growth Marketing" matched Soil and Plant Scientists because of the word "growth" (Method 5 fixes this); (b) joke titles built from real words sail through (Method 4 fixes this).

Method 4: The word-familiarity score (our surprise winner)

What it is. A tiny statistical model that answers: "does this sequence of words look like sequences that occur in real job titles?" It's what catches "Chief Happiness Wizard" — every word is real, but real titles never chain words this way. In our head-to-head test it beat the AI-embedding approach at this job, and it's absurdly cheap.

How it works. From your catalog of real titles (Method 3's data), count how often each word follows each other word: after "chief" you often see "executive", "financial", "marketing"... and never "happiness". For a new title, walk through its word pairs and average how probable each step is. Real titles score high; absurd combinations score low. Two technical details that matter: blend the pair-probability with the single-word probability (so a rare-but-real word isn't punished too hard), and let unknown words pull the score down naturally (a word appearing nowhere in millions of real titles is itself suspicious).

How to reproduce.

  1. Same input data as Method 3 — no new downloads.
  2. Count word pairs and single words over the catalog (~30 lines of Python, a dictionary of counts, builds in half a second).
  3. Score = average log-probability across the title's word transitions.
  4. Pick the flag threshold on a small labeled sample: take ~50 real titles and ~50 jokes, choose the score that catches most jokes while flagging few real ones.

What to expect (measured). Caught 90% of joke titles at a 6% false-alarm rate on data it had never seen — the previous system managed 82% at a 20% false-alarm rate. Speed: 74,000 titles/second on one core (this will never be your bottleneck). Limit: it judges language only. A perfectly fluent but false title ("Regional Development Manager" for a person who isn't one) passes — that's Method 6's job.

Method 5: AI embeddings — for understanding meaning, not for detecting garbage

What it is. An embedding model turns any text into a list of numbers (a "vector") such that texts with similar meaning get nearby numbers — regardless of shared letters. This fixes Method 3's wrong-occupation problem: by meaning, "Head of Growth Marketing" sits near "Marketing Manager" and nowhere near soil science.

How it works. Use a small pretrained model (ours: Model2Vec potion-base-8M — 30MB, free download, runs fast on any laptop CPU, no GPU). Embed all 64k catalog titles once and store the vectors. For each incoming title, embed it and find the catalog title with the closest vector (a "cosine similarity" — one line with numpy). Two uses: (a) the closest match by meaning becomes your canonical occupation; (b) a title far from everything in the catalog is suspicious.

How to reproduce.

  1. pip install model2vec and load minishlab/potion-base-8M (3 seconds, cached afterward).
  2. Embed the catalog once (~10 seconds for 45k titles), keep the matrix in memory (~44MB).
  3. For each title: embed, dot-product against the matrix, take the best match and its score.
  4. Practical rule from our tests: use the embedding match only when Method 3's match is weak (similarity below ~0.5) or when the two methods disagree wildly — the lexical match is fine most of the time, so the embedding is a repair tier, not a replacement.
  5. Do you need OpenAI/Claude/Gemini APIs? No. Local scoring did 20,000 titles in 4 seconds on one core of a budget server. API embeddings cost pennies but add rate limits and a network dependency for quality this task doesn't need. Do you need to train anything? Also no — the pretrained model worked untrained ("zero-shot").

What to expect (measured). Fixed our documented wrong-mapping bugs ("Pharmacovigilance Specialist": Landscaping Workers → pharmacologist). At detecting jokes it was good but lost to Method 4 — hence the split of jobs: Method 4 detects, Method 5 maps. Speed: 4,700 titles/second per core. Bonus for big datasets: your 10 million records contain maybe 500k unique titles — embed each unique string once and join back, and the whole thing takes minutes.

Method 6: Check against reality (the truth layer — designed, sources verified, not yet built)

What it is. Every method above judges the string. None can tell whether this person actually holds this title at this company. For that you need external evidence — and a lot of it is free and machine-readable. We verified each source below with live API calls.

The sources, plainly:

The golden rule of using evidence — asymmetry. Found the title at the company → strong positive. Didn't find it → almost meaningless (every source has holes — never punish absence). Found a contradiction (record says "CEO of X" but filings name someone else) → the strongest negative there is, and the only kind safe enough to auto-reject on.

How to reproduce (starter version). Pick your 100 most important companies, hit the three ATS endpoints with each company's name-slug, store returned titles in a SQLite table, and check incoming titles against it (using Method 4's spirit: match = boost, no match = ignore). One afternoon of work for a real signal.

Method 7: Judge the batch, not just the row (designed, not yet built)

What it is. Fake data is usually fake in bulk, and bulk has statistical fingerprints single rows don't.

Three checks, plainly:

How to reproduce. Group your data by company; count titles per seniority level (Method 3's seniority regex gives you the levels); apply the three checks as flags on the batch. Pure counting — no ML, no downloads, an hour of work.

Putting it together: the cascade (the actual production design)

Run cheap filters first; send only survivors to expensive checks:

every record   → Method 1 clean  →  Method 2 flags  →  Method 3 catalog match
                                        ↓ still unsure (a few %)
                 Method 4 familiarity + Method 5 meaning
                                        ↓ still unsure (~1–2%)
                 Method 6 evidence lookup → optionally an LLM as final judge

Each level is 10–100× more expensive and sees exponentially fewer records. Measured on our hardware, 10 million records clear Methods 1–5 in about an hour on one modest 4-core machine. An LLM (Claude/GPT) is worth using only on the final 1–2% — roughly $50–150 per 10 million records — and never in bulk.

If your data isn't English

Everything above works unchanged — you just swap the data, not the code: detect the language first (fasttext's language identifier: 1MB, microseconds per title), then use ESCO's labels for the 28 EU languages as your catalog, and rebuild Method 4's counts per language (it builds in half a second). Use a multilingual embedding model for Method 5. Without this step, an English-trained system will confidently flag every legitimate French or German title as garbage — the single biggest trap in going global.

The three lessons we paid for, so you don't have to

  1. Keep two test sets. One "frozen" set you tune against, and one "holdout" you score exactly once at the end. Our system looked 98% accurate at catching invented titles on the tuning set — and dropped to 12% on the holdout, because it had memorized examples instead of learning properties. Every number in this guide is a holdout number where it matters.
  2. Statistics survive; lists decay. Checks that measure properties (letter-pair statistics, word familiarity) held up on unseen data. Checks that memorize examples rotted immediately. Prefer properties.
  3. Bench, don't argue. The fanciest-sounding method we tried finished dead last (worse than random, in fact — see the bake-off post if curious), and a half-second word counter beat the neural net at detection. Every method earns its place by measurement on your data — the reproduce steps above are exactly how.