Publishing Your False Positive Rate: The Differentiator Almost Nobody Has

In fraud detection, everyone publishes the number that looks good. “We catch 98% of tampering.” Great. Now tell me how many legitimate documents you had to suspect to get there, and then we can talk.

Recall — the fraction of fraud you catch — is the easy part to measure and the easy part to inflate. You tighten the thresholds, fire more signals, catch more fraud, and the number goes up. What goes up with it, and nobody puts on the slide, is the false positive rate: the fraction of honest documents you accused. I came from KYC/AML, and the false positive is where real compliance bleeds — every legitimate document flagged as suspect is a customer stuck in a queue, an analyst burning an hour, an account that doesn’t open. A detector that screams at everything isn’t safe; it’s expensive and useless, and it trains the team to ignore it.

This post is about the discipline almost nobody practices: measuring the false positive rate across separate populations of documents you know are legitimate, and publishing the result even when it’s ugly. I’ll use real numbers from Tamperlens, including the ones that don’t flatter me, because those are precisely the ones that prove the method.

The standard most don’t meet — and the test that shows it

Let’s start with the size of the problem, with numbers that aren’t mine. In 2025 NIST published SP 800-63A-4; §3.13 requires a document validation system to have a Document False Accept Rate (DFAR) of 0.1 or better — at most one fraudulent document accepted per ten.

The same year, DHS (US Homeland Security) ran the Remote Identity Validation Rally (RIVR), testing seven commercial validation systems against genuine and forged documents. The measured DFAR ranged from less than 0.88% to less than 76.68%. Two of the seven systems accepted roughly seven in ten forged documents — 69.20% and 76.68%, something like seven hundred times the limit in that same year’s federal standard.

A neutral government evaluator, not a vendor, showing that “we already have document verification” is not a solved control. (A caveat that always travels with this number: RIVR tests identity documents via image capture, not bank statements in PDF via structural analysis. It proves the category of control is unreliable; it says nothing about our engine, and the numbers are not ours.)

Hold onto the lesson: if serious commercial systems miss this badly, the only thing separating an honest product from just another one of them is having measured its own error and shown the receipt.

The discipline: measurement comes before the code

At Tamperlens the rule is written before the feature exists. When I planned the family that checks whether the document’s own arithmetic holds (does the column sum match the stated total? is the running balance consistent? does the payslip reconcile?), the plan named the number that would decide whether it’s born: how many legitimate documents fail the check. If the false positive rate on real government and bank documents is above a few percent, the tolerances are wrong or the family does not ship.

Notice the inversion. The acceptance criterion isn’t “the check catches fraud.” It’s “the check does not accuse the legitimate,” measured before a line is written, as a number declared on paper. That changes everything, because now measuring the false positive rate well is the condition of existing — and a bad result kills the feature instead of becoming a footnote.

I ran that family against a corpus of 3,842 PDFs — government contracts, bank PDFs, Brazilian web documents, plus the tamper corpora. Three candidate checks. The result decided each one:

• Column sum vs stated total. Fired on 29 published documents — real, legitimate documents. 29 false positives is unacceptable for the value the check delivers. It did not ship. Measured, and shelved.
• Running balance. Practically zero false positives on the real statements. It shipped, as running-balance-break, at medium severity.
• Payslip identity. Population of three documents in 3,842. Sample too small to claim anything. Not built.

One of three made it in, and the corpus is the reason. Not intuition, not what I thought worked — what 3,842 legitimate documents said about how many of them each check would accuse for no reason.

When a false positive takes down a signal that “looked right”

The most instructive case is a signal the PDF standard itself seems to endorse. A PDF’s trailer has an /ID array with two elements: per spec, /ID[0] is permanent, assigned at creation, and /ID[1] changes on every save. So the two differing should mean “changed since creation.” A free tampering signal, straight from the spec.

Then you measure. Against 1,728 documents nobody curated — a random sample of .gov PDFs and a random sample of the Brazilian web — divergent /ID fired on 44.7% and 36.7% of them. And most tellingly: 83.4% and 88.6% of those files had a single revision. A file written once cannot have been “saved again after it was created.” The inescapable conclusion is that a mountain of PDF writers simply generate both elements fresh on first output, ignoring the spec. What the spec says and what writers do are different things.

What do you do with a signal like that? The lazy answer is to keep it at medium and live with ~40% false positives. The cowardly answer is to delete it. The honest answer is the one we took: divergent /ID reports medium only when the file’s own structure agrees it was written more than once. When it doesn’t, the observation is kept — it’s still a real fact about the trailer, and hiding facts is how a forensic tool becomes unauditable — but emitted at info, which scores zero and stays out of the signal count. The fact survives; the weight doesn’t.

Separate populations, and why that matters

Notice I didn’t say “I measured the false positive rate.” I said “across separate populations”: PNCP contracts, bank PDFs, US .gov PDFs, the Brazilian web. That’s not statistical fussiness. One PDF writer dominates a market (Word dominates the Brazilian government; iText dominates banking). If you measure the false positive rate in one pooled bucket, the dominant writer’s bias hands you a number that collapses on the first client whose software vendor is different. Measuring per population is what reveals whether a signal is robust or just lucky with Word.

And there’s an honest limit statistics imposes. When I validated the family that reads Brazilian identifiers (CNPJ and similar numbers that carry a check digit), the result was zero false positives over 156 real numbers, from PNCP contracts (53), Brazilian web PDFs (16), and the tamper corpora (34). “Zero” sounds perfect, but 156 is a finite sample: by the rule of three, zero errors in 156 cases bounds the false positive rate at roughly 1.9% with 95% confidence. The point estimate is nothing; the honest ceiling I can claim is ~1.9%. Saying “zero percent” would be lying about what the sample supports. A number without a denominator isn’t a measurement — it’s a claim.

Why publishing the ugly one is the differentiator

Back to PNCP, which I’ve told elsewhere through the lens of structure, now through the lens of measurement. We ran the engine over 1,317 public contracts and 77% of the signed ones score high — because of the countersignature, not fraud. That’s a false positive of our own product, on the most common legitimate document in that market. And it’s written on our own evidence page.

Think about how many “document verification” vendors publish the scenario where their own tool misses badly. The answer is: almost none, because the commercial incentive runs the other way. But that’s exactly why publishing is the differentiator. Someone buying a risk control is buying calibration, not optimism. A vendor who tells you where the tool is noisy has given you the information you need to operate it; one who only shows you recall has sold you a number and hidden the receipt.

The asymmetry is worth repeating for anyone in compliance and KYC: the cost of a false negative is one fraud that got through — real, painful, but rare and usually caught by other layers. The cost of a false positive is structural: it multiplies across your entire volume of legitimate documents, every day, and it’s where the whole operation jams. That’s why the false positive rate is the number that decides whether a detector is usable, and why not publishing it is the biggest red flag a vendor can raise without noticing.

What to take from this

If you’re evaluating a detection tool — for document fraud, identity, anything — the question isn’t “how much do you catch?” It’s:

1. What’s your false positive rate, and over which population? Without a denominator, the number is advertising.
2. Did you measure across separate populations or in one pool where the dominant writer saved you?
3. Did the measurement come before or after the decision to ship the feature? Measurement that only confirms a decision already made isn’t measurement.
4. Will you show me a case where the tool misses badly? If it won’t, it either didn’t measure or is hiding.

Measuring recall is the easy part. Measuring, publishing, and acting on the false positive rate — killing a feature with 29 false positives, demoting a 40% signal, admitting a 1.9% ceiling instead of singing “zero” — is the whole discipline. It’s the boring work that doesn’t fit on a slide. And it’s the only thing that separates a risk control from security theatre.