Why we publish our accuracy numbers, including the bad ones

On a construction takeoff project, the system we shipped reads drawing sets and produces device counts. Its measured mean accuracy across categories was 0.81. Five of six categories were above 0.91. One was noticeably worse.

We put all of that in the interface, visible to the estimators using it every day.

Why not just say "highly accurate"?

Because the estimator will find out. Not in a demo — in week three, on a real bid, when a number is wrong and they had no reason to check it. At that point you have not just lost the bid, you have lost the tool, because nobody trusts it again.

A published weak category means the estimator knows precisely where to spend their review time. It converts an unreliable black box into a reliable assistant with known limits. Those are very different products, and only one of them stays in use.

What "accuracy" has to mean

A single accuracy number is nearly meaningless without three things attached to it.

1. 01Measured against what. Ground truth from an expert on your real documents, not a curated public benchmark.
2. 02Broken down how. An average hides everything interesting. Per-category is where the decisions are.
3. 03Over what sample. Twenty documents is an anecdote.

The design consequence

Once you accept that the system is wrong some of the time, you design differently. You add reconciliation checks that catch errors arithmetically rather than trusting a model to rate its own confidence. You clamp outputs to declared bounds. You run multiple passes and take the median. You route the uncertain cases to a human with the source document beside the extracted value.

None of that is possible if you have told the client it is always right. Honesty about the failure rate is not a marketing concession — it is what makes the engineering possible.

The systems that get abandoned are not the ones that make mistakes. Every system makes mistakes. They are the ones that make mistakes the user was told would not happen.