Plausible Undercount

Context

An instrument answers a question of the form “here are the N things.” Findings in a codebase, callers of a function, vulnerable dependencies, uncovered branches, matching records, affected services.

Its input degrades — a build fails, a reference doesn’t resolve, a glob misses a directory, an index is stale, a crawl times out, a permission is denied. It does not crash. It produces a shorter list.

A short answer and a correct answer are indistinguishable from the outside. Nothing in the output says how much it couldn’t see, because the output format has no place to say it. The reader gets a confident list, acts on it, and the thing that mattered was in the part the instrument never reached.

Cost type

Behavioral, deferred, borne by whoever the missing item eventually hurts — which is rarely the person who ran the tool. The reader’s belief about coverage diverges from actual coverage, and the divergence is silent by construction.

What goes wrong

• A dependency scan runs against a project whose restore half-failed. Half the transitive graph is invisible. The report is clean and everyone believes it.
• A “find all callers” search misses the reflection-based, convention-discovered, or configuration-wired callers. The refactor ships and something breaks in a subsystem nobody thought was involved.
• An analysis tool runs before the build. Symbol resolution silently returns null for everything in the unresolved assemblies, and an entire category of finding — persistence, messaging, whatever those assemblies provided — vanishes from the report without a trace.
• A migration script processes the rows it can see. The ones behind a failed join are not reported as skipped; they are simply not in the count.
• The shorter list is more believable, not less. It looks focused. Long noisy output invites suspicion; a tight list of four findings reads like precision.

Prospective indicators

• The tool’s output is a list, and there is no field anywhere in it describing what was excluded, unreachable, or unresolved.
• Success is signalled by exit code alone, and the exit code reflects “the process completed,” not “the analysis was complete.”
• Degraded input produces a warning on stderr. Stderr is not in the artifact and will not be there tomorrow.
• Nobody can state what a floor on the answer would be — no independent way to know that N should be at least some number.
• The pipeline runs the analysis step without a hard precondition on the steps that produce its input.

Retrospective indicators

• Two runs against the same commit produce different counts and nobody investigates, because both looked plausible.
• An incident traces to something the tool was pointed at and didn’t report.
• The count changed sharply after an unrelated infrastructure fix, and the new number is the one everyone now uses without asking what the old one was.
• Someone can only tell the output was wrong because they happened to know the right answer independently.

Prevention

• Refuse rather than degrade. If the analysis cannot see everything it claims to cover, exit non-zero and write no artifact. A missing report is loud; a short report is silent. This is the whole entry — most of the rest is detail.
• Make preconditions a hard gate, not an assumption. Restore, build, resolve, then analyse. The role establish that the input is trustworthy is usually simply uncast, and everything downstream inherits the silence.
• If a degraded mode must exist, stamp the degradation into the artifact itself. In the file, next to the findings, not in a log. The artifact outlives the run.
• Report the denominator. “41 of 41 projects analysed” is a different statement from “41 findings.” Coverage is a separate fact from results and should never be inferable only by absence.
• Establish a floor independently. A second, cruder instrument — a grep, a count of files, a known-good baseline — that would notice if N collapsed. It need not be accurate; it needs to be differently wrong.

When this isn’t a failure mode

• The tool is explicitly a sample. “Here are ten examples” makes no completeness claim, and nobody infers one.
• Best-effort search where the user knows it’s best-effort. Interactive grep, IDE find-usages. The user is in the loop, iterating, and will notice.
• Degradation is loud by nature. If half the input being missing makes the output obviously absurd, the failure announces itself.
• The consumer independently knows the answer’s shape. Someone who knows there are eight services will notice a report covering three.

Note

The failure is a property of the output shape, not of the tool’s quality. Any instrument that answers with a list is exposed, and the better the tool’s reputation the worse the exposure, because a confident short list from a trusted source is exactly what nobody audits.

It is also the natural failure mode of AI-assisted analysis. Asked “where else is this used?”, an agent greps, reads what it found, and answers. The answer is usually right and occasionally short by exactly the amount that matters — reflection, dynamic dispatch, another repository — and it reads identically either way. Unlike a build, there is no restore step whose failure could have been gated.

Discovered concretely: a static-analysis probe reported 14 findings; the correct number was 35. The target’s dependencies had never been restored, every symbol lookup silently returned null, and an entire category of finding disappeared. No error. As its author put it — “the failure mode is a confident short list, which is exactly what this tool is supposed to produce.”

Related: Ceiling Reported as Floor is a tool reporting real findings at the wrong severity. This is a tool reporting fewer findings than exist, at the right severity, with no way to tell.