Four bugs my own checks caught — and one still open
I build AI and data products for organizations where a wrong answer harms someone. The way I argue for that work is by showing what my own verification catches. Here is the full list from one product, including the defect that would have inverted it while my most important test passed.
1. The defect that inverted the product while the test passed
Entity resolution never said what a record was matched against
Deciding whether this year's grantee is last year's grantee is a name-matching problem — Form 990-PF records a recipient's name and address but no tax ID. My specification said: exact normalised-name match, otherwise mark it unresolved and exclude it. What it never said was what the record was being matched against.
Read it the obvious way — match against records from other years — and a genuine first-time grantee matches nothing. It gets marked unresolved. And then it is deleted by the very rule written to prevent bias.
Every true new grantee disappears. Every rate collapses toward zero. Every foundation in the product reads as closed to new applicants. The tool would have told nonprofits not to apply anywhere.
The reason this is worth writing about is what happened to the test. I had a mandatory check asserting that an unresolvable record could not inflate the new-grantee rate. Under the broken reading, that check passes cleanly — corrupting a repeat grantee cannot raise a rate that is already zero, and the match rate does drop as expected. Green light, inverted product.
The fix was conceptual, not mechanical: resolution is a clustering problem, not a lookup. A grantee appearing once forms a valid cluster of one. "Unresolved" has to mean "cannot be assigned confidently", never "matched nothing". And the test gained a second half asserting that a genuine first-time grantee is counted — without which the first half is decoration.
A test that checks one direction is not half a test. It is a test that reports success while the product is upside down.
2. The three that were invisible in every aggregate
The remaining defects share a property worth naming: none of them moved a summary statistic in a direction that looked wrong. Two of them made the data quality metric look better.
The similarity function merged a university with its own nursing school
A standard fuzzy-matching measure, token-set ratio, returns a perfect 100 whenever one name's words are a subset of the other's. So:
UNIVERSITY OF MICHIGAN UNIVERSITY OF MICHIGAN SCHOOL OF NURSING → 100 BOYS & GIRLS CLUB BOYS & GIRLS CLUB OF DETROIT → 100
Same state, both clear any sensible threshold, and two distinct organizations collapse into one. That suppresses the new-grantee count and makes an open funder look closed.
No automated check could see it. Merging two records into one raises the match rate, so the data-quality metric improves while the output degrades. The fix blocks a merge whenever the extra words carry meaning, and the design rule is now explicit: prefer a false split, which is visible when you sample, over a false merge, which is not.
PS 128 and PS 129 are one character apart
Numbers are among the most distinguishing parts of an organization's name —
school districts, union locals, chapter numbers — and they are also nearly
identical as strings. Any character-similarity measure scores
PS 128 against PS 129 as almost the same thing.
A test fixture of a foundation that funds someone new every year scored 0.1 instead of the expected 0.8. The rotating grantees were merging with each other. Differing numbers now always block a match.
A grantee moved from Washington DC to New York
African Parks Foundation of America appears under DC in some filing years and NY in others. I required the state to match before treating two records as the same organization, which split one five-year repeat grantee into two entities — and counted it as new.
The match rate sat at 1.00 the entire time. Nothing in any summary would ever have surfaced it. State corroboration is now scaled to the strength of the name evidence: an exact name match survives a move, because organizations relocate; a fuzzy match still needs the state to agree.
A fourth, from reading the schema properly
Not a bug that shipped, but the same lesson. The 990-PF return has an element listing grants approved for future payment — sitting directly beside the paid-grants element, with an identical internal structure. Counting it invents grantee relationships that never received money.
In one foundation's filing that is 178 future-approved grants against the real ones. Any parser written from the element names alone would have swallowed them silently, and the totals would have looked plausible.
What I take from this
Aggregates hide the errors that matter. Three of these four moved no summary in a suspicious direction, and two improved the data-quality metric. If your verification is a dashboard, you are checking the errors that were already easy to see.
A checker that shares assumptions with the thing it checks proves little. The relocation bug surfaced only because a second implementation disagreed by one grantee. Independence is the property that did the work, not rigour.
Ask what a passing test would look like if the product were broken. That question, applied to my own mandatory check, is what exposed bug 1. It is now the first thing I ask of any evaluation I design.
The check that is still open
Nobody has read the filings by eye.
A sample of the published foundations has been re-derived from the raw XML by an independent implementation, and the two agree — six of six, with differences of nought to two grantees, all explained by the deliberate differences between the methods.
That is not the same as verification. Both implementations were written by one person from one reading of what Part XV of the return means. If that reading is wrong, they agree with each other and are both wrong. An independent recomputation catches parser and logic bugs. It cannot catch a shared misunderstanding.
So the foundation pages carry a preview label, and they will keep it until a person opens a handful of returns and counts by hand. If you find a figure about your own foundation that looks wrong, write to me — I would rather hear it than keep publishing it.
This is what I mean when I say evaluation before features. Not a principle — a list of the specific things it caught, and an honest account of what it has not caught yet.