How Reliable is Your “Reference Data”?

Before the Crisis of 2008 hit, main mortgage brokers on the Wall Street were confident of the way they were running their business. Their software programs were written by crackerjack developers from top Ivy League schools. When they fed their sub-prime lending numbers into these programs, the result was “positive.”
After a year of monitoring, the default rates on these “junk loans” were below the “historic benchmark” of two percent.
The lenders knew that the NINA (No Income, No Asset) mortgages they were dishing out at alarming rate would eventually lead to foreclosures but their software programs were suggesting  a maximum of 2%, which was a totally acceptable level of risk. Even if things went bad and the actual loan default rates turned out to be 5 or 6 times as bad, they’d still be looking at 10-12% default rate, which was still okay.
When the dust settled, most of those portfolios suffered a FIFTY percent or above default rates.
Which means — the reality turned out to be not 5 or 6 but 25 times as bad!
So what went wrong?
The “historic benchmark” reference data that the programmers have used belonged to a period where NINA loans did not exist and all loans were given out to people with solid and verifiable assets. That’s why the average of 2% default rate observed in a strict regulatory environment could not of course be applied to a totally different context in which an “everything goes” attitude prevailed.
As technical communicators we need to pay attention to the specific context within which the “reference data” is collected.
If we claim that a fuse operates within a safety range of X and Y volts, we want to make sure that the reference data behind that assertion were collected in conditions similar to our own. For example, the “safety range” data collected in Canada may fail to be a reliable reference if the gadget is exported to United Arab Emirates.
Or if we are looking at post-WW2 employment data benchmarks, we have to remember the impact of the “G.I. Bill” before using the software program built on those benchmarks to analyze the historic trends in another country.
A joke’s punch-line is as good as its setup.
Similarly, the worth of a technical analysis is as good as the reliability of the reference data that it rests on.