Survivorship bias

During the Second World War, the Statistical Research Group (SRG) at Columbia University in New York did quiet, high-stakes math for the U.S. war effort. One of its members was Abraham Wald, a mathematician who had fled Austria after the Anschluss and already had a reputation for unusually careful thinking.

Among the problems brought to the group was aircraft survivability. Planes came home with damage from enemy fire. Armor could help, but armor is heavy, and too much of it would make a bomber worse at its job. So the question was not whether there should be armor. It was where a limited amount of protection would do the most good.

Wald wrote a series of 1943 memoranda on how to estimate a plane's vulnerability from the damage observed on the planes that returned. The hard part, which he faced directly, is that the sample is incomplete. You can count holes in survivors. You cannot, in the same way, count holes in the planes that did not come back. If hits were spread somewhat randomly, a part that shows unusually little damage among the survivors is a part where a hit was more often fatal. Those planes are missing from the hangar. The empty spots are the dangerous ones.

One of the memoranda works through vulnerability by section of the aircraft. In a worked example using hypothetical data, a hit in the engine area comes out as especially serious. That is why later tellings so often say look at the engines, not the riddled wings. The insight is real, and it is in the technical work.

The memoranda were not published in a journal at the time. They were later reprinted, in 1980, by the Center for Naval Analyses, and in 1984 statisticians Marc Mangel and Francisco Samaniego published a clear exposition of the work in the Journal of the American Statistical Association. Related methods were used again in later wars. Wald died in a plane crash in 1950, at the age of 48, while traveling to India for lectures.

Survivorship bias is a sampling problem wearing a success story. You start with a question, where do impacts land, which businesses thrive, which habits do high achievers share, and you answer it using only the examples you can still see. The missing examples are not a blank. They are data that got filtered out by the very process you are trying to understand.

In the aircraft problem, the filter is lethality. A returning bomber is a plane that was not hit in a way that brought it down, or that was lucky. If you plot the holes on those planes and then armor every crowded cluster, you armor the places that could already take a hit and still fly home. The quiet parts of the diagram are quiet because those hits are underrepresented in the survivor sample. Wald's contribution was not a single punchline. It was a method for estimating, from incomplete damage data, how vulnerable different parts of a plane were, including parts you mostly see when the plane does not return.

The same shape appears far from hangars. If you study only the companies that still exist, their habits will look like a recipe, because the companies that used the same habits and failed are no longer in the brochure. If you study only the books that are still in print after a hundred years, old books will look unusually sturdy. The worn-out ones were thrown away. The question to keep asking is: what had to be true for this example to show up in front of me?

Whenever a chart, a tip sheet, or a here is how it is done story is built from the examples still standing, ask who is missing and why they are missing. If an outcome can knock a case out of the sample, a crash, a bankruptcy, a book going out of print, then a hole you do not see may be more important than a hole you can photograph. A practical habit is to write down the hidden group on purpose: the planes that did not return, the applicants who were rejected, the customers who never left a review. If you cannot get that data, lower your confidence instead of pretending the survivor sample is complete. And when you are the person presenting evidence, say clearly what did not make it into the room.