The Information We Never Get
I. Toward a Theory of Knowledge Under Information Loss#
Every explanation begins with an assumption that is almost never stated.
It assumes that the information available to us is sufficient to explain the event we are trying to understand.
Most of the time we never question this assumption. We simply ask why something happened and begin constructing an answer.
Why did one company become one of the most valuable in history?
Why did one civilization collapse?
Why did one relationship succeed while another failed?
Why did one entrepreneur build an empire while thousands of equally ambitious people disappeared into obscurity?
These questions appear different, but they share a hidden premise.
They assume the evidence left behind is capable of supporting the certainty we seek.
I am no longer convinced that it is.
The more I thought about history, biographies, business, politics, and even our own lives, the more I found myself returning to the same uncomfortable observation.
Perhaps our greatest limitation is not intelligence.
Perhaps it is information.
Not information we fail to discover.
Information that no longer exists.
We often imagine that history is a record of the past.
It is not.
History is the residue of the past.
Reality contained vastly more information than history preserved.
Every conversation that was forgotten.
Every opportunity that never appeared.
Every opportunity that appeared but was ignored.
Every illness that never came.
Every accident narrowly avoided.
Every competitor who almost entered the market.
Every friendship that almost formed.
Every decision that was considered and abandoned before anyone else knew it existed.
Every event unfolded within an enormous field of possibilities.
History preserves only one path through that field.
The rest disappears.
This matters more than it first appears.
Because explanation depends not only on what happened, but also on everything else that could have happened and did not.
When that information disappears, our certainty should change with it.
Yet it rarely does.
Imagine reading the biography of a famous founder.
The book contains childhood memories, decisive moments, difficult decisions, and the habits that supposedly separated that person from everyone else.
By the final page the reader often feels they understand success.
But what exactly have they understood?
They have observed one realized life.
Not the countless lives that same individual might have lived under slightly different circumstances.
Not the businesses that would have existed had one investor declined a meeting.
Not the versions of history where illness arrived at the wrong moment.
Not the versions where regulation changed one year earlier.
Not the versions where a competitor released a better product six months sooner.
Those histories are invisible.
Yet they are not irrelevant.
They are part of the explanation.
The biography cannot include them because they never became history.
The absence of evidence is not evidence of absence.
It is often evidence that reality discarded almost everything before we had the chance to examine it.
Science became powerful because it discovered a way to resist this problem.
It repeats.
If an experiment produces an unexpected result, we perform it again.
Then again.
We change one variable.
Observe.
Compare.
Gradually the fog lifts.
Confidence grows because reality allows us to ask the same question repeatedly.
Many of the questions that matter most do not grant us this privilege.
There is one Roman Empire.
One Industrial Revolution.
One childhood.
One marriage.
One life.
The events themselves cannot be repeated.
Only the stories about them can.
That distinction is easy to overlook.
It is also fundamental.
This difference suggests that there are at least two kinds of knowledge.
The first comes from repetition.
It is strengthened whenever reality permits another observation.
The second comes from unique events.
It is built from evidence that can never be expanded in the same way.
These two forms of knowledge deserve different levels of confidence.
Yet we often treat them as though they were the same.
We expect history to teach with the precision of experiment.
We expect biographies to produce principles with the reliability of scientific laws.
We expect one extraordinary life to reveal universal truths.
Sometimes it does.
Often it cannot.
Not because the people involved misunderstood their own experiences.
But because the event itself never contained enough recoverable information to support the certainty we later attach to it.
This is the problem I want to examine.
Not whether history teaches us.
It clearly does.
Not whether successful people possess valuable knowledge.
Many undoubtedly do.
The question is different.
It is whether we systematically overestimate what can be known from events whose causal information has been permanently lost.
If that is true, then many of our strongest explanations deserve to become weaker.
Not because they are false.
Because they are built from evidence with limits we rarely acknowledge.
The challenge, then, is not simply to become better at finding patterns.
It is to understand what kind of patterns reality is actually capable of revealing.
Before asking what history teaches us, we should first ask a quieter question.
How much of history survived long enough to become knowledge?
II. Why We Mistake Stories for Knowledge#
If history preserves only fragments of reality, an obvious question follows.
Why are we so comfortable treating those fragments as complete explanations?
The answer, I suspect, lies less in history than in ourselves.
Human beings did not evolve to understand civilizations.
We did not evolve to explain financial markets, multinational corporations, political systems, or technological revolutions.
For almost all of our existence, the problems we faced were immediate, local, and remarkably repetitive.
We learned where food could be found.
Which plants were dangerous.
Which paths predators preferred.
How to throw more accurately.
How to cooperate.
How to hunt.
How to survive.
These environments possessed an important characteristic.
They allowed repetition.
Failure was often followed by another attempt.
Success could be refined through practice.
Small changes produced observable differences.
Our understanding improved because reality repeatedly exposed us to similar problems.
Learning was not merely possible.
It was rewarded.
Imagine learning to throw a spear.
You adjust your grip.
Throw again.
You change your stance.
Throw again.
Gradually, your nervous system begins to distinguish useful changes from useless ones.
You do not need perfect knowledge.
You need feedback.
Repeated feedback allows the mind to separate signal from noise.
Much of human intelligence appears to rely on precisely this process.
We improve because the world lets us compare similar situations many times.
Now compare this with the kinds of questions that dominate history.
Why did one civilization collapse?
Why did one company reshape an industry?
Why did one scientific revolution occur when it did?
These are fundamentally different problems.
There is no second Roman Empire against which to compare the first.
There is no alternative version of history in which every condition remained identical except one.
There is no opportunity to repeat the twentieth century while changing only the invention of the transistor.
History presents us with outcomes.
It almost never presents us with controlled comparisons.
Yet our minds continue searching for them.
Perhaps this is because the same learning machinery that served us so well in repeatable environments continues operating when repetition disappears.
It searches for the decisive variable.
The defining habit.
The crucial decision.
The single turning point.
Not because reality necessarily contains one.
But because repeated learning trained us to expect that careful observation eventually reveals stable causes.
In many areas of life, that expectation is justified.
In history, it becomes far more difficult.
This is not because history is irrational.
Nor because causes do not exist.
It is because historical causes exist within systems that cannot be reconstructed in full.
Thousands of variables move together.
Technology changes.
Institutions evolve.
Individuals mature.
Cultures transform.
Unexpected events intervene.
Some influences become visible only decades later.
Others remain invisible forever.
The mind still attempts to isolate causes.
Reality no longer cooperates.
This creates a subtle illusion.
When enough information is missing, the explanation that feels most coherent often appears to be the explanation that is most complete.
The two are not the same.
Coherence is a property of stories.
Completeness is a property of evidence.
One should never be mistaken for the other.
This distinction becomes especially important because successful explanations often become simpler over time.
The complexity of an event gradually disappears beneath repeated retelling.
The founder becomes a visionary.
The empire becomes corrupt.
The invention becomes inevitable.
Each retelling removes details that appear unimportant.
Each simplification makes the story easier to remember.
Each simplification also removes pieces of the causal landscape.
Eventually the explanation becomes elegant.
It may also become misleading.
The irony is difficult to escape.
The better a story survives, the less information it often contains.
This is not a flaw of historians.
Nor of journalists.
Nor of memory itself.
Compression is unavoidable.
No mind can preserve reality in its full complexity.
The problem begins only when compressed representations are mistaken for complete representations.
The distinction is easy to state.
It is extraordinarily difficult to remember.
Perhaps that is why intelligent people disagree so passionately about history.
Not because they possess different facts.
But because the missing facts are larger than the surviving ones.
When the unseen exceeds the seen, confidence should become smaller.
Instead, it often becomes larger.
That inversion may be one of the defining features of human reasoning.
III. The Architecture of False Certainty#
If the argument so far is correct, then the consequences reach far beyond history.
The same mistake should appear wherever people attempt to explain unique events using incomplete evidence.
Once you begin looking for it, it becomes difficult not to see.
Consider the modern obsession with biographies.
Every generation searches for the habits of extraordinary people.
The books change.
The individuals change.
The questions remain remarkably similar.
What made them successful?
What separated them from everyone else?
What should we copy?
The assumption beneath these questions is rarely examined.
It assumes that one realized life contains enough recoverable information to identify the decisive causes of its outcome.
Perhaps it does.
Perhaps it does not.
The difficulty is that we have no access to the lives that never occurred.
We do not observe the equally talented founder born twenty years earlier.
Or the one who became ill before finishing the company.
Or the one whose first investor declined the meeting.
Or the one who built the better product but entered the market after it had already consolidated.
History gives us survivors.
Reality contained possibilities.
The two are not identical.
This does not mean biographies are useless.
It means we should understand what kind of knowledge they provide.
A biography is not an experiment.
It is one realization from an immense space of unrealized alternatives.
Its greatest value may lie not in proving causes, but in expanding our imagination about what is possible.
The same principle appears in business advice.
A founder explains why a company succeeded.
Another founder explains why a different company failed.
Each account feels persuasive.
Each is internally coherent.
Yet coherence alone cannot establish causality.
Even when many founders agree, another problem appears.
Agreement is not explanation.
Suppose a hundred successful founders share one habit.
What exactly have we learned?
Have they succeeded because of that habit?
Despite it?
Would they have succeeded without it?
Did the habit matter equally in every case?
Was it decisive for ten founders and almost irrelevant for ninety?
Would it produce the same result under today’s technological, economic, and cultural conditions?
The pattern is real.
Its meaning remains uncertain.
Resemblance is not replication.
This distinction deserves more attention than it receives.
We often treat repeated historical patterns as though they were repeated experiments.
They are not.
No two companies begin under identical conditions.
No two civilizations inherit the same institutions.
No two elections involve the same electorate.
Similarity increases confidence.
It does not eliminate uncertainty.
The same caution applies to politics.
Political disagreement often appears to concern values.
Sometimes it concerns evidence.
More often, I suspect, it concerns missing evidence.
Every election emerges from millions of interactions.
Economic conditions.
Cultural shifts.
Media environments.
Institutional incentives.
Individual personalities.
Unexpected events.
Long historical trajectories.
No observer can recover every relevant cause.
Yet political explanations frequently become more confident precisely where recoverable information becomes scarcer.
The same pattern appears in financial markets.
After every major movement, explanations quickly emerge.
Interest rates.
Innovation.
Consumer confidence.
Government policy.
Geopolitics.
Each explanation may contain part of the truth.
None has access to the complete causal landscape that produced the event.
Markets move once.
Explanations arrive afterward.
The order matters.
Perhaps nowhere is this tendency more personal than in our understanding of ourselves.
Human beings are natural storytellers.
We explain who we are by constructing narratives from remembered events.
This is unavoidable.
Memory itself is selective.
But a remembered life is not the same thing as a lived life.
We remember turning points.
We forget ordinary days.
We explain ourselves through the fragments that survived.
The self, in this sense, is partly an interpretation built upon incomplete evidence.
This is not a defect.
It may be unavoidable.
The point is not that our stories are false.
The point is that they are necessarily incomplete.
The same structure appears everywhere.
History.
Business.
Politics.
Markets.
Relationships.
Identity.
Different subjects.
The same epistemic problem.
Reality generates more causal information than observers can preserve.
Everything that follows is an attempt to reason from what remains.
Recognizing this does not weaken knowledge.
It changes the kind of confidence knowledge deserves.
Perhaps wisdom is not the accumulation of ever more convincing stories.
Perhaps it is learning to recognize the difference between an explanation that feels complete and one supported by all the information reality has left us.
Those are not always the same thing.
IV. A Different Way of Knowing#
If the difficulty is not that history lacks meaning, but that it preserves only fragments of the information that created it, then the question becomes practical.
How should we reason under these conditions?
The obvious answer is humility.
It is also insufficient.
Humility tells us to be cautious.
It does not tell us how to think.
What is needed is not less explanation.
It is better calibration between explanation and evidence.
The mistake we repeatedly make is allowing confidence to exceed what the available information can reasonably support.
This suggests a simple principle.
Confidence should never exceed the resolution of the evidence.
By “resolution,” I do not mean certainty in the mathematical sense.
I mean something simpler.
Every explanation is built from a certain amount of recoverable causal information.
Some explanations are built upon evidence that can be observed repeatedly, challenged, tested, and refined.
Others are built from singular events that left behind only partial traces of the processes that created them.
These two kinds of explanations should not command the same degree of confidence.
The distinction seems obvious once stated.
Yet we routinely ignore it.
Consider again the biography of a successful founder.
Most readers ask:
“What can I learn from this?”
A more careful question would be:
“What kind of evidence is this?”
It is one realized trajectory.
It contains insight.
It also contains irreducible uncertainty.
The goal is therefore not to dismiss the story.
The goal is to classify it correctly.
This way of thinking gradually changes the questions we ask.
Instead of asking:
“What caused this success?”
We begin by asking:
“How much of the causal landscape can we actually observe?”
Instead of asking:
“What is the lesson?”
We ask:
“How much confidence does this lesson deserve?”
Instead of asking:
“Did this pattern appear before?”
We ask:
“Did it appear independently, across genuinely different conditions?”
Even then, another difficulty remains.
Patterns themselves can mislead.
Suppose a characteristic appears repeatedly across successful people.
At first glance, this seems persuasive.
Yet repetition alone answers surprisingly few questions.
It does not tell us whether the characteristic produced success.
It does not tell us whether success occurred despite the characteristic.
It does not tell us whether the characteristic mattered equally in every case.
Nor does it tell us whether the conditions that once made the characteristic valuable still exist.
A repeated pattern is therefore not the end of inquiry.
It is the beginning.
Its appearance justifies attention.
Not certainty.
This changes the purpose of comparison itself.
We no longer compare events in order to discover formulas.
We compare them in order to identify principles that remain recognizable despite enormous differences in context.
A principle that survives across centuries, cultures, industries, technologies, personalities, and institutions deserves more confidence than one that appears only within a narrow historical setting.
Even then, confidence should remain proportional to the evidence available.
Not because knowledge is impossible.
But because reality often withholds the very information that would allow certainty.
This way of thinking also changes how we understand disagreement.
Two intelligent people may study the same historical event and arrive at different conclusions.
Ordinarily we assume one of them must have misunderstood the evidence.
Sometimes that is true.
Sometimes both are reasoning from evidence that is fundamentally incomplete.
Where the surviving information is sparse, disagreement should not surprise us.
It should be expected.
Indeed, complete agreement in such circumstances might deserve more suspicion than disagreement itself.
The aim, then, is not to eliminate uncertainty.
That would require recovering information reality has already discarded.
The aim is something more modest.
To discipline our confidence according to the structure of the evidence itself.
There is a quiet consequence to this way of thinking.
Many of the questions that dominate public life may simply admit less certainty than we demand from them.
Why did Rome fall?
Why did one company reshape an industry?
Why did one marriage endure while another failed?
Why did one election transform a nation?
These are meaningful questions.
But meaningful questions are not necessarily questions that permit precise answers.
Perhaps the highest form of intellectual discipline is not producing increasingly confident explanations.
Perhaps it is recognizing when reality has left us too little information to justify them.
Knowledge does not become weaker by admitting its limits.
It becomes more honest.
V. The Information We Never Get
There is a temptation, after an argument like this, to become skeptical of everything.
If history preserves only fragments, perhaps we cannot know anything.
That is not the conclusion.
The point is not that knowledge is impossible.
It is that different kinds of evidence deserve different kinds of confidence.
A laboratory experiment and the rise of an empire are not epistemically identical.
A clinical trial and a human life are not the same kind of observation.
A founder’s biography and a law of physics do not deserve the same certainty.
We often recognize these differences intuitively.
What we fail to do is allow those differences to discipline our confidence.
That failure has consequences.
It encourages hero worship.
It encourages political certainty where evidence is thin.
It encourages business folklore disguised as universal principle.
It encourages us to mistake elegant stories for complete explanations.
Most importantly, it encourages us to forget how much of reality has disappeared before we ever began asking questions.
Perhaps every explanation should begin with a different question.
Not:
“What happened?”
But:
“What information no longer exists?”
That question does not weaken explanation.
It protects explanation from becoming more confident than its evidence permits.
It reminds us that every event is surrounded by invisible alternatives we can never fully recover.
The conversation that almost happened.
The illness that never came.
The investor who almost declined.
The technology that arrived five years too early.
The regulation that was never passed.
The friendship that was never formed.
The war that almost began.
These unrealized possibilities are not fictional.
They are part of the causal landscape from which reality selected one path.
History remembers the path.
Reality contained the landscape.
The distinction matters.
Our civilization increasingly rewards certainty.
Algorithms reward certainty.
Political movements reward certainty.
Markets reward certainty.
Public debate rewards certainty.
Confidence is often mistaken for competence.
Yet confidence and knowledge are not the same thing.
Confidence is a psychological state.
Knowledge is constrained by evidence.
The two should never be confused.
If there is one habit worth cultivating, it is this:
Whenever an explanation feels complete, ask what information had to disappear before that explanation became possible.
Whenever a lesson feels universal, ask whether it emerged from repeated evidence or from a single surviving story.
Whenever a pattern appears obvious, ask whether it represents genuine causation or merely repeated resemblance.
These questions do not make us less capable of understanding the world.
They make us more careful about the difference between what the world reveals and what we infer.
Perhaps that difference has always existed.
Perhaps we simply underestimated its importance.
I began with a simple observation.
Every explanation assumes that the information available to us is sufficient to explain the event we are trying to understand.
I no longer believe that assumption deserves to go unquestioned.
Reality does not preserve itself for our convenience.
It leaves traces.
Fragments.
Residues.
From those fragments we build histories, theories, identities, and civilizations.
That is an extraordinary achievement.
It is also an invitation to humility.
Not because our explanations are worthless.
But because they are always explanations of the information that survived, never of reality in its entirety.
There is a final irony.
The more successful an explanation becomes, the more it tends to be repeated.
The more it is repeated, the more its complexity is compressed.
The more its complexity is compressed, the easier it becomes to mistake it for certainty.
Perhaps wisdom consists in resisting that compression.
In remembering that behind every elegant explanation lies a world that was richer, more contingent, and more complex than any story can preserve.
We do not reason from reality itself.
We reason from what reality leaves behind.
Everything else follows from that.