Most engineering organizations don't think of production changes as experiments.
They think of them as deployments.
That's a subtle distinction.
And I think it's costing us.
Before a scientist runs an experiment, they usually write down a hypothesis.
Not because they're required to.
Because without a prediction, it's difficult to know what was actually learned.
A typical experiment asks:
Now compare that to many production deployments.
A change request is opened.
Someone reviews the code.
CI passes.
The deployment succeeds.
Then everyone moves on.
Notice what's missing.
The prediction.
Suppose an engineer says:
> This change should improve performance.
What does that mean?
By how much?
For whom?
Under what workload?
Compared to what baseline?
Without explicit predictions, almost any outcome can be interpreted as success.
That isn't engineering.
That's hindsight.
Whether it's documented or not.
Consider a simple database index.
The implicit hypothesis is:
> This index will reduce query latency more than it increases write overhead.
Or a caching layer:
> The reduction in backend load will outweigh the operational complexity.
Or introducing a message queue:
> The benefits of decoupling exceed the additional failure modes.
These aren't implementation details.
They're hypotheses about how the system behaves.
Imagine six months after a deployment.
Someone asks:
Why did we introduce Redis?
You find the pull request.
You find the Terraform.
You find the Helm chart.
You find the deployment logs.
What you often don't find is:
> What did we expect Redis to accomplish?
That's the expensive part.
The code survived.
The hypothesis didn't.
Imagine every production change included three additional fields:
Prediction
Average API latency will decrease by approximately 20%.
Confidence
Moderate.
Evidence That Would Change Our Minds
If cache hit rates remain below 40%, or write amplification increases database latency beyond acceptable thresholds, we should reconsider this design.
Now the deployment has become an experiment.
Reality can answer it.
Most postmortems reconstruct what happened.
Good ones reconstruct why the team expected something different.
That's impossible without documented predictions.
A prediction creates an objective comparison:
Expected.
Observed.
Difference.
Learning.
Without the prediction, postmortems become stories.
With it, they become science.
As AI proposes increasingly sophisticated infrastructure changes, organizations will need a way to evaluate more than implementation quality.
They'll need to evaluate reasoning.
Predictions become one of the most effective ways to do that.
If an AI consistently makes accurate predictions, confidence grows.
If its predictions repeatedly diverge from reality, the organization knows exactly where improvement is needed.
This is a much stronger feedback loop than simply counting successful deployments.
The best engineering organizations don't win because they avoid mistakes.
They win because every deployment leaves behind more understanding than existed before it.
That's what experiments do.
They don't merely change systems.
They change models.
Every production change is already a hypothesis about reality.
The question is whether your organization is treating it that way.
Because if you're not recording predictions, comparing them to outcomes, and updating your understanding, you're not getting the full value of every deployment.
You're changing software.
But you're leaving knowledge on the table.