High accuracy is quietly costing you money. The advice has always been to optimize for the lowest loss function and the cleanest confusion matrix, and yet those perfect models keep falling over the moment they reach production. It is rarely the data quality or the feature engineering. It is that there is no Causal Inference anywhere in the pipeline.
The accuracy trap: prediction is not a decision
Machine learning is built for pattern recognition and it is very good at spotting associations. Feed a model enough data and it will find every correlation in there. Stakeholders rarely want to know what will happen, though. They want to know what they should do, and answering that with an associational model is how you end up pulling levers that have no causal link to the outcome.
A health-tech startup I worked with recently built a readmission model that hit 94% accuracy. Textbook work. Then they used the predictions to prioritize follow-up calls and readmissions went up. The model had found a correlation between older patients and risk, and missed what sat underneath it: those patients had no way to get to their follow-up appointments. A phone call does not solve transportation.
I went further into why these systems break in an earlier post on Causal Inference in Data Science.
Judea Pearl’s ladder of causation
Getting out of that means moving up the ladder. Judea Pearl, a pioneer in the field, defines three rungs of reasoning:
- Level 1, association, or seeing. What happens if I observe X? Most standard ML sits here, and it is pure correlation.
- Level 2, intervention, or doing. What happens if I do X? Answering that means knowing how changing one variable moves the rest of the system.
- Level 3, counterfactuals, or imagining. What would have happened if I had acted differently? This is the hardest of the three.
Level 1 data will not answer a Level 2 or Level 3 question, and that is where most production portfolios come apart. There is more on running these environments in my post about Machine Learning Engineering and environment success.
Implementation: causal inference in Python
The tooling has caught up. Microsoft Research released DoWhy, a Python library that makes you write your assumptions down before you run a single regression. The workflow has four steps: model, identify, estimate, refute.
# Example Causal Inference Workflow with DoWhy
import dowhy
from dowhy import CausalModel
# 1. Model the causal assumptions
model = CausalModel(
data=df,
treatment="marketing_spend",
outcome="revenue",
common_causes=["seasonality", "competitor_action"]
)
# 2. Identify the causal estimand
identified_estimand = model.identify_effect()
# 3. Estimate the causal effect
estimate = model.estimate_effect(
identified_estimand,
method_name="backdoor.propensity_score_matching"
)
# 4. Refute the result (CRITICAL STEP)
refutation = model.refute_estimate(
identified_estimand,
estimate,
method_name="random_common_cause"
)
print(refutation)That last step is the one people skip. You are not hunting for a p-value, you are trying to break your own conclusion. If dropping in a random common cause moves your estimate, the estimate was never real.
If causal inference work is eating your dev hours, hand it over. I have been working with WordPress and messy data integrations since the 4.x days.
What to take away
If your intervention changes the relationships the model learned from, the correlations it found will not survive deployment. Accuracy tells you how well you captured a pattern. Causal Inference tells you whether the lever you plan to pull moves the outcome, which is the question the business asked in the first place.
