Causal Inference Analysis is the part of measurement most teams skip. The standard advice for business owners and developers is to open the Google Analytics chart and see whether the line went up. That is storytelling with a chart attached. If you have ever had to explain why a feature “didn’t work” or why traffic spiked, you already know how messy the raw data gets, and that a line chart hides both selection bias and confounding variables.
I have watched dozens of projects where a client was certain a new plugin caused a 10% jump in conversions, only for it to turn out to be a seasonal trend or an email campaign that went out the same week. Correlation gets you to the wrong answer confidently. What you want instead is human-centered data analytics, aimed at the why behind the what.
The problem with naive estimates in causal inference analysis
When you measure the impact of an event, say a tube strike in London or a major site update, the naive approach is to subtract the average before from the average after. For the London tube strikes, that naive difference shows a +5.5% increase in bike usage. It is also wrong, because areas with tube stops are busier to begin with. That is selection bias.
What a real Causal Inference Analysis isolates is the treatment effect. The question is what bike usage would have been if the strike had never happened. That is the counterfactual, and without it you are reading noise.
Implementing Two-Way Fixed Effects (TWFE)
The usual tool for this is the Two-Way Fixed Effects (TWFE) model. It works on panel data, meaning you observe the same subjects across multiple time periods, and it lets you control for fixed effects on both axes: location characteristics, because some areas are always busier, and time characteristics, because weekends are always different.
In Python this goes through the statsmodels library. The naive version comes first, so the contrast with the fix is visible.
# The Naive Approach (Don't do this for critical decisions)
naive_diff = treated_mean - control_mean
print(f"Naive impact: {naive_diff:+.1f}%")That version conflates the strike effect with the location effect. Separating them takes a regression that accounts for both the cell (location) and the date. Doing so demeans the data within each group, which strips out the time-invariant advantages.
import statsmodels.formula.api as smf
# A robust TWFE model for Causal Inference Analysis
# C(h3_cell) controls for location; C(date_str) controls for time
model = smf.ols(
formula="log_trips ~ treated + temperature + precip + C(h3_cell) + C(date_str)",
data=df
).fit(cov_type="cluster", cov_kwds={"groups": df["h3_cell"]})
print(model.summary())Why clustering errors matters
Run standard OLS on this and your p-values come back far too optimistic, because observations from the same location on different days are correlated with each other. It is one of the more common data modeling mistakes. Clustering the errors at the cell level allows arbitrary correlation inside a cell, and the confidence interval you get back is one you can believe.
In the London tube strike study, that rigor dropped the estimated impact from a naive 5.5% to a statistically significant 3.95%. Smaller number, but it is a real one, and it is the one you would bet a budget on.
Assumptions and SUTVA gotchas
Then check your assumptions, because a Causal Inference Analysis that skips them is not finished. The one people trip over is the Stable Unit Treatment Value Assumption (SUTVA). If a tube strike in one area pushes people to bike into another area, the units are not independent. That normally pulls the estimate toward zero, so the +3.95% is probably a conservative floor.
If this Causal Inference Analysis work is eating your dev hours, hand it over to me. I have been wrestling with WordPress and complex data pipelines since the 4.x days.
The pragmatic takeaway
Raw averages are not worth trusting. Whether the subject is WooCommerce performance or urban transport trends, Causal Inference Analysis exists to isolate the mechanism. Going from a cross-sectional ML estimator to a fixed-effects panel regression is what decides whether the effect you found is real or an artifact of who happened to be in which group. The Statsmodels documentation covers the modeling side, and The Effect Book covers the fixed-effects theory.
