Counterfactual Analysis: Five Methods and Bet Logs for Product Teams
Learn how founders and product teams use five counterfactual methods, test pricing with a worked example, and log confidence to separate skill from luck.
On this page
- What is a counterfactual, and why can’t we just observe it?
- Which methods estimate counterfactuals in practice?
- How do you estimate a counterfactual step by step?
- How do DAGs help you choose what to control for?
- What does a full workflow look like on a pricing test?
- How do you read results without overclaiming?
- Why logging bets makes counterfactual work honest
- Keep the receipts on every bet your team makes
- FAQ
- Sources
A counterfactual is the outcome that would have occurred without the intervention, and estimating it is how you turn observed outcomes into causal claims. In potential-outcome terms, you compare Y(1), what happened with the change, against Y(0), what would have happened without it. Impact is the observed result minus that counterfactual, so build a credible comparison before you judge any outcome.
TL;DR:
- Randomize exposure when possible; use difference in differences only with parallel trends, regression discontinuity around real cutoffs, and matching only for observed traits.
- Before launch, define one outcome, record your hypothesis and kill criterion, collect baseline covariates, then test balance and design specific assumptions.
- Use a causal graph to adjust for shared causes of treatment and outcome, but exclude variables changed by treatment and colliders that can create false associations.
- Do not treat before and after or enrolled versus unenrolled differences as causal effects, because background trends and self selection distort those comparisons.
- Interpret average effects as group level, not customer guarantees; if the confidence interval includes zero, the estimated lift is not established.
What is a counterfactual, and why can’t we just observe it?
The potential-outcomes framework gives every unit two outcomes: one under treatment and one under control. The fundamental problem of causal inference is that you can never see both for the same unit. A customer either saw the new pricing page or didn’t. Because of that, teams rely on group averages and must build a statistically comparable control group. The potential-outcomes chapter of a causal inference textbook shows why randomized assignment makes those averages trustworthy.
In impact evaluation, the counterfactual is the outcome you would have seen without treatment, and establishing a valid one is the crux of rigorous work. That usually requires a theory of change and an understanding of how your data got generated: who got the treatment, and why.
A valid counterfactual group should be:
- Alike on everything that affects the outcome before treatment
- Exposed to the same background events and seasonality
- Different only in the treatment itself, like a clone without the change
Which methods estimate counterfactuals in practice?
Each design needs one core assumption. Match the method to what your operations allow.
- Randomized controlled trial (RCT): Randomization removes selection bias, so a simple difference in averages identifies the effect. Best for onboarding flows, emails and pricing displays.
- Difference-in-differences (DiD): Compare the before/after change in a treated group with the change in a comparison group. It needs parallel trends: both groups would have moved together without treatment. Combination designs like DiD help when a pure experiment is infeasible.
- Regression discontinuity (RD): Works when a rule assigns treatment at a cutoff, such as a usage threshold for a feature. Customers just above and below are nearly clones.
- Instrumental variables (IV): Uses something that nudges treatment but affects the outcome only through it, such as a randomized promotion.
- Matching and propensity scores: Pair treated and untreated units with similar observable traits. Useful when you only have historical data, but it cannot fix unobserved differences.
The Impact Evaluation in Practice handbook lays out this full menu. Tradeoffs come down to feasibility, sample size and data quality: small teams often lack the volume for subtle effects.
Pro Tip: Choose the design before you see results. Switching methods after the data arrives is how motivated reasoning sneaks in.
How do you estimate a counterfactual step by step?
- Define the treatment and the single outcome metric, then log your hypothesis and how confident you are that conversion will rise.
- Pick the identification strategy your constraints allow. Randomize if you can control exposure. Use DiD if you have a comparable untouched group. Use RD if a cutoff exists. Use matching only as a last resort.
- Collect baseline data and covariates, and confirm common support, meaning treated and comparison units overlap on key traits.
- Run validation checks: pre-treatment balance, a parallel-trends check for DiD, placebo cutoffs for RD, and falsification tests on outcomes the treatment should not move.
- Compute the effect, report it with a confidence interval, and close with a post-mortem that separates skill from luck.
Pro Tip: Write down what result would prove you wrong before launch. A kill criterion set in advance cannot be rewritten in hindsight. Our guide to outcome-blind reviews covers that last step in depth.
How do DAGs help you choose what to control for?
A directed acyclic graph (DAG) draws your causal assumptions as arrows between variables. It is a standard tool for telling good controls from bad controls, which prevents common adjustment errors. Structural models and graphs also yield formal rules, such as the backdoor criterion, that identify an adjustment set under which conditional regressions recover average causal effects.
Keep these rules in mind:
- Adjust for common causes of treatment and outcome, such as customer size or acquisition channel.
- Never adjust for anything the treatment itself changes, like post-signup engagement, when estimating the total effect.
- Avoid colliders, variables caused by both treatment and outcome. Conditioning on them can create fake associations, and graphical criteria and M-bias examples show how easily it happens.
- When a variable’s role is unclear, draw the graph first and argue about it before running any regression.
What does a full workflow look like on a pricing test?
Say a founder suspects that adding an annual-billing option will lift paid conversion. The example below is illustrative.
- The bet: Annual billing raises paid conversion among new trials. Primary metric: trial-to-paid rate. Confidence logged before the test. Kill criterion: no measurable lift after the planned sample.
- The design: Promotion happens by sign-up date, so the team randomizes which new trials see the annual option. Randomization avoids the selection problem of comparing customers who chose annual with those who didn’t.
- Validation checks: Trial size, region and channel are balanced across groups before launch. A falsification test confirms that trial activity in the first hour, which the option cannot affect, is similar in both groups.
- The estimate: Suppose the lift is positive, but the confidence interval includes zero. The honest reading: a small positive effect is plausible, but it is not established.
- The post-mortem: The team records that the 60% estimate was somewhat optimistic, notes that the test was underpowered, and updates how it sets future probabilities.
How do you read results without overclaiming?
An average treatment effect describes the group, not every customer, and a confidence interval tells you how much to trust its size. Check for heterogeneity, but treat subgroup findings as new hypotheses, not conclusions. External validity matters too: a result from one market or season may not carry to another.
The most common errors are false counterfactuals:
- Before/after comparisons blame the change for everything else that happened. Practitioners call these counterfeit counterfactuals because background trends get credited to your intervention.
- Enrolled vs. not-enrolled comparisons mix the effect with who chose to enroll. Use randomization or a quasi-experimental design instead.
Course materials on impact evaluation flag before/after and enrolled-versus-not-enrolled comparisons as common false counterfactuals, which can produce biased estimates.
If your evaluation needs go beyond what a small team can run, specialists such as monitoring and evaluation services can help with measurement workflows.
Why logging bets makes counterfactual work honest
Most bad causal stories start after the result arrives. Once you know the outcome, every explanation feels obvious. Writing the hypothesis, the probability and the kill criterion down first removes that wiggle room. It also forces the design question early: if you can’t say what the comparison group is, you aren’t ready to run the test. My favorite ritual is a premortem: ask the team to imagine the test failed and list why. It surfaces confounders while you can still fix them.
— Cesar
Keep the receipts on every bet your team makes
Counterfactual thinking only pays off if the reasoning survives until the results come in. We built Betlog so your team can record each hypothesis, confidence level, metric and kill criterion before launch, then close the bet with a post-mortem that separates skill from luck. Over time you can compare your stated confidence with how often your bets win. Start with our product page or check pricing.
FAQ
What is an example of a counterfactual?
A counterfactual is what would have happened had things gone differently, such as what a customer’s conversion would have been without the new pricing page. In analysis, you estimate it with a comparable control group, since you can never observe both outcomes for one customer.
Is counterfactual thinking good or bad?
It is both. Psychology research describes counterfactual thought as a process used to explain the past and prepare for the future, and it can improve performance. It turns harmful when it becomes hindsight storytelling that ignores luck.
What does “counterfactual” mean?
It means contrary to fact: a statement about a situation that did not occur. In causal analysis, it is the outcome a unit would have had without the treatment, which is the baseline for measuring impact.
How is counterfactual analysis used in machine learning?
It asks what minimal change to an input would flip a model’s decision, which supports interpretability. Microsoft documents this as what-if analysis for model decisions.
