Bayes & statistics

Causal inference: separating effects from associations

People receiving training perform worse than others. Did training cause harm, or were difficult cases more likely to receive it? A worked example shows why a raw comparison can mislead.

  • 3 min read
  • Updated on
  • smartlytics editorial team

The short answer

Causal inference asks how an outcome would change under an intervention. An observed difference between groups answers that question only under additional assumptions about common causes and comparable treatment options.

Explore the interactive exampleFree · in your browser · no sign-up

At a glance

  1. An association between an intervention and an outcome is not yet its causal effect.
  2. Different group composition can reverse a positive effect in the overall average.
  3. Standardisation helps only when the chosen variables form an appropriate adjustment set.

Training participants perform worse than nonparticipants. Does that mean the training was harmful? Not necessarily. Perhaps the most difficult cases were preferentially assigned to it. The causal question is: How would the same eligible cases have performed with and without training? Both outcomes cannot be observed for the same case. A comparison therefore needs a design and defensible assumptions.

The counterfactual behind the question

Write the potential outcome with training as Y(1)Y(1) and without training as Y(0)Y(0). For a specified target population, one quantity of interest is the average effect

ATE=E[Y(1)−Y(0)].\mathrm{ATE}=\E[Y(1)-Y(0)].

This differs from the observed gap between people who actually received training and those who did not. Hernán and Robins develop causal questions through such potential outcomes and the conditions that identify them (Hernán & Robins 2020). Here, identification means that the quantity can be determined from observable information under the assumptions, not merely estimated precisely.

A fictional example with a reversed sign

Half our population consists of easy cases and half of difficult cases. Without training, success rates are 70% and 20%, respectively. Training improves both groups by ten percentage points, to 80% and 30%. The model’s true effect is therefore positive and equal in both groups.

Now assign training to 80% of difficult cases but only 20% of easy ones. Difficult cases consequently account for 80% of trainees. Their combined success rate is 0.8⋅0.3+0.2⋅0.8=0.400.8\cdot0.3+0.2\cdot0.8=0.40. Easy cases dominate the untrained group, whose success rate is 0.2⋅0.2+0.8⋅0.7=0.600.2\cdot0.2+0.8\cdot0.7=0.60.

The raw gap is minus 20 percentage points, although training produces plus ten percentage points within each difficulty group. This is a form of Simpson’s paradox. It uses exact expected proportions, not a noisy small sample or real employee data.

Which variable creates the distortion?

The assumed graph contains three arrows: difficulty affects training, difficulty affects success, and training affects success. Difficulty is therefore a common cause of intervention and outcome, or a confounder. The raw comparison mixes the training effect with differences in group composition.

Pearl’s account of causal models explains why assumptions about the data-generating process are needed in addition to probability calculus (Pearl 2009). A graph is not proof automatically extracted from a correlation matrix. It makes substantive assumptions visible so that domain experts can justify or challenge them.

Standardising to the same population

Instead of using the groups’ differing observed composition, weight both treatment states using the same target-population proportions:

Δ=∑zP(Z=z)[E(Y∣A=1,Z=z)−E(Y∣A=0,Z=z)].\Delta=\sum_z P(Z=z)\big[\E(Y\mid A=1,Z=z)-\E(Y\mid A=0,Z=z)\big].

In the example this gives 0.5(0.8−0.7)+0.5(0.3−0.2)=0.100.5(0.8-0.7)+0.5(0.3-0.2)=0.10. The adjusted difference matches the causal effect by construction. The calculator changes allocation, effect and the share of difficult cases. With random allocation at 50% within each group, the raw difference also matches the effect.

Interactive

Interactive: Causal inference: separating effects from associations

The defaults give a raw difference of −20 percentage points and a standardised difference of +10 points. The table shows how group composition reverses the sign.

Why adjustment is not an automatic solution

A causal interpretation requires three central conditions here. Consistency: training denotes a sufficiently well-defined intervention. Conditional exchangeability: within difficulty groups, no further common causes remain unaccounted for. Positivity: both treatment options are genuinely represented in each relevant group.

These conditions hold by construction in the demonstration. Real observational data offers no such guarantee. Difficulty may be incompletely measured, motivated people may select themselves into training, or one group may never receive it. More data does not remove those identification problems. A narrow confidence interval can still surround a biased number.

Preparing a business investigation

Specify the target population, actual intervention, comparison and outcome window first. Then draw the suspected common causes. Where feasible, a properly planned randomised A/B test provides a more direct comparison. With observational data, assumptions, competing explanations and sensitivity checks belong alongside the conclusion.

Do not confuse this demonstration’s known effect with evidence about a real training programme. Its purpose is to show how identical within-group improvements can coexist with a misleading overall result. A real study must establish why its comparison is credible before calculating an effect.

Frequently asked questions

Why is a before-and-after comparison insufficient?

Other influences may change at the same time. A causal interpretation needs a clear account of what would have happened without the intervention and assumptions allowing that comparison (Hernán & Robins 2020).

Should we control for every available variable?

No. Some variables lie on the causal pathway or are common consequences of other variables. Adjusting for them can change the target effect or create bias. A causal model justifies the choice (Pearl 2009).

Sources and further reading

  1. PaperPearl, J. (2009): Causal inference in statistics: An overview.
  2. BookHernán, M. A. and Robins, J. M. (2020): Causal Inference: What If.

Content last reviewed on 14 November 2026. Please let us know about any errors: impressum@nyxai.com.

From NyxAI

smartlytics is the learning hub of NyxAI GmbH.

Explore the methods we work with, explained openly, with sources and calculations you can reproduce.

Decision Core

NyxAI is developing a Decision Core for Decision Intelligence that combines optimisation, simulation, statistics and reinforcement learning: compare options, make uncertainty visible and learn from outcomes, with people in control.

DeepSquare

Explainable chess AI with human playing styles: NyxAI’s testing ground for understandable decisions under time pressure.