< section id="summary" class="level1">

Summary

Who this is for: Data scientists and analysts who need to estimate what an intervention does (a call, a discount, a campaign) and want to know when causal ML methods are worth using over LM, GLM, and GBM. The basis is nine simulations with a known true effect, each with 40 repetitions.

The five most important findings

1. Data first, algorithm second. If the model contains a wrong variable or omits an important influencing factor, all methods are equally wrong, including the most flexible ones. Only domain knowledge or an instrument helps here (Cases 4, 5).
2. For the average effect, causal ML wins only when influencing factors act nonlinearly. In that case the LM is clearly biased and Double ML is almost right. The difference comes from the construction principle (first adjust, then estimate), not from a particular algorithm (Case 3).
3. A risk model is not an effect model. Calling the customers with the highest cancellation risk achieves less than a random selection, because precisely these customers are hardly retainable any more (Case 7).
4. The correct ranking is easier than the correct magnitude. A simple GLM can select the right units and still misstate the size of the effect (Case 9).
5. The causal forest pays off when the size of the effect per unit matters and its form is unknown (Cases 8, 9). In simple worlds, a well-specified GLM is better (Case 6).

< section id="guide-which-method-when" class="level1">

Guide: Which Method When?

Work through the steps from top to bottom. Stop at the first point where the answer is “not possible”: no algorithm decides steps 1–3; only step 4 chooses the method. “Unit” stands for anything that is treated (customer, person, contract, branch); the examples in the cases are set at a fictional online fitness subscription service (an app with training plans and classes).

The terms used in the guide:

• A/B test: Who is treated was determined at random (technical term: randomized). Treated and untreated units are therefore comparable.
• No random assignment (called “non-random” for short in the table): The treatment followed rules or judgments; the data simply accumulated (technical term: observational data). Treated and untreated units already differ beforehand.
• Influencing variable (confounder): A variable that influences who is treated and the outcome. Measured means it is in the data. Not measured: It is missing, and no algorithm can replace it.
• Instrument: A random nudge that shifts the treatment but affects the outcome only through it (e.g. a randomly sent invitation). Allows an estimate even though a confounder is missing.
• Average: A single effect for all units together (technical term: ATE). Answer to “Does the intervention work on average?”.
• Ranking (targeting): Deciding whom to treat, e.g. the 30% with the largest effect. Only who comes first matters; a model may over- or underestimate the effect by the same amount for everyone.
• Effect per unit (CATE): The exact size of the effect for each unit, e.g. to weigh costs and benefits per unit. Stricter than the ranking: A model can rank correctly and still estimate the size wrongly (Case 9).

Blue = question, green = recommendation, red = stop.

< section id="step-1-what-is-the-decision" class="level2">

Step 1 – What is the decision?

< section id="step-2-how-was-the-treatment-assigned" class="level2">

Step 2 – How was the treatment assigned?

• At random (A/B test, lottery): Identification is secured, continue with step 4.
• By rules or by judgment of the business department (no random assignment): continue with step 3.
• Randomly assigned, but not everyone received the treatment (e.g. call assigned, customer not reached): The comparison of means by assignment measures the effect of the assignment. For the effect of the treatment itself, the assignment serves as an instrument (2SLS, Case 5 [5]). Not simulated.

< section id="step-3-identification-only-without-random-assignment" class="level2">

Step 3 – Identification (only without random assignment)

This is where most projects fail, and no algorithm helps.

Three terms decide which variables belong in the model. They differ only in where the variable sits in the cause-and-effect picture ( = treatment, = outcome, = the variable in question):

Rule of thumb: What exists before the treatment and co-determines it belongs in the model. What arises after the treatment does not. Whether a variable is a confounder or a mediator is not revealed by the data – it is domain knowledge about the process.

1. Which variables go into the model? What exists before the treatment and co-determines it goes in (confounder). What arises after the treatment stays out (mediator, collider). The data do not say which is which; this is domain knowledge. In Case 4 all algorithms are equally biased when the mediator is included. Overview of good and bad control variables: [19]. Tool: Draw the process as a DAG in DAGitty; the program names the required variables [26].
2. Is a confounder missing? Cannot be proven from the data. Hints come from the placebo test (an outcome that the treatment cannot influence must show effect 0; Case 5) and the E-value (how strong a missing confounder would have to be to explain away the effect) [20]. DoWhy runs such tests automatically [27].
3. If one is missing, is there an instrument? A random nudge that acts only through the treatment. Yes: 2SLS for a constant effect, Instrumental forest for a varying effect (Case 5). Validity cannot be checked from the data. No: no causal statement. Label the number as an association, not as an effect. Exception: If before-after data or a threshold exist, see difference-in-differences and regression discontinuity under “Further methods (not simulated)”.
4. Overlap: Are there treated and untreated units in every group? Check with the propensity histogram (section “Overlap diagnostic”). Propensities near 0 or 1: restrict the population to the region with overlap. AIPW and other weighting estimators otherwise become unstable (Case 3, †). The restriction itself is not simulated here.

< section id="step-4-choose-a-method" class="level2">

Step 4 – Choose a method

First find the data situation (column 2) and the question (column 3); the terms are defined at the beginning of the guide.

As a decision tree, from top to bottom (blue = question, green = recommendation, red = stop; Var. = variables). It presupposes that step 3 is passed (including overlap) and that the question concerns an effect, not pure risk. With little data (a few hundred units), row 2 of the table applies. The table additionally lists a cross-check and what to avoid:

< section id="step-5-check-without-knowing-the-truth" class="level2">

Step 5 – Check without knowing the truth

• A/B test: Qini/uplift curve and calibration on a hold-out dataset are valid.
• No random assignment: R-loss ranks the methods only as long as step 3 holds. In addition, placebo test and overlap histogram.
• Agreement of several methods proves nothing: In Cases 4 and 5 all methods agree and are wrong.
• State the uncertainty: An estimate should come with a confidence interval (Methodology, “Confidence intervals”). It measures only the random error: If step 3 is violated, even a narrow interval misses the target.

Details in the section “Evaluation without knowing the truth”.

< section id="step-6-when-in-doubt" class="level2">

Step 6 – When in doubt

Keep the simplest correct model as the baseline. Adopt the more complex method only if it is clearly better on validated data. In Cases 1, 2 and 6, complexity brings nothing.

< section id="other-decision-aids" class="level2">

Other decision aids

A fully automatic decision tree that chooses the right method from the data alone does not exist, and cannot exist: Whether a variable is a confounder or a mediator is not in the data (Case 4). Any automation therefore needs a cause-and-effect picture (DAG) based on domain knowledge as input. What does exist:

• DAGitty [26]: You draw the DAG; the program names the variables that must go into the model, and possible instruments. Automates step 3.1.
• DoWhy [27]: From the DAG through identification and estimation to automatic checks (placebo, random additional confounder, subsamples). Closest to an automatic workflow.
• EconML Library Flow Chart [28]: Decision tree for choosing the estimator, similar to the one in step 4. Additionally asks about compliance and whether the effect depends linearly on the features.
• Causal Roadmap [24] and Target Trial Emulation [25]: standardized workflows from epidemiology. You first describe the experiment you would have liked to run, and then check how close the data come to it.
• Automatic model selection for CATE (e.g. causaltune based on EconML): chooses the estimator via R-loss or similar measures. This works only if identification holds (see “Evaluation without knowing the truth”).

< section id="methodology" class="level1">

Methodology

This part describes what is estimated and with which methods, without simulation results: the core idea, the terms, the methods studied with their formulas, and the question of how to check a method on real data (where the true effect is unknown).

< section id="core-idea" class="level2">

Core idea

LM, GLM, and GBM are normally trained to predict Y. CausalML models are built to estimate the effect of a treatment T on Y, either on average (ATE) or per unit (CATE). A good prediction model is not automatically a good effect model. Conversely, an effect model is not automatically better when the world is simple.

In real data the true effect is never known. That is why we simulate here: we define the formulas that generate the data ourselves. We therefore know the true effect of every single unit and can measure each method against it exactly.

Classification following Molnar [6]: the report takes the causal inference mindset (what causes what?) and uses supervised learning (prediction models such as GBM) only as a building block, e.g. in Double ML. Evaluation is frequentist (bias and spread over repeated samples). Other mindsets lead to other tools: Bayesian to BART and Bayesian causal forest (uncertainty per unit as a posterior), reinforcement learning to bandits, which assign the treatment continuously and learn while doing so. That is a different type of data than the one used here. The core message in this language: a good supervised-learning model is not automatically a good causal-inference model. Introductions to the field: [1, 2, 3].

< section id="terms" class="level2">

Terms

Potential outcomes. Every unit has two possible outcomes: if it is treated () and if not (). Only one of them is ever observed. The individual effect is .

In the diagrams (DAGs) below, an arrow means: “A enters the formula of B”. Dashed = unobserved variable.

< section id="methods" class="level2">

Methods

All methods receive the training data as input and return an estimated effect for every test customer as output. Exception: the “pure” GBM, which ignores T and only predicts a risk.

xgboost and LightGBM: depth 3, learning rate 0.1, number of rounds via 5-fold CV with early stopping, otherwise library defaults (LightGBM: num_leaves = 8, min_data_in_leaf = 20).

< section id="methods-in-detail" class="level2">

Methods in detail

The methods from the S-learner onward, each with the idea, the formula, and the typical weakness. Common notation: is the expected outcome, the propensity (probability of being treated), and denotes estimated.

< section id="gbm-s-learner-single" class="level3">

GBM S-learner (“Single”)

A single model for all units, with as one feature among many. The effect is the difference in the prediction when is set from 0 to 1 and everything else is held fixed.

Weakness: If has only a small influence, the GBM barely uses it (regularization), and the effect is pushed toward 0. Under strong confounding a residual bias remains (Case 3). Source: [9].

< section id="gbm-t-learner-two" class="level3">

GBM T-learner (“Two”)

Two separate models: one only on the treated, one only on the untreated. The effect is the difference between the two predictions.

Weakness: Each model sees only part of the data, and the two errors add up. With small n or groups of unequal size this is very noisy (Case 1). The X-learner (see “Further methods”) addresses exactly this. Source: [9].

< section id="pure-gbm-ignores-t" class="level3">

Pure GBM (ignores )

An ordinary prediction model for without , e.g. a churn model. The score is the predicted risk.

Weakness: It answers “who is at risk?”, not “for whom does the treatment work?”. Units with high risk may be unpersuadable (Case 7). On the difference between risk and uplift models: [21].

< section id="double-ml" class="level3">

Double ML

Two nuisance models predict from what is to be expected without the treatment: the outcome and the assignment . What remains is only what does not explain (the residuals). The effect is the slope between the two residuals.

Cross-fitting: The nuisance models predict each unit using models that have not seen it (5 folds). Otherwise overfitting would distort the residuals. Why it works: Errors in and enter only as a product, so the form of the confounders can be learned freely without biasing the effect. Weakness: a single effect for everyone (the average), no effect per unit. Source: [7]; textbook: [2].

< section id="r-learner-double-ml-with-per-unit-effect" class="level3">

R-learner (Double ML with per-unit effect)

The same residuals as in Double ML, but the effect is no longer a single number but a function learned by an arbitrary model. It is chosen so that the residuals of are explained as well as possible by :

Implemented as a weighted regression: an xgboost predicts from , with weight . Weakness: Units with (propensity near 0 or 1) contribute almost nothing; with a constant effect the model learns heterogeneity that does not exist (more variance than Double ML). Source: [8].

< section id="causal-forest" class="level3">

Causal forest

A forest of trees that are split so that the effect differs as much as possible between the leaves (not the outcome). For a new unit , the forest counts which training units often lands in the same leaf with (weight ) and estimates the effect only from these similar units, again using the residuals as in Double ML:

Honest: One half of the data determines the splits, the other half estimates the effect. This prevents the forest from learning random fluctuations as an effect. Weakness: needs a lot of data to find fine differences; with a constant effect or a linear world there is no advantage. Sources: [10, 11]; honest trees: [12].

< section id="aipw-doubly-robust" class="level3">

AIPW (doubly robust)

Combines two models. The outcome model predicts both outcomes for every unit. The weighted correction offsets deviations at the units actually observed, with rare cases (small propensity) counting more.

Doubly robust: The result is correct if either the outcome model or the propensity model is correct. Weakness: If is near 0 or 1, the weights explode (Case 3, individual runs fail). Average effect only. Source: [13]; TMLE as a more stable variant: [17].

< section id="sls-instrument-z" class="level3">

2SLS (instrument )

If a confounder is unobserved, an instrument helps: it shifts but has no other path to . Stage 1 isolates the part of that explains. Stage 2 uses only this part, which is free of the unobserved confounder.

Without and with a binary , this is the Wald estimator: the effect of on , divided by the effect of on .

Weakness: Whether is valid cannot be tested from the data. If is weak (barely shifts ), the denominator is near 0 and the estimate is very uncertain. Source: [5].

< section id="instrumental-forest" class="level3">

Instrumental forest

The same principle as 2SLS, but local as in the causal forest: for each unit , a separate IV estimator is computed from the similar units (weights of the forest). The effect may therefore depend on .

Weakness: With a constant effect it is merely noisier than 2SLS (Case 5). Source: [11].

< section id="strengths-and-weaknesses-by-method" class="level2">

Strengths and weaknesses by method

Summary of the simulation results per method. Which method fits which situation is given in the guide; here we give the reasons.

< section id="evaluation-methods" class="level2">

Evaluation methods

In real data the true effect is unknown. What can be checked depends on whether the treatment was assigned at random.

With randomized data (e.g., a randomly treated share of a campaign), the difference in means is unbiased and serves as the benchmark. On a hold-out set, any effect model can then be checked from the outside:

• Calibration: Group units into deciles by predicted effect and compute the actual difference in means within each decile. If prediction and measurement agree, the model is usable (grf: test_calibration).
• Uplift or Qini curve: Do the top X % according to the model really have the higher effect? This is the practical version of the top-30% measure in this report [21].
• Comparison with chance: The benefit of selecting by model is compared with random selection and with “treat everyone”.

A permanently small random share (e.g., 5–10 %) costs something, but it is the only clean answer.

With observational data there is no such benchmark. The following tools narrow down errors but prove nothing, and none of them is recommended across the board: each works only under its own assumptions or is too noisy to decide on its own. It is best to combine several.

• Test on your own data (“plasmode”): keep the real features and the real assignment , regenerate with an effect of your own choosing, and measure the methods against it. This is what this report does on a small scale, only with the structure of your own data. The result holds only for the chosen effect form, however.

• Loss function on hold-out data: The R-loss [8, 22] (or the analogous DR-loss [14]) compares effect models without knowing the true effect. It is noisy and depends on the nuisance models : usable for discarding poor models, but for picking the best one only when the differences are clear (see Simulation: “Evaluation without knowing the truth”).

  Step 1: nuisance models. Two prediction models are learned on the training data, the outcome model and the treatment model (propensity).

  Step 2: residuals on the hold-out set. From and , subtract what can already be predicted from :

  

  What remains is only the part of not explained by and the part of that is “random”. If the treatment acts with , then approximately .

  Step 3: loss. An effect model is then judged by how well it fits this relationship:

  

  Smaller is better. For comparison, (no effect) gives . A method is useful only if it falls below this value. The loss needs neither the true effect nor randomization, but it presupposes that contains all confounders (otherwise Cases 4 and 5).

• Agreement of several methods: Similar results from LM, Double ML and causal forest are reassuring; clear deviations point to a wrong functional form or weak overlap. It is not proof (Cases 4, 5: all equally wrong).

• Overlap and balance: Histogram of the estimated propensity (values near 0 or 1 are a warning sign, Case 3) and comparison of the features of treated and untreated units after weighting. This checks only measured variables.

• Placebo test: The treatment is applied to an outcome it cannot affect (or to a period before it). An “effect” indicates remaining confounding. It requires a credible placebo outcome, which is rarely available.

• Sensitivity analysis (e.g., E-value, Rosenbaum): How strong would an unobserved confounder have to be to explain the effect away? This yields an order of magnitude, not a decision [20].

• Comparison with an experiment: If a small randomized test exists somewhere, the observational estimator is measured against it. This holds only if population and treatment are similar enough.

Not testable is whether all confounders are measured and no mediators are in the model. No quality measure detects this; predictive accuracy can even look good (see “Predictive accuracy is not effect accuracy”). This is decided by domain knowledge of the process, not by the method.

< section id="confidence-intervals" class="level2">

Confidence intervals

A confidence interval shows how much an estimate would vary from sample to sample. A 95 % interval is usually the estimate ± 1.96 standard errors (SE). It measures only random error, not systematic error: if a wrong variable is in the model or a driver is missing, a narrow interval will surely surround the wrong value (see Simulation, “Do the confidence intervals hold?”).

Often more useful for decisions than an interval per unit: the interval for a group, e.g., the 30 % with the largest estimated effect. To obtain it, estimate the model on training data, select the group on hold-out data, and estimate the effect there: in an A/B test as the difference in means within the group, without random assignment as the mean of the group’s AIPW scores (SE = standard deviation / ). Because selection and estimation happen on separate data, the interval is honest.

When the standard errors are too small: with clustered data (several observations per customer or per branch: cluster-robust SE or bootstrap whole groups), with time series and panels (block bootstrap), with weak overlap (AIPW SE unstable) and with a weak instrument (2SLS SE unreliable).

< section id="further-methods-not-simulated" class="level2">

Further methods (not simulated)

The simulated methods cover the most common families, but not all. The table maps each further method to a place in the guide: the column “Alternative to” names the simulated method it can replace, and the row of the table in Step 4. Most change only the estimation. Two intervene earlier: difference-in-differences and regression discontinuity make estimation possible even when a driver is missing, and causal discovery helps with the choice of variables. Everything in the table is literature knowledge, not tested here. With a wrong choice of variables (Step 3), the estimators fail just like the simulated ones.

< section id="simulation" class="level1">

Simulation

Here the methods are measured on nine simulated worlds in which the true effect is known. First the setup, then the cases in four blocks, then the overall comparison and the conclusion.

< section id="setup" class="level2">

Setup

< section id="how-are-methods-scored" class="level3">

How are methods scored?

For each case, a training dataset with new random numbers is generated 40 times. Each method is trained on it and compared with the known true effect on a separate test dataset (n = 5000).

What this simulation can and cannot show. The nine worlds are deliberately simple and chosen by the author, each to make one mechanism visible. The methods run with little tuning (xgboost and LightGBM with depth 3 and number of rounds chosen by cross-validation, grf with default settings). The ranking of the methods therefore holds for these worlds, not in general; what transfers are the mechanisms, that is, why a method fails. Every number is a mean over 40 repetitions, and the result tables report the Monte Carlo standard error. Differences below about twice that are to be read as equally good. All methods run on the same datasets, so differences between them are more precise than the individual means.

< section id="overview-of-the-cases" class="level3">

Overview of the cases

The cases are arranged in four blocks. A shows the simple world. B asks whether the effect can be estimated at all with the available variables (identification) – there, usually no algorithm helps, but knowledge about the variables does. C asks how flexible the model must be when the effect varies from customer to customer (estimation). D combines both: observational data with confounding and a varying effect.

Every case follows the same structure: story → DAG → formulas that generate the data → model input → true effect → why simple methods fail → correct estimation → results → lesson.

< section id="a-simple-world" class="level2">

A – Simple world

< section id="case-1-small-n-constant-effect" class="level3">

Case 1 – Small n, constant effect

Story. A pilot trial with only 200 customers, randomized. The world is linear and the effect is the same for everyone.

Data generation (n = 200, randomized):

Model input: .

True effect: for everyone, ATE = 1.

Results:

< section id="b-which-variables-belong-in-the-model" class="level2">

B – Which variables belong in the model?

< section id="case-2-confounding-linear" class="level3">

Case 2 – Confounding, linear

Story. Customer service preferentially calls dissatisfied customers (T = call). But dissatisfied customers also have a lower satisfaction score Y three months later even without a call. Prior satisfaction (X1) is therefore a confounder: it determines who is called, and it determines Y.

Data generation (n = 1000, observational data):

Model input: (all observed).

True effect: for every customer, so ATE = 2.

Why the naive comparison is wrong: . Called customers are on average 0.93 standard deviations less satisfied, so the naive comparison is ≈ 2 + 3 · (-0.93) = -0.81. The call appears to harm, although it helps.

Results:

< section id="case-3-confounding-nonlinear" class="level3">

Case 3 – Confounding, nonlinear

Story. As in Case 2, but the confounder acts nonlinearly. A motivational call (T) goes preferentially to very young and very old customers (X1 = age, standardized). Precisely these groups also train less without a call (Y = activity score, e.g., training minutes per week). The relationship is U-shaped: it depends on , not on .

Data generation (n = 1000, observational data):

Model input: (all observed; nothing is missing, only the form is unknown).

True effect: for everyone, ATE = 1.

Why the LM fails even though the confounder is included: lm(Y ~ T + X1 + X2) can only fit a straight line in . The U shape is not in but in , and called customers have on average a 1.18 larger . The LM attributes this difference to the call: coefficient of T ≈ 1 − 1.5 · 1.18 = -0.78 (n = 200,000). The call appears to harm.

Results:

< section id="case-4-bad-control-mediator" class="level3">

Case 4 – Bad control (mediator)

Story. A campaign (T) is sent out at random. It affects additional revenue Y (booked premium courses) in two ways: directly, and indirectly because customers who received the campaign open the app more often afterwards (M = number of app visits in the following three months). M is in the data and predicts Y extremely well. But M is a mediator: it arises only after T and is part of the effect.

Data generation (n = 1000, randomized):

Model input: . For comparison, LM and causal forest are also run with only (“w/o mediator”).

True effect: The question “Is the campaign worthwhile?” refers to the total effect: direct 0.5 plus via M 1 · 1 = 1.5 for all customers.

Why adjusting for M shrinks the effect: For fixed M, the model compares customers who received the campaign with those who did not at the same number of app visits. This cuts off the path through M, and only the direct effect 0.5 remains (simulated with n = 200,000: 0.49). Bias = −1.0: two thirds of the effect disappear.

Results:

< section id="case-5-unobserved-confounding" class="level3">

Case 5 – Unobserved confounding

Story. Customers can voluntarily take part in an 8-week training program (T). More motivated customers (U) are more likely to take part and also train more without the program (Y = activity score). U is not recorded anywhere. As part of an experiment, however, an invitation Z was sent at random, which strongly increases participation. Z is an instrument.

Data generation (n = 2000, observational data with randomized invitation):

Model input: for all standard methods; for 2SLS and the instrumental forest. U is never in the input.

True effect: for everyone, ATE = 1.

Why all standard methods fail: Participants are more motivated (higher U). Without U, no model can separate which part of the difference comes from the program and which from U. The naive comparison yields 2.68 instead of 1. More flexibility (GBM, causal forest) does not help, because the missing information is not in the data.

Why the instrument helps: Z is assigned at random, hence independent of U. It changes the participation rate from 0.26 to 0.74 and acts on Y only through T. The difference in Y between invited and non-invited customers, divided by the difference in participation rates, is the effect (Wald estimator): 0.97. The three conditions: Z affects T (testable), Z is random (here by design), Z acts only through T (not testable, an assumption).

Results:

< section id="c-when-does-flexibility-pay-off" class="level2">

C – When does flexibility pay off?

< section id="case-6-linear-heterogeneity" class="level3">

Case 6 – Linear heterogeneity

Story. A discount (T) is assigned at random. It works the more strongly, the more price-sensitive the customer is (X1). Y is revenue. X1 is an effect modifier; the effect changes linearly with X1.

Data generation (n = 2000, randomized):

Model input: .

True effect: between −0.5 () and 2.5 (), ATE = 1.

Results:

< section id="case-7-risk-uplift-binary" class="level3">

Case 7 – Risk ≠ uplift (binary)

Story. Retention calls (T) are assigned at random. Y = 1 means the customer stays. X1 is a cancellation risk score: the higher it is, the more likely the customer is to cancel the subscription. The call works only at medium risk (“persuadables”). High-risk customers leave anyway, and low-risk customers stay anyway. X1 is both a risk driver and an effect modifier.

Data generation (n = 3000, randomized):

Model input: .

True effect (on the probability scale): . It is a bell shape with maximum 0.27 at 0.15 and falls to ≈ 0 for high and low risk. ATE = 0.10.

Why the usual approach fails: A churn model (“pure” GBM) predicts the risk correctly and selects the 30% with the highest X1. Precisely there the effect is almost 0. This is an assumption of this case: if the effect increases with risk instead of lying at medium risk, the risk model works passably.

Results:

< section id="case-8-many-variables" class="level3">

Case 8 – Many variables

Story. Randomized campaign (T). There are 20 customer features. The campaign works only if two conditions hold simultaneously (X1 > 0 and X2 > 0). This is not known in advance. The baseline depends nonlinearly on X3–X6; X7–X20 are noise.

Data generation (n = 2000, randomized, 20 variables):

Model input: .

True effect: 2 for the quarter of customers with and , otherwise 0. ATE = 0.5.

Why it is difficult: The effect is an interaction of two variables, and which two they are is stated nowhere. A linear T x X cannot represent a “corner”. The GLM with all two-way interactions can approximate it roughly, but needs about 400 parameters to do so.

Results:

< section id="d-both-together" class="level2">

D – Both together

< section id="case-9-observational-data-and-varying-effect" class="level3">

Case 9 – Observational data and varying effect

Story. As in Case 3: the motivational call (T) goes preferentially to very young and very old customers (X1 = age), who also train less without a call (activity score Y, U-shaped). In addition, customers benefit to different degrees from the call, depending on a second quantity (X2, e.g., pre-existing conditions). X1 is the confounder, X2 the effect modifier.

Data generation (n = 2000, observational data):

Model input: (all observed).

True effect: , ATE = 1.

Why the GLM fails: As in Case 3, the U shape is in , and called customers have a 1.18 larger . lm(Y ~ T * X2 + X1) attributes this to the call: the effect at is -0.78 instead of 1. The slope in , by contrast, is captured correctly by the GLM (1.51 instead of 1.5), because is independent of the confounding. The GLM thus ranks customers well but gets the level wrong.

Results:

< section id="overall-comparison" class="level2">

Overall comparison

< section id="cate-error-across-all-cases" class="level3">

CATE error across all cases

How to read: The number in each tile is the method’s CATE-RMSE, i.e., the typical error per customer (0 = perfect). Because the errors lie on very different scales depending on the case (Case 7 measures probabilities, Case 2 score points), the color is relative: it shows how much larger the error is than that of the best method in the same case. Green = best method in the case, red = at least 4 times the error of the best. Empty = the method does not estimate an effect (pure GBM) or does not appear in the case.

< section id="bias-of-the-average-effect" class="level3">

Bias of the average effect

Number = mean ATE error (estimate − truth, 0 = perfect). Color = magnitude of the error.

< section id="predictive-accuracy-is-not-effect-accuracy" class="level3">

Predictive accuracy is not effect accuracy

Each point is a method. x-axis: how well it predicts Y; y-axis: how well it estimates the effect per customer. For both, 0 = perfect. If good prediction implied a good effect estimate, the points would lie on a rising line.

In Case 4 (mediator) and Case 3 (nonlinear confounding), good prediction models give a poor effect estimate. The pure GBM predicts Y almost as well as the S-learner but is useless as a targeting model (Cases 7, 8). LightGBM vs. xgboost: practically identical in all cases. What matters is the construction (S, T, pure), not the library.

< section id="where-is-the-causal-forest-really-better" class="level3">

Where is the causal forest really better?

The causal forest is clearly the best method in Case 8 (unknown interaction among many variables) and in Case 9 (observational data with nonlinear confounding and a varying effect; there it is also nearly unbiased for the ATE). There, however, the S-learner and the R-learner deliver the same top-30% selection; the advantage lies in the more accurate magnitude of the effect per customer (e.g., for cost-benefit calculations), not in the ranking. In Case 3 (nonlinear confounding), the causal forest is nearly unbiased, but Double ML with ordinary xgboost estimates the average effect somewhat more accurately still. What wins there is the orthogonalized construction, not the forest.

< section id="evaluation-without-knowing-the-truth" class="level2">

Evaluation without knowing the truth

In real data, only the tools from the Methodology section “Evaluation methods” would be available. Here we know the truth and can check whether these tools would have found the right winner. To this end, they are computed on the same test data as the quality measures above, but without using the true effect.

< section id="r-loss-and-qini-do-they-find-the-best-method" class="level3">

R-loss and Qini: Do they find the best method?

Interpretation

• Identification holds (Cases 1–3, 6–9): The R-loss ranks the methods almost like the truth (rank correlation 0.98 in Case 6, 0.97 in Case 8), and anyone choosing by it loses little (regret at most 0.04). It does not always pick the single best method (Case 2: 35 %, Case 7: 48 %), because several methods are almost equally good there. The Qini value likewise ranks almost like the truth (Case 8: 0.95) but rarely names the best method exactly (30 %): causal forest, R-learner and S-learner are too close together. By contrast, pure ATE estimators (Qini around 0) and the pure GBM (Qini below 0, i.e., worse than chance) are reliably screened out.
• Identification fails (Cases 4, 5): The R-loss ranks the methods against the truth (rank correlation -0.27 and -0.18, respectively) and never selects the right method (regret 0.94 and 1.55, respectively). It rewards whatever fits the observed residuals, including the biased effect. As with predictive accuracy: a quality measure on observed data does not detect a wrong choice of variables.

< section id="calibration-case-8" class="level3">

Calibration (Case 8)

If a point lies on the dashed line, the prediction agrees with the experiment. The calibration slope (red line) summarizes this: 1 = calibrated, > 1 = predictions compressed too much, < 1 = spread too much. Averaged over 40 simulations in Case 8: causal forest 1.20, R-learner 1.23, S-learner 1.66, LM T x X 0.77. Calibration thus shows not only whether the effect is right but also in which direction a model is off. Constant estimators (Double ML, AIPW) have no spread and cannot be checked this way.

< section id="overlap-diagnostic" class="level3">

Overlap diagnostic

In Case 3, 6.6 % of customers lie outside 0.1–0.9 (Case 2: 2.8 %, randomized Case 6: 0.0 %). This is the warning signal that leads to failures in AIPW. If anything, the diagnostic underestimates the problem: the smoothed propensity from the forest does not reach the extremes, while the true propensity lies outside this range for 10.4 % of customers.

< section id="placebo-test-and-e-value-case-5" class="level3">

Placebo test and E-value (Case 5)

The placebo outcome reveals the error: although has no effect on it, an “effect” of 1.58 results. This almost equals the bias of the effect on (1.65). The placebo test thus detects the unobserved confounding where R-loss and quality measures fail. The instrument passes the same test (-0.07). The E-value of the measured effect is 4.4: an unobserved confounder would need a risk ratio of about 4.4 with both and to explain it away completely. That sounds like a lot but is in fact the case here ( shifts and strongly). The E-value says how strong a confounder would have to be, not whether it is that strong.

< section id="agreement-of-methods" class="level3">

Agreement of methods

In Cases 4 and 5, the methods agree with one another (range 0.179 and 0.150, respectively) and are all off by about 1.00 and 1.68, respectively. In Case 3, by contrast, the estimates diverge widely (range 1.74), and here that is a legitimate warning signal. Disagreement therefore points to a problem; agreement proves nothing.

< section id="do-the-confidence-intervals-hold" class="level3">

Do the confidence intervals hold?

For the methods with a standard formula (see Methodology, “Confidence intervals”), the 95% interval for the average effect is computed in every run. The table shows in how many of the 40 runs it contains the true effect. A correct interval should do so in about 95% of runs.

< section id="conclusion" class="level2">

Conclusion

< section id="when-does-causalml-pay-off" class="level3">

When does CausalML pay off?

CausalML methods pay off when predictors act nonlinearly (Cases 3, 9), when the effect depends on many variables in an unknown form and its magnitude per unit matters (Cases 8, 9), and, for selection, only when the model includes the treatment (Case 7). They bring no advantage with small samples and in linear worlds (Cases 1, 2, 6), with a wrong choice of variables (Case 4), and with a missing variable without an instrument (Case 5). How to choose a method from this is explained in the guide; what has not been tested is listed in the box at the end of the guide.

< section id="continuous-treatment" class="level3">

Continuous treatment

Example: discount size instead of discount yes/no. In all cases, is binary; what follows is methodological knowledge, not a simulation. The rules on confounder, mediator and collider apply unchanged. The question changes: the target is a dose-response curve or the slope , and “whom to treat?” becomes “how much?”. LM/GLM, S-learner, Double ML (residuals instead of the propensity), causal forest and 2SLS can be carried over to continuous ; the T-learner and the difference in means cannot, and AIPW only via a generalized propensity density (unstable with thin overlap). Double ML and causal forest assume the effect to be linear in . If the effect saturates, they return a mean slope that is correct at no dose; the model must then learn the shape of the curve (S-learner, splines). Overlap here means: every relevant dose must occur among comparable customers; outside the observed range, every estimate is extrapolation.

< section id="limitations" class="level3">

Limitations

• The results apply to these data-generating processes. The ranking of the methods depends on the setup, not on a fixed hierarchy.
• The ATE of the row “Causal forest” is the mean of its CATE predictions on the test data set; the doubly robust estimator is listed separately as “AIPW (causal forest)”. Double ML (without an effect model) and AIPW deliver only an average effect; their CATE-RMSE is therefore poor by construction when the effect is heterogeneous, and their top-30% uplift corresponds to a random selection.
• In Case 3, overlap is weak at the extremes (for , the propensity is close to 1). This makes the case harder for all methods, not easier.
• In real data, the true effect is never known; only proxy validation is possible there (e.g., uplift curves on randomized data).
• A much larger study of the same kind, with dozens of methods and teams, is the ACIC competition [23]. It reaches the same basic finding: flexible methods help with estimation, but no method wins everywhere.

< section id="references" class="level1">

References

The numbers in the text refer to this list.

Introductory reading

[1] Hernán, M. A., Robins, J. M. (2020). Causal Inference: What If. Chapman & Hall/CRC. Free: miguelhernan.org/whatifbook

[2] Chernozhukov, V., Hansen, C., Kallus, N., Spindler, M., Syrgkanis, V. (2024). Applied Causal Inference Powered by ML and AI. arXiv:2403.02467. Free: causalml-book.org

[3] Facure, M. (2022). Causal Inference for the Brave and True. Free: matheusfacure.github.io/python-causality-handbook

[4] Pearl, J., Glymour, M., Jewell, N. P. (2016). Causal Inference in Statistics: A Primer. Wiley.

[5] Angrist, J. D., Pischke, J.-S. (2009). Mostly Harmless Econometrics. Princeton University Press.

[6] Molnar, C. (2022). Modeling Mindsets: The Many Cultures of Learning From Data. christophmolnar.com/books/modeling-mindsets

Methods

[7] Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal 21(1), C1–C68. doi:10.1111/ectj.12097

[8] Nie, X., Wager, S. (2021). Quasi-oracle estimation of heterogeneous treatment effects. Biometrika 108(2), 299–319. doi:10.1093/biomet/asaa076

[9] Künzel, S. R., Sekhon, J. S., Bickel, P. J., Yu, B. (2019). Metalearners for estimating heterogeneous treatment effects using machine learning. PNAS 116(10), 4156–4165. doi:10.1073/pnas.1804597116

[10] Wager, S., Athey, S. (2018). Estimation and inference of heterogeneous treatment effects using random forests. JASA 113(523), 1228–1242. doi:10.1080/01621459.2017.1319839

[11] Athey, S., Tibshirani, J., Wager, S. (2019). Generalized random forests. Annals of Statistics 47(2), 1148–1178. doi:10.1214/18-AOS1709

[12] Athey, S., Imbens, G. (2016). Recursive partitioning for heterogeneous causal effects. PNAS 113(27), 7353–7360. doi:10.1073/pnas.1510489113

[13] Robins, J. M., Rotnitzky, A., Zhao, L. P. (1994). Estimation of regression coefficients when some regressors are not always observed. JASA 89(427), 846–866. doi:10.1080/01621459.1994.10476818

[14] Kennedy, E. H. (2023). Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics 17(2), 3008–3049. doi:10.1214/23-EJS2157

[15] Hill, J. L. (2011). Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics 20(1), 217–240. doi:10.1198/jcgs.2010.08162

[16] Hahn, P. R., Murray, J. S., Carvalho, C. M. (2020). Bayesian regression tree models for causal inference: Regularization, confounding, and heterogeneous effects. Bayesian Analysis 15(3), 965–1056. doi:10.1214/19-BA1195

[17] van der Laan, M. J., Rose, S. (2011). Targeted Learning: Causal Inference for Observational and Experimental Data. Springer.

[18] Athey, S., Wager, S. (2021). Policy learning with observational data. Econometrica 89(1), 133–161. doi:10.3982/ECTA15732

Pitfalls and evaluation

[19] Cinelli, C., Forney, A., Pearl, J. (2024). A crash course in good and bad controls. Sociological Methods & Research 53(3), 1071–1104. doi:10.1177/00491241221099552

[20] VanderWeele, T. J., Ding, P. (2017). Sensitivity analysis in observational research: Introducing the E-value. Annals of Internal Medicine 167(4), 268–274. doi:10.7326/M16-2607

[21] Gutierrez, P., Gérardy, J.-Y. (2017). Causal inference and uplift modelling: A review of the literature. Proceedings of Machine Learning Research 67, 1–13.

[22] Schuler, A., Baiocchi, M., Tibshirani, R., Shah, N. (2018). A comparison of methods for model selection when estimating individual treatment effects. arXiv:1804.05146.

[23] Dorie, V., Hill, J., Shalit, U., Scott, M., Cervone, D. (2019). Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition. Statistical Science 34(1), 43–68. doi:10.1214/18-STS667

Process models and tools

[24] Petersen, M. L., van der Laan, M. J. (2014). Causal models and learning from data: Integrating causal modeling and statistical estimation. Epidemiology 25(3), 418–426. doi:10.1097/EDE.0000000000000078

[25] Hernán, M. A., Robins, J. M. (2016). Using big data to emulate a target trial when a randomized trial is not available. American Journal of Epidemiology 183(8), 758–764. doi:10.1093/aje/kwv254

[26] Textor, J., van der Zander, B., Gilthorpe, M. S., Liśkiewicz, M., Ellison, G. T. H. (2016). Robust causal inference using directed acyclic graphs: the R package dagitty. International Journal of Epidemiology 45(6), 1887–1894. doi:10.1093/ije/dyw341. Online: dagitty.net

[27] Sharma, A., Kiciman, E. (2020). DoWhy: An end-to-end library for causal inference. arXiv:2011.04216. pywhy.org/dowhy

[28] PyWhy EconML: Library Flow Chart (decision tree for choosing an estimator). pywhy.org/EconML/spec/flowchart.html

< section id="appendix-software" class="level1">

Appendix: Software

40 simulations per case, seeds deterministic (1000 · case number + repetition). Code and data: https://github.com/PhilippPro/blog_philipppro/tree/main/posts/causalml-toy-cases. As of: 2026-10-26.