∧pex

Calibrated Impact

Calibrated impact is the lift number you stake your strategy on. It answers: how much better did your journey make things, vs. having done nothing?

Without it, "Adaptive Journeys are working" is a vibe. With it, you can rank journeys by causal contribution to revenue, retire ones that don't move the needle, and double down on the winners.

The math

For a journey with a configured optimization goal:

lift = (exposed_conversion_rate − holdout_conversion_rate) / holdout_conversion_rate

Where each rate is converted / triggered for its cohort.

That's it. No regression. No multi-touch model. Two cohorts, one event, one window.

This works because the global holdout is a deterministic, statistically-uniform sample of the same population the journey targets. The two groups differ only in whether they were exposed to the journey.

What "exposed" and "holdout" mean

  • Exposed — the trigger fired AND the user is NOT in the global holdout AND a journey execution started. They went through the journey.
  • Holdout — the trigger fired AND the user IS in the global holdout. The execution was suppressed; they got nothing from this journey. They're the calibrated control.

Critically, holdout users are still counted as triggered. Without them in the denominator, the holdout cohort would look like it has zero opportunity — and lift would be infinite. With them, both cohorts are anchored to the same trigger event and the comparison is honest.

What gets credited as conversion

The journey's goalEventName event firing for a subject within goalEventWindowDays after their trigger fired:

  • For exposed users: window starts at pointer.startedAt (the SFN execution start)
  • For holdout users: window starts at holdoutSubjectRecord.triggeredAt (the moment the trigger matched)

These are usually within seconds of each other for any given user — both are tied to the same trigger event.

Council-Stage-2 censoring

Following the runtime-correctness-as-estimand-precondition principle:

  • Exposed pointers with status failed, stopped, or timed_out are censored — they don't credit a conversion even if the goal event fires within the window. They also weren't counted in triggered_exposed's denominator. This avoids penalizing the journey for our infrastructure errors.
  • Holdout subjects are constrained to the valid window via TTL. Records older than the window auto-purge from DynamoDB.

Configuring the goal

Set the journey's goal in the canvas editor's toolbar (the Set goal button). Pick:

  • Goal event name — the binary outcome you care about (purchase, subscription.created, signup.confirmed)
  • Window days — how long after the trigger fires the user has to fire the goal. Default: the journey's attributionWindowDays (which defaults to 7).

A journey without a goal accumulates exposure data (you can see "X exposed, Y holdout" on the panel) but produces no lift number. Set the goal once you know what you're optimizing for.

Reading the panel

The CalibratedImpactPanel on the journey detail page has three tiers:

  1. Exposure split — visible bar showing % exposed vs % holdout. Confirms the holdout assignment is working as expected (~95/5 by default).
  2. Conversion + lift — appears once conversion data exists. Shows exposed rate, holdout rate, lift %.
  3. Note — when conversion data is zero, the panel surfaces an explanatory note pointing operators to set the goal.

Honest limitations

This methodology is intentionally simple. It doesn't:

  • Do multi-touch attribution (a goal event might be caused by multiple journeys; we credit each that has the user in its cohort)
  • Adjust for confounders (we rely on the holdout being statistically equivalent — no propensity scoring)
  • Estimate uncertainty (the dashboard surfaces the point estimate; precise confidence intervals live in the science dashboard)

For most workspaces, this is the right trade-off. Simpler math means fewer ways to be wrong. If you need rigorous causal inference for a regulatory filing, talk to us — we can layer doubly-robust estimators on the same exposure + holdout data.

Configuration knobs

  • journey.goalEventName — set per journey
  • journey.goalEventWindowDays — set per journey; defaults to attributionWindowDays
  • journey.attributionWindowDays — workspace default
  • JOURNEY_GLOBAL_HOLDOUT_PCT env — platform default (5%); per-workspace override is a 1b polish item

Related

PreviousGlobal HoldoutNextPredictions