Global Holdout

The global holdout is the system-wide control group for measuring Apex's incremental impact. A small fraction of every workspace's end-users — by default 5% — are assigned to the holdout. Apex leaves them completely alone: no marketing sends from any journey, AND they are forced to control in every experiment. They are the persistent "do-nothing" cohort.

The difference between holdout and exposed groups is the calibrated impact of everything Apex does.

One cohort, both surfaces ("Is our growth real?")

The same person is held out of journeys and experiments, because the holdout is hashed on a single identity. That means one do-nothing cohort can prove the cumulative lift of your entire Apex footprint — not just journeys. The Executive Summary's "Is our growth real?" panel reads these counters per goal and shows the lift vs the holdout with a 95% confidence interval.

This is the load-bearing axis of Apex's four-way measurement:

1. vs the control group — the other arm of a specific experiment (on the experiment's detail page).
2. vs the previous baseline — the same goal in the prior equal-length window.
3. vs the do-nothing global holdout — the only axis that answers "would this have happened anyway?" This is the proof.
4. vs your industry — an opt-in benchmark (locked until you opt in).

Honest by construction

• Forward-only. Experiments only began honoring the holdout once enforcement shipped, so the panel counts from that point on and never backfills pre-enforcement history (which would fabricate lift). Until cohorts are large enough for a confidence interval that excludes zero, each goal reads "Collecting since <date>."
• Held-out subjects are excluded from the experiment's own stats — they're the do-nothing cohort, not the experiment's randomized control — so they never bias a per-experiment result or trip the sample-ratio-mismatch check.
• The panel computes its interval with the same statistical engine as the rest of Apex (two-proportion z for rate goals; Welch for revenue goals), so the headline number can never contradict a per-experiment readout.

Why workspace-level (not per-journey)

Per-journey holdouts compose poorly. A user excluded from journey A is still exposed to journey B; the "control" stops being a control once two journeys overlap.

A single workspace-level holdout fixes this:

• Every journey's lift estimate uses the same control group
• Lift estimates are comparable across journeys
• Users have a stable holdout status (no flipping between in/out as journeys come and go)

Determinism

Assignment is computed as a stable FNV-1a hash of (workspaceKey, endUserId) modulo 100:

bucket = FNV-1a(workspaceKey + "::" + endUserId) % 100
inHoldout = bucket < holdoutPct

The hash is stable across deploys, retries, and re-evaluations. Same (workspaceKey, endUserId) pair always produces the same bucket. A user's holdout status never changes.

This determinism is what makes the holdout a valid control. If users could flip in and out, accumulated impact measurements would be invalidated.

What "no sends" means

Holdout users still:

• Receive transactional communications (2FA codes, password resets, receipts) — these aren't subject to holdout
• Show up in your event stream, audiences, and analytics
• Have their preferences honored normally
• Get their data erased per GDPR on request

Holdout users do not receive:

• Any marketing communication from any Adaptive Journey
• Any webhook step that would otherwise fire on their behalf
• Any experiment treatment variant — they are deterministically forced to control (recorded on the assignment as heldOut)

Default 5%

The default is 5% — small enough that you don't sacrifice much volume, large enough that lift detections converge in reasonable time for most workspaces.

For very small workspaces (a few thousand active end-users), 5% may not produce statistically significant lift estimates within a quarter. The auto-grow rule (Phase 1b polish) will adjust the holdout percentage based on the number of monthly active users targeting the configured minimum detectable effect.

You can override the default per platform via the JOURNEY_GLOBAL_HOLDOUT_PCT environment variable; per-workspace UI overrides land alongside the calibrated-impact dashboard.

Tradeoffs

5% is a common-case sweet spot. If you're running journeys you're confident produce a >5% lift, the default is comfortable.

Auditing the holdout

Each journey's detail page in the dashboard includes a Holdout audience panel that lists the subjects currently held out for that journey-version. The list is the auditable view of your control group:

• Every row is a subject who matched the journey's trigger but was suppressed by the workspace holdout
• Records are TTL'd to triggeredAt + goalEventWindowDays + 24h grace, so the panel reflects the currently-attributable control population, not the all-time history
• Email + name are hydrated from the contact store when available; users who exist only as anonymous endUserIds show as —
• A CSV export ships the loaded page so you can hand it to compliance or marketing-ops without an extra round trip

The panel hides itself on journeys that have never published (no version, no firings) so it doesn't show empty chrome. On a published journey with no current holdout records, you'll see a "no one is currently held out" note alongside an explanation of the lifecycle.

Holdout vs. Adaptive Branches

These measure different things:

• An Adaptive Branch measures arm A vs. arm B within users who reached the branch
• The global holdout measures journey-exposed users vs. anyone-not-exposed across the entire workspace

You need both. Adaptive Branches optimize which variant of a journey works best. The holdout tells you whether the journey works at all (vs. doing nothing).

Related

• Calibrated Impact — how the lift number is computed
• Adaptive Journeys — what produces the exposed group
• Beliefs — where the resulting impact estimates flow