Experiments

Experiments are how you test beliefs against reality. Apex supports experiments that run client-side through the tracking snippet or server-side through the SDK, with deterministic visitor assignment and built-in statistical analysis.

Experiment Types

Apex supports three variant types, each suited to different kinds of tests:

• Text change — Swap text content on a page without changing the URL. The snippet finds the target element and replaces its content for visitors in the variant group.
• Redirect — Send variant visitors to a completely different URL. Useful for testing entirely different page designs or flows.
• Code — Execute custom JavaScript (client-side) or use the SDK for server-side feature flags. The most flexible option for complex changes.

Experiment Structure

Every experiment has these core fields:

Traffic is split evenly between control and variant_b by default (50/50). You can adjust this, but even splits give you the fastest path to statistical significance.

How Assignment Works

Apex uses MurmurHash3 for deterministic visitor bucketing. Here's what that means in practice:

1. Each visitor gets a persistent anonymous ID (stored in a first-party cookie)
2. The hash of visitorId + experimentId produces a number between 0 and 99
3. That number determines whether the visitor sees control or the variant
4. The same visitor always gets the same assignment — no flickering between variants

This approach is fast (no network round-trip needed), deterministic (same input always produces same output), and statistically uniform (even distribution across buckets).

Running an Experiment

Results and Statistical Confidence

As visitors flow through the experiment, Apex tracks:

• Visitor count per variant
• Conversion count and conversion rate per variant
• Relative lift (how much better or worse the variant performs vs control)
• Statistical confidence (how likely the observed difference is real, not noise)

Apex picks the right estimator for your metric. Results are considered significant at 95% confidence by default. You'll see the confidence percentage climb as more data comes in.

The measurement contract

There is one measurement contract with the right estimator behind it for each metric type, so a goal is always scored honestly:

Whichever engine runs, the output is the same single field — the probability this winner is actually best — so the readiness gate and calibration read it identically.

Continuous metrics are handled honestly. Revenue and duration are heavy-tailed, so Apex winsorizes the extremes (a few whales can't decide the test) and sums value per subject, not per event. Revenue goals can also be net (e.g. purchase value minus refunds).

Power before you launch. When you pick a goal, Apex estimates how long the test will take to reach significance from your live event volume and warns when a design is underpowered ("~N weeks; reduce variants or raise the minimum detectable effect"). Testing more variants or more metrics tightens the bar automatically (a multiple-comparisons correction that scales with the number of arms and guardrails).

Value maturation. Revenue and other late-settling outcomes don't become decisive the moment they're statistically ahead — Apex holds the call through a maturation window (default 30 days) so a winner isn't crowned on revenue that could still be refunded. While it waits, the experiment shows a distinct maturing state.

Frozen at launch. When an experiment starts running, its primary metric and guardrails are frozen. Editing them requires forking a new experiment — this prevents moving the goalposts mid-test (HARKing).

Guardrails (protected metrics)

Every experiment is protected by default. Apex watches a set of business "vital signs" and blocks a variant from winning if it causes harm — you don't configure anything.

• What's watched is derived from your conversion model plus universal harms, and only attached when the event is actually firing: revenue (must not drop), refunds and errors/crashes (must not rise), plus model-specific signals (activation for PLG, demo requests for sales-led, supply/demand signups for marketplaces). Add extra guardrails under Advanced when a test has a specific risk.
• How a breach is judged. Guardrails are checked continuously, so Apex uses an always-valid (sequential) test — not a fixed-horizon one — so watching every poll doesn't inflate false alarms. A breach is only critical when the harm is past your safe-range margin AND statistically confirmed AND past a sample floor. Revenue/refund guardrails also respect the maturation window. Anything less shows as At risk, not a block.
• SRM. Apex also checks that traffic is splitting the way you intended (sample-ratio-mismatch). A broken split means the results are invalid, so Apex flags the experiment rather than trusting the numbers.
• What happens on a breach. Promotion is blocked and you get a notification. If you turn on auto-pause (Settings or the "Protect your experiments" setup step), Apex pauses the experiment automatically; otherwise it blocks + alerts and re-alerts daily until you decide.

Muting the notification never disables the protection — the block and (optional) auto-pause always apply.

Server-Side Experiments with the SDK

For experiments that require server-rendered changes (pricing logic, feature flags, API responses), use the Apex SDK:

import { Apex } from "@anthropic/apex-sdk";

const apex = new Apex({ workspaceKey: "YOUR_KEY" });

const variant = apex.getVariant("pricing-test", visitorId);

if (variant === "variant_b") {
  // Show the experimental pricing
}

The SDK uses the same MurmurHash bucketing as the snippet, so a visitor assigned to variant_b client-side will also get variant_b server-side.

On-device screenshots (mobile)

Apex auto-captures public web URLs server-side when you create an experiment, and the agent CLI (via attach_experiment_asset) covers localhost. But servers can't reach authed in-app mobile screens — so the Capacitor plugin can capture them on-device.

On the screen that renders the variant, call captureVariantScreenshot() keyed to the resolved variant:

const variant = useApexVariant(experimentId); // or: const { variant } = await Apex.getVariant({ experimentId })
useEffect(() => {
  if (variant) {
    Apex.captureVariantScreenshot({ experimentId, variantKey: variant });
  }
}, [variant]);

The captured shot lands on the dashboard experiment card (cover) and the detail Variant previews gallery, right alongside web and agent captures.

Grounded experiments (brand truth)

When an AI agent writes experiment copy, it should never invent facts. Apex ships four short, merchant-owned guardrail files you drop into your repo so the agent writes variants against your truth:

A fifth file, .cursor/rules/apex-experiment-guardrails.mdc, is an always-on rule that tells the agent to read the four files before writing any experiment.

Setting them up

In the dashboard, open Set up Apex → Intelligence → "Set up experiment guardrails." The recommended (Cursor) lane scaffolds all five files into your repo through your connected agent; a non-Cursor lane lets you download them from /shipped-guardrails (with a SHA-256 manifest to verify integrity). The templates ship blank — you fill in the facts. Apex never guesses your facts for you.

How enforcement works

There's no server-side fact-checker. Enforcement is by use: create_experiment and the new-experiment prompt instruct the agent to read .apex/brand-truth.md and cite the line each claim traces to before proposing copy. If a claim isn't backed there, the agent asks instead of inventing.

Where an experiment runs: data source + randomization unit

Every experiment belongs to a data source (a property: your website, your iOS app, your Android app, your backend) — the same registry that classifies where your events come from. This is what the experiment runs in, and it's separate from surface (the delivery channel: a DOM swap vs a native payload). A Capacitor app is one property, so all of its experiments group under the one mobile app — never split across "Website" and "Mobile app".

Each experiment also has a randomization unit — what it buckets on:

• visitor (default) — one anonymous device/browser.
• person — one stitched human; all their devices get the same variant (requires identify()).
• account / org — everyone in a B2B account gets the same variant.

The randomization unit is also the unit results are counted on, so pick account for account-level tests or the math treats four devices in one account as four independent draws.

Counting one user across web, iOS, and Android (golden path)

Apex can only treat one human as one subject if you give them one id. The golden path (same as Statsig/PostHog):

1. Mint one Apex visitor id and pass it everywhere. The SDKs persist it under apex_vid; for a web→app transition, carry it across and call Apex.setVisitorId(webVid) so the app reuses the site's id.
2. Call identify({ email }) on every platform after login. Apex links the per-device ids to one person on verified email — it never merges on device signals alone.
3. Server-side events: pass the browser's apex_vid as the visitor id so server events join the same person.

Assignment is sticky: once a subject is bucketed, that variant is locked — re-evaluations never re-bucket them.

Connecting to Predictions

Before running an experiment, log a prediction — what you think will happen. This builds your team's calibration score and makes experiment results more actionable.

After the experiment completes, Apex compares your prediction against actual results to calculate an accuracy score. Over time, this feedback loop makes your team better at anticipating outcomes.

Lifecycle