Apex

Running Experiments

This guide walks you through the full experiment lifecycle: creating an experiment, setting up variants, activating it, reading results, and promoting the winner.

Before You Start

Make sure you have:

  • The tracking snippet installed on your site
  • At least one conversion configured
  • A clear hypothesis about what change you expect to improve your metric

Tip

The best experiments start with a recorded belief. Even if you don't use the MCP server, writing down "I think X will improve Y" before testing keeps your team honest about assumptions.

Creating an Experiment

Navigate to Experiments

Go to Experiments in the dashboard sidebar and click New Experiment.

Name your experiment

Give it a descriptive name that includes the page and the change. Good: "Pricing page — social proof badges." Bad: "Test 1."

Set the page, then where on the page

Enter the URL of the page. Then pick Where on the page: the whole page, or one named part (hero, pricing, footer). A missing part is the whole page.

Two experiments on different named parts of the same page are not the same place. The same person can see both. Do not add the two lifts. A whole-page experiment plus any part on that page is one place.

Keep them apart only when they share a place. That gives each experiment fewer people, and it applies to new visitors only. People already in both stay as they are.

The live-experiment stop looks at the file, not the part. Split files if you want an edit of one part to name only that experiment.

Define your variants

Control: The current page as-is. No changes needed — this is the baseline.

Variant B: Describe or implement the change you want to test. When Apex applies the variant, this is a text or DOM change made at runtime. When your code applies it, you implement the change with useApexVariant.

Choose who applies the variant

Every website experiment answers one question: does Apex change the page, or does your code?

  • Apex changes the page (snippet mode) — Apex's snippet makes the change in the visitor's browser. No deploy needed. Best for headlines, copy, and simple visual changes; it can only change what's already on the page.

  • Your code changes the page — Your code calls useApexVariant and renders the variant itself. It can change anything: new components, interactive elements, backend logic. It starts collecting data once that code is deployed.

    First HTML is flicker-free only when the server already assigned the visitor and passed that arm in: useApexVariant(id, { initialVariant }) or <ApexProvider assignments={{ [id]: variant }}>. The hook still re-checks /assign after paint; a sticky assignment does not change what they see.

    If you use Next.js, npm install @apex-inc/next @apex-inc/react, then apexVisitorMiddleware, assignExperiments, <ApexProvider assignments>. The hook alone is not flicker-free.

    If the hook runs with no server arm, the first HTML is control. After /assign returns, React swaps to the assigned arm. That swap is the flicker. A client-only app that cannot assign before HTML has the same limitation.

Experiments created through the MCP server always let your code change the page: you're in an editor with the code open, which is where that approach is most reliable.

Configure traffic split

Set what percentage of visitors see each variant:

  • 50/50 (recommended) — Equal split, fastest to reach statistical significance
  • 80/20 — Conservative, lower risk but takes longer to get results
  • Adaptive — Apex automatically shifts traffic toward the winning variant over time

Link a conversion

Select the conversion that defines success for this experiment. This is what Apex measures to determine the winner.

Link a belief (optional)

If you've recorded a belief about why this change should work, link it. When the experiment completes, the belief's confidence will be automatically updated.

Activating the Experiment

After creating the experiment, it starts in Draft status. Review the summary, then click Activate to start splitting traffic.

Warning

Once activated, visitors are assigned to variants immediately. If your code changes the page, that code must be deployed before you activate.

Reading Results

While the experiment runs, the results page shows:

  • Visitors per variant — How many unique visitors saw each version
  • Conversions per variant — How many completed the conversion
  • Conversion rate — Per-variant conversion rate
  • Lift — Percentage improvement of Variant B over Control
  • Confidence — Statistical confidence that the difference is real, not noise
  • Days running — How long the experiment has been active

Wait for confidence to reach 95% or higher before making a decision. Calling experiments early leads to false positives.

Promoting the Winner

Verify confidence is sufficient

Ensure the experiment has reached at least 95% confidence. Below this, the result may not be reliable.

Choose the winner

Click Promote Winner and select either Control or Variant B.

Clean up (only when your code applies the variant)

Remove the useApexVariant conditional from your code. Keep only the winning variant's content and deploy.

Record the outcome

If the experiment was linked to a belief, Apex automatically updates the belief's confidence. Confirmed beliefs go up; contradicted beliefs go down.

Shipping onto a live test

If someone ships a change onto the page (or letter, Adaptive Journey, or Milestone) a test is already running on, Apex still splits people across every version. Those counts mix. Apex will not call a winner on mixed numbers, and it will not learn from them.

Apex does not pause the test for you. Pause is on the experiment page if you want everyone on the original. If the change was not the thing you were testing, say so on that page and Apex treats the numbers as fair again.

Before you edit a homepage, commit, or push, even if nobody said "experiment," have your agent call check_live_tests with the full file, not the diff hunk. Three outs: end the test then ship; wait; or say this change is not the thing under test.

On GitHub, Apex comments on the pull request and fails a check named Apex live tests. That check is not a merge gate. It does not start or pause the test.

Other Actions

  • Pause — Stop splitting traffic without losing data. All visitors see the control. Resume anytime with Activate.
  • Archive — Remove from active views while preserving all data and belief graph connections. Archived experiments still contribute to intelligence scoring.

Best Practices

  • One change per experiment. Testing multiple changes at once makes it impossible to know which one caused the result.
  • Don't peek too early. Statistical significance takes time. Checking results hourly and reacting leads to false conclusions.
  • Run for at least 7 days. Even if confidence is high after 2 days, day-of-week effects can skew results.
  • Document your learnings. The belief system is most valuable when you close the loop: believe, predict, test, update.

Next Steps

PreviousSetting up conversionsNextGraduation Pipeline