> For the complete documentation index, see [llms.txt](https://help.usejimo.com/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://help.usejimo.com/docs/analyze/a-b-test.md).

# A/B Test

Split a tour's audience between up to three versions, measure each one against its goal, and make the winner the default once the result is reliable.

An A/B test shows different versions of the same tour to different users at the same time, and measures which one reaches the goal more often. It answers "is this version better?" with data instead of opinion.

A/B tests run on tours, modals included. Each test compares up to three versions: the **Original** and up to two variants, **A** and **B**.

## When to run a test

Run a test when you have a specific doubt about one tour, and enough users to settle it.

| Good reasons                                                | Poor reasons                                 |
| ----------------------------------------------------------- | -------------------------------------------- |
| A tour has a clear drop-off and you have a fix in mind      | You have not looked at the tour's funnel yet |
| Two wordings or two step orders, and the team disagrees     | You want to change five things at once       |
| A bold change you want to try on part of the audience first | The tour reaches a few dozen users a month   |

{% hint style="info" %}
**Fix what is broken before you test.** If users drop off because a step's element is not found, a test only measures the bug twice. Check the **Issues** tab first, see [Analytics](https://help.usejimo.com/docs/analyze/analytics.md#what-to-do-with-a-drop-off).
{% endhint %}

## How a test works

A test is attached to the tour's [Goal](https://help.usejimo.com/docs/publish/mesure-outcomes-goal.md). The goal decides what counts as a conversion: completing the tour, or triggering a tracked event.

<figure><img src="https://2794860263-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKzAcDWQbK1gKbpra7bkb%2Fuploads%2FERD4scSDySAIW5r3141C%2FD-03-ab-split.png?alt=media" alt="The Who audience of a tour is split between the Original, Variant A and Variant B, each between 10% and 90%, all measured on the same Goal"><figcaption></figcaption></figure>

* The audience is the tour's [Who](https://help.usejimo.com/docs/publish/target-right-people-who.md). The split does not change who is eligible, it only divides them.
* Eligible users are dispatched at random into the versions, in the proportions you set.
* Each version gets at least 10% and at most 90% of the audience.
* For each version, Jimo counts the users exposed, the users who converted, and the conversion rate.
* One test per tour at a time. Changes made in one variant do not affect the others.

## Set up a test

<figure><img src="https://2794860263-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKzAcDWQbK1gKbpra7bkb%2Fuploads%2FX1jqEOofcRy8iFth6P3E%2FD2-07-ab-test-lifecycle.png?alt=media&amp;token=ed7615ad-777a-41c5-8bd2-aa6bfcebafed" alt="The six stages of an A/B test: start from Goal, edit one thing in the variant, set the split, publish, wait for a reliable confidence, set the winner as default"><figcaption></figcaption></figure>

{% stepper %}
{% step %}

### Start the test from the Goal

Open the tour's **Target & Publish** settings, go to **Goal**, and click **Start A/B Test**. Jimo creates the **Original**, your current tour, and **Variant A**, a copy of it. The test is not published yet.

<figure><img src="https://2794860263-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKzAcDWQbK1gKbpra7bkb%2Fuploads%2Fsy62uMaQ1x2tloScVL4j%2Fimage.png?alt=media&amp;token=4fdcd1d3-ecb8-46a7-ba29-bf48773798ce" alt="Start A/B Test button in the Goal section"><figcaption></figcaption></figure>
{% endstep %}

{% step %}

### Change one thing in the variant

Click **Edit** on Variant A to open it in the builder, and change the one thing you want to test. In the builder, the top bar switches between versions. To test a third version, click **Add variant** and duplicate the Original or Variant A.

<figure><img src="https://2794860263-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKzAcDWQbK1gKbpra7bkb%2Fuploads%2FyVAyF8awWyWEKQ1NOkU9%2Fimage.png?alt=media&amp;token=81c09269-2333-4488-9ba0-1f3b27d14891" alt="Variants list with Preview, Edit and Delete"><figcaption></figcaption></figure>
{% endstep %}

{% step %}

### Set the split

Use the slider to decide how much of the audience each version receives. The panel shows the approximate number of users per version.

<figure><img src="https://2794860263-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKzAcDWQbK1gKbpra7bkb%2Fuploads%2F8hozAYosNkExREJykO8V%2Fimage.png?alt=media&amp;token=70ff6f45-e39d-4ed2-b581-a054fef3ce5a" alt="Audience distribution slider"><figcaption></figcaption></figure>
{% endstep %}

{% step %}

### Publish

Click **Publish**. All versions are now served, each to its share of the audience. You can add a variant to a live test later, then publish again.
{% endstep %}
{% endstepper %}

{% hint style="warning" %}
**Cancel Test** before publishing deletes the variants and keeps only the Original. After publishing, cancelling also requires a new publish to go back to a single version.
{% endhint %}

#### How to choose the split

Start at 50/50: it reaches a reliable result fastest. Use an uneven split, such as 80/20, only when the variant is risky and you want few users exposed while you learn. The smaller share takes longer to collect enough users.

## Read the results

Results appear on the tour's **Statistics** tab, in the **A/B Test** block.

<figure><img src="https://2794860263-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKzAcDWQbK1gKbpra7bkb%2Fuploads%2FQwk1PCZiNIxLvQ4CnGnv%2Fimage.png?alt=media&amp;token=6aa2fc47-40bb-4a5c-bfb8-f8e174954063" alt="A/B Test block with confidence gauge and variant table"><figcaption></figcaption></figure>

| Column              | What it counts                                                                                                                                                                     |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Started**         | Users exposed to this version. Not the **Started** of [Analytics](https://help.usejimo.com/docs/analyze/analytics.md#how-jimo-counts), which counts users who began the experience |
| **Goal reached**    | Users exposed to this version who converted                                                                                                                                        |
| **Conversion rate** | Goal reached divided by Started                                                                                                                                                    |

The confidence gauge says whether the difference between versions is reliable, or that it is still too early. The other statistics of the tour can also be filtered by variant, to compare where each version loses users. See [Metrics by experience type](https://help.usejimo.com/docs/analyze/analytics/metrics-by-experience-type.md#tours-and-modals).

## When to conclude

A higher conversion rate is not a result until the test says it is reliable.

#### Wait for enough users

The confidence comes from a statistical test on the conversion rates of the versions. With few users, the test cannot tell a real difference from chance, and the result shows as too early. Your assistant applies the same rule and gives no confidence below 30 users per version.

#### Do not stop on the first good day

Early numbers move a lot. A variant leading 4 conversions to 2 is not a winner. Decide in advance how long the test runs, and look at the confidence at the end, not every morning.

#### Then pick the winner

Click **Set as Default** on the version that won. It becomes the only version shown to all users, and the others are retired. This cannot be undone. The test and its numbers stay in the block's history.

<figure><img src="https://2794860263-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKzAcDWQbK1gKbpra7bkb%2Fuploads%2FJ4lhhZMlMfZy5q8v61lK%2Fimage.png?alt=media&amp;token=bf156ada-576f-436b-9ca0-af8e8ce71974" alt="Set as Default on the winning variant"><figcaption></figcaption></figure>

#### And test again

To keep improving, start a new test from **Goal**. The current tour becomes the new Original, and you add a challenger.

## Good practice

* **One change per test.** Copy, step order or design, not all three. Otherwise you will not know what made the difference.
* **Tell your team.** While a test runs, two users can see two different tours. Support should know before a customer describes the "wrong" one.
* **Keep the goal stable.** Changing the goal during a test changes what you are measuring.

## From your assistant

With the [Jimo MCP](https://help.usejimo.com/docs/integrations/mcp/what-to-ask.md), your assistant reads the test for you, and says plainly when it is too early to call.

> "Did variant A of the pricing tour win?"

<figure><img src="https://2794860263-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKzAcDWQbK1gKbpra7bkb%2Fuploads%2F5GX5RPwXwtbdP7bEsFGe%2Fmcp-12-ab-test.png?alt=media&amp;token=9f80ce2a-3046-472b-aa2b-fda0174de714" alt="An assistant answer on an A/B test: exposure and conversions per version, lift over the Original, and whether the result is reliable yet"><figcaption></figcaption></figure>

It returns exposure and conversions per version, the lift over the Original, and whether the difference is reliable yet. It can also set up a test: "Set up an A/B test on the Pricing modal headline" creates the variant, edits it and sets the split, with a preview before anything changes. See [What to ask?](https://help.usejimo.com/docs/integrations/mcp/what-to-ask.md).

## Verify

* The A/B Test block lists every version with its share of the audience.
* After a few days, each version shows a **Started** count roughly in line with its share.
* The confidence gauge shows a percentage, not too early, before you click **Set as Default**.

## Go further

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th><th data-hidden data-card-cover data-type="files"></th></tr></thead><tbody><tr><td><strong>Goal</strong></td><td>The goal every test is measured against</td><td><a href="https://help.usejimo.com/docs/publish/mesure-outcomes-goal.md">Goal: measure outcomes</a></td><td><a href="https://2794860263-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKzAcDWQbK1gKbpra7bkb%2Fuploads%2Fq6boiAUVnaRmBVrPk36I%2FB-c05-goal.png?alt=media&amp;token=4eab78cb-945f-46bb-a99a-6ea294d3e3aa">B-c05-goal.png</a></td></tr><tr><td><strong>Analytics</strong></td><td>Find the drop-off worth testing</td><td><a href="https://help.usejimo.com/docs/analyze/analytics.md">Analytics</a></td><td><a href="https://2794860263-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKzAcDWQbK1gKbpra7bkb%2Fuploads%2Fgm50QqSDYzPc0Np1BN9F%2FD2-c01-analytics.png?alt=media&amp;token=766f0de0-cac3-4ac5-873a-30d565fa299c">D2-c01-analytics.png</a></td></tr><tr><td><strong>Who</strong></td><td>The audience a test splits</td><td><a href="https://help.usejimo.com/docs/publish/target-right-people-who.md">Who: target the right users</a></td><td><a href="https://2794860263-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKzAcDWQbK1gKbpra7bkb%2Fuploads%2F33oQUR818lKt8U4mhvt9%2FB-c03-who.png?alt=media&amp;token=fc5c73b6-2e2d-45df-8d65-eab12dff76c4">B-c03-who.png</a></td></tr></tbody></table>

**Next page:** [Users & Segments](https://help.usejimo.com/docs/analyze/users-and-segments.md)
