Skip to main content

Set up A/B tests in Alia

Test two or more popups against each other in a campaign, read the results, and pick a winner.

Written by Rob D

An A/B test splits the visitors who match a campaign's targeting between two or more popups, so you can see which performs best. You can test designs, offers, timing, copy, a floating button, and more.

💡 Tip: Change one thing at a time. If variants differ in both headline and delay, you won't know which change made the difference.

Step 1: Choose your campaign

Go to Campaigns and open the campaign you want to test. Every test lives inside one campaign.

Step 2: Add a variant

Click Add test variant. Alia copies your control popup into Drafts, so you can edit the copy and test it against the original.

The arrow next to Add test variant has more options:

  • Duplicate another variant

  • Build variant with Lia, to have Lia build a new one

  • Build variant from scratch

💡 Note: If you leave the page before Lia finishes building the variant, Alia removes it from the campaign.

To test three or more versions, such as 10%, 15%, and 20% offers, add a variant for each. There's no limit on the number of variants.

Step 3: Edit the variant

Open the new draft and change the one thing you want to test, such as the design, the delay, or the offer. For example, test a 5-second delay against a 10-second delay.

Rename your popups clearly, such as "5s delay" and "10s delay". The names appear in the results, the AI summary, and the A/B Tests history.

A draft has to be Ready before it can go into a test. Alia marks it ready when it saves, as long as the popup passes its checks. If a draft shows Not ready, open it and click Mark as ready to see what needs fixing.

💡 Note: Starting a test puts every popup in the campaign into it: the live popup and all drafts. Archive any draft you don't want in the test first.

Step 4: Start the test

  1. Click Start new A/B test.

  2. Give the test a Title, such as "Time delay test – 5s vs 10s".

  3. Check the Control, the popup every other variant is compared to. Alia picks a popup with "control" in its name, or otherwise the live popup.

  4. Choose the Traffic weight. Equal traffic split is on by default. Turn it off to set a percentage for each popup. The percentages must add up to 100%.

  5. Click Start test if the campaign is live. If it isn't, choose Create and publish or Create and schedule.

Example: test on only 10% of traffic

To test a new variant while keeping most visitors on your current popup, duplicate the control and split traffic like this:

  • Control: 90%

  • Copy of the control: 5%

  • New variant: 5%

10% of visitors take part in the test, split evenly between the control copy and the new variant. Compare those two, with Adjust all stats for visitors on.

How visitors are assigned

When a visitor first matches the campaign's targeting, Alia assigns them to one variant at random, using the traffic weights. They keep that variant for the rest of the test, even on later visits. Visitors who never match the campaign's targeting aren't part of the test.

If you change traffic weights during a test, only visitors who haven't been assigned yet use the new weights.

Read your results

On the campaign, click View test results.

The results page shows one column per variant, with the control first:

  • Percentage lift: the change compared with the control. Green is better, red is worse.

  • Check mark: the difference is statistically significant. Hover to see the p-value.

  • Clock: not enough data yet.

  • Star: the best variant for that metric so far.

  • prob. to win: Alia's estimate of how likely each variant is to be the best overall.

The more variants you run, the more traffic you need, so plan to run tests with three or more variants for longer. Give every test at least a full week, so weekday and weekend shoppers are both included.

To export the data, open the ... menu on the results page and select View report. This opens the A/B test performance report, which you can export. You can also open it from Reports.

Hide a weak variant

To stop sending new visitors to a weak variant without ending the test, click the arrow next to Pick as winner and choose Hide. Visitors already assigned to it keep seeing it. You can only hide a variant while at least two others are still getting traffic.

End the test

Click Pick as winner under the winning variant, then End test. The winner keeps running as the campaign's live popup, and the other variants move to Drafts.

You can't add a variant to a running test. Pick a winner first, then start a new test.

Testing a floating button and Smart Trigger

Floating button vs no floating button

  • Create a variant with a floating button and one without.

  • For a teaser-only variant, set the popup to open on the floating button, so the full popup only appears when visitors click it.

  • Keep Smart Trigger off during this test, so you only measure the floating button.

Smart Trigger on vs off

Smart Trigger opens the popup when a visitor looks likely to sign up, alongside your other triggers. Test it on vs off on its own, in a separate test from other changes.

Popup vs no popup

To measure what your popup does to sales compared with showing nothing, copy the popup, delete all its triggers (including Smart Trigger), mark it as ready, and test it against the original. Visitors assigned to the copy are counted but never see a popup.

Compare the Sitewide metrics and Bounce rate, not signups. See How to A/B test popup vs no popup.

For developers

Detecting the campaign in the browser

Alia fires the alia:campaignMatched browser event when a campaign's popup is loaded for a visitor on a page. It also fires for no-popup variants, before any trigger.

Splitting traffic with an outside tool

To pass test data to tools like Google Analytics 4, or to compare Alia with another popup tool using Intelligems, Shoplift, or Convert, split visitors outside Alia. For example, set a browser variable such as popupVariant = 'A', and read it with a Custom code targeting condition. Alia's A/B report won't be the source of truth for that split.

Manual A/B tests and Smart Testing

The tests in this guide are manual: you choose the variants and pick the winner. A Smart Test runs a series of tests for you, and can hide losers and pick winners on its own. See Setting up Smart Testing in Alia.

Still have questions?

Start a chat from inside the Alia app, or use the chat in our Help Center.

Did this answer your question?