The A/B test results page shows how each popup in your test performs against your control, so you can pick a winner with confidence. This article explains how to open it, what each number means, and how to end the test.
Open your test results
You can open results from the campaign or from the A/B Tests page.
From the campaign:
In Alia, go to Campaigns.
Select the campaign that's running the test.
Select View test results.
From the A/B Tests page:
In the Alia sidebar, under Data, select A/B Tests.
Find the test. Each one shows its dates, campaign, and outcome: Winner, Tie, or Inconclusive.
Select View Results.
To review a finished test from the campaign, select the arrow next to View test results, then select View past tests.
How the results page is laid out
Each variant gets its own column, with the control first. Each row is a metric, with a short description under its name. The date range defaults to Full test duration.
What you see | What it means |
Percentage lift | How the variant compares with the control. Green is better and red is worse. For metrics where lower is better, like bounce rate, a decrease shows as green. |
Check mark | The difference from the control is statistically significant. Hover over it to see the p-value. |
Clock | There isn't enough data yet to tell whether the difference is real. Only shown while the test is running. |
Star | The best-performing variant for that metric so far. |
prob. to win | Under each variant's name. Alia's estimate of how likely the variant is to be the best overall, combining signups, conversions, and engagement. Shows "-" until there's enough data. |
Once there's enough data, Alia may also show AI recommendations. A Recommended winner badge marks the variant most likely to win. A View AI recommendation badge suggests hiding a variant that's performing poorly.
At the top of the page, the AI summary describes in plain language how the variants compare.
š”Note: The AI summary is written from the full test's numbers when you open the page. It doesn't change when you filter or change the date range, so trust the numbers and check marks over the summary.
What each metric tells you
Some rows have a drop-down so you can switch what they show, such as submit rate or opt-in rate.
Metric | What it tells you |
Visitors matching targeting | How many visitors were assigned to each variant. |
Popup views | How many times each variant was seen. |
Email and phone submit rate or opt-in rate | The share of popup views that led to a signup. |
Email and phone submissions or opt-ins | The total number of signups. |
Attributed sales or average order value | Sales from visitors who signed up, within the attribution window you choose. You can show order value before or after discounts. |
Attributed conversion rate, orders, or time to purchase | How many signups went on to order, and how quickly. |
Sitewide conversion rate, sales, orders, or average order value | Orders from every visitor assigned to the variant, whether or not they signed up. |
Bounce rate | The share of visitors who left after one page. |
Learning completion rate | For Education campaigns: how many visitors who started a lesson finished it. |
Attributed and sitewide metrics won't match, even with the same window length. Attributed metrics count from the moment a visitor signs up. Sitewide metrics count from the visitor's first visit, within the Conversion window (20 days by default). For full definitions, see Understanding your Alia statistics.
Compare variants with uneven traffic
If your variants got different amounts of traffic, a bigger variant will naturally have bigger totals. Turn on Adjust all stats for visitors in the first row to see counts and sales per visitor instead. Alia turns this on automatically when your traffic split isn't even.
Filter results by segment
Select Show filters to break results down by the same dimensions as your home dashboard, including:
UTM source, medium, and campaign, initial path, and domain
Country, region, city, Shopify market, and locale
Device type, browser, and operating system
Email submissions, reward variants, and purchased products or collections
Survey responses, mini quiz answers, and custom user properties
prob. to win isn't shown while filters are applied.
š”Tip: A test that looks even overall can have a clear winner within a segment, such as mobile shoppers or paid traffic. Check your most valuable segments before you decide.
Decide whether you have a winner
Early results are noisy, and a weaker variant often leads at first. Before you pick a winner:
Wait for check marks on the metrics that matter most to you.
Run the test for at least a full week, so both weekday and weekend shoppers are included.
Expect some metrics to settle sooner than others. Bounce rate draws on far more data than conversion rate, so it settles first. If the two seem to disagree early on, that's usually a difference in data volume, not a real conflict.
If a variant is clearly underperforming, you can stop sending it new visitors without ending the test. Select the arrow next to Pick as winner on that variant, then select Hide. Visitors already assigned to it keep seeing it until the test ends. You can only hide a variant while at least 2 others are still getting traffic.
End the test and pick a winner
On the results page, make sure the date range is set to Full test duration.
Under the winning variant, select Pick as winner.
In the End test and pick winner window, select End test.
You can also select Pick as winner on the campaign page.
When the test ends:
The winner keeps running as the campaign's only live popup.
The other variants move to Drafts, so you can test them again later.
The test and its results stay on the A/B Tests page, marked with the winner.
š”Note: You can't add a new variant to a running test. Pick a winner first, then start a new test that includes the new variant.
For how to set up your next test, see Setting up A/B tests in Alia.
FAQ
My variant has a higher submit rate but fewer signups. Which one is better?
Usually the one with more total signups. Submit rate only counts visitors who saw the popup. A variant that shows later, or only on scroll or exit intent, reaches fewer visitors. Its rate can be higher while it collects fewer signups overall. Compare total email and phone submissions, and attributed sales.
Signups went up, but attributed conversion rate went down. Is that bad?
Not necessarily. Attributed conversion rate is orders divided by signups. When a variant brings in many more signups, a slightly smaller share of them may order, even while total orders and sales go up. Check attributed sales and sitewide conversion rate to see the full effect.
Conversion rate went up, but orders went down. How is that possible?
The same math works in reverse. If a variant collects fewer signups but a similar number of orders, its conversion rate rises even though it brought in fewer customers. Look at total signups and sales alongside the rate.
Why does prob. to win show "-"?
The test hasn't collected enough data yet, you have filters applied, or the variant is hidden. It also doesn't load during a test's first day. For tests that have run longer than 30 days, hover over the variant on the campaign page to load it. Popup vs no popup tests never show it, because the no-popup variant has no popup views.
Why is Pick as winner greyed out?
You're looking at a date range other than the full test duration. Change the date range back to Full test duration.
Why do my variants have very different visitor counts?
Check Test settings for an uneven traffic split, or for a variant that was hidden partway through. To find Test settings, select the arrow next to View test results on the campaign. If the split is uneven, turn on Adjust all stats for visitors.
Is sitewide sales shown before or after discounts?
Sitewide sales is a single total, with no before-and-after-discount breakdown on this page or in Custom Reports. The before-or-after option applies to attributed sales and average order value.
Still have questions?
Start a chat with Lia, our virtual agent, from inside Alia. Lia can hand you off to our support team. Fully Managed customers can also ask their Customer Success Manager.