Skip to main content

How to A/B test popup vs. no popup

Measure what your popup does to sales and bounce rate by testing it against showing nothing.

Written by Rob D

A popup vs. no popup test (also called a holdout test) shows whether your popup affects sales or makes shoppers leave. Half your visitors see the popup and half see nothing, and you compare how each group shops. Alia doesn't have a separate "no popup" option. Instead, you make a copy of your popup with no triggers, so it never opens.

Set up the test

  1. On the Campaigns page, open the campaign you want to test.

  2. Select Add test variant. Alia copies your live popup into Drafts.

  3. Open the draft in the popup editor and rename it "No popup" so it's easy to spot in the results.

  4. Click an empty area of the canvas so nothing is selected, then open the Triggers tab in the right sidebar.

  5. Delete every trigger, including Smart Trigger if it's on.

  6. If the draft shows Not ready, select Mark as ready.

  7. Back on the campaign, select Start new A/B test. Enter a Title, check the Control and Traffic weight, then select Start test.

Visitors assigned to the "No popup" variant count as part of the test but never see a popup.

💡 Note: For Subscription Upsell campaigns, see Understanding your Subscription Upsell analytics.

Read the results

Nobody can sign up in the no-popup variant, so signup numbers don't help here. Compare these rows instead:

  • Sitewide conversion rate, sales, orders, and average order value. These count every visitor in each variant, whether or not they saw a popup, within the Conversion window (20 days by default).

  • Bounce rate.

prob. to win and AI recommendations never appear in this kind of test, because they wait for every variant to have popup views. Judge the result from the check marks on the sitewide and bounce rate rows.

Early results can be noisy. Bounce rate usually settles before conversion rate because it has more data behind it. Give the test at least a full week before you draw conclusions.

💡 Tip: Lia, Alia's built-in AI assistant, can explain your results, including why Alia's numbers differ from Shopify's. See How to ask Lia about your results.

Want a gentler test?

If you'd rather not remove the popup completely, test a longer delay or an exit-intent trigger against your current trigger. That still reduces friction while keeping the popup.

Frequently asked questions

Why does the no-popup variant show visitors but no popup views?

That's expected. Its visitors are assigned to it but never see a popup.

Why doesn't prob. to win show?

It needs popup views in every variant, and the no-popup variant has none.

Why are attributed orders empty for the no-popup variant?

Attributed metrics count orders after a signup, and nobody signs up in that variant. Use the Sitewide rows instead.

Still have questions?

Start a chat from inside the Alia app, or use the chat in our Help Center.

Did this answer your question?