A/B Testing for In-App Surveys

A/B testing compares two or more versions of a survey to see which one performs better. For surveys, that usually means a higher response rate.

Refiner doesn’t include a dedicated A/B testing tool. You can run a simple A/B test for Web and Mobile App surveys with the steps below.

Create a variation

Start by duplicating your existing survey. Open the three-dot menu on the survey and select Duplicate.

Name the original survey “Variation A” and the duplicate “Variation B” so you can tell them apart. The duplicate is created in draft mode. Leave it in draft until you finish the steps below, so users don’t see an unfinished variation.

Make changes to the variation

Edit the duplicate and make the change you want to test. You might reword a question, add or remove a question, or adjust the survey design.

Change one thing at a time. If you change several elements at once, you can’t tell which change affected the response rate.

Keep the Target Audience and launch trigger the same as on the original survey. Both variations need to reach the same users in the same situations. The change you are testing should be the only difference between them.

Prevent overlap between variations

When two surveys match at the same time, Refiner shows one of them at random. That split is your A/B test. Use Prevent Overlap so the same user doesn’t see the other variation later.

Open the Advanced Options tab. Under Cross-Campaign Behavior, enable Prevent Overlap on both surveys:

• On Variation A, select Variation B in Seen surveys.
• On Variation B, select Variation A in Seen surveys.

Leave the lookback period on All time, so a user who already saw one variation stays excluded from the other.

Let the test run

Publish both variations. Each survey collects its own responses. Open a user’s details panel to confirm they saw only one variation.

Wait until each variation has enough views before you compare them. We recommend at least 1,000 survey views per variation.

Create an ad-hoc dashboard for each variation and compare the response rates. If one variation performs clearly better, keep that version and stop showing the other.