How to Run an A/B Test

Samantha Brown Updated by Samantha Brown

Creating an A/B Test

Starting Your A/B Test

As you grow throughout the implementation of the referral program you can experiment with multiple variants to determine the optimal conversion experience. Learn how you can utilize A/B testing to create the best program for your company and customers!

Creating an A/B Test

Step 1

Navigate to Referral > Campaigns > Select or create a Campaign > A/B Testing tab, and click Create A/B Test.

Step 2

Select the variable you'd like to test! You can A/B test either the Advocate Reward, the Friend Offer, or both. For the purposes of this help article, we'll be setting the test variable as both the Advocate Reward + Friend Offer. Click the Next Step button to continue.

Note: Be sure to update the campaign content as necessary in order for the A/B testing to match the variables you are testing. This may include:

-Advocate Widgets Copy
-Advocate Reward Email
-Share Email
-Share Messages
-Friend Widgets Copy
-Friend Offer Email
-Terms and Conditions
-OG Tags
-Ribbon Copy
What if I don't see a relevant option to select for testing?
We’re still working on adding additional variables for you to select from, but in the meantime, you may select any of the above 3 options and edit the relevant assets for your test as needed, such as the Settings, Widgets, Emails, etc.
Step 3

In the General Settings step, enter the Test Name you would like to call the A/B test, along with the number of Variants (2, 3 or 4) you would like to test, and any Hypothesis about the details or purpose of your test. Click the Next Step button to continue.

Step 4

In the Weights section, you can control what percentage of customers will see the Control variant or the Challenger variant when landing on one of your tested widgets. For the Original Campaign box, you can enter in the percentage of traffic that you'd like to see the original offer instead of your running A/B test. In the Control & Challenger B boxes, enter the percentage of traffic you would like to have directed to the Control (Variant A) versus the Challenger (Variant B).

To Run an A/B Test on 100% of Your Overall Website Traffic:

Set the Original Campaign value to 0%, as this will ensure that all new customers who arrive on the site will be included in part of the A/B test. In the Control and the Challenger B boxes, enter in what percentage of customers you'd like to see one variant over the other. Below, we have an example of a setup for a standard 50%/50% weight A/B test.

Note: If the original offer offer in your campaign is $10, and you’d like to test a $20 offer, 50% of new users will receive $10 and 50% will receive $20, all referral links prior that were created prior to this test will not be included in the test and they will still see $10. When setting up this configuration, you can leave the Weights as shown in the screenshot above.
To Run an A/B Test on a Percentage of Your Overall Website Traffic:

Determine what percentage of your overall traffic that you'd like to see the A/B test. For example, if you'd like to test 30% of your total traffic, you'd want to set the Original Campaign value to 70%. In the Control and the Challenger boxes, split the remainder of the testing percentage between the two variants. Using the same 30% testing example, you might set the Control and the Challenger B variants to 15% each to test the remainder of the traffic equally.

Note: All referrals created prior to the A/B test will always be shown the original offer

Adding the Final Details & Starting Your Test!

Step 1

After clicking Finish, your new draft campaign will be shown in the A/B Testing tab. Click Edit to get started!

Step 2

Upon clicking Edit, you'll be taken to an overview of the draft you created during your A/B Testing configuration. From this screen, you can Start Test, Edit Test, or Delete the draft. This is also where you can make edits to the Control and the Challenger variants set up for the test.

Step 3

The next step to getting the test started would be editing the variants as needed. For this help article, we'll be testing both the Friend Incentive as well as the Advocate Reward, so we'll be making a few changes to the Challenger variant prior to starting our official A/B test.

If you need to edit your Control variant, first disable Use Original Campaign Settings. For now, we'll start off by clicking Edit Variant under the Challenger (Variant B).

Step 4

After clicking Edit Variant, you'll be taken to the following screen. The Edit Variants screen is your overview of all of the different items being tested and will help you understand what items would need to be configured prior to starting your test.

By default, the Control variant settings will be locked unless the Use Original Campaign Settings feature toggle is disabled. For the sake of this help article, we will be enabling the Original Campaign Settings for the Control variant.

Step 5

Once in the Edit Variant menu, you can start setting up the configuration for the A/B test! For our test, we will be making changes to both the Advocate Reward as well as the Friend Incentive. Along with this, make sure we edit Widgets, Emails, and OG Tags as necessary to match the tested Reward and Offer Amounts. Be sure to navigate through each tab to check that all configurations match the desired testing scenarios.

When everything has been set up and is ready to go, remember to click Save Changes in the upper right corner.

Note: When A/B testing a Friend Incentive or a Reward using a Code Bank, be sure to create a new Code Bank as necessary to be distributed specifically for the new tested offer(s).
Step 6

You are now ready to start your A/B test! We would recommend doing one last quick review of the settings to ensure that everything is correct then click Start Test and confirm again after you receive the pop-up modal to Start Test Now. Then all changes made to the Campaign will be published immediately.

Congratulations! You have launched an A/B test for your referral program, and now you can sit back, relax, and collect the results to further improve your growth with the Friendbuy Referral Program. Be sure to check out real-time results of the A/B test in the Analytics > Dashboard > A/B Tests section once it is running.

Note: Once the A/B test has been started, it can only be either Ended or Paused. If you accidentally start an A/B test early but realize an error right away, you can Pause the A/B test to make any changes as necessary before deciding to Resume the test again.

We would only recommend doing this if this was realized immediately after starting the A/B test so that analytics are not skewed by improper data.
Review: Referral links that are created before an A/B test starts are not included in the test. Referred Friends that click on those original links will continute to see the original campaign offer and will not be counted in the A/B test results.

Similarly, referral links that are created during the A/B test will continue to preserve the A/B test offer(s) for referred Friends after an A/B test has ended or is paused.

How did we do?

Shopify Coupon Codes Integration

How to Change Your Referral Offer

Contact