A/B Testing

Overview

A test splits a widget's traffic between two variations, A and B, each running its own algorithm, so a retailer can see which one performs better. Tests live inside the Experiences section of the Recs console, run at the widget level, and are available on request only, not on every site key.

How Widget A/B Testing Works

Compare two algorithms on the same widget to identify which performs better.

Recommendation Widget

↓ ↓

Variation A

Algorithm A

Results A

Variation B

Algorithm B

Results B

↓

Compare Results

Identify which algorithm performs better.

📘
Important Point to Remember

A/B testing is available under Experiences in the Recs console.

Testing runs at the widget level, comparing two variations (A and B), each using a different algorithm.

Enablement is available on request and is not supported for every site key.

Set Up a A/B Testing

In the Recs console, navigate to Experiences and click Create experience. The A/B testing banner appears.
Name the experience, pick the page type, select the widget, and set the duration.
Turn on A/B test. The configuration panel opens.

Set the traffic split, winning KPI, test duration, and preferred variation from the table below, then click Apply.

Setting	Desscription
Traffic split	Percentage of widget traffic sent to variation A versus variation B
Winning KPI	Click Rate, Cart Rate, Conversion Rate, Average Order Value or Revenue Per Visit
Test duration	How long the test runs
Preferred variation	A or B, used if there is no clear winner
Algorithm per variation	The algorithm each variation runs

Pick an algorithm for variation A and a different algorithm for variation B.
Click Publish. The test shows as Upcoming or Live depending on the dates set, and Expired once it ends.

Recommendation A/B VS Search A/B Testing

Recs A/B tests have no default variation, unlike Search, and that is deliberate.
In Search, a base ranking returns products whether or not a merchandising strategy exists, so a default option is meaningful. In Recs, a widget shows products only when an experience exists, so there is no baseline to compare against.
A Recs test compares one experience against another experience within the same widget.

Reporting and Metrics

Once a test is live, results are available in two places.

The Experiences listing page shows each test's status as

Live
Upcoming
Expired

The algorithm runs on variation A and variation B, and these metrics: impressions, clicks, carts, orders, revenue and conversion rate.

To refer to the detailed reports per experience results navigate to Reports > A/B Tests.