We're sorry but our site requires JavaScript

How to prioritize A/B testing hypotheses

In the process of product development, different teams have various improvement ideas to test. However, the launch of the experiment requires resources and time. Obvious that it is not possible to test absolutely all ideas at once, and it is crucial to understand which of the hypotheses are most important and in which sequence they will be tested.

In this article, we will share what logic we use to prioritize the hypotheses and how this helps us to create a hypotheses roadmap.

Essential hypotheses prioritization parameters

article image

Hypothesis implementation complexity

This is a simple one. The more complicated the hypothesis development and implementation process, the more time and money required.

Traffic volume

In order to achieve representative results, a certain amount of data required. It is important to look at the amount of traffic on a particular funnel step where we are going to test the hypothesis, not at the total product traffic amount. It is critical to estimate in advance how long it will take to test the hypothesis, because one experiment can take several months.

Metric type

Different metrics can validate experiments. Not every experiment validated by conversion rate. Such metrics as time per scenario / time per page / number of pages viewed and similar do not require accumulation, because these metrics do not have a business cycle. If the experiment metric is conversion / revenue / average check, then these metrics have a business cycle, but typically not too long. However, if our experiment aimed to test metrics like Life Time Value, then it’s more challenging, and you have to wait long enough before getting to analysis.

Growth potential

This is perhaps the most ambiguous parameter that appears in the priority assessment. It can even be called cumulative, and usually answers the following questions:

  1. how was the hypothesis formed? User research, support requests, data analysis, previous A/B testing insights or just an idea?
  2. whether similar features have been tested before?
  3. which segment of the audience are we trying to influence with this experiment?

Based on these 4 parameters, we are able to prioritize an extensive list of hypotheses. Let's try this with an example.

article image

Let's look at the table in details. We describe the hypothesis so that all team members are aware of what is being verified and where. Based on the knowledge gained, we prioritize one or another hypothesis. When all priorities set, we find the average between all quantitative estimates. The hypothesis with the highest SCORE is the most important, and then descending. According to this logic, our hypotheses will have the next priority.

article image

Now the hypotheses are prioritized and can be executed.

Important points to consider:

  • Prioritization logic and methodology should be discussed with the team so that the process could be clear and transparent.
  • The product development vector may change from time to time, and the hypothesis roadmap should be changing too.
  • The roadmap could contain hypotheses that are somehow similar to each other

Obviously, hypothesis prioritization is an important process. This document is an excellent way to align all team members and make decisions together based on the data you have. The format we presented is easily adapted to different teams and products, and you can easily add different requirements you think are necessary.