When they talk about a/b tests, most often we are talking about conversion. Conversion is a brilliant metric for monitoring a product and validating experiments. The conversion is easy to understand, easy to interpret, and simple enough to count. In fact, conversion is just one of the indicators and is not the main metric for all types of businesses. If we imagine an average online store, then it is much better for it to increase the volume of the average receipt or the number of purchases per user than to focus solely on the conversion to the first purchase. There are quite a large number of growth metrics and behavioral metrics that form an entire hierarchical system. If the product focuses on different indicators, it is logical that experiments should not be limited only to conversion.Conversion is exceptionally popular due to its pivotal role in determining the success of various initiatives, particularly in the realm of business and marketing. The concept of conversion revolves around turning potential customers or users into actual buyers or subscribers, reflecting a tangible impact on a company's bottom line. 
Several factors contribute to the popularity of conversion!
Measurable Impact: Conversion provides a clear and measurable metric to evaluate the effectiveness of strategies. It offers a quantifiable way to assess the success of marketing campaigns, user experiences, and product offerings.
Direct Revenue Generation: Conversions directly influence revenue generation. Increasing the conversion rate means more sales or sign-ups, leading to enhanced profitability and growth opportunities.
Resource Efficiency: Focusing on improving conversions allows businesses to allocate resources more efficiently. By honing in on optimizing conversion rates, companies can target specific areas for improvement, leading to higher return on investment (ROI).
Enhanced Customer Insights: Conversion analysis provides valuable insights into customer behavior, preferences, and pain points. This information can guide refinements in marketing strategies and product development to better cater to customer needs.
Comparative Analysis: Conversion rates enable comparative analysis of different strategies, channels, or designs. This helps businesses identify which approaches are most effective and make informed decisions based on empirical evidence.
A/B Testing Possibilities: The discrete nature of conversion data makes A/B testing and experimentation simpler, allowing businesses to test variations and optimize outcomes.
Goal Clarity: Conversions represent a clear and common goal across diverse industries and campaigns. This shared objective makes it easier to communicate and align efforts within teams.
Decision-Making Tool: Conversion data serves as a valuable tool for strategic decision-making. It informs businesses about the viability of their initiatives and guides adjustments for better results.
Continuous Improvement: The pursuit of higher conversion rates fosters a culture of continuous improvement. Businesses strive to refine their processes, user experiences, and offerings to consistently enhance their performance.
Competitive Edge: Given the competitive landscape, companies that effectively master conversion optimization gain a competitive edge by outperforming peers and capturing a larger share of the market.Experiments can test completely different hypotheses, such as the impact of site loading speed or the impact of a new recommendation system. Different experiments can be aimed at improving completely different indicators. If all tests are validated only by conversion, then it is often possible not to notice improvements where they really are, and vice versa, if all attention is focused only on conversion, then we simply do not see if the experiment had a bad effect on other indicators. Imagine that you are in a dark room and illuminate only one point with a flashlight. In this case, you perfectly see what is happening in the illuminated area, but you have no idea what is happening in other places. In one of the previous articles, we talked about a fairly typical case in our work when the experiment was validated by conversion and as a result, the client could make the wrong decision.Any experiment is a data set. The data has different characteristics and shape. There are continuous metrics, and there are discrete ones, as in the case of conversion. Discrete data always takes the form of a binomial distribution, which is also a normal distribution. What is the form of distribution and why it is important can be read here. In short, completely different calculation methods should be used for metrics with different distribution forms. If the data has a normal distribution form, then you can determine the winner in the test by applying parametric methods, for example, the T test. Parametric methods rely on the mean, standard error, and imply a normal distribution. If we apply the parametric method on an abnormal sample, then there is a very high probability of error. In a future article, we will look at an example of what happens if a parametric method is applied to non-normally distributed data. So why are conversion experiments so much more popular? Because they are much easier to calculate and determine the difference between two independent samples.
In this series of articles, we will look at examples of how to determine the impact of an experiment on other metrics:There are three main ways to analyze data that is not normally distributed.
-Use nonparametric methods
-Use data transformation methods - logarithm as an option
-Using bootstrap is a living example of pain information.
In the following articles, we will look at how to analyze the above metrics in 3 different ways.
When we hear something about A/B testing, most often it's about the conversion rate. The conversion rate is an excellent indicator for monitoring the product and confirming the results of experiments. The transformation is easy to understand, easy to interpret and quite simple to calculate. In fact, conversion is just one of the indicators and is not the main indicator for all types of businesses. For an average online store, it is much better to increase the average receipt or the number of purchases per user, rather than focus solely on the conversion rate. There are quite a large number of growth and behavior indicators that form a hierarchical system of indicators. Thus, if a product measures success based on various indicators, then it is logical that experiments should also be confirmed by various indicators.There are a number of tools on the market that allow launching A/B tests which are mostly aimed to improve the conversion rate. In my opinion, these companies laid the foundation for how most people perceive the concept of AB testing nowadays.
Deeper look
Another stereotype is that A/B testing aimed to improve only visual website elements, like buttons, colors, text. In fact, any hypotheses could be tested. As an example, we’ve run experiments to determine the influence of website page loading speed or a new recommendation system on various product indicators. Different experiments can be aimed at improving completely different indicators. If all tests are validated only by conversion rate, then you miss growth opportunities. Another thing is that, if all attention is focused on conversion rate, then we simply do not see if the experiment had a negative impact on other metrics. Imagine that you are in a dark room and a flashlight illuminates only one spot. In this case, you perfectly see what is happening in the illuminated area, but you have no idea what is happening in other room areas. In one of the previous articles, I talked about a fairly typical case in our work, when an experiment was validated by conversion rate and as a result, a client almost made a wrong decision.
Any test is just a data set. The data have different characteristics and forms. There are continuous metrics with values from minus infinity to plus infinity, and discrete, with values, 0 or 1, yes or no. As you can see the conversion rate is a discrete type of data. Discrete data always takes the binomial form of distribution, which is also a normal form of distribution. What is a form of distribution and why it’s important, you can read in the article A/B testing: the importance of Central limit theorem. In a nutshell, different statistical methods should be applied to data with a different type of distribution. If the data has a normal form of distribution, then you can determine the winner in the experiment using parametric methods like Student T-test. Parametric methods rely on the mean, standard error and assume a normal distribution. If we apply a parametric method on a not normally distributed sample, then there is a very high probability of making an error. In a future article, we will walk through a case study showing what happens if a parametric method applied to non-normally distributed data. So, why the experiments with conversion rate are much more popular? It is because the discrete type of data is easier to calculate and to determine a difference between two independent samples.
In this series of articles, we will study how to analyze experiments aimed at improving different metrics.
- The Average Revenue Per User
- Average Time on Page
- Purchase Frequency Per User
I picked these 3 metrics because all of them are united by one property - these metrics have continuous data and most often do not have a normal form of distribution. That means, we can’t use parametric methods to assess experiments.
How to analyze continuous metrics with non-normal distribution
There are mainly three ways to analyze non-normally distributed data.
-Use nonparametric methods
-Use data transformation methods - logarithm, for instance
-Use bootstrap - a live example of the Law of large numbers.
In the following articles, we will look at how to analyze all 3 metrics in 3 different ways.