Ways to Optimize Android Application Perfomance

After a sufficient amount of data collected and AB test completed, some of you might think it’s time for data analysis, but before we start to analyze we need to ETL the data. ETL stands for Extract, Transform, Load. In this article, we will talk about the transform part, which is in our case cleaning. Often the collected data contains outliers, and let’s say, if we want to apply a parametric method to compare means, then this method requires robust data. Let’s talk about an experiment with “average check” metric. In the average check dataset, we want to find mean value. In this case, outliers could have a significant influence on our mean value. Let’s say if the average check of a regular customer is $30, but there are customers who typically spend $300, the mean value will be skewed. One of the options is to find and remove the outliers from the dataset manually. Sounds easy, right? However, the challenge is if our dataset contains millions of observations.Obviously, we have many outliers in this dataset. Most likely, these outliers are the clients who spent significantly more as compared to an average customer, these are also known as so-called “whales”. These users may generate a significant revenue share, and they must be analyzed separately, and one of the reason is that their behavioral patterns are most likely to differ. We’ll uncover this topic in future posts. So, how do we deal with such a dataset, to stabilize and clean the outliers? Let’s delve a little bit into the theory. Three sigma rule — almost all values of a normally distributed random variable lie within (x̅-3σ; x̅+3σ). With a probability 0.9973, the value of a normally distributed random variable lies in the specified interval (if the value is not obtained as a result of sampling).We can use the following function in R language to exclude all those data that lies outside of three standard deviations from the mean. After applying this function, you will clean a significant amount of outliers in your dataset. Worth to note though, that you should use this method very carefully because it can remove too much data in your sample. This number could be %30 or even more. There are many more data cleaning approaches. In this article, we have reviewed only one of them which is pretty simple and quite popular. In practice, a data cleaning method depends on the data quality and nature. Couple less conservative methods to mention the Box-Cox transformation or methods using the median to compare samples, such as Kruskal-Wallis criterion. We will discover other data cleaning methods in the coming articles.

When they talk about a/b tests, most often we are talking about conversion. Conversion is a brilliant metric for monitoring a product and validating experiments. The conversion is easy to understand, easy to interpret, and simple enough to count. In fact, conversion is just one of the indicators and is not the main metric for all types of businesses. If we imagine an average online store, then it is much better for it to increase the volume of the average receipt or the number of purchases per user than to focus solely on the conversion to the first purchase. There are quite a large number of growth metrics and behavioral metrics that form an entire hierarchical system. If the product focuses on different indicators, it is logical that experiments should not be limited only to conversion.Conversion is exceptionally popular due to its pivotal role in determining the success of various initiatives, particularly in the realm of business and marketing. The concept of conversion revolves around turning potential customers or users into actual buyers or subscribers, reflecting a tangible impact on a company's bottom line.  Several factors contribute to the popularity of conversion! Measurable Impact: Conversion provides a clear and measurable metric to evaluate the effectiveness of strategies. It offers a quantifiable way to assess the success of marketing campaigns, user experiences, and product offerings. Direct Revenue Generation: Conversions directly influence revenue generation. Increasing the conversion rate means more sales or sign-ups, leading to enhanced profitability and growth opportunities. Resource Efficiency: Focusing on improving conversions allows businesses to allocate resources more efficiently. By honing in on optimizing conversion rates, companies can target specific areas for improvement, leading to higher return on investment (ROI). Enhanced Customer Insights: Conversion analysis provides valuable insights into customer behavior, preferences, and pain points. This information can guide refinements in marketing strategies and product development to better cater to customer needs. Comparative Analysis: Conversion rates enable comparative analysis of different strategies, channels, or designs. This helps businesses identify which approaches are most effective and make informed decisions based on empirical evidence. A/B Testing Possibilities: The discrete nature of conversion data makes A/B testing and experimentation simpler, allowing businesses to test variations and optimize outcomes. Goal Clarity: Conversions represent a clear and common goal across diverse industries and campaigns. This shared objective makes it easier to communicate and align efforts within teams. Decision-Making Tool: Conversion data serves as a valuable tool for strategic decision-making. It informs businesses about the viability of their initiatives and guides adjustments for better results. Continuous Improvement: The pursuit of higher conversion rates fosters a culture of continuous improvement. Businesses strive to refine their processes, user experiences, and offerings to consistently enhance their performance. Competitive Edge: Given the competitive landscape, companies that effectively master conversion optimization gain a competitive edge by outperforming peers and capturing a larger share of the market.Experiments can test completely different hypotheses, such as the impact of site loading speed or the impact of a new recommendation system. Different experiments can be aimed at improving completely different indicators. If all tests are validated only by conversion, then it is often possible not to notice improvements where they really are, and vice versa, if all attention is focused only on conversion, then we simply do not see if the experiment had a bad effect on other indicators. Imagine that you are in a dark room and illuminate only one point with a flashlight. In this case, you perfectly see what is happening in the illuminated area, but you have no idea what is happening in other places. In one of the previous articles, we talked about a fairly typical case in our work when the experiment was validated by conversion and as a result, the client could make the wrong decision.Any experiment is a data set. The data has different characteristics and shape. There are continuous metrics, and there are discrete ones, as in the case of conversion. Discrete data always takes the form of a binomial distribution, which is also a normal distribution. What is the form of distribution and why it is important can be read here. In short, completely different calculation methods should be used for metrics with different distribution forms. If the data has a normal distribution form, then you can determine the winner in the test by applying parametric methods, for example, the T test. Parametric methods rely on the mean, standard error, and imply a normal distribution. If we apply the parametric method on an abnormal sample, then there is a very high probability of error. In a future article, we will look at an example of what happens if a parametric method is applied to non-normally distributed data. So why are conversion experiments so much more popular? Because they are much easier to calculate and determine the difference between two independent samples. In this series of articles, we will look at examples of how to determine the impact of an experiment on other metrics:There are three main ways to analyze data that is not normally distributed. -Use nonparametric methods -Use data transformation methods - logarithm as an option -Using bootstrap is a living example of pain information. In the following articles, we will look at how to analyze the above metrics in 3 different ways. When we hear something about A/B testing, most often it's about the conversion rate. The conversion rate is an excellent indicator for monitoring the product and confirming the results of experiments. The transformation is easy to understand, easy to interpret and quite simple to calculate. In fact, conversion is just one of the indicators and is not the main indicator for all types of businesses. For an average online store, it is much better to increase the average receipt or the number of purchases per user, rather than focus solely on the conversion rate. There are quite a large number of growth and behavior indicators that form a hierarchical system of indicators. Thus, if a product measures success based on various indicators, then it is logical that experiments should also be confirmed by various indicators.There are a number of tools on the market that allow launching A/B tests which are mostly aimed to improve the conversion rate. In my opinion, these companies laid the foundation for how most people perceive the concept of AB testing nowadays. Deeper look Another stereotype is that A/B testing aimed to improve only visual website elements, like buttons, colors, text. In fact, any hypotheses could be tested. As an example, we’ve run experiments to determine the influence of website page loading speed or a new recommendation system on various product indicators. Different experiments can be aimed at improving completely different indicators. If all tests are validated only by conversion rate, then you miss growth opportunities. Another thing is that, if all attention is focused on conversion rate, then we simply do not see if the experiment had a negative impact on other metrics. Imagine that you are in a dark room and a flashlight illuminates only one spot. In this case, you perfectly see what is happening in the illuminated area, but you have no idea what is happening in other room areas. In one of the previous articles, I talked about a fairly typical case in our work, when an experiment was validated by conversion rate and as a result, a client almost made a wrong decision. Any test is just a data set. The data have different characteristics and forms. There are continuous metrics with values from minus infinity to plus infinity, and discrete, with values, 0 or 1, yes or no. As you can see the conversion rate is a discrete type of data. Discrete data always takes the binomial form of distribution, which is also a normal form of distribution. What is a form of distribution and why it’s important, you can read in the article A/B testing: the importance of Central limit theorem. In a nutshell, different statistical methods should be applied to data with a different type of distribution. If the data has a normal form of distribution, then you can determine the winner in the experiment using parametric methods like Student T-test. Parametric methods rely on the mean, standard error and assume a normal distribution. If we apply a parametric method on a not normally distributed sample, then there is a very high probability of making an error. In a future article, we will walk through a case study showing what happens if a parametric method applied to non-normally distributed data. So, why the experiments with conversion rate are much more popular? It is because the discrete type of data is easier to calculate and to determine a difference between two independent samples. In this series of articles, we will study how to analyze experiments aimed at improving different metrics. - The Average Revenue Per User - Average Time on Page - Purchase Frequency Per User I picked these 3 metrics because all of them are united by one property - these metrics have continuous data and most often do not have a normal form of distribution. That means, we can’t use parametric methods to assess experiments. How to analyze continuous metrics with non-normal distribution There are mainly three ways to analyze non-normally distributed data. -Use nonparametric methods -Use data transformation methods - logarithm, for instance -Use bootstrap - a live example of the Law of large numbers. In the following articles, we will look at how to analyze all 3 metrics in 3 different ways.

Data can bring a huge value into product development and growth. For example with data, you can launch experiments to define what product feature is going to resonate with your customers before you implement it or build classification models which based on a user 1st session can identify if the user is going to be loyal or going to churn. Sounds pretty cool and powerful, right? However, before you start using the data, you need to collect and store it properly. Most people think that data scientists spend most of their time building algorithms and machine learning models, but the truth is that vast amount of time consumed just to get, prepare and clean the data for analysis.  Building a reliable process to collect and robust architecture to store the data is a crucial component of product success. The easier it is to get the data from the data warehouse, the faster you get results. With а pace of product growth, you get more and more data from different sources. All that data needs to be combined and stored in one place. In this case, Universal Analytics becomes insufficient. Such option as building own data storage warehouse is quite an expensive, lengthy and complicated process, so the question of how to quickly build an affordable, but high-quality data aggregation and storage solution often pops up. Snowplow is a JavaScript based tracker that allows sending events to a cloud database using an API and data transfer protocols. One of the Snowplow’s advantages is a free license and easy implementation. Also, Snowplow has a similar to Universal Analytics events logic and syntax, and by just changing few parameters we can migrate from UA. The same as in Universal Analytics, Snowplow allows collecting e-commerce data, advanced parameters as well as a wide range of various indicators. After the data is collected, and stored in Amazon redshift, one of the options is to use Amazon's internal interface to write SQL queries to get the data, but this is not the most convenient process, and we recommend the following: 1) BI analytics tools Amazon has various cloud BI tools integrations with the ability to pull the data using SQL / R / python. Couple tools we want to highlight are Modeanalytics and Redash. Both tools are very affordable but have a lot of useful features to visualize data and build clear and meaningful dashboards. 2) Raw data To work with raw data, build models and statistical data processing there is an option to connect to Amazon directly through IDEA R or Python. This R library allows connecting to Amazon Redshift so you can get the data and process it according to your requirements. Data architecture is a complicated thing, and it must continually evolve along with the product. The solution described in this article is a great start which allows focusing resources on the product instead of diving into a long, complicated development. In conclusion, I want to say obvious and very logical, but the most important thing that every digital product company must follow since a very early stage. The most important in data analysis is the data quality. Not algorithms, not magic data science stuff. If you don’t have data collected and stored properly, there is no way you can get any insights of that.

If you use experiments to evaluate a product feature, and I hope you do, the question of the minimum sample size required to get statistically significant results is often brought up. In this article, we explain how we apply mathematical statistics and power analysis to calculate AB testing sample size. Before launching the experiment, it is essential to calculate the experiment potential, ROI and time required to get statistical significance. The experiment can not last forever. However, if we don’t collect enough data, our experiment gets small statistical power, which doesn't allow us to determine the winner and make the right decision. Let's start with terminology. Statistical power is the probability that one or another statistical criterion can correctly reject the null hypothesis H0, in the case when the alternative hypothesis H1 is true. The higher the power of the statistical test, the less likely you can make type II error. Type II error is tightly related to 1) Difference magnitude between the samples - Effect size 2) the number of observations 3) the spread of data The power analysis allows you to determine the sample size with a specific confidence level which is required to identify the effect size. Also, this analysis makes it possible to estimate the probability of detecting the given value effect size with a specified degree of certainty with a given sample size. The most important is the number of observations: the larger the sample size, the higher the statistical power. With "sufficiently" large samples, even small differences are statistically significant, and vice versa, with small samples, even large differences are difficult to identify. By knowing these patterns, we can determine in advance the minimum sample size required to get a statistically significant result. In practice, usually, a test power equal to or greater than 80% is considered acceptable (which corresponds to a β-risk of 20%). This level is a consequence of the so-called "one-to-four trade-off" relationship between the levels of α-risk and β-risk: if we accept the significance level α = 0.05, then β = 0.05 × 4 = 0.20 and the power of the criterion is P = 1-0.20 = 0.80. Now let's look at the effect size. There are two approaches to calculating the required sample: 1) Calculating using the confidence level, the effect size, and the power level 2) Applying statistical sequential analysis, which allows calculating required sample size during the experiment Let's investigate the first case, and apply t.test for two independent samples. Let's assume, we test a hypothesis aimed to improve “item to wishlist” conversion rate. Delta, which covers costs of the experiment with a six months return >= 5% gain of the mentioned conversion rate. This >= 5% gain results in additional profit, which covers all the resources invested in the experiment. In addition to this, you want to be 90% sure that you will find the differences if they exist, and 95% - that you do not accept the differences that are random fluctuations. d - delta, siglevel - confidence level, power - power level pwr.t.test(d=.05, sig.level=.05, power=.9, alternative="two.sided") Two-sample t test power calculation n = 8406.896 d = 0.05 sig.level = 0.05 power = 0.9 alternative = two.sided NOTE: n is number in each group As a result, 16,814 participants required to get statistically significant results for A / B experiment. Imagine, that this test affects the funnel step with approximately 8000 unique visitors a month. In this case, the test requires 2 months. Let's take a look at another case when stakeholders want to get results in a couple of weeks. In this case, we have an approximate sample size of 4000 visitors and the delta >=5%. We want to know the probability to get statistically significant results under the mentioned circumstances. Add n, remove power pwr.t.test(d=.05, n=2000, sig.level=.05, alternative="two.sided") Two-sample t test power calculation n = 2000 d = 0.05 sig.level = 0.05 power = 0.3524674 alternative = two.sided NOTE: n is number in each group The probability to determine the difference, if any, is 35%, which is not too low and the probability of missing the desired effect is 65%, which is too high. Let’s look at the chart below. It shows clearly the higher the effect size, the lower sample required for a significant result.