Our Blog

AWSMD is a renowned UI/UX design & web development company offering mobile development, branding, website design & development services for ambitious companies.  AWSMD is a renowned data-driven UI/UX and development company offering mobile development, branding, website design & development services with 7+ years of expertise for tech companies, ambitious startups, and seasoned experts. We take pride in delivering flexible partnerships, creating reliable web solutions, implementing robust processes, and delivering timely quality results. AWSMD is a client-focused web design & development company with a headquarters in San Francisco that started its work in 2015. We create website solutions that deliver tangible business results, both local and international. Our skilled web development staff has a distinguished track record and experience of working with Intel, UBER, Oracle, Nutanix, GoFundMe, Upside Travel, CrossLead, New Balance, and State Farm. We are delighted to declare that our experienced and professional team quickly went to work to make all your ideas a reality. Our Front End & Backend Developer Team provides elite website development services to help your brand in the ever-changing digital world. No matter how complex or hard to implement your idea is, we are here to create a custom development project and achieve your key businesses goals to: stand out from the competitors; enhance the brand functionality; fully right towards your niche; build trust with customers. Custom website development We are focusing on what we do best, no matter what challenges your business faces, our experts develop customized web development solutions truly unique and tailored to your needs. No cookie-cutter projects! Just websites that can be competitive, ideally designed, user-friendly, stable, and with convenient navigation. Our team is always flexible and proactive and combines interactive components with end-to-end web development solutions, SEO-optimized design, and customized branding to level your company among others and deliver exceptional web results. AWSMD is always engaged in AB tests, and we know what solutions will be business-oriented because the data-driven approach is our prerogative. Graphic design and branding Any custom website or mobile app is not complete without custom branding or graphic design elements because it gives your product a sense of identity and value. That is why our custom web development company organizes a full cycle of web development to increase the potential of your business ideas. We offer responsive and adaptive web design services, intuitive navigation development, amazing graphic design, branding, and beyond to enable your brand to competitive abilities on the market and increase customer satisfaction and loyalty. Custom web design Elevate your product's appeal with our exceptional custom web design services. Our dedicated team creates captivating online experiences that reflect your brand's essence. From seamless navigation to captivating design, we optimize customer satisfaction and leave a lasting impression. CMS Integration Our expert team is continually learning the best WordPress practices for developing and successfully running your business. We use our WordPress expertise, handbooks, official code references, and extensive API integrations, and also offer you scalability, full data migration, design and development, performance, and long-term maintenance. We can take your business to the next level and satisfy all your WordPress needs by integrating it with different CMS platforms like WordPress, Shopify, Webflow, and others. Support and maintenance We make it easy to add a wide range of functionality to your website and check whether your site is using the right tools for the job. We provide end-to-end web development opportunities from strategy to support and maintenance to successfully running your business. Advanced technologies and complete solutions AWSMD - a web design and development agency provides a wide range of web services for creating and developing your business. Our specialists guarantee: to apply the most up-to-date web technologies in development; to make your brand visible to the world and build trust with a target audience; to be functionally superior to the projects of your competitors; to have the technological potential for development. Top-rated team of experts We are ready to make your ideas a reality! It doesn't matter if you have already developed ideas or need to create a project from scratch. The experience of our team was gained over 7+ years and allows us to provide website design and development services of any complexity. Close developers & designers' cooperation We are highly customer oriented and by considering user needs we produce functional products in close cooperation between developers and designers. By selecting our company, you can have confidence in receiving expert web services. Comprehensive services and premium quality Our main specialization is website design & development services. Working on your project, we meet your company's challenges, understand your business, and tailor a comprehensive premium quality service to meet our client's expectations. FAQ What web development services does AWSMD offer? For each project we choose individual solutions, and find the specifics of your business, to make your company stand out against the competitors. Working with every business solution, our team makes a whole complex of actions, which provides high efficiency and quick boosting of your product. What are the deadlines for app development? It will take at least 2 months to create a mobile or web application. But each project requires its own terms. Please contact our specialists, they will be able to guide you on the terms. You will get exact dates only after finishing the discovery stage. Do you provide UI/UX web and mobile development services? Our company provides UI/UX design and development services to develop and enhance your customers' satisfaction. Boost your business with an e-commerce development agency. Use professional custom e-commerce development services to attract new customers and deliver results. Elevate your customer's shopping level and make it easier with custom e-commerce web development services. Our certified front-end and back-end services can help to grow your business with a custom e-commerce website. Our expert team is skilled at building E-commerce solutions of any scale and increasing your profitability. Attract more sales and online traffic with a professionally developed e-commerce website. At AWSMD, we offer solutions optimized for mobile apps, blazing-fast speeds, and responsive web design for platforms like Shopify, Magento, WooCommerce, Prestashop, and others. Convenient cart and checkout Push the limits of the simple purchasing process. Shape your customer's purchasing experience at every step by designing thoughtful shopping cart solutions. Product Catalog navigation Make it easy to find the product in a few clicks and create a new revenue stream. Strengthening your brand with a custom ecommerce website design and a user-friendly product catalog. Wishlist and favorites Make your customers' shopping experience adorable and exciting with existing eCommerce technologies. Attract more sales by using the wishlist and favorites options. With 7+ years of expertise for both tech novices and ambitious startups, and seasoned experts, we take pride in delivering flexible partnerships, creating reliable web solutions, implementing robust processes, and delivering quality results on time. Get a preview of our e-commerce website design process now: Market and target research Custom e-commerce development begins with the market and targeting audience analysis. We create e-commerce sites with UX/UI trends and modern development technologies in mind to make your store work fast and meet the target demands. UX/UI design building Our development team always uses a data-driven approach, implement the latest technological tools, and focuses on UX/UI trends to create interactive and user-friendly web interfaces for your business. Testing the prototype Our team commits to ensuring the quality of the product we create. By testing e-commerce sites, we optimize their performance, fix all bugs, and define the most winning solutions. Website launch Once we make sure that the site prototype is flawless, we launch it on the market, hosting it on your server for you to use. We also provide maintenance and support if you notice any performance issues. E-commerce Expertise Our professional team of over 60 people has 7+ years of expertise for tech novices, ambitious startups, and seasoned experts. We use a variety of e-commerce platforms and technologies to meet your business needs. Stunning design Our e-commerce web design solutions can take your business to the next level. By choosing AWSMD, you get ongoing design services: theme upgrades, user interface optimization, user interface design, and more to help your business achieve greater success. Latest Technologies We will help you improve your website functionality and increase sales using modern solutions like Prestashop, Magento, Shopify, Broadleaf, and WooCommerce. Individual approach Our clients have included such leading companies as Intel, Oracle, UBER, GoFundMe, Upside Travel, Nutanix, CrossLead, New Balance, State Farm, and beyond. We have a vested interest in helping your business grow. Stay digitally competitive with our trusted eCommerce website development services.

These days, to make a business successful, you need to find and create your path, show your uniqueness among your competitors, and build your reputation from the ground up. Instead of chasing after marketing trends and following the crowd, apply the best logo design company and use smart branding services to turn an ordinary consumer product into a successful brand. We pay great attention to the development of your brand - we create a unique logo, design, and packaging, and pay attention to corporate identity to help you connect with your target audience.  Creating a company's brand or a separate product is necessary for every market player if you want to make a profit and stay competitive. Competent branding helps solve such business problems: Bring a new product or service to market. Adapt your brand to international markets. Create effective communication with your target audience. Help you update your website or app and stand out from your competitors. Provide consistency and integrity in communication with customers. We make your business successful by designing captivating interfaces, creating your brand identity concept, and implementing it for your digital products. Our experts create a brand designed for the end consumer. Using modern technology, we convey the tone of voice of your brand, products, and services. Our aim is to provide you with an effective tool for business development, such as: Logo design Logo guidelines Competitor analysis Typography Brochure and stationery design Packaging and product design The marketing strategy task is to select effective marketing solutions to create a positive company image in the eyes of the target audience, partners, and stakeholders. Using a number of modern methodologies we help the client create a value proposition, define the essence of the brand, and scale the business. Here we can offer clients an analysis of the target audience and competitors, finding a brand identity and creative directions. Expertise in branding With over 10 years of experience in branding services, we have collaborated with Intel, UBER, Oracle, Nutanix, GoFundMe, Upside Travel, CrossLead, New Balance, State Farm, and other companies across various niches. Our team of experts possesses extensive knowledge and skills in delivering compelling branding services that captivate users and grow your business. Individual approach Besides the creative design process, our experts also carry out other important processes such as company research, product and industry data collection, and industry trends. We take a holistic and personal approach to showing your business values through branding. To achieve the project’s desired outcomes, regardless if it is the branding of an app from scratch or rebranding an existing website, we always focus on providing exceptional workflow and harnessing innovative technology. Our step-by-step branding process includes these 7 pillars: Market research and strategy planning The most significant stage is where we get to know your business values and goals, your target audience and industry, and your audience. We research the market to gain clarity about your business model and to choose the best branding development process. That will help your brand identity and strategic direction aligns with your internal culture, business goals, and customers` needs. Brand Discovery We guide you to discover your brand values, to show your uniqueness and personality to carve your space in the market. We bring all the elements together to build a brand strategy and develop a unique selling proposition and a base for its visual identity that connects it with consumers. Marketing strategy To stand out your company from the noise, we conduct a top-level analysis of competitors. We establish a common understanding of your identity and use this information in marketing strategy creation. For positioning your business effectively in the market, we create your logo, message, and tagline and use them in digital marketing and hard copy. This solidifies your market position and your brand will be the first thing that customers recognize. Brand Development & Campaign Launches We laid the groundwork for your brand establishment, and now it's time to complete the visual language. Our experienced team develops your business image - update your logo, website fonts, colors, and visuals. After planning and graphic updates, we launch the marketing strategy. We look at every detail, support your needs and provide you with all brand materials to ensure the launch's success.

Want to build a digital product with a beautiful and smooth UI/UX design? At AWSMD, we create UI/UX design products aligned with your business goals. Our UI UX agency offers UI design services to create digital products that users will enjoy. Our team establishes a straightforward design process, delivers a spot-on end result, and meets deadlines.UI design Every detail matters when it comes to UI design, from fonts and colors to button placement and visual elements. A high-quality UI design enhances user interaction, converting potential visitors into loyal customers. Hiring a UI design company ensures the creation of a visually appealing and user-friendly digital product that users will enjoy using. UX design UX design focuses on the user experience, taking into account user needs, desires, and expectations. By prioritizing user-centric design, a user interface design company creates products that are necessary and useful, resulting in higher conversion rates, lower churn rates, and increased customer loyalty. Collaborating with our UI/UX design agency not only enhances the quality of your products, but also helps your business thrive. A well-designed interface builds a memorable and recognizable brand, expands your audience, improves conversion rates, boosts user retention and loyalty, and enables effective data collection and analysis. Diverse Skill Set: UI/UX agencies typically have a team of designers with a diverse range of skills and expertise. This allows them to handle various aspects of your project, from user research and wireframing to visual design and prototyping. With a collective skill set, agencies can provide comprehensive solutions and ensure that all aspects of your UI/UX design are covered. Collaboration and Teamwork: Working with an agency means that you have access to a team of professionals who collaborate closely on your project. Each team member brings their unique perspectives and insights, contributing to the overall success of the design. This collaborative approach ensures that your project benefits from a collective brainstorming process, resulting in more innovative and effective design solutions. Scalability and Flexibility: UI/UX agencies are well-equipped to handle projects of various sizes and complexities. They have the resources and flexibility to scale up or down based on your project requirements. Whether you need additional designers or specialized expertise, agencies can quickly adapt to meet your changing needs, ensuring that your project progresses smoothly and efficiently. Quality Assurance and Industry Standards: Agencies have established processes and quality assurance measures in place to ensure that the final design meets industry standards and exceeds client expectations. They conduct user testing, incorporate feedback, and follow best practices to deliver a polished and user-friendly design. By leveraging their expertise, agencies can help you create a UI/UX design that is not only visually appealing but also aligned with your business goals. Long-term Support and Maintenance: UI/UX agencies often provide ongoing support and maintenance services, even after the initial design phase is complete. They can assist with updates, bug fixes, and improvements to ensure that your design remains effective and up to date. Establishing a long-term partnership with an agency allows you to benefit from their continuous support and expertise, ensuring the longevity and success of your design. AWSMD professionals offer UI/UX design services for products built from scratch and redesigning existing digital products. We give you the customer trust, control, power, and flexibility you need to deliver an aesthetic, engaging, and intuitive experience to the end-users of your digital product. UI/UX Design for Web We are customer focused and pay great attention to the details. Our experts specialize in designing complex interfaces and UX services. Following proven best practices in accessibility, usability, and compatibility, you will get a convenient, user-friendly, and intuitive UX and UI-designed product. UI/UX Design for Mobile Our UI/UX design company focuses on creating a favorable user experience. The main goal of our experienced team is to balance between smooth product solutions and easy-to-use interfaces. Our UX design company applies 7+ years of experience to create authentic and unique spot-on designs for iOS and Android that make your mobile apps smarter and more functional. Wireframing and Prototyping At our UI/UX design agency, our team uses a powerful technique in digital application design and creates wireframes and prototypes. We create and test wireframes using cut-edge tools and techniques to represent the page structure and layout visually, and illustrate the structural arrangement of different page components and the relationships between them. Don't waste your time on features that users do not actually need. We are customer-oriented and pay great attention to the details that make you stand out from the competition. We are a professional team, motivated to do a great job and achieve a great result. Therefore, our UI/UX design process has such steps: Market research: We delve into market trends, competitor analysis, and user preferences to identify the unique features that will captivate your users. Brainstorming Our team conducts strategic brainstorming sessions to devise a brand promotion strategy and define the communication goals that will be translated into the product design. Design Style We meticulously curate the UI design, focusing on color schemes, typography, and icons to create a visually appealing and cohesive user interface. Prototyping We bring your vision to life by creating an interactive prototype that showcases the user experience and ensures seamless interaction with the product. Testing We conduct rigorous usability testing to validate the effectiveness of the design, ensuring it not only captivates users but also provides intuitive navigation and usability that sets you apart from competitors. Launch With a comprehensive and intuitive UI and UX design, your mobile app is ready to make a lasting impression on your target audience. To achieve the project’s desired outcomes and maximize your business growth, regardless if it is the UI/UX design of an app from scratch or redesigning an existing website, we always focus on providing exceptional workflow and harnessing innovative technology. To embark on the journey of developing a captivating website tailored to your specific business needs, simply follow these steps: Contact us The first step is to contact us and initiate the process. Fill out this form{Форма} to kick-start the process. Submit request We create UI/UX designs tailored to specific niches, so it's crucial for our designers to understand your needs. Get result Expect to receive a meticulously designed and fully functional digital project that not only keeps existing clients but also attracts new ones. Our aim is to deliver a resource that effectively serves your business goals.

Any digital product that sells something, whether it is an e-commerce website, subscription-based cloud tool, an online grocery store have something in common, they all exist within some economics model. Ignoring the economic metrics increases risks such as investing in inefficient ad channels and funding useless product features or blindly communicating with your customers. In this article, we will review the fundamental product economic principles and try to figure out why it’s so important for business. The economic model can make a profit on the very first sale or it can demand hard work with users before the users become profitable for the business. There aren’t that many product business metrics and they are all calculated at the level of grade school math. However, there is great business value behind their simplicity. Each user from different marketing and sales channels falls into your product funnel. Every channel requires a specific budget. Users complete a certain quantity of macro and micro-actions and it is important to evaluate how much the completion of each funnel step costs you. CPA allows you to calculate how much you are paying for a certain user action. For example, we brought in 10,000 users from an AdWords campaign and only 1,000 of them added an item to their shopping cart. The given advertisement campaign cost us $500. CPA- $500/1,000= $0.5 Every time someone puts an item into their shopping cart we spend $0.5. However, it would be more valuable to know how much a successful order checkout costs us (I want to make a note that in some cases an order checkout does not signify an order that has been paid) Out of 1,000 users that added an item into their cart, only 500 successfully completed checkout. CPA2- $500/500 = $1. Remember this value because we will need it later. Let’s examine the funnel further. It is very important to understand how many users paid for their order. We know that 500 users completed checkout, but only 120 of them have paid for their order. CAC = $500/120 = $4.17 Note how much our CPA differs from our CAC. Nevertheless, we still can’t say whether these numbers are good or bad for the business. It is obvious that we are missing some variable in this formula. This is an average value of how much money we get from a user who made a purchase. Let’s imagine that the revenue from 120 successful purchases is $1,500. The average bill is $12.5. On average, therefore, the order service cost is $5. Now we can calculate our ARPPU. ARPPU = (1500/120) - 5 = $7.5 Now everything becomes more interesting. On average the profit from one order is $7.5 - $4.17 = $3.33 We are definitely making a profit on the first purchase and that’s a good news. Additionally, we see that there is a big gap between completed checkouts and paid orders, which indicates a clear opportunity for business growth. Also, we have to take into account the fact that customers can make repeatable purchases. Imagine that on average one user makes 1.5 purchases during their entire life cycle. LTV = (($1,500/120) - 5)1,5 = $11.25 In the classic formula of product economics, LTV should be 3 times higher than CAC, but you must take into account that every business is different and you need to set this ratio based on the product specifics. In this article, we investigated the most basic ways of calculating metrics without using predictive models or mathematical statistics. For example, using general linear models, we can predict what the LTV most likely will be in 6 months. We will talk about this in more detail in the future articles, but for now keep an eye on the economics of your product. Eventually, it is the foundation of your business.

Metrics are the representation of user behavior or product success. Metrics values are obtained by aggregation: addition, division, fraction, subtraction, multiplication, logarithm, etc. There are continuous metrics with values from minus infinity to plus infinity, or nominative, for example, 0 or 1, yes or no. Basically, we categorize metrics by behavioral metrics, which are related to user experience (UX) and growth metrics which are related to business and product economics. Some examples of growth metrics MAU - monthly active users DAU - daily active users Profit - how much does the product makes during the reporting period versus with the previous reporting periods User actions - numbers and ratios of sign-ups, purchases, services usage, etc. are metrics that indicate how users interact with our product and based on this data, we can gather insights on whether the users interact effectively with the product or not. Examples of behavioral metrics Task completion error ratio % of users who left a particular funnel step The time required to complete a particular scenario Relationships between actions and time CTR - interactions with the specific product elements Metrics are formed based on individual product characteristics and its business model. Interviews with stakeholders Stakeholders are aware of the strategic and business goals. The interview helps to understand the business and build a proper economic model as well as form growth metrics hierarchy. Interviews with team members Customer success, support and dev teams are user-focused and can tell a lot about customer pains and gains. Also, the metrics and its structure must be clear, complete and useful for the whole team, so the team can use the metrics as a reference point. It should be noted that not all metrics are equally valuable. Metrics should be prioritized and structured. Prioritized metrics help to understand how successful was an A/B testing or a new feature release. Also, metrics prioritization helps to focus on the most important things for the product and business. We will uncover more details on metric prioritization in the next article. Stay tuned. In the Choosing the relevant product metrics (Part 1) article, we reviewed product metrics classification to distinguish performance and user experience as well as a fundamental approach to build a metric map aligned with business goals and product team. More growth, more experiments results in more metrics to follow. It is crucial to prioritize metrics and stay focused on the most important things especially when our creativity goes wild. Priority 1 metrics These are the highest priority metrics. Top priority metrics are the product growth main indicators. Though, with the pace of growth influencing key metrics is getting more difficult. When a product is new, such indicators can grow rapidly. It is essential to keep an eye on the highest priority metrics, but it makes no sense to expect their constant growth. Why is A / B testing culture important since day 1? Experimentation explains how each product change is influencing your product growth and economics. If one does not know to which port one is sailing, no wind is favorable. Priority 2 metrics These are critical metrics but mainly reflect user experience. Priority 2 metrics not directly relate to product KPIs, but can have a substantial impact. Influencing priority 2 metrics is easier than highest priority, but still challenging. Priority 3 and the above metrics can be classified as “micrometrics”. These metrics also describe user experience and behavior, but the product shouldn't be too sensitive to fluctuations. These metrics are especially helpful in validating A/B tests which didn't affect priority 1 and 2 metrics. The more micrometrics, the better. Though, every micrometric should be directly or indirectly, related to priority 2 metrics. In conclusion I’d like to say that metrics map is a versioned document. Product changes and experimentations is an ongoing process and the metrics must evolution together with the product. As you can see there is no magic trick behind the metric prioritization, but rather simple logic. Although this approach helping us to stay focused, while being creative to push the product limits.

When you have a new product, you can find a lot of significant hypotheses to test and grow your product. However, with every passing month and year, you will soon realize that you must test more and more specific and even minor hypotheses, especially if the growth of your business slows down significantly. In this article, we will review how to determine the moment when your business is in need of crucial changes at which to take that inevitable leap of growth. Let's look at a service known to everyone, booking.com as an example and work through it as a case study. Booking.com product has existed since 1996, and I am sure that those who work for the company test a lot of hypotheses every day. We assume that booking.com team launch around 300 different experiments daily. With such a mature product it could be difficult to find something new and original that would provide additional and substantial growth. With a product like booking.com, in general, even a small increase such as a hundredth of one percent in global metrics and also micro-indicators can be meaningful. Let's imagine that booking.com team decided to make significant changes in the design or even launch a completely updated product. For the purposes of this case study, we will describe how we would approach this process. It is essential to understand how a product works and where its weaknesses lie. First of all, we would explore the product funnels for each segment to learn where the product pain-points are. Next, it is just important to look at the user journey to determine any non-obvious behavior scenarios. This would make it possible to understand what funnel steps can be optimized and whether the core audience would be hurt with the new update. Also, we would machine learning classification algorithms, such as a decision tree and random forest algorithm, to name a few. This would give us an understanding of the most significant factors affecting the key metrics in various segments. We will review how we use machine learning algorithms for UX analysis in future articles. At this stage, we can now identify the problems and develop hypotheses, but we would not immediately get an answer as to what causes these problems in the product. On the previous step, we received data indicating where the product has problems. At this next stage, we need to get qualitative feedback from users and learn what exactly causes most of the issues. We would choose the following scenario for user research: Phase 1 - Moderated, in-person Usability Research Having conducted a series of studies with different types of users, we will have successfully gained valuable insights into what exactly the users found to be incomprehensible and unusable about the product. Based on our analysis of the quantitative and qualitative data, we now would form our primary hypotheses. Phase 2 - Remote Testing After the first phase, it makes sense to make prototypes and run remote usability tests, where users will be able to complete tasks and give feedback without a moderator's supervision. The pros of these types of usability studies are that they allow us to collect a generous amount of respondents in a short period, as well as quickly and cheaply test the prototype. Before implementing any new design or code, we would run experiments to test the boldest hypotheses. Moreover, we would first look at the segment of "loyal" customers, because, most likely, they form a fundamental part of the product economy. Of course, we would not forget about the new users, too, but the main point here is that we should analyze these two groups separately. In the results of the study, we expect to get the following insights: - What hypotheses make a negative impact. If the obvious to us hypothesis in the first and second phases showed a negative impact, it makes sense to return to step 2 and chat with users again. - We would learn the hypotheses that do not impact users at all - if a complex hypothesis somehow has no effect on the audience, this in itself is a reason to abandon it. The implementation and support of such hypotheses could potentially be quite expensive, and it would not benefit the business. - a solution that would reflect positively on the users and increase business metrics. The process of creating and developing a product requires unbiased attention. The product is not someone's personal ambition; it is what secures company growth. If the manager or designer bases decisions on opinions or tastes, while ignoring audience insights, the outcome most likely will be disappointing. Consequently, data and research are always needed, and the method outlined above will help create a truly useful and highly-demanded product.

In the process of product development, different teams have various improvement ideas to test. However, the launch of experiment requires resources and time. Obvious that it is not possible to test absolutely all ideas at once, and it is crucial to understand which of the hypotheses are most important and in which sequence they will be tested. In this article we will share what logic we use to prioritize the hypotheses and how this helps us to create a hypotheses roadmap. Hypothesis implementation complexity This is a simple one. The more complicated the hypothesis development and implementation process, the more time and money required. Traffic volume In order to achieve representative results, a certain amount of data required. It is important to look at the amount of traffic on a particular funnel step where we are going to test the hypothesis, not at the total product traffic amount. It is very important to estimate in advance how long it will take to test the hypothesis, because one experiment can take several months. Metric type Experiments can be validated by different metrics. Not every experiment validated by conversion rate. Such metrics as time per scenario / time per page / number of pages viewed and similar do not require accumulation,because these metrics do not have a business cycle. If the experiment metric is conversion / revenue / average check, then these metrics have a business cycle, but typically not too long. However if our hypothesis testing metrics like Life Time Value, then it’s more challenging and you have to wait long enough for such experiments before getting to analysis. Growth potential This is perhaps the most ambiguous parameter that appears in the priority assessment. It can even be called cumulative, and usually answers the following questions: - how the hypothesis was formed? User research, support requests, data analysis, previous A/B testing insights or just an idea? - whether similar features were tested before? - which segment of the audience we are trying to influence with this experiment? Important points to consider: - Prioritization logic and methodology should be discussed with the team so that the process could be clear and transparent. - The product development vector may change from time to time, and the hypothesis roadmap should be changing too. - The roadmap could contain hypotheses that are somehow similar with each other Obviously, hypothesis prioritization is an important process. This document is an excellent way to align all team members and make decisions together based on the data you have. The format we presented is easily adapted to different teams and products, and you can easily add different requirements you think are necessary.

If you use experiments to evaluate a product feature, and I hope you do, the question of the minimum sample size required to get statistically significant results is often brought up. In this article, we explain how we apply mathematical statistics and power analysis to calculate AB testing sample size. Before launching the experiment, it is essential to calculate the experiment potential, ROI and time required to get statistical significance. The experiment can not last forever. However, if we don’t collect enough data, our experiment gets small statistical power, which doesn't allow us to determine the winner and make the right decision. Let's start with terminology. Statistical power is the probability that one or another statistical criterion can correctly reject the null hypothesis H0, in the case when the alternative hypothesis H1 is true. The higher the power of the statistical test, the less likely you can make type II error. Type II error is tightly related to 1) Difference magnitude between the samples - Effect size 2) the number of observations 3) the spread of data The power analysis allows you to determine the sample size with a specific confidence level which is required to identify the effect size. Also, this analysis makes it possible to estimate the probability of detecting the given value effect size with a specified degree of certainty with a given sample size. The most important is the number of observations: the larger the sample size, the higher the statistical power. With "sufficiently" large samples, even small differences are statistically significant, and vice versa, with small samples, even large differences are difficult to identify. By knowing these patterns, we can determine in advance the minimum sample size required to get a statistically significant result. In practice, usually, a test power equal to or greater than 80% is considered acceptable (which corresponds to a β-risk of 20%). This level is a consequence of the so-called "one-to-four trade-off" relationship between the levels of α-risk and β-risk: if we accept the significance level α = 0.05, then β = 0.05 × 4 = 0.20 and the power of the criterion is P = 1-0.20 = 0.80. Now let's look at the effect size. There are two approaches to calculating the required sample: 1) Calculating using the confidence level, the effect size, and the power level 2) Applying statistical sequential analysis, which allows calculating required sample size during the experiment Let's investigate the first case, and apply t.test for two independent samples. Let's assume, we test a hypothesis aimed to improve “item to wishlist” conversion rate. Delta, which covers costs of the experiment with a six months return >= 5% gain of the mentioned conversion rate. This >= 5% gain results in additional profit, which covers all the resources invested in the experiment. In addition to this, you want to be 90% sure that you will find the differences if they exist, and 95% - that you do not accept the differences that are random fluctuations. d - delta, siglevel - confidence level, power - power level pwr.t.test(d=.05, sig.level=.05, power=.9, alternative="two.sided") Two-sample t test power calculation n = 8406.896 d = 0.05 sig.level = 0.05 power = 0.9 alternative = two.sided NOTE: n is number in each group As a result, 16,814 participants required to get statistically significant results for A / B experiment. Imagine, that this test affects the funnel step with approximately 8000 unique visitors a month. In this case, the test requires 2 months. Let's take a look at another case when stakeholders want to get results in a couple of weeks. In this case, we have an approximate sample size of 4000 visitors and the delta >=5%. We want to know the probability to get statistically significant results under the mentioned circumstances. Add n, remove power pwr.t.test(d=.05, n=2000, sig.level=.05, alternative="two.sided") Two-sample t test power calculation n = 2000 d = 0.05 sig.level = 0.05 power = 0.3524674 alternative = two.sided NOTE: n is number in each group The probability to determine the difference, if any, is 35%, which is not too low and the probability of missing the desired effect is 65%, which is too high. Let’s look at the chart below. It shows clearly the higher the effect size, the lower sample required for a significant result.

Data can bring a huge value into product development and growth. For example with data, you can launch experiments to define what product feature is going to resonate with your customers before you implement it or build classification models which based on a user 1st session can identify if the user is going to be loyal or going to churn. Sounds pretty cool and powerful, right? However, before you start using the data, you need to collect and store it properly. Most people think that data scientists spend most of their time building algorithms and machine learning models, but the truth is that vast amount of time consumed just to get, prepare and clean the data for analysis.  Building a reliable process to collect and robust architecture to store the data is a crucial component of product success. The easier it is to get the data from the data warehouse, the faster you get results. With а pace of product growth, you get more and more data from different sources. All that data needs to be combined and stored in one place. In this case, Universal Analytics becomes insufficient. Such option as building own data storage warehouse is quite an expensive, lengthy and complicated process, so the question of how to quickly build an affordable, but high-quality data aggregation and storage solution often pops up. Snowplow is a JavaScript based tracker that allows sending events to a cloud database using an API and data transfer protocols. One of the Snowplow’s advantages is a free license and easy implementation. Also, Snowplow has a similar to Universal Analytics events logic and syntax, and by just changing few parameters we can migrate from UA. The same as in Universal Analytics, Snowplow allows collecting e-commerce data, advanced parameters as well as a wide range of various indicators. After the data is collected, and stored in Amazon redshift, one of the options is to use Amazon's internal interface to write SQL queries to get the data, but this is not the most convenient process, and we recommend the following: 1) BI analytics tools Amazon has various cloud BI tools integrations with the ability to pull the data using SQL / R / python. Couple tools we want to highlight are Modeanalytics and Redash. Both tools are very affordable but have a lot of useful features to visualize data and build clear and meaningful dashboards. 2) Raw data To work with raw data, build models and statistical data processing there is an option to connect to Amazon directly through IDEA R or Python. This R library allows connecting to Amazon Redshift so you can get the data and process it according to your requirements. Data architecture is a complicated thing, and it must continually evolve along with the product. The solution described in this article is a great start which allows focusing resources on the product instead of diving into a long, complicated development. In conclusion, I want to say obvious and very logical, but the most important thing that every digital product company must follow since a very early stage. The most important in data analysis is the data quality. Not algorithms, not magic data science stuff. If you don’t have data collected and stored properly, there is no way you can get any insights of that.

When they talk about a/b tests, most often we are talking about conversion. Conversion is a brilliant metric for monitoring a product and validating experiments. The conversion is easy to understand, easy to interpret, and simple enough to count. In fact, conversion is just one of the indicators and is not the main metric for all types of businesses. If we imagine an average online store, then it is much better for it to increase the volume of the average receipt or the number of purchases per user than to focus solely on the conversion to the first purchase. There are quite a large number of growth metrics and behavioral metrics that form an entire hierarchical system. If the product focuses on different indicators, it is logical that experiments should not be limited only to conversion.Conversion is exceptionally popular due to its pivotal role in determining the success of various initiatives, particularly in the realm of business and marketing. The concept of conversion revolves around turning potential customers or users into actual buyers or subscribers, reflecting a tangible impact on a company's bottom line.  Several factors contribute to the popularity of conversion! Measurable Impact: Conversion provides a clear and measurable metric to evaluate the effectiveness of strategies. It offers a quantifiable way to assess the success of marketing campaigns, user experiences, and product offerings. Direct Revenue Generation: Conversions directly influence revenue generation. Increasing the conversion rate means more sales or sign-ups, leading to enhanced profitability and growth opportunities. Resource Efficiency: Focusing on improving conversions allows businesses to allocate resources more efficiently. By honing in on optimizing conversion rates, companies can target specific areas for improvement, leading to higher return on investment (ROI). Enhanced Customer Insights: Conversion analysis provides valuable insights into customer behavior, preferences, and pain points. This information can guide refinements in marketing strategies and product development to better cater to customer needs. Comparative Analysis: Conversion rates enable comparative analysis of different strategies, channels, or designs. This helps businesses identify which approaches are most effective and make informed decisions based on empirical evidence. A/B Testing Possibilities: The discrete nature of conversion data makes A/B testing and experimentation simpler, allowing businesses to test variations and optimize outcomes. Goal Clarity: Conversions represent a clear and common goal across diverse industries and campaigns. This shared objective makes it easier to communicate and align efforts within teams. Decision-Making Tool: Conversion data serves as a valuable tool for strategic decision-making. It informs businesses about the viability of their initiatives and guides adjustments for better results. Continuous Improvement: The pursuit of higher conversion rates fosters a culture of continuous improvement. Businesses strive to refine their processes, user experiences, and offerings to consistently enhance their performance. Competitive Edge: Given the competitive landscape, companies that effectively master conversion optimization gain a competitive edge by outperforming peers and capturing a larger share of the market.Experiments can test completely different hypotheses, such as the impact of site loading speed or the impact of a new recommendation system. Different experiments can be aimed at improving completely different indicators. If all tests are validated only by conversion, then it is often possible not to notice improvements where they really are, and vice versa, if all attention is focused only on conversion, then we simply do not see if the experiment had a bad effect on other indicators. Imagine that you are in a dark room and illuminate only one point with a flashlight. In this case, you perfectly see what is happening in the illuminated area, but you have no idea what is happening in other places. In one of the previous articles, we talked about a fairly typical case in our work when the experiment was validated by conversion and as a result, the client could make the wrong decision.Any experiment is a data set. The data has different characteristics and shape. There are continuous metrics, and there are discrete ones, as in the case of conversion. Discrete data always takes the form of a binomial distribution, which is also a normal distribution. What is the form of distribution and why it is important can be read here. In short, completely different calculation methods should be used for metrics with different distribution forms. If the data has a normal distribution form, then you can determine the winner in the test by applying parametric methods, for example, the T test. Parametric methods rely on the mean, standard error, and imply a normal distribution. If we apply the parametric method on an abnormal sample, then there is a very high probability of error. In a future article, we will look at an example of what happens if a parametric method is applied to non-normally distributed data. So why are conversion experiments so much more popular? Because they are much easier to calculate and determine the difference between two independent samples. In this series of articles, we will look at examples of how to determine the impact of an experiment on other metrics:There are three main ways to analyze data that is not normally distributed. -Use nonparametric methods -Use data transformation methods - logarithm as an option -Using bootstrap is a living example of pain information. In the following articles, we will look at how to analyze the above metrics in 3 different ways. When we hear something about A/B testing, most often it's about the conversion rate. The conversion rate is an excellent indicator for monitoring the product and confirming the results of experiments. The transformation is easy to understand, easy to interpret and quite simple to calculate. In fact, conversion is just one of the indicators and is not the main indicator for all types of businesses. For an average online store, it is much better to increase the average receipt or the number of purchases per user, rather than focus solely on the conversion rate. There are quite a large number of growth and behavior indicators that form a hierarchical system of indicators. Thus, if a product measures success based on various indicators, then it is logical that experiments should also be confirmed by various indicators.There are a number of tools on the market that allow launching A/B tests which are mostly aimed to improve the conversion rate. In my opinion, these companies laid the foundation for how most people perceive the concept of AB testing nowadays. Deeper look Another stereotype is that A/B testing aimed to improve only visual website elements, like buttons, colors, text. In fact, any hypotheses could be tested. As an example, we’ve run experiments to determine the influence of website page loading speed or a new recommendation system on various product indicators. Different experiments can be aimed at improving completely different indicators. If all tests are validated only by conversion rate, then you miss growth opportunities. Another thing is that, if all attention is focused on conversion rate, then we simply do not see if the experiment had a negative impact on other metrics. Imagine that you are in a dark room and a flashlight illuminates only one spot. In this case, you perfectly see what is happening in the illuminated area, but you have no idea what is happening in other room areas. In one of the previous articles, I talked about a fairly typical case in our work, when an experiment was validated by conversion rate and as a result, a client almost made a wrong decision. Any test is just a data set. The data have different characteristics and forms. There are continuous metrics with values from minus infinity to plus infinity, and discrete, with values, 0 or 1, yes or no. As you can see the conversion rate is a discrete type of data. Discrete data always takes the binomial form of distribution, which is also a normal form of distribution. What is a form of distribution and why it’s important, you can read in the article A/B testing: the importance of Central limit theorem. In a nutshell, different statistical methods should be applied to data with a different type of distribution. If the data has a normal form of distribution, then you can determine the winner in the experiment using parametric methods like Student T-test. Parametric methods rely on the mean, standard error and assume a normal distribution. If we apply a parametric method on a not normally distributed sample, then there is a very high probability of making an error. In a future article, we will walk through a case study showing what happens if a parametric method applied to non-normally distributed data. So, why the experiments with conversion rate are much more popular? It is because the discrete type of data is easier to calculate and to determine a difference between two independent samples. In this series of articles, we will study how to analyze experiments aimed at improving different metrics. - The Average Revenue Per User - Average Time on Page - Purchase Frequency Per User I picked these 3 metrics because all of them are united by one property - these metrics have continuous data and most often do not have a normal form of distribution. That means, we can’t use parametric methods to assess experiments. How to analyze continuous metrics with non-normal distribution There are mainly three ways to analyze non-normally distributed data. -Use nonparametric methods -Use data transformation methods - logarithm, for instance -Use bootstrap - a live example of the Law of large numbers. In the following articles, we will look at how to analyze all 3 metrics in 3 different ways.

If you validate all experiments by macro metrics (conversions, average check, signups, searches, etc.), then you can miss valuable insights. Not every change in your product can be described by a “big” metric that shows the success of your business. This article is based on our own experience working on some projects in the past. Let’s imagine that we decided to test our website search. We decided to make changes in the algorithm which shows search suggestions to users when they start typing the search query. This change is not too noticeable for the users. Analyzing the experiment data, we noticed that there is no statistical difference between the control and experimental groups. This could be the end of the data analysis and the article, but let’s dive deeper into what this change might also affect: Users are more likely to use the suggestions because they are more relevant to their search The time between typing a search query and browsing search results reduced Returning users are using search more frequently As you can see, now we have three new micro metrics that are not related to product economic indicators. Not every change in the product aimed at maximizing profits. User experience is also extremely important and good UX may affect repeating purchases, user loyalty and many more. Metric Priority Search retention rate: priority — 1 The time between query and browsing search results: priority — 2 Suggestions usage: priority — 2 Revenue: priority — 4 1 — Highest priority, 4 — Lowest priority Let’s analyze one of the micro metrics — The time between query and browsing search results.It’s obvious, that the type of distribution for the data is not normal. After applying the nonparametric Mann-Whitney criterion, we get a p-value = 0.003. Test variant B time decreased by 30%. We can conclude that the new search is more convenient for users and they switch from query to results quicker. Product growth is not limited to pushing up macro metrics. if you focus on the “big” macro metrics, you can miss valuable insights as well as a large number of great hypotheses. Looking deeper into the metrics helps you better understand your users and their behavior.

After a sufficient amount of data collected and AB test completed, some of you might think it’s time for data analysis, but before we start to analyze we need to ETL the data. ETL stands for Extract, Transform, Load. In this article, we will talk about the transform part, which is in our case cleaning. Often the collected data contains outliers, and let’s say, if we want to apply a parametric method to compare means, then this method requires robust data. Let’s talk about an experiment with “average check” metric. In the average check dataset, we want to find mean value. In this case, outliers could have a significant influence on our mean value. Let’s say if the average check of a regular customer is $30, but there are customers who typically spend $300, the mean value will be skewed. One of the options is to find and remove the outliers from the dataset manually. Sounds easy, right? However, the challenge is if our dataset contains millions of observations.Obviously, we have many outliers in this dataset. Most likely, these outliers are the clients who spent significantly more as compared to an average customer, these are also known as so-called “whales”. These users may generate a significant revenue share, and they must be analyzed separately, and one of the reason is that their behavioral patterns are most likely to differ. We’ll uncover this topic in future posts. So, how do we deal with such a dataset, to stabilize and clean the outliers? Let’s delve a little bit into the theory. Three sigma rule — almost all values of a normally distributed random variable lie within (x̅-3σ; x̅+3σ). With a probability 0.9973, the value of a normally distributed random variable lies in the specified interval (if the value is not obtained as a result of sampling).We can use the following function in R language to exclude all those data that lies outside of three standard deviations from the mean. After applying this function, you will clean a significant amount of outliers in your dataset. Worth to note though, that you should use this method very carefully because it can remove too much data in your sample. This number could be %30 or even more. There are many more data cleaning approaches. In this article, we have reviewed only one of them which is pretty simple and quite popular. In practice, a data cleaning method depends on the data quality and nature. Couple less conservative methods to mention the Box-Cox transformation or methods using the median to compare samples, such as Kruskal-Wallis criterion. We will discover other data cleaning methods in the coming articles.