| type | quantitative |
|---|---|
| number of participants | percentage of total user base (n>=1000) |
| product | web, mostly. |
An (automated) comparison between two or more variants of solutions to the same problem.
If you know the problem and you've made a potential solution to it, but you're not sure if it's the right one. The problem should be measurable. Here are a few examples of indicators you can measure:
Some of these examples are a bit more into the usability spectrum. Still, from the results one can only guess the intent of the user so be careful (and maybe test) before drawing conclusions.
If your website has too little traffic, it is not possible to do this test. To test if running an A/B test may at all be useful for your website can be done using this tool: https://abtestguide.com/calc/. Note that it is the number of conversions (or people who will reach a goal) that determines how significant your test is, not the total amount of visitors.
Next to this, a lot of variables are not measurable. Any qualitative data will be impossible to extract with an A/B test only.
Lastly, A/B testing is time consuming. A test may have to be run for a long period of time before the results are significant.
Setup your tools for testing. If you are using a framework (e.g. React), you have the choice to either test within the framework, or outside the framework. It is advisable to test outside of the framework if possible: serving content from a random container, each container having its own test case. If you do want to implement this within your front-end code, there are packages available for this. Please search on npm.
Write your test. Be warned that as you change more variables in your test, it will be more difficult to draw general conclusions. You don't have to draw general conclusions though: if you just want to compare to options and pick the best, that is fine.
If your result is a single value for each case, you may just compare each case, pick the best and be done with it. If you design your test in such a way that you have multiple input (country, screen resolution, test variant, browser type, etc.) and output variables (number of clicks, did the user succeed?, number of mistakes the user made), you may choose to visualize them. For this you can Tableau for example.
Look at the right indicators.
T.B.D.