Rookies deploy their A/B tests based on tactics and best practices. The pros use a process and framework before they start their first A/B test.
Collecting and analyzing data is the single most important task before running an A/B test on your website. Simply because you need to know what’s happening on your site before you can actually make things better.
I have encountered many companies who run dozens of A/B tests without analyzing their data first. This approach won’t make you rich!
In my opinion A/B tests are ideally based on data, psychology, usability and UX guidelines.
In this post I will share Google Analytics reports and strategies to turn data into valuable insights for A/B testing.
Traffic and Conversion Numbers
All too often people run A/B tests without looking at their numbers first. They run a test, find a winner and implement it, but nothing changes.
The reason why is that the website doesn’t get high enough numbers to run reliable A/B tests.
As a rule of thumb you want to run an A/B test for two to four weeks and have at least 300 to 500 conversions per variation.
With a conversion rate of 2% you are already talking about 15,000 to 25,000 unique visitors per variation. If you are lucky to have a conversion rate of 10%, lower numbers will do.
“You have to calculate the traffic, conversion and conversion rate overall, but also per individual page (or template) on your website. Since on some of your pages you may have better numbers than on others.”
Do you have much lower numbers, like 5.000 unique visitors per month? No worries, there are other ways to improve your website. For example, tools like SessionCam can help you uncover potential blockages to a higher conversion rate or leaking funnel.
In Google Analytics you can find your numbers in two reports: audience report and goal overview (or ecommerce) report.
Report 1: Audience >> Overview.
In this case the number of users (four week period) is sufficient to run proper A/B tests.
Report 2: Conversions >> E-commerce >> Overview.
Although the ecommerce Conversion Rate is pretty low, the total number of transactions is high enough for A/B testing.
Note: I recommend to run A/B tests on unique numbers, so you want to work with buyers instead of transactions.
Get this number by applying a segment (Made a Purchase) to you Audience overview report.
Step 1: select segment (Made a Purchase) in Google Analytics.
Step 2: go to Audience report and write down the number of users that made a purchase.
The results are:
- Transactions = 5,876
- Buyers = 5,547
In this case the average number of transactions per buyer is low.
However, if you run an ecommerce website with low involvement products that people buy on a regular basis this can make a big difference.
Tip: use this A/B testing tool to find out whether your traffic and conversion numbers are high enough for A/B testing.
Do you have a responsive website or do you have a separate domain for mobile traffic?
In both cases you want to get a better overview on how your traffic and conversions are divided among different device types.
Step 1: navigate to Audience >> Mobile >> Overview.
Step 2: decide upon your A/B testing strategy.
In this example the CR varies dramatically:
- CR on desktop is two times higher as CR on tablet
- CR on desktop is five times higher as CR on mobile
It seems that desktop traffic is dominant in both the number of sessions as well as conversions. In A/B testing this device should get your main focus.
You won’t be able to draw reliable conclusions on mobile and tablet if the numbers are too low.
“One of the most important things in Google Analytics is to get the goal setup right. Read this strategic goals guide if you want to learn all about setting up goals in Google Analytics.”
Page Value of Individual Pages
By now you have learned a lot about whether your website is suitable for running A/B tests and on which devices you could test in a reliable way.
Another crucial step is to define where to run your next A/B test.
In this example I will concentrate on finding interesting pages for A/B testing.
One way to do this, is by calculating the value of each of your pages in relation to the macro conversion on your website.
Step 1: define a goal value for your main conversion.
For each of the goals that you define in Google Analytics you can choose to apply a goal value.
“You run a lead generation website. In one year you have 100 leads that generate €100,000 in revenue. It means that the average revenue per lead is €1000. In that case you fill in 1000 in the goal value field in Google Analytics.”
There is one exception where you should refrain from using the goal value metrics. And that’s when you run an ecommerce website. In that case you should use ecommerce tracking and direct revenue metrics instead.
Your metrics will get inflated if you use both a goal value and ecommerce tracking on the “thank you” page of a transaction.
Step 2: learn about page value in Google Analytics.
The page value metric collects data after you have set up your goals and goal values (and if applicable ecommerce tracking) in Google Analytics.
“The page value metric assesses the individual pages on your website and determines the value of each of your pages in relation to your goals. Google Analytics divides the goal and ecommerce value in a session where a visitor converts among the pages a visitor has seen before converting.”
Read this article to learn more about how page value is calculated.
Step 3: analyze the page value of each of your pages.
Navigate to Behaviour >> Content >> All pages report to see the page value of each of your unique pages.
You can see a few things here:
- The average page value is €20.77.
- The page value of your top 10 pages varies from €2.48 to €24.51.
- The number of different pages is above 60,000.
The difference in page value per page illustrates that somes pages are visited more often before someone converts than others.
In general you want to run A/B tests on pages:
- That receive a lot of page views
- That are often visited before someone converts
Note: be careful with specific buying funnel pages. These pages naturally have a high page value, because a visitor must visit these pages before converting.
Make sure to combine page traffic, page value and common sense to find pages for running A/B tests.
Keep in mind that A/B tests are not always necessary. E.g. technical issues should be fixed immediately, don’t waste an A/B test on that.
Setting up goal values and analyzing individual page values are a great help in determing where to test. However, I recommend to use another feature in Google Analytics if you run a website with thousands of individual pages.
The feature is called content grouping and it can be extremely valuable in determining the best templates for A/B testing.
Page Value of Content Groups
Let’s have a look at the definition from Google:
“Content Grouping lets you group content into a logical structure that reflects how you think about your site or app, and then view and compare aggregated metrics by group name in addition to being able to drill down to the individual URL, page title, or screen name.”
For a website with thousand of pages, this feature can be extremely beneficial. By setting up content groups you get an aggregated overview on the performance of your website.
Here is an overview of content groups on a blogger’s website:
As you can see, the content is aggregated in different groups. The owner of the website didn’t set up page values yet, but if she did, she could analyze the traffic and page value of each of her content sections. This information could help her to deploy A/B tests on the template level of her website.
I recommend to read this in-depth guide on ontent groupings if you want to learn more about leveraging them in Google Analytics.
“Besides analyzing page content groups, you can also analyze your landing page content groups. Extremely useful to find out how your different content sections perform as a landing page, both in terms of traffic and conversions.”
Practical Example of Content Grouping
You might wonder, how do I determine the different content sections of my website.
For an ecommerce site it’s most often straightforward:
- Search results / listing page
- Category page
- Subcategory page (optional)
- Product page
- Cart confirmation (optional)
- Funnel step 1
- Funnel step 2, 3 etc. (optional)
This is how it often looks like.
But what if you run a different site?
I will give you some tips based on the website of SessionCam.
The primary goal / conversion of this website is a free trial. SessionCam may run A/B tests to optimize their trial conversion. The “Chat with us” function may also play an important role in the sales process.
Now the question is, what templates could they define on their website?
You can use a spider tool (e.g. Screamingfrog) to quickly crawl the URL structure of the website. This can be a tremendous help in mapping out the different pages and site structure.
In this case you could also head over to the sitemap.
The difficulty here is that the sitemap doesn’t contain all pages (e.g. blogposts are not included) on the website. In addition to that, the blogposts are all located in the root itself. For example:
- https://sessioncam.com/sessioncam-sponsors-new-shift-conference/ (there is no subdirectory “blog” defined)
Let’s demystify the website structure a bit more.
I don’t have access to the data of the website, but here are my initial thoughts about content structure and groups:
- The “Try SessionCam” page is the prior step to conversion and should be defined as a separate content group.
- The “Solution” page is a separate content group.
- The different benefit pages of the “Solution” belong to a separate content group.
- The different feature pages of the “Solution” belong to a separate content group.
- The “Contact” page is a separate content group.
- The “About” page is a separate content group.
- The “blog posts” belong to a separate content group.
This is not a complete list, but it gives you a good indication on how this content grouping feature works and how you can apply it to your website.
Content Grouping in Google Analytics
The most easy way, if your website is well structured, is to set up content groups by using page definitions.
Here is one example:
In one content grouping you have to define all groups that you want to measure separately. Pages that are not defined in one of your groups will be attributed to the category (not set). You want to keep the numbers in this category as low as possible.
Tip: use a negative match if you have all your blogposts reside in the root of your website. It’s more easy to exclude all other content groups instead of grouping all the different blog posts URLs.
- Google Analytics (or another web analytics tool) can provide a ton of insights that help you approach A/B testing in the most effective way.
- Always combine different data sources when defining A/B tests on your website.
- Not all websites are suitable for A/B testing. It depends on the traffic, conversions and conversion rate whether A/B testing is appropriate for you.
- Leverage qualitative analytics tools like SessionCam to complement the quantitative analytics data.
- Small traffic websites can be optimized even without A/B testing.
- Invest sufficient time and resources to set up your Analytics tools and to transform the data into insights for your organization. It will pay off big!
Please leave your comment below if you have any questions or remarks related to Google Analytics or A/B testing. And a share or tweet is much appreciated if you like the article!
About Paul Koks
Paul Koks is an Analytics Advocate at Online Metrics and the author of Google Analytics Health Check. He helps companies to capture valuable insights from simple and reliable data.