Monday, January 27, 2014

Rishi pal Singh: The A/B Testing Process

Rishi pal Singh: The A/B Testing Process: The correct way to run an AB testing experiment (or any other experiment for that matter) is to follow the Scientific Method. The steps of...

The A/B Testing Process

The correct way to run an AB testing experiment (or any other experiment for that matter) is to follow the Scientific Method. The steps of the Scientific Method are:

  1. Ask a question: "Why is the bounce rate of my website higher than industry standard?"
  2. Do background research: Understand your visitors' behavior using Google Analytics and any other analytics tools running on your website.
  3. Construct a hypothesis: "Adding more links in the footer will reduce the bounce rate".
  4. Calculate the number of visitors/days you need to run the test for: Always calculate the number of visitors required for a test before starting the test. r 
  5. Test your hypothesis: You create a site wide A/B test in which the variation (version B) has a footer with more links. You test it against the original and measure bounce rate.
  6. Analyze data and draw conclusions: If the footer with more links reduces bounce rate, then you can conclude that increased number of links in the footer is one of the factors that reduces bounce. If there is no difference in bounce, then go back to step 3 and construct a new hypothesis.
  7. Report results to all concerned: Let others in Marketing, IT and UI/UX know of the test results and insights generated.

Analyze The Relationship Between Ad Spend & Revenue For Each Channel



major sections in Google Analytic

My major sections in Google Analytic are:
  • Timing Target: What time of year are you looking to book, sell, etc.? (If there is no timing target, just leave this alone.)
  • Top Converting Traffic Sources: Look at the All Traffic report under the Acquisition tab in Google Analytics.
  • Best Converting Landing Page: Look at the Landing Pages report under Behavior –> Site Content and view it with Ecommerce info sorted by transactions descending.
  • Highest Visit Count Landing Page: Look at same report as above, but sorted by Visits descending.
  • Top 3 Keywords Driving Conversions: This is trickier because we live in “Not Provided” land. You will need to use PPC or Webmaster Tools Search Query data to help you get an idea of keyword phrases that convert.
  • Top 3 Converting States/Cities/Metros: Under Audience –> Geo –> Locations, look at cities or states and Ecommerce, then sort by transactions descending.
  • Male/Female Conversion Ratio: Under Audience, look at Demographics and then Gender. Apply the Ecommerce info and sort by transactions descending to see who converts more.
  • Top Converting Age Group Demographic: Under Audience, look at Demographics and then Age. Again, apply the Ecommerce info and sort by transactions descending.
  • Purchase/Booking Window: This is somewhat difficult to gauge in Google Analytics. I recommend looking at Time to Purchase under Conversions –> Ecommerce. The tricky piece is attribution — if you are looking at “Last Click Attribution,” which is the default