A/B Testing in R – Part 1

November 29, 2011

(This article was first published on Abraham Mathew » R, and kindly contributed to R-bloggers)

A/B testing is a method for comparing the effectiveness of several different variations of a web page. For example, an online clothing retailer that specializes in mens’ streetwear may want to examine whether a black or pink background results in more purchases from visitors to the site. Lets say that our online store is just a single web page, and we run this experiment by randomly showing one variation (pink background) of the page to half the visitors and the control background to the other half. After running the experiment for one week, we find that the pink background resulted in 40% purchase rate with 500 visitors while the black background resulted in a 30% purchase rate with 550 visitors. So which background is more effective at generating purchases from visitors to the online store. One way to examine this problem is by calculating confidence intervals of the conversion rates for each variation of the site. In the following R code, I construct a function which calculates the confidence intervals for the purchase rate of each site at a 80% significance level. In this example, the purchase rate for the pink background is significantly higher than the purchase rate for the black background.

site1 = c(.40, 500) # pink
site2 = c(.30, 550) # black

abtestfunc <- function(ad1, ad2){
      sterror1 = sqrt( ad1[1] * (1-ad1[1]) / ad1[2] )
      sterror2 = sqrt( ad2[1] * (1-ad2[1]) / ad2[2] )
      minmax1 = c((ad1[1] - 1.28*sterror1) * 100, (ad1[1] + 1.28*sterror1) * 100)
      minmax2 = c((ad2[1] - 1.28*sterror2) * 100, (ad2[1] + 1.28*sterror2) * 100)
      print( round(minmax1,2) )
      print( round(minmax2,2) )

abtestfunc(site1, site2)

To leave a comment for the author, please follow the link and comment on their blog: Abraham Mathew » R.

R-bloggers.com offers daily e-mail updates about R news and tutorials on topics such as: Data science, Big Data, R jobs, visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...

If you got this far, why not subscribe for updates from the site? Choose your flavor: e-mail, twitter, RSS, or facebook...

Tags: , , ,

Comments are closed.


Mango solutions

RStudio homepage

Zero Inflated Models and Generalized Linear Mixed Models with R

Dommino data lab

Quantide: statistical consulting and training




CRC R books series

Six Sigma Online Training

Contact us if you wish to help support R-bloggers, and place your banner here.

Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)