The Netflix Prize, Big Data, SVD and R

May 31, 2011

(This article was first published on Revolutions, and kindly contributed to R-bloggers)

One of the key data analysis tools that the BellKor team used to win the Netflix Prize was the Singular Value Decomposition (SVD) algorithm. As a file on disk, the Neflix Prize data (a matrix of about 480,000 members' ratings for about 18,000 movies) was about 65Gb in size — too large to be read into the standard in-memory data model of open-source R directly. But in the video below, Brian Lewis shows us how to use the sparse Matrix object in R to efficiently store the data (about 99 million actual movie ratings) and the irlba package (which features a fast and efficient SVD algorithm for big data) to perform SVD analysis on the Netflix data in R.


Big Computing: Bryan Lewis's Vignette on IRLBA for SVD in R

To leave a comment for the author, please follow the link and comment on their blog: Revolutions. offers daily e-mail updates about R news and tutorials on topics such as: Data science, Big Data, R jobs, visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...

If you got this far, why not subscribe for updates from the site? Choose your flavor: e-mail, twitter, RSS, or facebook...

Tags: ,

Comments are closed.

Search R-bloggers


Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)