Tracking the Frequency of Twitter Hashtags with R

February 21, 2011

(This article was first published on Michael Bommarito » r, and kindly contributed to R-bloggers)

 I’ve posted three examples of Twitter hashtags datasets in the last week: one on China, one on Iran, and one on Algeria.  In order to build these datasets, I needed to obtain older tweets; this is slightly more difficult than simply filtering the streaming feed for your hashtag of choice.  The original code I wrote for this task is in Python and is well-parallelized.  However, the code isn’t commented and looks more complicated than it is due to parallelization choices.  

  As part of my recent exercise to replace Python with R for entire tasks, I decided to rewrite this code using R tonight.  The code is pretty simple, well-commented, and consists of two functions – loadTag and downloadTag. Both of these methods are embedded in the script below:

  There is one significant issue with the code, however. At the moment, neither rjson nor RJSONIO seem to support Unicode data in JSON responses.  Furthermore, when character vectors of "unknown" encoding are written to file with a function like write.table, they produce output that cannot be reliably read back into R.  As a result, the code above does not retain the text of a tweet – only the id, date, and username.  This script can be easily modified to include other JSON variables by modifying the lambda function in line 70.

  Once you’ve downloaded some data, producing figures like the ones in the posts above is only two lines away: tweets <- loadTag(tag) and ggplot(data=tweets, aes(x=as.POSIXct(date))) + geom_bar(aes(fill=..count..), binwidth=60*5).  Here’s the current figure of 5-minute frequencies since the 20th (x-axis is EST unlike the previous post, in which it was UTC. Pass tz="UTC" to as.POSIXct to change this).

Email Google Buzz Facebook Twitter Share

To leave a comment for the author, please follow the link and comment on their blog: Michael Bommarito » r. offers daily e-mail updates about R news and tutorials on topics such as: Data science, Big Data, R jobs, visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...

If you got this far, why not subscribe for updates from the site? Choose your flavor: e-mail, twitter, RSS, or facebook...

Tags: , , ,

Comments are closed.


Mango solutions

plotly webpage

dominolab webpage

Zero Inflated Models and Generalized Linear Mixed Models with R

Quantide: statistical consulting and training




CRC R books series

Six Sigma Online Training

Contact us if you wish to help support R-bloggers, and place your banner here.

Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)