Some Code for Dumping Data from Twitter Gardenhose

March 30, 2010
By

(This article was first published on Byte Mining, and kindly contributed to R-bloggers)

Gardenhose is a Streaming API feed that continuously sends a sample (roughly 15% according to Ryan Sarver at the 140tc in September 2009) of all tweets to feed recipients. This is some code for dumping the tweets to files named by date and hour. It is in PHP which is not my favorite language, but works nonetheless. I received a few requests to post it, so here it is.

 <?php

//gardenhosedump.php
$username = '';
$password = '';

while(true) {
         $file = fopen("http://" . $username . ":" . $password . "@stream.twitter.com/1/statuses/sample.json","r");

         while($data = fgets($file))
         {
             $time = @date("YmdH");
             if ($newTime!=$time)
             {
                 @fclose($file2);
                 $file2 = fopen("{$time}.txt","a");
             }
             fputs($file2,$data);
             $newTime = $time;
         }
        //need to close the file, but only if it is open!
        try {
                @fclose($file);
        } catch (MyException $e) {}
        try {
                @fclose($file2);
        }
        catch (MyException $e) {}

}
?>

To leave a comment for the author, please follow the link and comment on his blog: Byte Mining.

R-bloggers.com offers daily e-mail updates about R news and tutorials on topics such as: visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...



If you got this far, why not subscribe for updates from the site? Choose your flavor: e-mail, twitter, RSS, or facebook...

Tags: ,

Comments are closed.