Web-scraping, or web-crawling, sounds like a seedy activity worthy of an Interpol investigative department. The reality, however, is far less nefarious. Web-scraping is any procedure by which someone extracts data from the internet. Given that it’s possible to get the internet on computers these days; web-scrapping opens an array ... [Read more...]

pie( table( whence.i.tweet )) qplot( whence ) + coord_polar() pie( log( table( whence )))+RColorBrewer ggplot (see below) plot( density( tweets.len )) qplot(... stat="density") + geom_density qplot(...stat="bin") + geom_text(...) tweeple tweep... [Read more...]

[Read more...]

Scraping table from any web page with R or CloudStat: You need to use the data from internet, but don’t type, you can just extract or scrape them if you know the web URL. Thanks to XML package from R. It provides amazing readHTMLtable() function. For... [Read more...]

Google scholar has now made Google Scholar Citations profiles available to anyone. You can read about these profiles and set one up for yourself here. I asked John Muschelli and Andrew Jaffe to write me a function that would download my Google Scholar... [Read more...]

I will assume you have read through part 1 and have the csv file loaded. While we covered some basic graphing in the last post i hope to get into a little more of the data crunching. Specifically I am interested in the names which where driven by a spe... [Read more...]

Kay Cichini recently wrote a word-cloud R function called GScholarScraper on his blog which when given a search string will scrape the associated search results returned by Google Scholar, across pages, and then produce a word-cloud visualisation. This was of interest to me because around the same time I posted ... [Read more...]

[Read more...]

This is a followup to a post I uploaded earlier today about web scraping data off Google Scholar. In that post I was frustrated because I’m not smart enough to use xpathSApply to get the kind of results I wanted. However fast-forward to the evening whilst having dinner with ... [Read more...]

I wanted to scrape the information returned by a Google Scholar web search into an R data frame as a quick XPath exercise. The following will successfully extract  the ‘title’, ‘url’ , ‘publication’ and ‘description’.  If any of these fields are not available, as in the case of a citation, the ... [Read more...]

Google slightly changed the html code it uses for hyperlinks on search pages last Thursday, thus causing one of my scripts to stop working. Thankfully, this is easily solved in R thanks to the XML package and the power and simplicity of XPath expressions: Lovely jubbly! P.S. I know ... [Read more...]

[Read more...]

[Read more...]

by Earl F. Glynn, Kansas Watchdog The goal of this article is to show how to visit 105 online web pages programmatically and “scrape” data from them to form a statewide summary of election data in Kansas. An earlier article gave details of ... [Read more...]

by Earl F. Glynn, Kansas Watchdog The goal of this exercise is to show how to “screen scrape” data from an online web page using R. Additional articles will extend this example to scrape data from 105 Kansas county pages to form a statewide... [Read more...]

Plenty of people have been scraping data from the web using R for a while now, but I just completed my first project and I wanted to share the code with you.  It was a little hard to work through some of the “issues”, but I had some great help ... [Read more...]