Robotstxt Update – v0.6.0 on CRAN

February 11, 2018

(This article was first published on petermeissner, and kindly contributed to R-bloggers)

I just got the news from the CRAN-team that the robotstxt version update 0.6.0
was accepted and now is available.

The robotstxt package aims at working with robots.txt files from within R by
providing a parser as well as a permission checker – and some convenience
goodies working behind the scenes.

This new version switches the default checking backend from an pure R
implementation (done by myself) to a C++ implementation (done by Moz and wrapped by Bob Rudis) that is both executing much faster
and also much more rigourous in interpreting the the standard (RFC). While it was hard to throw
away so much of my own work it also is very liberating to now have to
maintain lesser code and leaving the package with the feeling that it now is
in a better state (more robust) than before.

Thanks CRAN, thanks Bob, thanks Moz.

To leave a comment for the author, please follow the link and comment on their blog: petermeissner. offers daily e-mail updates about R news and tutorials on topics such as: Data science, Big Data, R jobs, visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...

If you got this far, why not subscribe for updates from the site? Choose your flavor: e-mail, twitter, RSS, or facebook...

Comments are closed.

Search R-bloggers


Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)