Blog Archives

New R Package – domaintools (access the DomainTools.com WHOIS API)

August 9, 2015
By

We just did a github release for an R package that provides an interface to the DomainTools API. It provides access to the core API functions that aren’t restricted (i.e. the ones we have access to): domaintools_api_key: Get or set DOMAINTOOLS_API_KEY value domaintools_username: Get or set DOMAINTOOLS_API_USERNAME value domain_profile: Domain Profile hosting_history: Hosting History parsed_whois: Parsed Whois reverse_ip: Reverse IP reverse_ns: Reverse Nameserver shared_ips: Shared IPs whois: Whois Lookup whois_history: Whois History Each function has a full...

Read more »

The New and Improved R Shodan Package

August 7, 2015
By

For those not involved with all things “cyber”, let me start with a description of what Shodan is (though visiting the site is probably the best introduction to what secrets it holds). Shodan is—at it’s core—a search engine. Unlike Google, Shodan indexes what I’ll call “cyber” metadata and content about everything accessible via a public IP address. This means things like routers,...

Read more »

RBerkeley Was Just Pining For The Fjords

July 27, 2015
By

If you made it to Chapter 8 of Data-Driven Security after ~October 2014 and tried to run the BerkeleyDB R example, you were greeted with: Warning in install.packages : package ‘RBerkely’ is not available (for R version ) That’s due to the fact that it was removed from CRAN at the end of September, 2014 because the package author &...

Read more »

Introducing the cymruservices R Package

July 22, 2015
By

The R world has come a long way since Jay & I wrote Data-Driven Security. We had to make a conscious decision to stick with R 2.14.0 (R is at version 3.2.1 now) and packages such as knitr and dplyr either didn’t exist or were in their infancy. In Chapter 4, we showed some very basic exploratory data analysis and...

Read more »

R 101 – Aggregate By Quarter

July 14, 2015
By
R 101 – Aggregate By Quarter

We were asked a question on how to (in R) aggregate quarterly data from what I believe was a daily time series. This is a pretty common task and there are many ways to do this in R, but we’ll focus on one method using the zoo and dplyr packages. Let’t get those imports out of the way: library(dplyr) library(zoo) library(ggplot2) Now, we need...

Read more »

“Just the text ma’am” – Web Site Content Extraction with XSLT & R

July 9, 2015
By

Sometimes you just need the salient text from a web site, often as a first step towards natural language processing (NLP) or classification. There are many ways to achieve this, but XSLT (eXtensible Stylesheet Language) was purpose-built for slicing, dicing and transforming XML (and, hence, HTML) so, it can make more sense and even be speedier use XSLT...

Read more »

Chart makeover – Unisys Security Insights Survey

July 2, 2015
By
Chart makeover – Unisys Security Insights Survey

It seems that not a day goes by without some information security vendor releasing a report based on a survey. Thankfully, this post is not about the efficacy of survey-based reports or their findings. Today, we’re doing a makeover for Unisys, who just released their Findings from the 2015 Unisys Security Insights Survey. I started reading their U.S....

Read more »

iptools 0.2.0 is now on CRAN

July 1, 2015
By

We are happy to announce that the iptools package is now on CRAN. Formerly only available on GitHub, iptools now compiles under Debian/Ubuntu, Fedora/CentOS/RedHat and Mac OS X (we’re still working on that other operating system). Oliver (the package co-author and on-CRAN instigator) wrote some excellent vignettes that cover the functionality of the package in-depth,...

Read more »

Short is the new Long with longurl for R (plus working with weblogs & URLs in R)

June 23, 2015
By

Necessity is the mother of invention and I had the opportunity today to take an R package from development to CRAN in less than the span of 24 hours. Despite being on vacation, I answered an R question on StackOverflow pertaining to the use of decode_short_url from the twitteR package. That function has no option for validity checking (i.e. whether the...

Read more »

Introducing passivetotal – R Package To Work With the PassiveTotal API

June 14, 2015
By

As a precursor to releasing Episode 18 of DDSec Podcast, we’re releasing a really basic R package to interface with the PassiveTotal API. We asked Brandon Dixon to be on the podcast to talk about his new visualization for users of PassiveTotal, which is a “threat research platform created for analysts, by analysts.”. PT...

Read more »