Features How to get How to use Screenshots Code In a previous post, I wrote about what I use association rules for and mentioned a Shiny application I developed to explore and visualize rules. This post is about that app. The app is mainly a wrapper around the arules and ... [Read more...]

Tips and tricks learned along the way 1. DATA STRUCTURES & ASSIGNMENT =__ Columns of lists =__ Suppressing intermediate output with {} =__ Fast looping with set =__ Using shift for to lead/lag vectors and lists =__ Create multiple columns with := in one statement =__ Assign a column with := named with a character object 2. BY =__ Calculate a function ... [Read more...]

Workflow This post is really about workflow. Specifically a data-science workflow, although it should be relevant for others. It will probably resonate most (if at all) with those who have some experience (mostly positive) generating reports from Rmarkdown files with knitr, but might have some gripes. Maybe not gripes, maybe ... [Read more...]

use knitr One way to blog using R and Jekyll is to copy and paste every code chunk, output and plot into a plain vanilla markdown file by hand. This is cumbersome, especially for plots which need to be saved as images and embedded. I tried this for my first ... [Read more...]

LDA for mortals I’ve been intrigued by LDA topic models for a few weeks now. I actually built one before I really knew what I was doing. That kind of frightens and excites me. On one hand, LDA provides rich output which is easy for the humanist researcher to ... [Read more...]

Motivation Accessing NYT API Extracting and parsing the article body text Writing to MongoDB Pipeline Results Motivation I’ve learned a little about a lot of different corners of the text mining and NLP world over the last few years… which sometimes makes me feel like I know nothing for ... [Read more...]

If you’ve ever… felt like you’re playing Simon Says with mouse clicks when repeatedly extracting data in chunks from a front-end interface to a database on the web, well, you probably are. There’s probably a better solution – Selenium. ever used XML or httr in R or urllib2 ... [Read more...]

Why create a personal R package? As a consulting data scientist, I write a lot of R code in a lot of different places – physically and virtually. Different computers, servers, evironments, VPNs, operating systems, all of the above. Even when I have the luxury of working with the same client (... [Read more...]

So I’ve been messing around with Shiny for a year or so now. It’s great tool and getting greater. the good: capability to rapidly build an interactive visualization with the full universe of R packages to choose from for the computation engine and visualization options. ability to code ... [Read more...]

So now that we’ve collected data from the web and wrangled it into something useful, what can we say about how accurate Next Bus is? This post is about the “making of the analysis” … which might be rather boring to those non data geeks (normal people). If you’re ... [Read more...]

Goal: determine how accurate Next Bus predictions are Step 1: Create a timeseries Step 2: Flag arrivals and departures Step 2: Filter Step 3: Calculate error in Next Bus predictions Step 4: Write out data to csv In a the previous post we extracted Next Bus predictions from the Wmata API. If that’s too ... [Read more...]