Blog Archives

Dealing with heteroskedasticity; regression with robust standard errors using R

Dealing with heteroskedasticity; regression with robust standard errors using R

First of all, is it heteroskedasticity or heteroscedasticity? According to McCulloch (1985), heteroskedasticity is the proper spelling, because when transliterating Greek words, scientists use the Latin letter k in place of the Greek letter κ (kappa). κ sometimes is transliterated as the Latin letter c, but only when these words entered the English language through French, such as scepter. Now that this is out of...

Read more »

Missing data imputation and instrumental variables regression: the tidy approach

Missing data imputation and instrumental variables regression: the tidy approach

In this blog post I will discuss missing data imputation and instrumental variables regression. This is based on a short presentation I will give at my job. You can find the data used here on this website: http://eclr.humanities.manchester.ac.uk/index.php/IV_in_R The data is used is from Wooldridge’s book, Econometrics: A modern Approach. You can download the data by clicking here. This is the variable description: 1....

Read more »

Forecasting my weight with R

Forecasting my weight with R

I’ve been measuring my weight almost daily for almost 2 years now; I actually started earlier, but not as consistently. The goal of this blog post is to get re-acquaiented with time series; I haven’t had the opportunity to work with time series for a long time now and I have seen that quite a few packages that deal with time series...

Read more »

Getting data from pdfs using the pdftools package

Getting data from pdfs using the pdftools package

It is often the case that data is trapped inside pdfs, but thankfully there are ways to extract it from the pdfs. A very nice package for this task is pdftools (Github link) and this blog post will describe some basic functionality from that package. First, let’s find some pdfs that contain interesting data. For this post, I’m using the diabetes country profiles from...

Read more »

{pmice}, an experimental package for missing data imputation in parallel using {mice} and {furrr}

Yesterday I wrote this blog post which showed how one could use {furrr} and {mice} to impute missing data in parallel, thus speeding up the process tremendously. To make using this snippet of code easier, I quickly cobbled together an experimental package called {pmice} that you can install from Github: devtools::install_github("b-rodrigues/pmice") For now, it returns a list of mids objects and not a...

Read more »

Imputing missing values in parallel using {furrr}

Today I saw this tweet on my timeline: For those of us that just can't wait until RStudio officially supports parallel purrr in #rstats, boy have I got something for you. Introducing `furrr`, parallel purrr through the use of futures. Go ahead, break things, you know you want to:https://t.co/l9z1UC2Tew— Davis Vaughan (@dvaughan32) April 13, 2018 and as a heavy {purrr} user,...

Read more »

Get basic summary statistics for all the variables in a data frame

I have added a new function to my {brotools} package, called describe(), which takes a data frame as an argument, and returns another data frame with descriptive statistics. It is very much inspired by the {skmir} package but also by assist::describe() (click on the packages to be redirected to the respective Github repos) but I wanted to write my...

Read more »

Getting {sparklyr}, {h2o}, {rsparkling} to work together and some fun with bash

This is going to be the type of blog posts that would perhaps be better as a gist, but it is easier for me to use my blog as my own personal collection of gists. Plus, someone else might find this useful, so here it is! In this blog post I am going to show a little trick to...

Read more »

Keep trying that api call with purrr::possibly()

Sometimes you need to call an api to get some result from a web service, but sometimes this call might fail. You might get an error 500 for example, or maybe you’re making too many calls too fast. Regarding this last point, I really encourage you to read Ethics in Web Scraping. In this blog post I will show you...

Read more »

Importing 30GB of data in R with sparklyr

February 15, 2018
By
Importing 30GB of data in R with sparklyr

Disclaimer: the first part of this blog post draws heavily from Working with CSVs on the Command Line, which is a beautiful resource that lists very nice tips and tricks to work with CSV files before having to load them into R, or any other statistical software. I highly recommend it! Also, if you find this interesting, read also...

Read more »

Search R-bloggers


Sponsors

Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)