Introduction In the previous part of this series, we looked at writing R functions that can be executed directly by Spark without serialization overhead with a focus on writing functions as combinations of dplyr verbs and investigated how the SQL is generated and Spark plans created. In this third part, ...

Problem Especially in a project with many different scripts, it can be challenging to keep track of all the packages you need to load. It’s also easy to lose track of whether or not you’ve incorporated package loading into the script itself until you switch to a new ...

A paper I worked on was just published in a medical journal. This is quite an odd thing for me to be able to say, given my academic background and the career path I have had, but there you go! The first author of this paper is a long-time friend ...

Recap We’ve covered various approaches in explaining model predictions globally. Today we will learn about another model specific post hoc analysis. We will learn to understand the workings of gradient boosting predictions. Like past posts, the Clevaland heart dataset as well as tidymodels principle will be used. Refer to ...

Problem The ifelse() function only allows for one “if” statement, two cases. You could add nested “if” statements, but that’s just a pain, especially if the 3+ conditions you want to use are all on the same level, conceptually. Is there a way to specify multiple conditions at the same ...

Introduction - Part I Earlier this year, Reviewing Fairfield County Municipal Fiscal Indicators Since 2001 looked at data from 17 annual Connecticut Municipal Fiscal Indicators reports to better understand the current financial condition of all 169 towns. Those reports included only began including the net pension liability over the last three years and ...

rpy2 for using R in Python

rpy2 for using R in Python

Within the field, Stata is the dominant software package for economists. I suspect a large portion of universities, like mine, still do alot of their teaching using it, and given its outsized influence, it’s probably still important to know how to us...

Simulation was the key to to achieve world beer dominance. ‘Scientific’ Brewing at scale in the early 1900s Beer bottles cheers This post is an explainer about the small sample experiments performed by William S. Gosset. This post contains some R code that simulates his simulations1 and the resulting determination ...

Machine learning (ML) and AI has become the new buzz word in town. With that being said, there is a lot of demand for data scientists and machine learning engineers across various industries including IT, telecom, automotive, manufacturing and many more. Today, there are hundreds to thousands of machine learning ...

What is the support vector machine (SVM) algorithm? Imagine you would like to predict whether your boss will be in a good mood or not (a very important machine learning application). Over a couple of weeks, you record the number of hours you spend playing games at your desk and ...