This morning, Stéphane asked me tricky question about extracting coefficients from a regression with categorical explanatory variates. More precisely, he asked me if it was possible to store the coefficients in a nice table, with information on the variable and the modality (those two information being in two different ... [Read more...]

[Read more...]

I have been working with R for some time now, but once in a while, basic functions catch my eye that I was not aware of… For some project I wanted to transform a correlation matrix into a covariance matrix. Now, since cor2cov does not exist, I thought about “... [Read more...]

Are your for loops too slow in R ? Are loops that should take seconds actually taking hours ? As I found out recently, how you structure your code can make a huge difference in execution times. Fortunately making a few small changes to your code can speed up these loops by ... [Read more...]

One of the most common pattern in programming languages is to ability to iterate over a given set (a vector usually) by using 'for' loops. In most modern scripting languages range operations is a build in data structure and trivial to use with 'for' lo... [Read more...]

So, you’ve written some code and you use it routinely. Now you’ve written some code and you’d like to use version control to ensure that development continues in a robust fashion. You do that and you use Github or something so that not only are changes tracked, ... [Read more...]

There is a mechanism that allows variability in the arguments given to R functions.  Technically it is ellipsis, but more commonly called “…”, dots, dot-dot-dot or three-dots. Basics The three-dots allows: an arbitrary number and variety of arguments passing arguments on to other functions Arbitrary arguments The two prime cases are ... [Read more...]

There was quite a lot of buzz around when the guys from Rstudio launched Shiny, a new web framework for R that promises to “make it super simple for R users like you to turn analyses into interactive web applications … Continue reading → [Read more...]

[Read more...]

There are many situations in R where you have a list of vectors that you need to convert to a data.frame. This question has been addressed over at StackOverflow and it turns out there are many different approaches to completing this task. Since I encou... [Read more...]

It's that moment we all know and love, somewhere in our code something has gone wrong. We think we have done everything right, but instead of expected glory we find only terse red text lain below our lintel. This can be very frustrating, and trouble shooting these issues can often ... [Read more...]

In my post yesterday comparing efficiency in joining two data frames, I overlooked the computing cost used to convert data.frames to data.tables / ff data objects. Today, I did the test again with the consideration of library loading and data conversion. After the replication of 10 times in rbenchmark package, ... [Read more...]

I’ve always had a special fondness for my name, which — according to Ryan Gosling in “Lars and the Real Girl” — is a scientific fact for most people (Ryan Gosling constitutes scientific proof in my book). Plus, the root … Continue reading → [Read more...]

The tagline for O'Reilly Strata conference series — Making Data Work — has meant that it's always been popular with practitioners, primarily data scientists working with Big Data in real-world environments. Recent Strata events have also attracted more business-oriented attendees, with events focused more on processes and outcomes than on the implementation ... [Read more...]

Big Data is the latest technology wave impacting C-Level executives across all areas of business, but amid the hype, there remains confusion about what it all means. The name emphasizes the exponential growth of data volumes worldwide (collectively, 5 Exabytes/ day in the latest estimate I saw from IDC), but more ... [Read more...]

Update 2: Put up a second graph showing debt of Spain.Update: I just read the most recent article of Stiglitz. I mostly agree. However, I'm not quite sure what he means when he says thatSpain and Ireland had fiscal surpluses and low debt/GDP ratios bef... [Read more...]

A friend of mine told me the secret of making money at the stock market. "It's easy", he said.All I would have to do is to buy a big jar of ants. Then I should observe the ants movement on my kitchen table, while following the stock market. I ... [Read more...]

Second update on my research, details here. [Read more...]

In R, there are multiple ways to merge 2 data frames. However, there could be a huge disparity in terms of efficiency. Therefore, it is worthwhile to test the performance among different methods and choose the correct approach in the real-world work. For smaller data frames with 1,000 rows, all six methods ... [Read more...]

I’ve been playing around with the ‘twitteR’ package for R ever since I heard of its existence. Twitter is great and easy to mine because the messages are all-text and most people’s profiles are public. This process is made even easier with the ‘twitteR’ package, which takes advantage ... [Read more...]