Data Science – learn the lessons of software

[This article was first published on alexfarquhar's posterous, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

We’re starting to see a deluge of companies who businesses are all about making data analysis/science/insight “easy for the non-expert”. We’ve been here before, quite a few times sadly. When I started writing software 12 years ago, there was great excitement in the air – finally we could use tools to design software, then press a button that would create our whole beautiful design in code! Then we could just hire some barely-sentient code monkeys to fill in the ‘easy bits’ like method definitions and those pesky database access routines.

It was a disaster. The fundamental problem was that by the time you’d crafted your beloved design and polished it to a high shine, the world had moved on. What may have worked on day 1 of the project was now hopelessly inadequate. We should always remember the maxim “no plan survives contact with the enemy”, the enemy here being the shifting reality of what your software needs to deliver.

Another major problem with this approach was the proliferation of so-called Software Architects, beings of such insight and experience that they didn’t even need to code anymore! Since they didn’t code, they couldn’t experience the grinding pain of trying to jam their grandiose designs into a reality-shaped hole.

Fast-forward to today – data is big, Data Science is even bigger (as a buzzword anyway), and we’re all short of the right people. The answer, however, is not to make tools that hide the complex, ever-shifting reality of the analytical process. It’s to make people better at doing this stuff. And there’ll be no magic off-the-shelf solution that can achieve this, any more than giving a terrible golfer great clubs will make them win The Masters.

Permalink | Leave a comment  »

To leave a comment for the author, please follow the link and comment on their blog: alexfarquhar's posterous. offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)