Data Science – learn the lessons of software

March 13, 2012
By

(This article was first published on alexfarquhar's posterous, and kindly contributed to R-bloggers)

We’re starting to see a deluge of companies who businesses are all about making data analysis/science/insight “easy for the non-expert”. We’ve been here before, quite a few times sadly. When I started writing software 12 years ago, there was great excitement in the air – finally we could use tools to design software, then press a button that would create our whole beautiful design in code! Then we could just hire some barely-sentient code monkeys to fill in the ‘easy bits’ like method definitions and those pesky database access routines.

It was a disaster. The fundamental problem was that by the time you’d crafted your beloved design and polished it to a high shine, the world had moved on. What may have worked on day 1 of the project was now hopelessly inadequate. We should always remember the maxim “no plan survives contact with the enemy”, the enemy here being the shifting reality of what your software needs to deliver.

Another major problem with this approach was the proliferation of so-called Software Architects, beings of such insight and experience that they didn’t even need to code anymore! Since they didn’t code, they couldn’t experience the grinding pain of trying to jam their grandiose designs into a reality-shaped hole.

Fast-forward to today – data is big, Data Science is even bigger (as a buzzword anyway), and we’re all short of the right people. The answer, however, is not to make tools that hide the complex, ever-shifting reality of the analytical process. It’s to make people better at doing this stuff. And there’ll be no magic off-the-shelf solution that can achieve this, any more than giving a terrible golfer great clubs will make them win The Masters.

Permalink

| Leave a comment  »

To leave a comment for the author, please follow the link and comment on their blog: alexfarquhar's posterous.

R-bloggers.com offers daily e-mail updates about R news and tutorials on topics such as: Data science, Big Data, R jobs, visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...



If you got this far, why not subscribe for updates from the site? Choose your flavor: e-mail, twitter, RSS, or facebook...

Comments are closed.

Search R-bloggers


Sponsors

Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)