by Yanchang Zhao, RDataMining.com Compared with many other programming languages, such as C/C++ and Java, R is less efficient and consumes much more memory. Fortunately, there are some packages that enables parallel computing in R and also packages for processing … Continue reading → [Read more...]

As I mentioned in a previous post, I compiled the data set that I’m currently working on in PostgreSQL.  To get this massive data set, I had to write a query that was massive by dint of the number of … Continue reading → [Read more...]

The European Court of Justice has published its decision in SAS v WPL; the title of the press release says it all “The functionality of a computer program and the programming language cannot be protected by copyright”. To summarise the background, World Programming Ltd developed a system that was capable ... [Read more...]

2nd CALL FOR CHAPTERS: proposals due by 31 May 2012 Data Mining Applications with R A book to be published by Elsevier http://www.RDataMining.com/books/book2 Introduction —————— R is one of the most widely used data mining tools in scientific and business … Continue reading → [Read more...]

The task today was to find what variables had significant relations with an important grouping variable in the big dataset I’ve been working with lately.  The grouping variable has 3 levels, and represents different behaviours of interest.  At first I … Continue reading → [Read more...]

I have just got back from the 24 hour Data Science Global Hackathon; I was an on-site participant at Hub Westminster in London (thanks to Carlos and his team for doing such a great job looking after us all {around 50 turned up from the 100 who registered; the percentage was similar in ... [Read more...]

Yesterday I tried to do some data processing on my really big data set in MS Excel. Wow, did it not like handling all those data!! Every time I tried to click on a different ribbon, the screen didn’t even … Continue reading → [Read more...]

The R language has passed another milestone, a paper aimed at the academic programming language community (or at least one section of this community) has been written about it, Evaluating the Design of the R Language by Morandat, Hill, Osvald and Vitek. Hardly earth shattering news, but it may have ... [Read more...]

by Yanchang Zhao, RDataMining.com There are more than 100 R tips at http://pj.freefaculty.org/R/Rtips.html, which provide quick examples to small challenges in everyday R programming, especially for users switching from other languages to R. There is also a .PDF version for … Continue reading → [Read more...]

Now that I have one enormous project on the go and one smaller one, I find it’s helping me considerably to have each project stored in separate RStudio projects.  So, each project has its own scripting that I’ve been working … Continue reading → [Read more...]

At work I’ve started to work with the biggest data set I’ve ever seen!  First, let me qualify my use of the term “Big Data”.  The number of rows in the resultant data set (after much transformation and manipulation in … Continue reading → [Read more...]

Both R and Python have facilities where the coder can write a script which requests a user to input some information. In Python 2.6, the main function for this task is raw_input (in Python 3.0, it’s input()). In R, there are a series of functions that can be used to ... [Read more...]

Last week, I read a chapter out of Visualize This by Nathan Yau.  I was, of course, delighted to see that he was championing the use of R.  One really cool thing that I learned from his book, and was very … Continue reading → [Read more...]

In my last post I was struggling with getting a big memory version of the distance matrix to work fast. Nick and other readers had some suggestions and after puttering around with Nicks code I’ve adapted it to big memory and not impacted the run time very much. For ... [Read more...]

After my last post, I came across a few articles supporting the opinion that if you have a good reason to take random samples from a “big” dataset, you’re not committing some kind of sin: Big Data Blasphemy: Why Sample? … Continue reading → [Read more...]

The process of working on metadata and temperature series gives rise to several situations where I need to calculate the distance from every station to every other station. With a small number of stations this can be done easily on the fly with the result stored in a matrix. The ... [Read more...]

Yanchang Zhao, RDataMining.com Obama administration unveiled a Big Data Research and Development Initiative with $200 million on March 29, 2012, to improve the ability to extract knowledge and insights from large and complex collections of digital data. Six Federal departments … Continue reading → [Read more...]

The R Foundation has been accepted for Google’s summer of code and I thought I would suggest a few ideas for projects. My interests are in optimization and source code analysis, so obviously the suggestions involve these topics. There are an infinite number of possible optimizations that can be ... [Read more...]

By Yanchang Zhao, RDataMining.com If you have tried social network analysis or graph mining with R, you might have already come across package igraph before. The package is designed for graphs and network analysis in R. It can handle large … Continue reading → [Read more...]

I’m in the process of remaking all the metadata from scratch and looking once again at the question of UHI. There are not any global conclusions we can draw from the data yet; I’m just in the process of checking out everything that is available that could be ... [Read more...]