Site icon R-bloggers

How to Create Cross-Validation Partitions with Plain Text in R?

[This article was first published on #untitled, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

In my last blog post, I talked about Random Sampling for Plain Text in R. Now when you are performing some supervised learning on your data, if you follow the cross-validation principle which divides the data into train, holdout and test, you want to make sure each subset is different. The great caret package has some neat partitioning techniques however for my use, I wanted a minimal function without the need to load extra libraries.

Solution
createPartition <- function (filename, trainPortion=0.6, holdOutPortion=0.2) {  
  con <- file(filename)
  total <- length(readLines(con))

  sample <- floor(total * trainPortion)
  sample_holdout <- floor(total * holdOutPortion)

  corpus <- scan(con, what="character", skip= 0, nlines=total, sep="\n", fileEncoding = 'UTF-8')
  close(con)
  train <- corpus[1:sample]
  holdout <- corpus[(sample + 1): (sample + sample_holdout)]
  test <- corpus[(sample + sample_holdout + 1) : total]

  result <- list(train=train, holdout=holdout, test=test)
  return(result)
}

What you need to pass is only a path to filename. I set the partitions to be 60/20/20 rule which is suggested by the great Andrew Ng.

Here’s how to use it:

data <- createPartition("path_to_file.format")  
data$train  
data$holdout  
data$test  

I hope this helped! ???

To leave a comment for the author, please follow the link and comment on their blog: #untitled.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.