I’m happy to announce that version 0.10.0 beta of the dplyrXdf package is now available. You can get it from Github:
This is a major update to dplyrXdf that adds the following features:
- Support for the tidyeval framework that powers the latest version of dplyr
- Works with Spark and Hadoop clusters and files in HDFS
- Several utility functions to ease working with files and datasets
- Many bugfixes and workarounds for issues with the underlying RevoScaleR functions
This (pre-)release of dplyrXdf requires Microsoft R Server or Client version 8.0 or higher, and dplyr 0.7 or higher. If you’re using R Server, dplyr 0.7 won’t be in the MRAN snapshot that is your default repo, but you can get it from CRAN:
The tidyeval framework
This completely changes the way in which dplyr handles standard evaluation. Previously, if you wanted to program with dplyr pipelines, you had to use special versions of the verbs ending with “_”:
select_, and so on. You then provided inputs to these verbs via formulas or strings, in a way that was almost but not quite entirely unlike normal dplyr usage. For example, if you wanted to programmatically carry out a transformation on a given column in a data frame, you did the following:
This is prone to errors, since it requires creating a string and then parsing it. Worse, it's also insecure, as you can't always guarantee that the input string won't be malicious.
The tidyeval framework replaces all of that. In dplyr 0.7, you call the same functions for both interactive use and programming. The equivalent of the above in the new framework would be:
!! symbol is a special operator that means to get the column name from the variable to its right. The verbs in dplyr 0.7 understand the special rules for working with quoted symbols introduced in the new framework. The same code also works in dplyrXdf 0.10:
New features in dplyrXdf
Copy, move and delete Xdf files
The following functions let you manipulate Xdf files as files:
move_xdfcopy and move an Xdf file, optionally renaming it as well.
rename_xdfdoes a strict rename, ie without changing the file’s location.
delete_xdfdeletes the Xdf file.
HDFS file transfers
The following functions let you transfer files and datasets to and from HDFS, for working with a Spark or Hadoop cluster:
copy_touploads a dataset (a data frame or data source object) from the native filesystem to HDFS, saving it as an Xdf file.
computedo the reverse, downloading an Xdf file from HDFS.
hdfs_downloadtransfer arbitrary files and directories to and from HDFS.
Uploading and downloading works (or should work) both from the edge node and from a remote client. The interface is the same in both cases: no need to remember when to use rxHadoopCopyFromLocal and rxHadoopCopyFromClient. The hdfs_* functions mostly wrap the rxHadoop* functions, but also add extra functionality in some cases (eg vectorised copy/move, test for directory existence, etc).
HDFS file management
The following functions are for file management in HDFS, and mirror similar functions in base R for working with the native filesystem:
hdfs_dirlists files in a HDFS directory, like dir() for the native filesystem.
hdfs_file_existstest for existence of a directory or file, like dir.exists() and file.exists().
hdfs_file_removecopy, move and delete files in a vectorised fashion, like file.copy(), file.rename() and unlink().
hdfs_dir_removemake and delete directories, like dir.create() and unlink(recursive=TRUE).
in_hdfsreturns whether a data source is in HDFS or not.
As far as possible, the functions avoid reading the data via rxDataStep and so should be more efficient. The only times when rxDataStep is necessary are when importing from a non-Xdf data source, and converting between standard and composite Xdfs.
as_xdfimports a dataset or data source into an Xdf file, optionally as composite.
as_composite_xdfare shortcuts for creating standard and composite Xdfs respectively.
is_composite_xdfreturn whether a data source is a (composite) Xdf.
local_execruns an expression in the local compute context: useful for when you want to work with local Xdf files while connected to a remote cluster.
dplyrXdf 0.10 is tentatively scheduled for a final release at the same time as the next version of Microsoft R Server, or shortly afterwards. In the meantime, please download this and give it a try; if you run into any bugs, or if you have any feedback, you can email me or log an issue at the Github repo.