In my work, we have come across problems where we wanted to visualize data in 4 dimensions (4D). The data came as brains, which are represented as 3D volumes and we wanted to visualize them at different time points (hence 4D). We were using images that are acquired one image per visit (CT structural scans, particularly), where each patient had a maximum of 10 time points. (The methods below have not been tested or adapted for fMRI scans that have hundreds of time points).

As a result, we created the package brainR (CRAN package) to fit our needs with this. We strive to make figures such as this 4D example.

This figure can be rotated and zoomed. The slider provides a changing opacity for the overall brain (in order to see the structures inside). The buttons for each “visit” represent enhanced lesions in patients with MS detected by SuBLIME (Sweeney et al. 2013). I feel that these figures can convey results that an orthographic figure such as

cannot, nor can a single slice over time as these enhancing lesions are all over the brain.

The steps to make images such asin brainR is this:

1. Read (neuro)imaging data (usually using oro.dicom or oro.nifti packages).
2. Have 1) a brain to plot and 2) a region of interest (ROI) in the same space as the brain
3. Create a scene using contour3d from misc3d package (Feng & Tierney, 2008).
4. Write the contents of this scene into 3D objects of formats STL) or OBJ using wrappers for writeSTL or writeOBJ from rgl package (Adler et al. 2014).
5. Use the X ToolKit (XTK) and simple JavaScript controls to visualize these structures and switch between them.

The last step is what I call 4D visualization, being able to move around space with rotation, zooming, and translation while moving through another dimension, usually different time points, ROIs, structures, etc. The reason we went this route is a few reasons, but particularly:

1. I know R
2. rgl and misc3d already exist
3. JavaScript is “accessible” and has a lot of docs/support
4. Going further into 4D visualization in software is essentially video game programming or using a language I don’t know

Number 4. was interesting as I looked into tools like Blender, Paraview, or CINEMA 4D, which are are highly powerful. Many of them have a GUI-only interface and some used for much visualizing more extensive than (and not adapted so well for) biomedical data, such as full scenes and landscapes. Most solutions, such as those for MATLAB, require the user either have MATLAB programming experience or download software, which we found to be a large and unneccesary burden – so we use JavaScript in a browser. Granted, to run brainR you need to know R, but the end user just needs to know how to open a browser. If the end user cannot do that, I don’t think 4D visualization is for them anyway.

Now, let’s rundown some of the functions:

• write4D takes in a “scene”, which is a collection of surfaces, and writes them to an html, along with the STL or OBJ files needed to render it.
• makeScene is an adaptation of contour3d from misc3d that does mutually exclusive leveling. For example, if you have an image that has values {1, 2, 3}, then you can create an individual surface for 1, one for 2, one for 3. With contour3d, they take the values \(\geq\) to the level for the surface, which means that the surface for 2 would contain 2 and 3.
• writeWebGL_split is an adaptation of rgl‘s writeWebGL but cuts the objects into individual objects of only 65535 vertices (see here for discussion of WebGL number of vertices, see here for discussion with Duncan Murdoch). This is a limitation of WebGL rendering and we simply split the data using the code given by Duncan Murdoch in the second link. (writeWebGL has a note in the details, but I missed it). Why is this important? If you don’t do the splitting, your surface will not render properly, if at all.

Overall, this is my first package and the paper is submitted to R Journal. The admins at CRAN were very helpful in getting the package ready for production. It wasn’t easy, but it’s done.

The development version of brainR is located at this gitHub package.

Note, there is some interesting work done with 3D objects (and some 4D functionality) into PDF recently and there is interesting overview of methods tried located here if you’re interested. We chose the web because it rendered faster and had easier integration with JavaScript.

Citations were done using knitcitations (Adler et al. 2014).