Operating systems are often excluded from “toolkit” discussions, presumably because most popular tools are cross-platform and abstract away a lot of the differences across operating systems. Nevertheless, operating systems are not all created equal, and in my opinion Linux-based operating systems are currently the best option for working with data.

Linux is similar in many ways to OS X (they are both UNIX-like), but without the annoying restrictions and limitations. Linux gives you a package manager and the freedom to install whatever tools you need, and to configure them however you like. All this freedom can make it easier to shoot yourself in the foot, but I find that preferable to being restricted by technical limitations (Windows) or greedy corporate policy (Apple OS X).

It is worth noting that the world of Linux is diverse and varied, and that there can be significant differences among different Linux distributions. One frequent issue is that many Linux distributions are very conservative about releasing software updates. This might make sense for embedded devices or servers (though I have my doubts) but it makes absolutely no sense on a laptop or personal machine. Generally speaking, I want to run the latest stable release of all the programs installed on my computers. Arch Linux is one of the few Linux distributions that makes it easy to keep your applications up-to-date. If you are new to Linux you may wish to start with Manjaro Linux, a pre-configured Arch Linux derivative.

I use the Emacs text editor because I haven’t yet found anything that I like better. It has a lot of legacy baggage that makes it feel alien and intimidating at first, but I got used the these quirks after a using it for a week or two. The defining feature of Emacs is that it can be customized and extended without limit. It is both a text editor and a toolkit to building your own editing environment. A large and robust ecosystem of community developed packages makes it easy to use Emacs not only as a text editor, but also as a Git and GitHub front-end, an email client, and StackOverflow browser (to give but a few examples).

Most of the documents I produces these days are technical or training materials that include lots of example code. I often use literate programming techniques to keep the examples and the output produced by those examples together in a single document. The markup language I use for this purpose depends on the complexity of the document.

Markdown is the simplest and most ubiquitous of the markup languages I use. I often use it for simple documents, or those on which non-emacs users are collaborating. Org mode is a more powerful markup language for which adequate support is available only in Emacs. I use it for many things, including most of my workshop notes. Finally LaTeX is the most powerful, complex, and verbose of the markup languages I use. It is useful when you need more control over the appearance of the resulting document.

I use R and Python for working with data because they provide a good balance of flexibility and convenience. I prefer them to statistics packages like SPSS or SAS because both R and Python are full-fledged programming languages that give me the power and flexibility I need to address unusual or complicated tasks. At the same time, their substantial standard libraries and huge package repositories make it easy to accomplish standard or common data management and analysis tasks.

Notably, I do not choose these languages because the core language design – in both cases I don’t particularly like the languages. It really is the ecosystem of packages that keeps me using these tools instead of running off to a shiny new thing like Julia or Go, or even from wandering off to an old but more interesting environment like Haskell.

It has been my experience that kids these days don’t really like shells. Honestly I don’t blame theme. Shell technology has been stuck in the 80’s for far too long. That situation is starting to change, but bash (a shell from the 80’s!) is still by far the most commonly used. I think it is time for a change, and fish is the most well-developed at the moment. It doesn’t make using the command line fun exactly, but it feels a lot less like being forced to time-travel 40 years into the past.

I use the Terminology emulator because unlike other terminal emulators it can display images and video directly in your terminal. This also helps avoid the forced-time-travel feeling commonly induced by using bash in a typical terminal emulator.

I don’t particularly like git, but since everyone uses it I don’t feel like I have much of a choice. If it were up to me I would use something simpler like mercurial but it’s not so I use git. It is much more complicated and frustrating than it needs to be, but it doesn’t suck once you get the hang of it.

If you’ve read this far you must be really interested in tools for working with data! While I hope it was interesting to read about my choices, I encourage you to try out some alternatives and pick a set of tools that works well for you.

In reflecting on my own tool choices I notice that customization and community activity are key values for me. For example, I like R not so much because of the design of the language, but because it is flexible and has an active community building and sharing tools in the form of R packages. Similarly, I value Emacs because it is easy to configure and because there is an active community developing Emacs packages. I value these tools not because of their design per se., but because they are actually platforms that their user and developer communities have built tools on top of. The downside of this preference for power and flexibility is that these tools are often complex. Some people prefer simpler tools, e.g., Stata instead of R or Jupyter Notebooks instead of Emacs with Org mode, and that is perfectly reasonable.

My data science tools workshop notes describe some alternative tools and is a good place to start if you’re not sure what you should use for a particular task.