Site icon R-bloggers

Darwinazing biodiversity data in R

[This article was first published on Vijay Barve, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

“Darwin Core (DwC) is a standard maintained by the Darwin Core maintenance group. It includes a glossary of terms (in other contexts these might be called properties, elements, fields, columns, attributes, or concepts) intended to facilitate the sharing of information about biological diversity by providing identifiers, labels, and definitions. Darwin Core is primarily based on taxa, their occurrence in nature as documented by observations, specimens, samples, and related information.” Darwin Core website

DwC is an evolving community-developed biodiversity data standard (Wieczorek et al. 2012). In simple words, it’s a list and definitions of common biodiversity data terms, ~200 of them, for more details see DwC quick reference guide. Today, hundreds of millions of biodiversity records from around the world are published in DwC format and aggregated into various portals (e.g. GBIF, vertNet, iDigBio). Nonetheless, data publishers still struggle with the essential step of mapping fields in their data to the terms in Darwin Core (Wieczorek et al. 2017a). Doing so requires a good understanding of both the data set and Darwin Core (Wieczorek et al. 2017b).

Related work

The remarkable Kurator project creates biodiversity data quality workflows, and via it web interface (Kurator-Web)– data quality-control is highly accessible. Thanks to this invaluable project, we have easy access to different lookup tables, that aggregates rare and highly valuable data regarding DwC vocabularies. For example in the Darwin Cloud table, knowledge is being accumulated about variations in DwC field names. Fully utilizing these precious data in the R environment can significantly enhance our ability to address more biodiversity data quality issues.

Details of project

Darwinizer workflow in R:

While DwC was adopted by most biodiversity data publishers, it implementation is somewhat incomplete. Imposing controlled vocabulary on millions of records is a complex and daunting task. For example, there are inconsistencies regarding field names between different data publishers. The CSV File Darwinizer Kurator workflow standardizes field names to the DwC standard name, thanks to the Darwin Cloud lookup file. By generating this workflow in R, we can easily input a wider range of data from different publishers. This module needs to work on various data files downloaded from different biodiversity portals, and handle all of them.

Data checks:

Data checks must be specifically tailored around the structure of the data, in our case- the DwC standard. Under this module we will address three major data checks collections:

R based Kurator actors:

Following a communication with the Kurator development team, we will explore the development of R actors (functions), that hopefully, will be seemingly integrated into the kurator infrastructure. Handling some Java and Python code will be required.

Expected impact

Improving the quality of biodiversity research, in some measure, is based on improving user-level data cleaning tools and skills. Adopting a more comprehensive approach for incorporating data cleaning as part of data analysis will not only improve the quality of biodiversity data, but will impose a more appropriate usage of such data.

If you are a full time University student and would like to participate in Google Summer of Code 2018 helping us build this, the project idea is listed here which has further details on how to start.

To leave a comment for the author, please follow the link and comment on their blog: Vijay Barve.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.