Pre-conference tutorial about R/exams at useR! 2019 (The R User Conference) in Toulouse: Slides, example code, and links to further information.

Tutorial at useR! 2019

Today, the pre-conference tutorials kick off this year’s useR! 2019 conference. The organizers kindly invited R/exams for one of these tutorials. This blog post provides some accompanying resources which may be of interest to those who do not attend the conference/tutorial. The plan is to roughly follow the outline below.

Time	Content
14:00 14:30 15:00	Introduction (overview, installation, written exam) Dynamic exercises (numeric, single choice, multiple choice, cloze) One-for-all (plain PDF, plain HTML, Moodle XML)
15:30	— Break —
16:00 16:30 17:00	E-Learning (Moodle, Canvas, ARSnova, …) Written exams (NOPS) Outlook

Most importantly the presentation slides are available in PDF format (under CC-BY) along with an R script (under GPL-2 or GPL-3) for the examples:

Below, the examples, code, and links from the slides and R script are also provided which might be more convenient for some readers.

Introduction

To set up your computer for this tutorial, follow Steps 1-4 in the tutorial http://www.R-exams.org/tutorials/installation/.

Subsequently, we start out with setps described in more detail in http://www.R-exams.org/tutorials/first_steps/. The package is loaded and first we create an “exams skeleton” with:

• demo-*.R scripts
• exercises/ folder with all .Rmd/.Rnw exercises
• templates/ folder with various customizable templates
• nops/ folder (empty) for exams2nops() output

library("exams")
exams_skeleton()

To check whether everything works we render a single-choice question (swisscapital, knowledge quiz about the Swiss capital) into a plain HTML and PDF format, respectively.

exams2html("swisscapital.Rmd")
exams2pdf("swisscapital.Rmd")

And then we do the same for a numeric question with mathematical notation: (deriv, product rule for derivatives). By default exams2html() uses MathML for the mathematical notation and expects the browser to render it which works ok in Firefox and Safari. But for Chrome it is better to use MathJax, see this post about mathematical notation in web exams for more details.

exams2html("deriv.Rmd")
exams2html("deriv.Rmd", converter = "pandoc-mathjax")
exams2pdf("deriv.Rmd")

Finally, a useful trick for checking whether the meta-information associated with an exercise is processed correctly:

exams_metainfo(exams2html(c("swisscapital.Rmd", "deriv.Rmd")))

Dynamic exercises

This section gives a tour of the dynamic exercises shipped along with the package. See http://www.R-exams.org/intro/dynamic/ for the supported exercise types and http://www.R-exams.org/templates/ for a list of all example exercises.

Additionally, there is a simple knowledge quiz question about useR! conferences: conferences.Rmd, conferences.Rnw.

And the following exercises are used to illustrate how to go from a static to a dynamic numeric or single-choice question, see http://www.R-exams.org/tutorials/static_num_schoice/ for more details. While the tutorial uses a basic economic example about computing elasticities, the exercises below are basic arithmetic exercises about derivatives.

#	Exercise templates	Dynamic?	Type	Description
1	expderiv1.Rmd expderiv1.Rnw	No	num	Fixed parameters and numeric solution.
2	expderiv2.Rmd expderiv2.Rnw	No	schoice	As in #1 but with single-choice solution (five answer alternatives).
3	expderiv3.Rmd expderiv3.Rnw	Yes	num	Randomly drawn parameters with dynamic computation of correct solution, based on #1.
4	expderiv4.Rmd expderiv4.Rnw	Yes	schoice	Randomly drawn parameters (as in #3) with dynamically-generated single-choice solution (as in #2), computed by num_to_schoice().
5	expderiv5.Rmd expderiv5.Rnw	Yes	schoice	As in #4 but with the last alternative: None of the above.

All of these extra exercise files can be downloaded from within R and stored in the skeleton set up above:

extra <- c("conferences", paste0("expderiv", 1:5))
extra <- c(paste0(extra, ".Rnw"), paste0(extra, ".Rmd"))
for(i in extra) download.file(paste0("http://www.R-exams.org/assets/posts/2019-07-09-user2019/", i), i)

To take a tour of the available exercise files, we switch to the exercises/ folder:

setwd("exercises")
dir()

Numeric exercises

exams2html(c(
  "deriv.Rmd",        ## product rule for derivatives
  "tstat.Rmd",        ## 1-sample t-test statistic
  "dist.Rmd",         ## distances and the Pythagorean theorem
  "regression.Rmd",   ## simple linear regression (by hand)
  "fruit.Rmd",        ## image-based systems of linear equations (numeric)
  "lagrange.Rmd",     ## method of Lagrange multipliers
  "currency8.Rmd"     ## convert currencies (UTF-8 encoding)
))

Single-choice exercises

exams2html(c(
  "swisscapital.Rmd", ## knowledge quiz question about the Swiss capital
  "Rlogo.Rmd",        ## knowledge quiz question about the R logo
  "deriv2.Rmd",       ## product rule for derivatives (single-choice, via num_to_schoice)
  "tstat2.Rmd",       ## 1-sample t-test statistic (single-choice, by hand)
  "dist3.Rmd",        ## distances and the Pythagorean theorem (single-choice, via num_to_schoice)
  "fruit2.Rmd",       ## image-based systems of linear equations (single-choice, via num_to_schoice)
  "hessian.Rmd",      ## 2x2 Hessian matrix (single-choice)
  "logic.Rmd"         ## interpretation of logic gates (using TikZ)
))

Multiple-choice exercises

exams2html(c(
  "switzerland.Rmd",  ## knowledge quiz question about Switzerland
  "gaussmarkov.Rmd",  ## knowledge quiz question about Gauss-Markov assumptions
  "anova.Rmd",        ## 1-way analysis of variance
  "boxplots.Rmd",     ## interpretation of 2-sample boxplots
  "scatterplot.Rmd",  ## interpretation of a scatterplot
  "ttest.Rmd",        ## interpretation of 2-sample t test
  "relfreq.Rmd",      ## interpretation of relative frequency tables
  "cholesky.Rmd",     ## Cholesky decomposition
  "automaton.Rmd"     ## interpretation of automaton diagrams (using TikZ)
))

String exercises

exams2html(c(
  "function.Rmd",     ## string question about R functions
  "countrycodes.Rmd"  ## string question about ISO country codes
))

Cloze exercises

These combine several num/schoice/mchoice/string parts:

exams2html(c(
  "lm.Rmd",           ## simple linear regression (with CSV data, schoice/num)
  "boxhist.Rmd",      ## univariate exploration of a CSV file (schoice/num)
  "confint2.Rmd",     ## 2-sided confidence interval (num)
  "dist2.Rmd",        ## three distances in a Cartesian coordinate system (num)
  "fourfold.Rmd"      ## fourfold table (num)
))

Stress testing dynamic exercises

The function stresstest_exercise() can help to do some basic quality control of dynamic exercise templates. It runs the exercises n times (default: 100) with seeds 1 to n and stores run time, solution, and accompanying variables (of length 1). This can help to spot problems or patterns in the exercises.

A simple example for a numeric exercise is:

s3 <- stresstest_exercise("expderiv3.Rmd")
plot(s3)
plot(s3, type = "solution")

And for a single-choice exercise:

s4 <- stresstest_exercise("expderiv4.Rmd")
plot(s4)
plot(s4, type = "solution")

To focus on typical errors the plots against the variables err1 and err2 can either be created by the plot() method or “by hand” with some customization:

plot(s4, type = "solution", variables = c("err1", "err2"))
plot(res ~ err1, data = s4$objects)
abline(0, 1)
plot(res ~ err2, data = s4$objects)
abline(0, 1)

Finally, we switch back to the original folder:

setwd("..")

One-for-all

A basic design idea for R/exams is that the same exercise can, in principle, be rendered into a wide range of different output formats. For standalone files the most important formats are PDF, HTML, and possibly also Docx:

demo_exam <- c("swisscapital.Rmd", "deriv.Rmd")
exams2html(demo_exam, edir = "exercises")
exams2pdf(demo_exam, edir = "exercises")
exams2pandoc(demo_exam, edir = "exercises")

The layout can be customized using suitable templates, e.g.,

exams2pdf(demo_exam, edir = "exercises", template = "templates/exam.tex")
exams2pdf(demo_exam, edir = "exercises", template = "templates/solution.tex")

Also, exams2nops() internally sets up a template on the fly and then calls exams2pdf(). And e-learning exams (Moodle XML, QTI 1.2/2.1, …) are illustrated in the next section.

E-Learning (Moodle, Canvas, ARSnova, …)

Output for various e-learning formats can be generated in pretty much the same way as the standalone files above. The main difference is that not one file per exam is generated but a combined output file comprising the desired number of exam replications. For example for Moodle:

elearn_exam <- c("conferences.Rmd", "deriv.Rmd", "ttest.Rmd", "boxplots.Rmd",
  "function.Rnw", "lm.Rmd", "fourfold2.Rmd", "confint3.Rmd")
set.seed(2019-07-09)
exams2moodle(elearn_exam, n = 30, name = "useR-2019",
  dir = "output", edir = "exercises")

For the registered participants of the course the resulting online quiz is made available at https://eeecon.uibk.ac.at/~moodle/. You can login with your e-mail address and the numeric code sent to you.

Analogously, the numeric and single-/multiple-choice questions (but not yet cloze) can be prepared for Canvas by:

set.seed(2019-07-09)
exams2canvas(elearn_exam[1:5], n = 3, name = "useR-2019",
  dir = "output", edir = "exercises", duration = 15)

For OpenOLAT all exercises except one Moodle-specific exercise can be used as well:

set.seed(2019-07-09)
exams2openolat(elearn_exam[1:7], n = 3, name = "useR-2019-olat",
  dir = "output", edir = "exercises")

For ARSnova only single- and multiple-choice questions have good support:

choice_exam <- list(
  "conferences.Rmd",
  "deriv2.Rmd",
  c("ttest.Rmd", "boxplots.Rmd")
)
set.seed(2019-07-09)
exams2arsnova(unlist(choice_exam), n = 1, name = "useR-2019",
  dir = "output", edir = "exercises")

Written exams (NOPS)

The written demo exam from the tutorial can be replicated by:

set.seed(2019-07-09)
exams2nops(choice_exam, n = 35, name = "useR-2019",
  dir = "nops", edir = "exercises",
  reglength = 8, blank = 1, date = "2019-07-09",
  title = "Demo Exam", institution = "useR! 2019")

More details about creating and evaluating NOPS exams is provided in the tutorial at http://www.R-exams.org/tutorials/exams2nops/.