Experiences with transitioning to R/exams for multiple automated and randomized online exams in a soil mechanics course at Universität Innsbruck (UIBK).
Guest post by Wolfgang Fellin (Universität Innsbruck, Division of Geotechnical and Tunnel Engineering).
Our exercises in soil mechanics are attended by about 120 students every year. Students receive their grades based on a final exam, with a multiple-choice part and additional small exercises with calculations. Due to the COVID-19 restrictions we were forced to switch from the written exams to online exams at rather short notice, for which we decided to use R/exams to randomize the exercises. The possibility that the exercises produced can be used in the following (hopefully COVID-free years) as part of a blended learning course made the decision for the (not so small) effort to switch to R/exams easy.
Transition of multiple-choice exercises
To leverage the about 100 multiple-choice exercises from our old question pool, we had to convert them to the R/exams format. The pool was stored in a MySQL data base including LaTeX text for the questions, answer alternatives, images for some exercises (in EPS format) and some additional meta-information regarding the topic category and the difficulty. Due to the standardized format of the question pool – and the kind support by Achim Zeileis and Christiane Ernst – it was possible to extract all questions, embed them into the same template for R/LaTeX questions (.Rnw), and convert all EPS figures to both SVG and PDF format. A typical example is shown below (in a screen shot from OpenOLAT, the learning management system at UIBK).
The corresponding R/LaTeX exercise file is grundwasser-1-068.Rnw which we also converted to R/Markdown for this blog post: grundwasser-1-068.Rmd. Both rely on the corresponding image file being available in the same directory or a sub-directory. Different graphics formats are used for different exam output formats: image68_1.svg, image68_1.pdf, image68_1.eps.
Essentially, the exercises consist of:
- some R code for including the right graphics file (SVG vs. PDF) depending on the type of output format (HTML vs. PDF),
- the (static) question text and answer alternatives in LaTeX format,
- meta-information including
exextratags for our custom topic category and difficulty level,
exshuffletag is used for randomly subsampling the available answer alternatives.
New numeric exercises
While the subsampling approach in the multiple-choice exercises provides a certain level of
randomization, I wanted to set up new numeric exercises in which all students have to calculate
the correct solution(s) based on their personalized parameters. Therefore, I set up new numeric
cloze format where several quantities have to be computed based on the same
question text. A typical example is shown below.
Despite being a newcomer to both R and R/exams, setting up these exercises was quite easy because I have been using MATLAB for a long time and could quickly grasp basic R code. Also the exercise templates available online helped as did the friendly e-mail support by Achim for resolving many of the small “details” along the way.
Following the R/exams tutorial on summative online exams in OpenOLAT
I prepared the online exam using the
exams2openolat() function. The R code I used along with a couple
of comments is provided below.
## load package library("exams") ## exercises for the exam ## (the actual exam had 16 exercises overall, not just 2) exm <- c("grundwasser-1-068.Rnw", "num-setzung-1-steinbrenner.Rnw") ## date used as seed and file name dat <- 20200528 set.seed(dat) rxm <- exams2openolat(exm, edir = "/path/to/exercise/folder/", n = 200, ## number of random versions for each exercise name = paste("BmGb1-Onlinetest", dat, sep = "_"), points = 1, ## all exercises yield 1 point cutvalue = 8, ## threshold for passing the exam ## evaluation rule for multiple-choice exercises (standalone and within cloze): ## every wrong answer cancels a correct answer mchoice = list(eval = exams_eval(partial = TRUE, negative = FALSE, rule = "true")), cloze = list(eval = exams_eval(partial = TRUE, negative = FALSE, rule = "true")), solutionswitch = FALSE, ## do not show full solution explanation maxattempts = 1, shufflesections = TRUE, ## sequence of exercises is shuffled randomly navigation = "linear", ## disable switching back and forth between exercises duration = 120, ## 2 hours stitle = "Exercise", ## section title (for group of exercises) ititle = "Question" ## item title (for individual exercises) )
Challenges and outlook
The main problem by producing randomized numeric examples was to find limits for the input so that the output is physically reasonable. Especially for newly created exercises, it is absolutely necessary that other colleagues act as test calculators, such that errors can be detected before students find them in their exams. The stress testing facilities in R/exams can also help.
Since the course will continue being online in 2020/21 due to ongoing COVID-19 restrictions, I use R/exams now also for the practice exercises which students have to calculate during the semester. Thus, I produced about 70 new exercises, which are offered in OpenOLAT in 12 weekly online tests. After COVID-19 I will use these examples as addition to written ones, where longer and more complex task could be trained.