# Occupancy model fit & AUC

July 30, 2013
By

(This article was first published on Ecology in silico, and kindly contributed to R-bloggers)

Occupancy models are used to understand species distributions while
accounting for imperfect detection. In this post, I’ll demonstrate a
method to evaluate the performance of occupancy models based on the
area under a receiver operating characteristic curve (AUC), as published
last year by Elise Zipkin and colleagues in
Ecological Applications.

Suppose we are to fit a multi-year occupancy model for one species. We
will evaluate the fit based on how well the model predicts occupancy in
the final year of the project. Start by simulating some data
(for details on the structure of these simulated data, refer to
this post and
references therein):

For illustration, I included a strong interaction between treatment
and the continuous site level covariate (could be elevation, area, etc). As such, a
measure of model fit such as AUC ought to identify a saturated model as
the best fitting. Handily, AUC is a derived parameter, and common
occupancy model parameters can be used to estimate a posterior.
To generate a posterior AUC, we need predicted occupancy probabilities
($\psi$) and realized occupancy states ($Z$) in the final year. Predicted occupancy
probabilities can be produced using data from previous years, and
realized occupancy states are assumed to be represented by the posterior
for $Z$ generated from a single-year model, fit to the data from the final
year of the study.

### Fitting a saturated model

Begin by modeling occupancy probabilities as a function of both covariates
and their interaction, predicting $\psi$ in the final year:

Now that we have our posteriors for $\psi$ at each site in the final
year, we can fit a single-year model to the final year’s data to
estimate $Z$.

To set up the data for AUC calculations, produce site by
iteration arrays for $\psi$ and $Z$:

Now generate the posterior for AUC and store data on the true and false
positive rates to produce ROC curves.

### Fitting a simpler model

Having fitted a saturated model, we can now fit a simpler model that
includes only main effects:

### Comparing models

How well did our models predict occupancy in the final year of the study,
and was one better than the other?
We can answer this question by inspecting posteriors for AUC (larger
values are better), and the ROC curves.

As expected, the model that generated the data fits better than the
model that excludes the strong interaction term. Note that AUC reflects
the accuracy of model predictions, and does not penalize model
complexity.

R-bloggers.com offers daily e-mail updates about R news and tutorials on topics such as: Data science, Big Data, R jobs, visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...