R FOR HYDROLOGISTS

CORRELATION AND INFORMATION THEORY MEASUREMENTS (Part 2)

Proposed back in the 40’s by Shannon Information theory provide a framework for the analysis of randomness in time-series, and information gain when comparing statistical models of inference. Information theory is based on probability theory and statistics. It often concerns itself with measures of information of the distributions associated with random variables. Important quantities of information are Entropy, a measure of information in a single random variable, Mutual information, a measure of information in common between two random variables, and Relative entropy that measure how one probability distribution diverges from a second expected probability distribution.

In this tutorial we will estimate these measurements in order to characterize the river dynamic. If you don’t have the data please first see the first part of the tutorial here and Install and load ggplot2 and reshape2 packages
if(!require(ggplot2)){install.packages(ggplot2, dep=T)} if(!require(reshape2)){install.packages(reshape2, dep=T)}
Answers to the exercises are available here.

All information measurements are derivate from the join and marginal distributions of two variables. To estimate this empiric distribution we will use histograms; in this opportunity we will geom_bin2d. Let’s do it step by step.

Exercise 1
First please create a geom_point plot of the LEVEL against the RAIN

Exercise 2
Now please overlap a 2D histogram with the function geom_bin2d()

Exercise 3
We have to get the joint probability matrix. So please set the number of bins =10 and plot the joint probability distribution of the LEVEL and the RAIN then assign it to an object p.

Exercise 4
Extract from the object p the data of the first layer with the function layer_data and assign it to pxy_m.

Exercise 5
As you can see ggplot return a column based data frame with the x, y and the value of the density index as columns. Please convert it to a rectangular matrix with the function acast and sign it to pxy

Exercise 6
Please guarantee the natural restriction the probability distribution sum(pxy)==1

Exercise 7
Estimate the marginal probabilities px and py.

Exercise 8
Great now we have everything we need. Please estimate the entropy in bits (log2) for each variable Hx and Hy.

Exercise 9

Estimate the Joint entropy in bits (log2) with the formula: Hxy=-sum(pxy*log2(pxy)). Remember that in order to avoid numerical error you have to use just positives probabilities pxy>0 before applying the formula

Exercise 10
Last step, please calculate the mutual information Hint: MI=Hx+Hy-Hxy