TLDR

We sometimes have the option to purchase additional information (like paying for an experiment to be performed, or for a report from an expert consultant) to help us in problems of decision making under uncertainty. Deciding whether or not such services are worthwhile is challenging – how do we know how to value some information before we get it? A powerful Bayesian procedure, which unfortunately goes by many names (such as preposterior decision analysis, multi-stage decision analysis, and Bayesian experimental design) allows us to quantify this expected value – in other words, it tells us how much we should be willing to pay for information, before receiving it. This result will be context-dependent, and informed by whatever our state of knowledge happens to be at the time. This post is a largely non-technical outline of the key concepts of Value of Information (VoI) analysis.

Decision Making under Uncertainty

Throughout industry, and even in our personal lives, we are required to make decisions without a perfect understanding of the problem. My PhD considers the problem of deciding whether or not risk mitigation measures were worth investing in, for structures in an uncertain state of damage. I quantified the expected value of inspection activities in this context, but this is quite a specific problem.

For the purposes of this post, I’ll use the example of a garage (auto repair shop if you’re in the USA) discussing some maintenance options for your car. I do not know enough about cars to talk about any specific problems, so imagine you have received the following generic statement from your mechanic:

“This particular model of car, of this age, often develops a critical problem. We recommend repairing / replacing the component that wears out, because if you don’t, it will be much more expensive to fix after a breakdown.”

Should you pay for the component to be replaced now? Or take your chances with a possible breakdown? Unsurprisingly, the expected optimal solution depends on the following:

• How likely do you think it is that your car has this critical problem?
• How much will the repairs cost: (a) now? and (b) after a breakdown?

This problem can be visualised using an influence diagram. The below uses the RHugin library, an R API for the Hugin software, although there are many suitable alternatives, such as the HydeNet library.

Elliptical nodes represent an uncertain outcome (the current condition of your car, and whether it will breakdown), rectangular nodes represent decisions (whether or not to pay for the repair) and diamond shaped nodes are the utilities (or costs) associated with each outcome. The arrows indicate dependency, for instance, the probability of a breakdown is dependent on whether or not the repair is performed (and the unknown condition of the car).

If the above diagram looks confusing, it could alternatively be represented as a decision tree. However, for larger (more complex) decision making problems, I’ve found influence diagrams to be much more interpretable. This is because where each possible action or outcome would need it’s own ‘branch’ in a decision tree, they can all be represented within a table that is hidden behind every node in the influence diagram.

If we had a perfect understanding of the condition of our car, then we would know if it was going to breakdown and deciding whether this repair was worth doing would be easy. But we’re interested in solving more challenging problems, where it is sometimes a case of…

Decision making under uncertainty.

We may be very unsure of the condition of the car. This implies that we consider there to be a relatively high probability that it will breakdown (because we do not know enough to be confident that everything is OK). Even so, we need some sort of quantitative approach to allow us to make coherent decisions. The uncertainty in the problem can be described using probability, and we can also put numbers to the consequences.

Firstly, after some online research (statements from the manufacturer, reviewers websites and forums), we’ve seen evidence that the problem that the mechanic described is real, and that 30% of cars like yours are affected:

(Note that we assuming if we do pay for a repair, then it will be perfect – in other words, once the garage repairs the problem component, there is no way the car will breakdown due to this problem. We could change this probability if we wanted to account for the reliability of the job.)

Secondly, after your best efforts to present yourself as a car aficionado, who can’t be fooled into paying an unfair price, the garage provides you with the following quotes:

Bartering with mechanics.

By the way, the above values have been entirely made up (there is nothing special about them and the validity of this calculation does not depend on them).

Here’s how you can populate the table of a node in a Hugin influence diagram from within R:

repair_cost <- -600

repair_cost_table <- get.table(domain = prior_analysis, node = 'Cost_Repair')
repair_cost_table['Utility'] <- c(0, repair_cost)

set.table(domain = prior_analysis, node = 'Cost_Repair', data = repair_cost_table)

We can then do the same to populate the other nodes (not shown here as it is more of the above), before asking RHugin to evaluate the influence diagram.

RHugin::compile(object = prior_analysis)

RHugin::propagate(object = prior_analysis)
expected_costs <- get.utility(domain = prior_analysis, node = 'Repair')

And our expected costs are shown for each of our decision options:

This shows that we can expect to pay less if we get the repair done now, so it’s reasonable to go with that option on the basis of this.

The Value of Information

Now that we’ve set the scene (and after that brief digression), remember that the problem we are interested in, is:

How (and to what extent) can some additional information assist in the decision making process?

Let’s extend the influence diagram, showing the option we have to perform a diagnostic test to improve our understanding of the condition of the car, before we decide whether to pay for the repair or not.

For the time being, I’ve not included a cost for the test, but I’ll come back to this point shortly. Now to compile the model…

RHugin::compile(object = prepost_analysis)

RHugin::propagate(object = prepost_analysis)
expected_costs <- get.utility(domain = prepost_analysis, node = 'Test')

And our expected costs are shown for each of our decision options:

This tells us that our expected cost is reduced when we have more information (from the test) to support our decision of whether or not to pay for the repair. Uncertainty makes decision making more difficult, and so improving our understanding of the problem can never hurt.

Specifically, the expected value of performing this test is equal to the difference between the expected costs, with ($180.00)

…and without (-$600.00)…

this information. This difference (our expected value of information) is equal to $420.00.

As I mentioned earlier, I have not included a cost of the test in the second influence diagram. This means we can interpret this value as the maximum amount we would be willing to pay for this test. If the garage offered the test for a fee less than $420.00, then we would be minimising expected costs by paying for the test. Otherwise, we would not expect the test to be worthwhile, since we’d be better off just paying for the repair.

Finally, what we have really calculated here, is the expected Value of Perfect Information. This is because we are assuming this test is completely accurate, i.e. if the test indicates the car is damaged then we know for sure that it is. No test is ever perfect, and that means the result won’t be as informative as we have assumed it to be here. The value of perfect information is therefore really just an upper bound. Quickly identifying an upper bound can be very useful, but for cases where we need a better estimate, we need to think about the imperfect features of the information we are considering buying.

In Practice: The Value of Imperfect Information

Imagine we now know a little bit more about the diagnostic test that the garage is offering. We know that if it reports damage, there is a 10% chance that this is a mistake (sometimes referred to as a false-positive error). Conversely, we have also learnt that this test misses damage (gives a false-negative error) 5% of the time. Since this will be less useful than the perfect test that we considered above, it can’t be any more valuable than that.

Based on these new expected costs, the expected value of this imperfect test is $342.11. We can use this calculation to perform a sensitivity analysis, showing how (for example) the false negative error rate influences the expected value of the test:

As expected, tests that are more likely to miss the damage are not worth as much to us. The discrepancy between the value of the perfect and imperfect tests without a false-negative error (\(\Pr(Pass\;Test \mid Car\;damaged) = 0\)), is due to the false-positive error of 10%, which I left unchanged.

More Realistic Models (Working on Continuous Scales)

By discretising uncertain outcomes, we simplify the the calculations that we need to do to identify our expected optimal decisions. However, this may be an inaccurate representation of the true problem. Predicting whether or not a car will breakdown based on the condition of a component may itself require a reliability assessment (probabilistic calculation). Similarly, it may not be appropriate to represent features of the diagnostic test on a discrete scale either. For instance, the probability of missing damage may be a function of the extent of the damage, i.e. it is generally less likely that greater deterioration is missed. The measurements may also need to be a continuous scale, which should account for precision and possible bias.

Summary of Key Principles & Final Thoughts

Here are the most important ideas to remember from the above:

The act of collecting information (such as the the diagnostic test on the car) does not itself save you money or reduce risk. What it does do, is allow us to re-evaluate the options available to us with a better understanding of the problem. Therefore…

to quantify the value of information, you need to relate it to the improved decision making that it facilitates.

This is exactly what we have done in this example. We have compared our expected costs with and without the diagnostic test, allowing us to quantify the expected benefit of performing the test, in terms of money. This benefit arises from the simplification of the decision problem of whether to pay for the repair.

One example of aimlessly collecting information was shown in the episode of The Simpsons, Homer’s Enemy. Bart buys an old factory at an auction. It’s in terrible condition, so he hires Milhouse as a night watchman, so that they can keep from collapsing and they can continue to play in it. As shown below, Milhouse carries out his watchman duties….

Monitoring, then not acting - the Milhouse approach.

This is the equivalent of paying for information just for the sake of it. Milhouse’s observations would only be of use, if they had a plan for what to do when it ‘started falling over’. An evaluation of inspection, monitoring, or testing options should be done in the context of the underlying decision problem that you are trying to solve. I don’t think this is a straightforward idea - but hopefully this discussion has demonstrated the concept.

Secondly:

The value of information is context dependent.

Sometimes, more expensive, higher quality information is worth paying for. Sometimes it isn’t. When we already have a good idea of what we will find, or if it doesn’t really matter what we find - then we may get little to no value from additional information. When there is a lot of uncertainty in a problem and it pays to know exactly what’s going on, it may be worth spending big for high quality data.

I’d suggest that it’s only really at these extremes where we would intuitively know our best course of action.

Uncertainty on top of uncertainty on top of…

The value of the diagnostic test on the car is dependent on the initial (prior) probability that the car was damaged. So the expected value of some data is based on what we think it will say? You may be understandably sceptical of what might seem fragile foundations of this method, but allow me to convince you otherwise:

Sceptical?

Remember - we are using this method to tackle problems of decision making under uncertainty and we are representing that uncertainty with probability. We are not doing this to complicate things. Uncertainty exists, whether we choose to account for it or not. We are can identify the expected best course of action, on the basis of our current partial understanding of the problem. If we knew more, we could do better, but we need to work with what we’ve got.

Also, when an organisation or an individual makes a difficult decision without thinking about it formally like we have here, then this entire process is just implicit. They are basing their decision on something, but we don’t know what exactly. This makes it very difficult to challenge (or audit) any particular aspect of their decision making process. For problems that do not have intuitive answers, relying on a gut feeling will not always be helpful.

If we see the result of a value of information calculation and it doesn’t feel right, this could mean a few things:

• The problem may not be intuitive. Our feeling for what the result should be is actually overlooking some complexity. If we suspect this is the case, then maybe it is better to trust the result. Since it can easily be documented, then each element can be critiqued and re-visited whenever we like.
• We’ve not included enough expert judgement. Fortunately, this can easily be fixed as Bayesian methods are happy to accommodate prior information. In my post on Bayesian logistic regression I showed how prior predictive simulation can help identify sensible priors, even in more complicated statistical models. A great feature of this, is that the expert knowledge that goes in, is part of the model. This is much smarter than throwing out a calculation that an expert doesn’t agree with, or discussing experience, judgement, or expertise as separate, unrelated topics.