4.13 Bayes’ theorem

Evidence is usually observed in the opposite direction to the way the probabilities are known. A medical test is described by how often it is positive for people who have a disease, but a patient wants to know how likely it is that they have the disease, given that their test was positive. A factory knows the fault rate of each machine, but an inspector who finds a faulty item wants to know which machine probably made it. Bayes' theorem converts one kind of conditional probability into the other.

The theorem also describes how a probability should be updated when new evidence arrives: a prior probability, held before the evidence, is combined with the probability of the evidence to give a posterior probability. This idea is central to modern statistics, spam filtering and machine learning.

From conditional probability to Bayes' theorem

The problem of reversing a conditional

In 4.11, conditional probability was defined as

and a reverse conditional was found from a tree diagram by dividing one path by the sum of the paths that end in the given event. Bayes' theorem is this calculation written as a single formula. Before deriving it, the denominator, the probability of the observed event, needs a rule of its own.

Partitions and the law of total probability

A tree diagram whose first stage has two or three branches splits the sample space into pieces that cannot overlap and that leave nothing out. Such a collection of events is called a partition.

When $B_1, B_2, B_3$ form a partition, any other event $A$ is split into the three pieces $A \cap B_1$

Each piece can be written with the multiplication rule, $P(A \cap B_i) = P(B_i)P(A \mid B_i)$

On a tree diagram, the law of total probability is the familiar rule of adding the probabilities of every path that ends in $A$.

The total probability is always between the smallest and largest of the conditional probabilities, here between $0.02$ and $0.05$, because it is their average weighted by $P(B_1)$, $P(B_2)$

Bayes' theorem

Deriving the theorem

The definition of conditional probability can be applied in either direction:

Substituting the second into the first gives

Finally, the law of total probability replaces $P(A)$ in the denominator. This expresses $P(B \mid A)$ entirely in terms of the probabilities on the branches of a tree drawn with $B$ first.

The structure of Bayes' theorem is easier to remember than the symbols: the numerator is the one path through the event wanted, and the denominator is the total of all the paths that end in the observed event $A$.

YouTube

Prior and posterior probabilities

In Bayes' theorem, $P(B)$ is the probability of $B$ before the evidence $A$ is observed, and $P(B \mid A)$ is its probability afterwards.

The prior probability is often called the base rate when it is the proportion of a population with some property. The posterior probability can be larger or smaller than the prior: evidence that is more likely under $B$ than under $B'$ raises the probability of $B$, and evidence that is less likely under $B$ lowers it.

Bayes' theorem with three events

With three possible causes, the denominator has three terms, one for each branch of the first stage. The posterior probabilities of the three causes always add up to $1$, which is a useful check.

Bayes problems can also be set without a context, in terms of bags or dice that select another trial. The method is the same: draw the tree, find the total probability of the observed result, and divide.

Finding an unknown prior

Bayes' theorem is an equation linking the prior, the two conditional probabilities and the posterior. If the posterior is given, the equation can be solved for an unknown prior.

Medical testing

Sensitivity, specificity and false results

A diagnostic test gives a positive or a negative result, and the person tested either has the condition or does not. There are four possible combinations, two correct and two incorrect.

A test's accuracy is described by two conditional probabilities, measured in trials where the true status of every person is known.

The sensitivity and specificity are the probabilities on the second-stage branches of a tree drawn with "condition" first. A false positive occurs with probability $1 - \text{specificity}$ for each healthy person tested, and a false negative with probability $1 - \text{sensitivity}$ for each person with the condition. The proportion of the population with the condition, the prevalence, is the prior probability.

The probability of a condition given a positive result

The question a patient asks is the reverse of the sensitivity: given a positive result, how likely is it that they have the condition? This is a Bayes' theorem calculation, and the answer depends strongly on the prevalence.

In November 2020 an evaluation of a rapid lateral flow test for COVID-19 reported a sensitivity of $76.8\%$ and a specificity of $99.68\%$.

Source: University of Oxford and Public Health England, evaluation of the Innova SARS-CoV-2 lateral flow test, 11 November 2020.

Even with a specificity above $99.6\%$, almost $30\%$ of positive results in this population are false positives. The reason is that $99\%$ of the people tested are not infected, so a small false positive rate applied to a very large group produces many false positives.

Natural frequencies

The same calculation is easier to understand with a large imaginary population. Out of $1\,000\,000$ people, $1\%$ (that is, $10\,000$) are infected. The test is positive for $76.8\%$ of these and for $0.32\%$ of the $990\,000$

The positive results come from two groups: $7680$ true positives and $3168$ false positives. The proportion of positives that are true is

exactly the result of Bayes' theorem. Natural frequencies show directly why the false positives matter: there are fewer of them than true positives, but not by much.

The base-rate effect

The probability $P(C \mid +)$ is not a fixed property of the test: it depends on the prevalence $p$. For this test,

The graph shows this as a function of the prevalence.

When only $0.1\%$ of those tested are infected, a positive result means infection with probability only about $0.194$: four out of five positives are false. This is the base-rate effect. A test used to screen a whole population, where the condition is rare, produces a much larger proportion of false positives than the same test used on people with symptoms, where the prevalence is higher. For this reason a positive screening result is usually followed by a second, more accurate test.

YouTube

Updating with a second test

A posterior probability can become the prior for the next piece of evidence. If a person who tested positive takes the test again, and the results are independent given the person's true status, Bayes' theorem is applied a second time with the new prior.

Exam-style questions on Bayes' theorem

Bayes' theorem questions almost always involve a tree diagram, either given or to be drawn. Most marks are for setting up the right products and sums, so write the formula with letters first and then substitute.

The second question uses a partition into three events and is set without a calculator, so the numbers are chosen to simplify.

A Bayes calculation can be set out in three ways that always agree: as a tree with "path over total", with the formula from the booklet, or with natural frequencies. Choose the one that makes the question clearest, and use another as a check.

Self-check

• To do:

  I can explain what a partition is and state the law of total probability for two or three events.

• To do:

  I can derive Bayes' theorem from the definition of conditional probability.

• To do:

  I can use Bayes' theorem for two events and for three events, and check that posterior probabilities add up to $1$.

• To do:

  I can set out a Bayes problem as a tree diagram, with the formula, and with natural frequencies.

• To do:

  I can define sensitivity, specificity, false positive and false negative.

• To do:

  I can calculate the probability of a condition given a positive or negative test, and explain the base-rate effect.

• To do:

  I can use a posterior probability as a new prior when further evidence is observed.

• To do:

  I can solve for an unknown prior probability when a posterior probability is given.