4.4 Linear correlation and regression

Many questions in science, economics and everyday life are about how two quantities are related: whether heavier cars use more fuel, whether a house uses less gas in warmer weather, whether richer countries have longer life expectancies. Answering them needs data in which both quantities are recorded for each individual, and tools for measuring how closely the two are linked.

Linear correlation measures how close the points of such data lie to a straight line, and linear regression finds the equation of the best straight line through them. Together they turn a cloud of points into a linear model that can be used to make predictions, with a clear sense of how far those predictions can be trusted.

Bivariate data and scatter diagrams

Bivariate data

Most of the statistics in 4.2 and 4.3 is about a single variable, such as the heights of a group of students. When two variables are recorded for each member of a group, the data are bivariate. Each individual then gives an ordered pair $(x, y)$.

Examples include the weight and fuel consumption of each of a set of car models, the outside temperature and gas used by a house in each of several weeks, and the height of each father and his adult son. The pairing matters: the values in each pair belong to the same individual, so the two lists cannot be reordered separately.

Independent and dependent variables

In a bivariate study one variable is often used to explain or predict the other. The independent variable $x$ is the one that is controlled, or that is known first and used to make predictions. The dependent variable $y$ is the one that responds to it, or that is to be predicted.

When neither variable clearly depends on the other (for example, the lengths of a person's left and right feet), either can be placed on the horizontal axis. The choice then depends on which variable is to be predicted from the other.

Drawing a scatter diagram

A scatter diagram shows bivariate data as a set of points, one for each pair. The pattern of the points shows at a glance whether the variables are related and how.

To draw a scatter diagram by hand:

1. Put the independent variable on the horizontal axis and the dependent variable on the vertical axis.

2. Choose a scale for each axis that covers the range of the data and uses most of the grid. An axis does not need to start at $0$.

3. Label both axes with the variable and its units.

4. Plot each pair as a small cross or dot. Do not join the points and do not draw a line yet.

In the scatter diagram of 32 car models tested in the 1970s, each point is one car, with its weight across and the fuel it uses to travel 100 km up.

Describing correlation

Direction, form and strength

When the points on a scatter diagram follow a trend, the variables show correlation. A full description of the correlation covers three features.

• Direction: positive (the points rise from left to right), negative (they fall) or no correlation (no trend).

• Form: linear (the points cluster around a straight line) or non-linear (they follow a curve).

• Strength: strong if the points lie close to the line or curve, weak if they are widely scattered about it.

Six scatter diagrams show the most common patterns. In the last one the variables are closely related, but the relationship is a curve rather than a line, so there is no linear correlation.

Outliers and clusters

A point that lies well away from the overall pattern is an outlier. On a scatter diagram an outlier need not be extreme in either variable on its own: a car of average weight with very low fuel consumption would stand out only because of the combination of its values. Outliers should be checked, since they may be recording errors, but they should not be deleted just because they do not fit. Points can also form separate clusters, which often means the data contain two different groups (for example, cars with petrol engines and cars with diesel engines).

Pearson's product-moment correlation coefficient

What r measures

Words such as "strong" and "weak" are a matter of judgement. Pearson's product-moment correlation coefficient, written $r$, puts a number on the strength and direction of a linear correlation.

Its key properties are:

• $r = 1$ means every point lies exactly on a straight line with a positive gradient; $r = -1$ means every point lies on a line with a negative gradient. These are perfect correlations.

• The sign of $r$ gives the direction of the correlation, and the size $|r|$ gives its strength.

• $r$ close to $0$ means there is little or no linear correlation. It does not rule out a curved relationship.

• $r$ has no units. Changing units (centimetres to inches, or kilograms to pounds) does not change it, and neither does swapping the roles of $x$ and $y$.

• $r$ is sensitive to outliers. A single point far from the rest can make it much larger or much smaller.

The value of $r$ is found with a graphic display calculator (GDC). Its formula involves sums of products of deviations from the mean, which is where the name "product-moment" comes from, but it is never calculated by hand in this course.

Interpreting the value of r

Different books use slightly different boundaries, but the descriptions in the table are widely accepted and are a sound rule of thumb.

For the 32 cars, a GDC gives $r = 0.891$ (3 s.f.), confirming the strong positive linear correlation seen on the scatter diagram.

The scatter diagrams A to D show four correlations of different directions and strengths.

Finding r with a GDC

On any GDC the steps are the same in outline:

1. Enter the $x$-values in one list and the corresponding $y$-values in a second list, keeping each pair in the same row.

2. In the statistics menu, choose two-variable statistics or linear regression of the form $y = ax + b$.

3. Read off $r$ (and, for regression, $a$ and $b$). On some models $r$ is shown only after a "diagnostics" or "statistics display" setting has been switched on.

The coefficient of determination

Squaring $r$ gives the coefficient of determination $r^2$. It is not needed for every question, but it gives a useful interpretation of the strength of a correlation.

For the cars, $r^2 = 0.891^2 \approx 0.794$: about $79\%$ of the variation in fuel consumption between the 32 cars is accounted for by the differences in their weight. The remaining $21\%$ comes from other factors, such as engine size and design.

Correlation and causation

A strong correlation between two variables does not show causation, that is, that changing one variable produces a change in the other.

There are three common reasons why two variables can be correlated without one causing the other.

• A third variable. Both variables may depend on something else, sometimes called a lurking or confounding variable. Across a summer, ice-cream sales and the number of people needing rescue at beaches rise and fall together because both increase in hot weather.

• Reverse direction. The cause may run the other way from the one suggested. Cities with more police officers tend to have more crime, but mostly because cities with more crime employ more police.

• Coincidence. Any two quantities that both grow over the same period, such as a country's mobile-phone use and the average height of its children, will be correlated even if they have nothing to do with each other.

Causation can be established only by other means, usually a controlled experiment in which one variable is changed deliberately while everything else is kept the same.

Lines of best fit

When a scatter diagram shows linear correlation, a straight line drawn through the middle of the points summarises the trend. Such a line of best fit can be drawn by eye, or found exactly as the regression line.

The mean point

The means of the two variables give a single point at the centre of the data.

The mean point is the balance point of the data, and every good line of best fit passes through it. Plot it first, marked differently from the data points (for example with a circle), and draw the line through it.

Drawing a line of best fit by eye

1. Calculate the mean point $(\bar{x}, \bar{y})$ and plot it.

2. Place a ruler through the mean point and turn it until it follows the trend, with roughly equal numbers of points on each side along the whole line (not all of one side's points at one end).

3. Draw the line across the range of the data only.

4. To find its equation, read the coordinates of a second point on the line, far from the mean point, and use the methods of 2.1: the gradient from the two points, then $y - \bar{y} = m(x - \bar{x})$.

A line drawn by eye depends on the person drawing it, so two people will usually get slightly different equations. It is useful for a quick estimate, or when no GDC is available.

A house owner recorded the average outside temperature, $x$ °C, and the gas used to heat the house, $y$ thousand cubic feet, for ten weeks of one winter.

Data: D. Whiteside, UK Building Research Station, house in south-east England, 1960s (Hand et al., 1993).

The regression line of y on x

Residuals and least squares

To find a single best line that everyone agrees on, there needs to be a precise meaning of "best". For a line used to predict $y$ from $x$, the errors that matter are the vertical distances between the points and the line, because these are the errors in $y$.

For the ten weeks of gas data, the residuals are the vertical segments between the points and the regression line. Some are positive and some negative; for the best line they add up to $0$.

The least squares principle chooses the line that makes the sum of the squares of the residuals as small as possible. Squaring makes every term positive, so positive and negative errors cannot cancel, and it penalises large errors more heavily than small ones. The line found this way is the regression line of $y$ on $x$.

The values of $a$ and $b$ come from formulas involving the means and sums of products of the data, but in this course they are always found with a GDC, in the same menu as $r$. Different calculators may write the line as $y = ax + b$ or $y = a + bx$, so check which letter is the gradient.

Finding the regression line with a GDC

Interpreting a and b in context

In the regression line $y = ax + b$, both coefficients have a meaning in context.

• The gradient $a$ is the change in $y$ for each increase of one unit in $x$, on average. Its units are "units of $y$ per unit of $x$".

• The y-intercept $b$ is the predicted value of $y$ when $x = 0$. It has a sensible meaning only if $x = 0$ is a realistic value that lies inside, or very close to, the range of the data.

For the gas data, $a = -0.399$ means that for each extra 1 °C of average outside temperature, the house uses about $0.399$ thousand cubic feet less gas per week. The intercept $b = 6.95$ means that in a week with an average temperature of 0 °C the house would use about $6.95$ thousand cubic feet. Since 0 °C is inside the data range ($-0.8$ °C to $10.2$ °C), this is a meaningful prediction.

The regression line for the 32 cars, by contrast, has an intercept with no practical meaning.

The regression line passes through the mean point

Because the regression line always passes through $(\bar{x}, \bar{y})$, substituting $x = \bar{x}$ into its equation always gives $\bar{y}$. This allows questions without a GDC in which a mean or a missing data value is found from the equation of the line.

Making predictions

Interpolation and extrapolation

The main use of a regression line is prediction: substitute a value of $x$ to estimate $y$. How far a prediction can be trusted depends on where the value of $x$ lies.

Interpolation is usually reliable when the correlation is strong, because the data show that the linear pattern holds there. Extrapolation assumes the same straight line continues beyond the data, and there is no evidence for that. Many real relationships level off, curve or stop making sense outside the range observed, so extrapolated predictions can be badly wrong even when $r$ is very close to $\pm 1$.

Predicting x from y

The regression line of $y$ on $x$ is designed to predict $y$. It minimises the errors in $y$, not in $x$, so rearranging it to predict $x$ from a given $y$ does not in general give the best estimate. To predict $x$ from $y$, a different line is used, the regression line of $x$ on $y$, which is covered in 4.10. The two lines are the same only when $r = \pm 1$.

When linear regression is appropriate

A GDC will produce a value of $r$ and a regression line for any set of pairs, whether or not a straight line makes sense. Before using a regression line, check the scatter diagram and the value of $r$.

Plot the data before trusting r

Anscombe's quartet is a set of four small data sets constructed in 1973 to show why. All four have the same means, the same correlation coefficient ($r = 0.816$) and the same regression line, $y = 0.5x + 3$ (to 2 significant figures), yet they look completely different.

Curved relationships

Real data can show the same problem as set B. Across the world, life expectancy rises with national income, but not in a straight line: among the poorest countries a small increase in income goes with a large increase in life expectancy, while among rich countries extra income makes little difference.

For these 142 countries $r = 0.679$, a moderate positive correlation, and the regression line of life expectancy on GDP per person (in thousands of dollars) is $y = 0.637x + 59.6$. The value of $r$ alone would suggest a reasonable linear model, but the scatter diagram shows that the line overestimates life expectancy for the poorest and the richest countries and underestimates it for many countries in between. A prediction from this line would be systematically wrong, however carefully it was calculated.

A checklist before using the regression line

A regression line of $y$ on $x$ is appropriate for prediction when:

1. the scatter diagram shows a linear pattern, with no clear curve;

2. the correlation is strong (as a rough rule, $|r| \ge 0.75$; with a moderate correlation predictions are much less precise);

3. there are no outliers or isolated points controlling the line;

4. the prediction is of $y$ from a value of $x$ (otherwise use $x$ on $y$, in 4.10);

5. the value of $x$ is inside the range of the data (interpolation).

Exam-style questions on correlation and regression

Questions on this subtopic usually appear on the calculator paper, with data given in a table. Parts typically ask for $r$, a description of the correlation, the regression line, a prediction, and a comment on the reliability of a prediction or on causation. Questions without a GDC give the equation of the line, or $r$, and test interpretation and the mean point.

The table gives GDP per person, $x$ thousand US dollars, and life expectancy at birth, $y$ years, for ten European countries in 2007.

Data: Gapminder, 2007.

Self-check

• To do:

  I can draw a scatter diagram with the independent variable on the horizontal axis and describe the correlation by direction, form and strength, in context.

• To do:

  I can find $r$ with a GDC and describe its value in words, and I know that $r$ measures linear correlation only.

• To do:

  I can explain, with an example of a third variable, why correlation does not show causation.

• To do:

  I can calculate the mean point and draw a line of best fit by eye through it.

• To do:

  I can find the regression line of $y$ on $x$ with a GDC and interpret its gradient and $y$-intercept in context.

• To do:

  I can use the fact that the regression line passes through $(\bar{x}, \bar{y})$ to find a missing mean or data value.

• To do:

  I can make a prediction from the regression line and say whether it is interpolation or extrapolation and how reliable it is.

• To do:

  I can decide from the scatter diagram and $r$ whether a linear regression model is appropriate.

$14.7\ldots, \; 13.7\ldots, \; 11.6\ldots$ (A1)

rounding to the nearest integer gives a total of 41, so one group must be rounded down

Year 10: 15, Year 11: 14, Year 12: 11 A1

Note: Accept any choice of three integers totalling 40 with each within 1 of the exact values, provided the student explains the adjustment, e.g. 15, 13, 12. Award A0 for 15, 14, 12.

3. The line by eye, $y = -0.408x + 7.0$, has almost the same gradient and intercept. Both pass through the mean point: for the regression line,

The regression line is the one that would be produced by anyone with the same data, which is why it is preferred.

3. $1.5$ tonnes lies inside the data range, so substitute (using the unrounded values $a = 7.77721\ldots$, $b = 1.40013\ldots$ from the GDC):

A car weighing 1.5 tonnes is expected to use about $13.1$ litres per 100 km.

The regression line passes through $(\bar{x}, \bar{y})$, so

The five $y$-values must therefore add up to $5 \times 11 = 55$:

4. For each additional thousand dollars of GDP per person, life expectancy increases by about $0.196$ years (roughly 2.4 months) on average.

5. $x = 25$ is inside the data range ($5.9$ to $40.7$), so substitute the unrounded values:

6.

This is unreliable: $49.4$ is outside the range of the data ($5.9$ to $40.7$), so it is extrapolation. The correlation is also only moderate. (Norway's actual life expectancy in 2007 was 80.2 years, lower than the prediction: among the richest countries life expectancy levels off.)

7. The line is the regression line of $y$ on $x$, which is designed to predict $y$ from $x$. Estimating $x$ from $y$ needs the regression line of $x$ on $y$.

The café sells a mean of $84.4$ hot drinks per day.

2. For each 1 °C increase in maximum temperature, the café sells about $4.2$ fewer hot drinks on average.

3. $12$ °C lies inside the range 8 °C to 28 °C:

About $110$ hot drinks.

4. $40$ °C is outside the range of temperatures recorded, so the prediction is an extrapolation and there is no evidence that the linear pattern continues. In fact the line gives $-4.2 \times 40 + 160 = -8$, a negative number of drinks, which is impossible.

5. $r^2 = (-0.91)^2 = 0.8281 \approx 0.828$. About $83\%$ of the variation in the number of hot drinks sold is accounted for by the linear relationship with temperature.