4.1 Populations, samples and sampling

Populations and samples

The population

Every statistical investigation starts by deciding exactly which individuals or items it is about. This group is the population. A population need not be a group of people: it can be every light bulb made on a production line in a week, every day of the year in a city's weather record, or every eruption of a geyser.

A population must be defined precisely, because a vague population leads to a vague answer. "Students" is not a population; "all students enrolled at a particular school on 1 September 2026" is. The precise definition also decides who can be included in the data and who cannot.

Census or sample

A survey that collects data from every member of the population is a census. Many countries run a national census every ten years. A census gives complete information but is slow and expensive, and for many populations it is impossible.

A sample is taken instead of a census for several reasons:

• Cost and time. Questioning 1000 people is far cheaper and quicker than questioning 60 million.

• Destructive testing. Measuring how long a battery lasts, or the force a cable can hold before it breaks, destroys the item. Testing every item would leave nothing to sell.

• Very large or unlimited populations. All the fish in an ocean, or all the eruptions a geyser will ever produce, cannot be listed, let alone measured.

• Accuracy of handling. A smaller data set can be collected and checked more carefully, so it may contain fewer errors than a rushed census.

A sample is useful only when it represents the population: the features of the sample (its proportions, its average, its spread) should be close to those of the population. A value calculated from a sample, such as the mean height of 50 students, is then an estimate of the corresponding value for the whole population.

Random samples and the sampling frame

To choose a sample fairly, a list of every member of the population is needed. This list is the sampling frame: a school register, an electoral roll, the serial numbers of a batch of components.

A sample is random when every member of the population has a known, non-zero chance of being chosen, and the choice is made by a chance process rather than by a person's judgement. Random selection is the main defence against bias, because no one decides which individuals end up in the sample. If the sampling frame leaves out part of the population, however, no amount of random selection from it can put that part back.

Types of data

Qualitative and quantitative data

Data describe a variable, a characteristic that changes from one member of the population to another. Qualitative (categorical) data are descriptions such as eye colour, blood group or whether a mother smoked during pregnancy. Quantitative data are numbers that come from counting or measuring. Only quantitative data can be averaged and used in the calculations of 4.2 and 4.3, and they divide into two kinds.

Discrete and continuous data

Discrete data can take only separate values, usually whole numbers, because they come from counting. Continuous data come from measuring, and can take any value in an interval.

A useful test is to ask whether a value halfway between two possible values makes sense. A family can have 2 or 3 children but not 2.5, so the number of children is discrete. A baby's mass can be 3.2 kg, 3.25 kg or any value in between, so mass is continuous. Discrete data need not be whole numbers: shoe sizes (7, 7.5, 8, …) are discrete, because there are no sizes between the listed ones.

The table classifies variables recorded for 189 births at a hospital in Massachusetts.

Variables from the low birth weight study, Baystate Medical Center, Springfield, Massachusetts, 1986 (Hosmer and Lemeshow, Applied Logistic Regression).

Rounded continuous data

When a continuous value is recorded to a given accuracy, the recorded value stands for a whole interval of true values. A length recorded as 17 cm to the nearest centimetre lies in the interval $16.5 \le l < 17.5$. A time recorded as 12.4 s to the nearest tenth of a second lies in $12.35 \le t < 12.45$. These intervals become class boundaries when continuous data are grouped in 4.2.

Reliability of data

Judging a data source

Data are only as good as the way they were collected. Before using a data set, ask:

• Who collected the data, and why? A drug company's study of its own product, or a survey by a pressure group, may present results selectively.

• How were the data collected? Measured by instruments, recorded by observers, or reported by the people themselves? People often round their own heights upwards and their weights downwards.

• How large was the sample, and how was it chosen? A small or self-selected sample gives less reliable results.

• When were the data collected? Figures from twenty years ago may no longer describe the population.

• Are the definitions clear? "Unemployed" or "household income" can be defined in several ways.

Official statistics agencies, peer-reviewed studies and well-documented data sets that state their methods are generally more reliable than data with no stated source.

Bias

The most serious problem in data collection is bias: a fault in the method that pushes the results in one direction. Bias differs from the random variation between one sample and the next. Random variation can be reduced by taking a larger sample; bias cannot, because every extra individual is collected in the same faulty way.

Bias enters a study in several ways.

The 1936 Literary Digest poll of the United States presidential election is the standard warning. The magazine mailed about 10 million ballots, using telephone directories and car registration lists as its frame, and received about 2.4 million replies, which predicted a clear win for Alf Landon. Franklin Roosevelt won with about 61% of the vote. In the 1930s, owning a telephone or a car was linked to wealth and political views, and the people who chose to reply differed again from those who did not. The very large sample did not compensate for the biased frame and the non-response.

Missing data and errors in recording

Real data sets are rarely complete or perfectly accurate. A participant may leave a question blank, an instrument may fail, or a value may be typed wrongly. Each problem should be found and dealt with before any calculation.

Missing data (a blank entry, or a code such as "NA" or $-99$) should never be treated as zero, because a zero is a genuine value that changes the results. The usual choices are:

• leave the individual out of any calculation that uses the missing variable, and report how many were left out;

• if many values are missing, ask whether the people who did not answer differ from those who did, since this is a form of non-response bias.

Recording errors often show up as impossible or implausible values: a height of 1630 cm, a negative time, a percentage over 100, an age of 4 for a parent. Common causes are a misplaced decimal point, a value in the wrong units, two digits swapped or a value typed into the wrong column. If the true value can be recovered from the original record, correct it; if not, remove it. In every case, state what was changed and why, so that someone else could repeat the analysis.

Outliers and what to do with them

An outlier is a value that lies unusually far from the rest of the data. In 4.2 an outlier is defined precisely: a value more than $1.5 \times \mathrm{IQR}$ below the lower quartile or above the upper quartile. Finding an outlier is only the first step; the important question is what it means.

An outlier has one of two causes. Either it is an error (a mistake in measuring or recording, or an individual who does not belong to the population), or it is a genuine extreme value. The response to each is different, as the flowchart shows.

Genuine outliers often carry the most interesting information in a data set. In a list of the lengths of 141 major rivers in North America, the Mississippi–Missouri system at 3710 miles and the next longest at 2533 miles lie far above the rest, the middle half of which are between 310 and 688 miles long. They are real rivers measured correctly, so they stay in the data. Removing them would understate both the spread of the lengths and the length of the longest rivers on the continent.

Data: lengths of 141 major rivers in North America, The World Almanac and Book of Facts, 1975.

When a genuine outlier is kept, the choice of summary matters. The median and interquartile range barely change when an extreme value is added, while the mean, range and standard deviation can change a great deal (see 4.3). Reporting both kinds of measure, or stating the results with and without the outlier, shows the reader how much the conclusion depends on it.

Sampling techniques

A sampling technique is a rule for choosing which members of the population go into the sample. Random methods (simple random, systematic, stratified) give every member a known chance of selection, which protects against selection bias. Non-random methods (quota, convenience) leave the choice to the person collecting the data; they are cheaper and quicker but open to bias.

Simple random sampling

Simple random sampling is the basic random method: every member of the population has the same chance of being chosen, and so does every possible group of the chosen size.

To take a simple random sample of size $n$ from a population of size $N$:

1. Obtain a sampling frame and number its members from $1$ to $N$.

2. Generate random integers between $1$ and $N$, using a graphic display calculator (GDC), a computer or a random number table.

3. Select the members with those numbers, ignoring any number that repeats, until $n$ different members have been chosen.

Drawing names from a hat is the same process done by hand. Simple random sampling is free of selection bias and easy to describe, but it needs a complete sampling frame, and by chance it can give an unrepresentative sample, for example one with very few people from a small group.

Systematic sampling

Systematic sampling chooses members at a fixed interval from an ordered list. For a sample of size $n$ from a population of size $N$, the sampling interval is

A starting point is chosen at random from the first $k$ members, and then every $k$th member after it is selected.

Systematic sampling is quick, spreads the sample evenly through the list, and works well when the population passes by in order, such as items on a production line or customers leaving a shop. Its weakness is a hidden pattern in the list. If a production machine has 10 heads and produces a faulty item every 10th item, a systematic sample with $k = 10$ could select every faulty item or none of them. When $\frac{N}{n}$ is not a whole number, $k$ is usually rounded down, which makes the sample a little larger, or rounded to the nearest whole number with a check that the list does not run out.

Stratified sampling

Many populations contain groups that differ in ways that matter to the investigation: year groups in a school, age bands in a town, departments in a company. These groups are called strata (singular stratum). Stratified sampling makes sure each stratum appears in the sample in the same proportion as in the population.

For a population of size $N$ and a sample of size $n$, the number chosen from a stratum of size $N_i$ is

Every stratum is sampled with the same fraction $\frac{n}{N}$, so the sample is a scaled-down copy of the population. Within each stratum, members are chosen by simple random sampling.

Stratified sampling gives a more representative sample than simple random sampling when the strata really do differ, and it allows each stratum to be described separately. It needs a sampling frame for every stratum and information about which stratum each member belongs to. The strata must not overlap, and together they must cover the whole population.

Quota sampling

Quota sampling also divides the population into groups, but the individuals are not chosen at random. Each interviewer is given a quota, for example "interview 18 women aged 18 to 30 and 15 men aged 18 to 30", and fills it with whoever is available and willing.

Quota sampling needs no sampling frame, so it is fast and cheap, and it guarantees the right number from each group. Because interviewers choose whom to approach, they tend to pick people who look approachable and who are in busy places at convenient times, and this introduces bias. In the 1948 United States presidential election, the major polling organisations used quota sampling and predicted that Thomas Dewey would beat Harry Truman. Truman won by more than four percentage points, and the polls' failure led the industry to move towards random methods.

Convenience sampling

Convenience sampling uses whoever is easiest to reach: friends, classmates, the first 30 people through a door, or volunteers who reply to an advertisement.

Convenience sampling is the quickest and cheapest method, and it is useful for a pilot study that tests a questionnaire before the real survey. It is almost always biased, because the people who are easy to reach usually share characteristics, so its results cannot safely be generalised to the population.

Comparing the techniques

Choosing an effective method

The effectiveness of a sampling method depends on the population, the question being asked and the resources available. Stratified sampling is most effective when the population falls into groups that are likely to answer differently. Systematic sampling suits a population that arrives in order. Simple random sampling suits a population with a complete list and no important subgroups. Quota and convenience samples are acceptable only when speed matters more than accuracy, or for a first trial of a survey.

Exam-style questions on sampling

Sampling questions mix short definitions with a calculation (usually stratum sizes or a sampling interval) and a request to comment on a method. The comment earns credit only when it refers to the context of the question.

Self-check

• To do:

  I can explain the difference between a population, a census and a sample, and give reasons for sampling.

• To do:

  I can classify data as qualitative or quantitative, and quantitative data as discrete or continuous.

• To do:

  I can identify selection, non-response, self-selection, response and measurement bias in a description of a survey.

• To do:

  I can explain why a larger sample does not remove bias.

• To do:

  I can decide what to do with missing values, recording errors and outliers, and justify each decision.

• To do:

  I can describe how to take simple random, systematic, stratified, quota and convenience samples.

• To do:

  I can calculate stratum sizes in a stratified sample, adjusting for rounding so they add up to the sample size.

• To do:

  I can find a systematic sampling interval and list the members chosen from a random start.

• To do:

  I can evaluate how effective a sampling method is in a given context.

junior: $360 \times \frac{1}{24} = 15$ A1

adult: $36$ and senior: $9$ A1

Note: Award the second A1 only if both adult and senior values are correct.