Judge how far research findings can be trusted and applied elsewhere, using reliability, validity, bias, credibility, reflexivity, generalisability and transferability, and evaluate studies such as Rosenhan’s with them.
Each is in the Psychology glossary, with every other term in the subject.
The tendency of some participants to agree with statements in a questionnaire or interview regardless of their content; also called yea-saying.
A detailed record of every step and decision in a qualitative study, from data collection to analysis, that allows another researcher to check how the findings were reached.
The extent to which qualitative findings give a true and believable picture of the participants' experiences and perceptions; a part of trustworthiness.
Cues in a study that suggest to participants what the researcher expects, which can lead them to change their behaviour to fit their interpretation of the study's purpose.
The extent to which the findings of a study can be applied to behaviour in real-life settings, which depends on how natural the setting, task and materials were.
Reflection on how the researcher's choices of research question, method and analysis may have shaped the findings and what other choices could have shown.
A form of researcher bias in which the researcher's expectations about the results unintentionally influence participants' behaviour or the way data are recorded.
Explain how reliability and validity (internal, external, ecological, content, face and construct) apply to judging a study.
Describe participant biases (demand characteristics, social desirability, the Hawthorne effect, acquiescence) and researcher biases (expectancy, confirmation, interviewer effects), and how each can be reduced.
Explain credibility, transferability, dependability and confirmability as criteria for the trustworthiness of qualitative research.
Evaluate strategies for ensuring credibility, including triangulation, member checking, thick description, audit trails and reflexive journals.
Explain the role of reflexivity, both personal and epistemological, in checking for unconscious bias.
Discuss the factors to consider when generalising quantitative findings or transferring qualitative findings to other populations and contexts.
Explain how ethical considerations shape research designs and can affect the results they produce.
Evaluate a published study, such as Rosenhan’s study of psychiatric diagnosis, using research considerations.
Research considerations are the questions used to judge how far a study’s findings can be trusted and how far they apply beyond the people studied. Two studies can use the same method and reach opposite conclusions; research considerations explain why one result deserves more confidence than the other.
Quantitative research is judged mainly by reliability, validity and generalisability; qualitative research by credibility, transferability and the researcher’s reflexivity. Both kinds of research are vulnerable to
With a free account, Augment remembers where you stopped and what you have practiced.
A change in participants' behaviour, often an improvement in performance, that occurs because they know they are being observed or studied rather than because of the variable being tested.
A credibility strategy in qualitative research in which participants are asked to check the researcher's transcripts or interpretations to confirm that they represent their views accurately; also called respondent validation.
A distortion of findings caused by participants changing their behaviour or answers because they know they are being studied, for example to help the researcher or to look good.
Reflection on how a researcher's own values, experiences, culture, gender and relationship with participants may have shaped the research.
A researcher's ongoing, critical awareness of how their own background, beliefs and choices of method may have influenced the research process and its findings.
Repeating a study, using the same or a very similar procedure with new participants, to check whether the original findings occur again.
A distortion of findings caused by the researcher's expectations, beliefs or behaviour, which may affect how a study is designed, how participants are treated, or how data are recorded and interpreted.
The tendency of participants to give answers or behave in ways that they think will be seen favourably by others rather than in ways that reflect what they really think or do.
A detailed account of a study's participants, setting, context and procedure, which allows readers to judge whether qualitative findings can be transferred to other contexts.
The extent to which the findings of a qualitative study can be applied to other people, settings or contexts, judged by readers from a detailed description of the original study.
The overall quality of qualitative research, judged by its credibility, transferability, dependability (consistency of the research process) and confirmability (findings shaped by the data rather than the researcher).
Reliability and validity are the two basic qualities of any measure. A measure is reliable if it gives consistent results: the same score when nothing has changed, and the same score whoever applies it. It is valid if it measures what it claims to measure. The two are linked but not the same, and a study can fail on either.
Researchers check three kinds of consistency. Test–retest reliability asks whether the same people get similar scores when measured twice, a few weeks apart; it suits stable traits such as personality, not states such as mood. Inter-rater reliability asks whether two observers or coders give the same ratings to the same behaviour; it is essential for observations and for coding interviews. Internal consistency asks whether the items of a scale that are meant to measure the same construct agree with each other; a self-esteem questionnaire whose items do not correlate is probably measuring several different things.
Reliability is improved by standardisation (the same procedure and instructions for everyone), clear operationalisation of variables, training observers and coders, and testing a measure in a pilot study before the real study.
Validity is a family of questions rather than one. Some concern the measure (does this questionnaire really measure anxiety?); others concern the study (did the IV really cause the change?) or its reach (does the result apply elsewhere?).
Kind of validity | The question it asks | Example of a problem |
|---|---|---|
Face validity | Does the measure look, on the surface, as if it measures the construct? | A “stress scale” made of questions about favourite foods |
Content validity | Does the measure cover all aspects of the construct? | A depression scale that asks about mood but not sleep, appetite or energy |
Construct validity | Does the measure really capture the theoretical construct, behaving as theory predicts? | An “intelligence” test that mainly measures familiarity with Western schooling |
Internal validity | Did the IV, and nothing else, cause the change in the DV? | A confounding variable such as time of day differing between conditions |
External validity | Do the findings apply to other people, places and times? | Results from 19-year-old students applied to older adults |
Ecological validity | Do the findings apply to real-life settings? | Memory for word lists in a laboratory applied to memory in daily life |
Construct validity is the deepest of the three measurement questions, because psychological constructs cannot be observed directly. Face validity is the weakest: a measure can look right and still be wrong, and some measures deliberately have low face validity so that participants cannot guess what is being measured.
Ecological validity is part of external validity and is closely related to mundane realism, the extent to which a study’s setting and task resemble everyday life. A study can have high mundane realism and still low ecological validity if, for example, participants know they are being studied and behave differently. Laboratory experiments usually have high internal validity but may have low ecological validity; field and natural experiments reverse the balance.
YouTubeIn research, bias is any systematic influence that pushes the results away from the truth. Unlike random error, which averages out over many participants, bias pushes in one direction and does not disappear with a larger sample. It can come from the people being studied or from the people doing the studying.
Orne (1962), who introduced the term, showed how far participants will go to cooperate with what they believe a study requires. In one demonstration, volunteers were given sheets of random numbers to add up and told to tear each completed sheet into 32 pieces before starting the next one. Many continued for hours. In the setting of an experiment, a pointless task appeared to have a purpose, and participants did their best to be “good participants”.
Demand characteristics are especially likely in repeated measures designs, where seeing both conditions makes the aim easier to guess, and in studies where the expected answer is obvious. In Tajfel’s minimal group studies, for example, critics have argued that boys asked to divide points between their own group and another may have assumed that favouring their group was what the experimenter wanted, because group membership was the only information they had.
Archive research by Le Texier (2019) found that guards in the Stanford prison experiment were briefed and encouraged to be tough, which suggests they were acting out what they took to be the study’s aim rather than responding spontaneously to their role.
People under-report behaviours that are disapproved of (drinking, prejudice, cheating) and over-report approved ones (exercise, helping, voting). The bias is strongest in face-to-face interviews and on sensitive topics. It is reduced by anonymity, by assuring participants that there are no right or wrong answers, by indirect questions, and by behavioural measures that do not depend on self-report.
Acquiescence inflates scores on scales where every item is worded in the same direction. Reverse-worded items detect it: a participant who agrees both that “I enjoy meeting new people” and that “I avoid meeting new people” is not reading the items carefully. Response styles also vary across cultures, which complicates cross-cultural comparisons of questionnaire scores.
The effect is named after studies at the Western Electric Hawthorne Works near Chicago between 1924 and 1932, in which workers’ productivity was reported to rise whether lighting was increased or decreased, apparently because the workers were being studied. When Levitt and List (2011) found the original data and reanalysed them, they found only slight evidence of such an effect, much weaker than the textbook story. The name is therefore attached to a weak original finding, although reactivity to being observed is well established in many other settings.
Doyen, Klein, Pichon and Cleeremans (2012) tested a famous claim that reading words linked to old age makes people walk more slowly. With 120 Belgian students and walking speed measured by infrared sensors, they found no effect. In a second experiment, they told some experimenters to expect slower walking: primed participants then walked more slowly, but only when the experimenter expected them to, and experimenters timing by stopwatch recorded the speeds they expected. The original effect may have depended partly on what the researchers believed.
Expectancy effects are controlled by double-blind procedures, in which the person who runs the session and records the data does not know the hypothesis or the participant’s condition, and by automated measures that leave no room for judgement.
Confirmation bias in research is the tendency to design studies, select data and interpret results in ways that support the researcher’s existing hypothesis: stopping data collection when the result looks good, reporting the measures that “worked”, or reading ambiguous interview statements as support. Interviewer effects arise when the interviewer’s age, sex, ethnicity, tone or reactions shape participants’ answers. Pre-registering hypotheses and analyses before collecting data, having several researchers code the data independently, and reflexivity all reduce these biases.
Bias | Source | Typical effect | Main control |
|---|---|---|---|
Demand characteristics | Participants guess the aim | Behaviour fits the guessed hypothesis | Single-blind procedure, deception with debriefing, independent samples |
Social desirability | Participants want to look good | Under-reporting of disapproved behaviour | Anonymity, indirect or behavioural measures |
Acquiescence | Tendency to agree | Inflated scores on scales | Reverse-worded items |
Hawthorne effect | Being observed | Performance changes when watched | Habituation, covert observation, control groups |
Expectancy effect | Researcher’s expectations | Participants or records shift towards the hypothesis | Double-blind procedures, automated measures |
Confirmation bias | Researcher’s beliefs | Selective analysis and interpretation | Pre-registration, several coders, reflexivity |
Interviewer effects | Interviewer’s characteristics and behaviour | Answers adjusted to the interviewer | Training, standard prompts, reflexivity |
Qualitative research cannot be judged by quite the same standards as experiments. An interview study of eight people cannot be statistically representative, and a researcher interpreting transcripts cannot be completely objective. Lincoln and Guba (1985) proposed a parallel set of criteria, which together describe the trustworthiness of qualitative research.
Dependability asks whether the research process was consistent and could be followed by another researcher; it is supported by an audit trail. Confirmability asks whether the findings come from the data rather than from the researcher’s preferences; it is supported by reflexivity and by showing the data (quotations) behind each conclusion.
Credibility is the qualitative counterpart of internal validity. It asks whether the researcher has understood and represented participants accurately, rather than imposing their own interpretation. It is threatened by brief contact with participants, by a single source of data, and by a researcher who sees what they expect to see. Several strategies increase it, and strong studies use more than one.
Triangulation means approaching the same question from more than one direction and checking whether the answers agree. Method triangulation combines different methods, such as interviews and observation. Data triangulation uses data from different sources, times or places. Researcher triangulation has more than one researcher collect or analyse the data. Theory triangulation interprets the data through more than one theory. Agreement increases confidence; disagreement shows where the picture is more complicated than one source suggested.
After an interview study, the researcher may return a summary of themes to participants and ask whether it reflects their experience. Member checking corrects misunderstandings and gives participants a voice in how they are represented, which is especially important when researchers study communities they do not belong to. Its limitation is that participants may agree out of politeness, or may reject an accurate interpretation they find uncomfortable.
Prolonged engagement means spending enough time with participants and in the setting to build trust, so that people move beyond polite or socially desirable answers. A reflexive journal records the researcher’s reactions, assumptions and decisions throughout the study, which supports both confirmability and reflexivity. Detailed, rich reporting of the context and of participants’ own words, thick description, lets readers see the evidence for each interpretation.
All research is carried out by people with particular backgrounds, and in qualitative research the researcher is the main instrument of data collection. Because the researcher’s influence cannot be removed, reflexivity aims to make it visible so that readers can take it into account. It is the main way of checking for unconscious bias.
A researcher who has experienced depression, interviewing others about depression, may build rapport more easily but may also expect their own experience to be shared. A researcher from outside a community may miss meanings that an insider would see. Personal reflexivity asks the researcher to state these positions and consider their effects.
A question framed as “Why do refugees fail to integrate?” already assumes that integration is the goal and that failure lies with refugees. Choosing interviews rather than a questionnaire, or thematic analysis rather than counting words, also shapes what can be found. Epistemological reflexivity asks the researcher to consider how the design limits and directs the knowledge it produces.
Generalisability depends on several separate factors, and a study can be strong on one and weak on another.
Population. Was the sample representative of the people the conclusion is about? Asch’s conformity studies used 123 male students from three American universities in the 1950s; whether women, older adults or people from collectivist cultures behave the same way cannot be assumed from that sample.
Setting. Do the findings apply outside the setting of the study? This is ecological validity.
Time. Do the findings still apply? Norms, technology and institutions change; research on conformity or on psychiatric hospitals from the 1950s–1970s may describe a different social world.
Culture. Do the findings apply across cultures? Segall and colleagues’ cross-cultural studies of the Müller-Lyer illusion found that susceptibility to it varied across societies, so even a basic perceptual effect measured in Western samples did not generalise to all humans.
Generalisability is strengthened by representative sampling, by field settings, and above all by replication in different populations and places.
Replication is the most direct test of whether a finding is reliable and generalisable. When the Open Science Collaboration (2015) repeated 100 studies published in leading psychology journals, about 97% of the originals had reported significant results, but only about 36% of the replications did, and the effects found were on average about half the size of the originals. This replication crisis led to reforms such as pre-registration and the sharing of data and materials, which make research more transparent and easier to check.
YouTubeA qualitative study of how eight nurses in one hospital cope with night shifts does not claim to represent all nurses. Instead, it describes the participants, the hospital and the context in detail (thick description), so that a reader working in another hospital can judge how similar their own situation is and therefore how far the findings might apply. When findings are generalised, the researcher claims that they apply to a population. When findings are transferred, readers judge the fit to their own setting from the information the researcher provides.
Factors to consider when transferring findings include how similar the new context is in culture, setting and the characteristics of the people involved; whether the original study described its context fully enough to judge; and whether the original findings were credible in the first place.
The ethical principles that govern research, such as informed consent, protection from harm, the right to withdraw and confidentiality, are a matter of responsibility. They also shape research designs and can affect the results those designs produce.
Consent and demand characteristics. Telling participants the full aim may change their behaviour. Researchers sometimes use partial deception, such as a cover story, followed by full debriefing, which an ethics committee must judge to be justified by the study’s value and low risk.
Consent and sampling. Only people who agree take part, so every sample is partly self-selected; those who refuse may differ in important ways.
The right to withdraw and attrition. If the participants who find a study most uncomfortable leave, the remaining data under-represent the effect being studied.
Anonymity and honesty. Guaranteeing anonymity can increase the validity of self-report on sensitive topics by reducing social desirability bias.
Protection from harm and what can be studied. Harmful variables cannot be manipulated, so research on trauma or abuse uses correlational, retrospective and natural-experiment designs, with weaker causal conclusions.
Covert research. Covert observation avoids reactivity but removes consent; it is acceptable only in limited circumstances, such as public settings.
Responsibility also extends beyond the study. Research on socially sensitive topics, such as mental illness or group differences, can increase stigma if it is reported carelessly, and researchers have a duty to communicate uncertainty and to avoid claims their designs cannot support.
Rosenhan’s 1973 study is one of the most widely cited studies in psychology and one of the most contested. It makes claims about the reliability and validity of psychiatric diagnosis, and investigations published since 2019 have raised doubts about its own credibility. It therefore shows research considerations being applied both by the study and to it.
To test whether psychiatric staff could distinguish people who were mentally healthy from people with mental illness, and to describe how patients were treated once they had been given a diagnosis.
Eight “pseudopatients” (Rosenhan and seven associates) sought admission to 12 psychiatric hospitals in five US states. Each reported hearing an unfamiliar voice saying “empty”, “hollow” and “thud”, and otherwise described their real life histories truthfully. Once admitted, they stopped reporting any symptoms and behaved normally. The study was a covert participant observation: the pseudopatients recorded their experiences and the behaviour of staff, while staff did not know they were being studied.
In a follow-up, staff at a research and teaching hospital who had heard of the study were told that one or more pseudopatients would try to gain admission over the next three months. Staff rated each new patient on the likelihood of being a pseudopatient. In fact, Rosenhan sent no one.
All eight pseudopatients were admitted. Seven were diagnosed with schizophrenia and one with manic-depressive psychosis, and all but one were later discharged with a diagnosis of schizophrenia “in remission”. Stays ranged from 7 to 52 days, with a mean of 19. No staff member was reported to have detected a pseudopatient, but in the first three hospitalisations 35 of 118 fellow patients expressed suspicion. Contact with psychiatrists averaged 6.8 minutes a day, and normal behaviour was interpreted through the diagnosis: one nurse recorded a pseudopatient’s note-taking as “writing behavior”, as if it were a symptom.
In the follow-up, 193 new patients were rated. Forty-one were judged to be impostors and 42 to be suspect, although none had been sent.
Rosenhan concluded that psychiatric diagnosis was unreliable and that the hospital setting and the diagnostic label shaped how patients’ behaviour was interpreted.
It is clear that we cannot distinguish the sane from the insane in psychiatric hospitals.
— Rosenhan (1973), Science, 179, as quoted in Wikipedia, “Rosenhan experiment”
The study contributed to criticism of psychiatric diagnosis in the 1970s and is often linked to the move towards more explicit, operationalised diagnostic criteria.
The journalist Susannah Cahalan, in The Great Pretender (2019), tried to trace the original records. She could identify only two of the eight pseudopatients, Rosenhan himself and a graduate student whose account of his stay was inconsistent with Rosenhan’s description, and she found apparent inconsistencies in the data and inaccurate or fabricated quotations. The historian Andrew Scull, writing in History of Psychiatry in 2023, supported her allegations and described the study as a “successful scientific fraud”. Without an audit trail, the original data cannot be checked, and the study’s credibility is now seriously in doubt.
Rosenhan was both the designer of the study and one of its participants, and he selected, recorded and interpreted the observations. The study had a clear hypothesis that diagnosis is unreliable, so there was a strong risk of confirmation bias in what was noted and reported, with no independent observers to provide researcher triangulation. The pseudopatients were also not ordinary help-seekers: they lied about a symptom, which is exactly the information doctors rely on, so the study arguably tested whether psychiatrists can detect deliberate deception rather than whether they can recognise mental health.
The hospitals were in the United States in the late 1960s and early 1970s, under diagnostic systems and practices that have since changed, so the findings cannot simply be generalised to psychiatric diagnosis today. Ethically, the covert design deceived hospital staff and used hospital resources without consent. Its defenders argued that the question, whether people were being wrongly confined, justified the deception, which is a cost-benefit judgement. If the data were partly invented, the harm extends further: a fraudulent study shaped public views of psychiatry and of people with mental illness for decades.
Sources: Rosenhan, D. L. (1973). On being sane in insane places. _Science, 179(4070), 250–258. Cahalan, S. (2019). The Great Pretender. Grand Central Publishing. Scull, A. (2023). Rosenhan revisited: successful scientific fraud. History of Psychiatry, 34(2), 180–195._
Research considerations are where the concepts become practical tools. A strong evaluation of a study usually draws on two or three of them together, showing, for example, how an ethical choice created a measurement problem.
Bias: participant and researcher bias, countered by blinding, anonymity and reflexivity.
Measurement: reliability and the kinds of validity, from face validity to ecological validity.
Causality: internal validity and the control of confounds.
Perspective: reflexivity, the insider view that member checking provides, and the cultural limits of generalisation.
Responsibility: ethical design choices and honest reporting, including replication and data sharing.
Research considerations are the main tools for Paper 2 Section B, where an unseen study is evaluated through the concepts, and for Paper 2 Section A questions that ask how a concept applies to a class practical. At HL, Paper 3 Question 3 asks about credibility, avoiding bias or transferability in a qualitative source.
Reliability is consistency (test–retest, inter-rater, internal consistency); validity is accuracy and has several kinds: face, content, construct, internal, external and ecological.
Participant biases include demand characteristics, social desirability, acquiescence and the Hawthorne effect; researcher biases include expectancy effects, confirmation bias and interviewer effects.
Blinding, anonymity, standardisation, reverse-worded items, pre-registration and several independent coders reduce bias.
Qualitative research is judged by trustworthiness: credibility, transferability, dependability and confirmability.
Credibility is increased by triangulation (method, data, researcher, theory), member checking, prolonged engagement, thick description, audit trails and reflexive journals.
Reflexivity makes the researcher’s influence visible: personal reflexivity concerns the researcher’s own position; epistemological reflexivity concerns the choice of question and method.
Generalisability depends on population, setting, time and culture and is tested by replication; transferability is judged by readers from thick description.
Ethical requirements shape designs and can affect results, through demand characteristics, self-selection, attrition and the limits on what can be manipulated.
Rosenhan (1973) questioned the reliability of diagnosis, but doubts about its data since 2019 make it a lesson in credibility as well.
I can explain the kinds of reliability and validity and apply them to a study.
I can describe participant and researcher biases and suggest how to reduce each.
I can explain credibility, transferability, dependability and confirmability.
I can evaluate strategies for ensuring credibility, such as triangulation and member checking.
I can explain personal and epistemological reflexivity with an example.
I can discuss what to consider when generalising or transferring findings.
I can explain how ethical considerations shape research designs and results.
I can evaluate Rosenhan’s study and the later challenges to its credibility.