24:52
Reliability
Mike Sliter
Overview
This video explains the concept of reliability in measurement, distinguishing it from validity. Reliability refers to the consistency or stability of a measure. The video details how to assess reliability, focusing on three main types: test-retest reliability, internal consistency reliability (including split-half, Cronbach's alpha, and item-total correlations), and inter-rater reliability. It emphasizes that reliability is a prerequisite for validity and provides examples and guidelines for reporting reliability statistics.
How was this?
Save this permanently with flashcards, quizzes, and AI chat
Chapters
- Reliability is the consistency or stability of a measure.
- Validity is whether a measure assesses what it is intended to measure.
- Reliability is a necessary but not sufficient condition for validity; a measure can be reliable but not valid (consistently wrong).
- Observed score = True score + Measurement error.
Understanding the difference between reliability and validity is crucial for evaluating the quality of any measurement, ensuring that the data collected is both consistent and accurate.
A conscientiousness test that yields the same score each time a person takes it is reliable. However, if that test actually measures agreeableness instead of conscientiousness, it is not valid.
- Every measurement includes a true score and measurement error.
- The true score represents the actual value of the variable being measured.
- Measurement error is the deviation of the observed score from the true score.
- Reliability helps assess the amount of measurement error; higher reliability means lower error.
Recognizing that all measurements contain error helps in understanding why perfect accuracy is impossible and how to interpret scores with a degree of uncertainty.
Using a dartboard analogy, the bullseye represents the true score. Reliable but invalid measures consistently hit the same spot, but far from the bullseye. Reliable and valid measures consistently hit near the bullseye.
- Measures the consistency of results when the same individuals are tested at two different points in time.
- Assumes the variable being measured is stable over time.
- Vulnerable to issues like memory effects (participants remembering previous answers) and maturation (participants changing over time).
- More appropriate for stable traits like personality than for transient states.
This method helps determine if a measure produces stable results over time, which is important for variables expected to remain constant.
Giving a math test to students, waiting a month, and then giving them the same test again to see if their scores are highly correlated.
- Assesses the consistency of items within a single measure.
- Assumes all items on a scale are measuring the same underlying construct.
- Common methods include split-half reliability, Cronbach's alpha, and item-total correlations.
- Cronbach's alpha is widely used and indicates how well items correlate with each other; a common threshold is .70 or higher.
- Item-total correlations help identify and remove poorly performing items.
This is the most frequently reported type of reliability, ensuring that different parts of a measurement tool are all measuring the same thing.
For a conscientiousness scale with 10 items, checking if each item correlates well with the total score of the other 9 items.
- Measures the degree of agreement between two or more independent raters or observers.
- Used when subjective judgments are involved in scoring or coding.
- Cohen's Kappa is a common statistic that adjusts for chance agreement between raters.
- Ensures that observations or ratings are consistent across different people.
This is essential when multiple people are involved in data collection or scoring, ensuring that their judgments are consistent and objective.
Two researchers observing primate behavior independently and then comparing their coded observations to see if they agree on specific actions.
- Reliability statistics should be reported clearly in research.
- Cronbach's alpha is typically reported using the Greek letter alpha (α) followed by its value.
- Test-retest reliability is reported as a correlation coefficient (r).
- Inter-rater reliability (like Cohen's Kappa) is reported using its specific statistic (e.g., κ).
- The number of items in a scale influences reliability scores (e.g., longer scales tend to have higher Cronbach's alpha).
Properly reporting reliability statistics allows other researchers to understand the quality of the measures used and to critically evaluate the findings.
Reporting 'Cronbach's alpha = .85' for a scale, or 'test-retest reliability, r = .72', or 'Cohen's Kappa = .93'.
Key takeaways
- Reliability is about consistency; validity is about accuracy.
- A measure must be reliable to be considered valid.
- Measurement error is an inherent part of all measurements.
- Test-retest reliability assesses stability over time.
- Internal consistency reliability checks if different parts of a measure assess the same construct.
- Inter-rater reliability ensures agreement between different observers.
- The choice of reliability measure depends on the nature of the data and the research design.
- Reliability statistics should be interpreted in context, considering factors like the number of items in a scale.
Key terms
ReliabilityValidityTrue ScoreMeasurement ErrorObserved ScoreTest-Retest ReliabilityInternal Consistency ReliabilityCronbach's AlphaItem-Total CorrelationInter-Rater ReliabilityCohen's Kappa
Test your understanding
- What is the fundamental difference between reliability and validity in measurement?
- How does measurement error affect the reliability of a test?
- Under what circumstances would test-retest reliability be an appropriate measure of a psychological construct?
- Why is internal consistency reliability often preferred for multi-item scales?
- How can you determine if two observers are consistently applying the same criteria when rating behaviors?