THREATS TO CONCEPTUAL VALIDITY
Unnecessary duplication
Theoretical isolation
THREATS TO CONSTRUCT VALIDITY
Construct validity is the quality of choices about the particular forms of the independent and dependent variables. These choices will affect the quality of research findings. Threats to construct validity can arise from the choice of treatment (the operationalization of the IV, and the administration of the treatment), the choice of oucome measure (the operationalization of the DV, and the administration of the measurement).
I. The independent variable
a. Inadequate operationalization of the IV:
Lack of reliability of the independent variable
Treatments lack reliability if they vary so much from one occasion to another that they introduce variability that obscures the relationship being studied.
Lack of representativeness of the independent variable
A treatment must be an adequate operational representation of the theoretical construct that is of interest.
Lack of impact of the independent variable
A treatment should produce a realistic impact on research participants; neither too much nor too little impact.
b. Treatment artifacts:
Treatment artifacts are threats to the validity of an experiment that arise in the actual presentation of treatments to research participants. They are the result of participant and experimenter motivations and actions, and of the research setting.
Demand characteristics in the research setting
Participants respond to unstated demands from the research setting. Desire to cooperate, and anxiety about evaluation are two such demands.
E.g., the Hawthorne Effect: workers at the Hawthorne plant of the Western Electric Co. in the 1930s showed increased productivity simply because they were studied.
Experimenter expectancy effects
Researchers provide unconscious cues to participants about the results that are expected from them. The Rosenthal effect: Rosenthal and Jacobson deliberately established expectations.
Pretest sensitization to the treatments ("practice effect")
In designs that use a pretest (a measurement before the treatment is administered), it can make the participants sensitive to the aims of the study. And it can simply provide an opportunity to practice that results in improved performance.
Order effects ("carryover effects")
If a study uses repeated measures or exposure to more than one treatment...
II. The dependent variable:
a. Inadequate operationalization of the DV:
Lack of reliability of the dependent variable (instrumentation)
Every measure gives results that vary. If the variation in measurements is large, the usefulness of the measure is compromised.
Interrater reliability: Changes may occur from pretest to posttest because the nature of the measurement has changed. For example, raters’ ability to assess patients may improve over time.
Lack of representativeness of the dependent variable
Measures need to have adequate content validity.
Lack of sensitivity of the dependent variable
Measures need to be sensitive enough to detect differences in outcome.
b. Measurement artifacts:
Strategic responding by the research participants
Participants may develop strategic response strateges: ‘helping’ or ‘challenging’ the researcher.
Linguistic/cultural bias in the dependent measure
Can participants understand the questions, and communicate their responses accurately?
THREATS TO INTERNAL VALIDITY
Extraneous effects (history)
Are participants exposed to events, other than the treatments, whose effects on their behavior could obscure the effects of the independent variable?
Temporal effects (maturation; fatigue)
Do the participants change with the passage of time in ways unrelated to the effects of the independent variable?
Group composition effects (selection)
If different groups are used to compare the effects of treatments, could pre-existing differences among the groups obscure the effects of the independent variable?
Interaction of temporal and group composition effects
Could changes in the participants’ behavior over time that are related to pre-existing differences among groups obscure the effects of the independent variable?
Selective sample attrition (mortality)
Do participants drop-out of the groups during the study in a systematic or selective way? This could create differences among groups that would obscure the effects of the independent variable.
Statistical regression effects (regression to the mean)
Regression toward the mean: the tendency of extreme (very high or very low) scores to fall closer to the mean on re-testing. Could changes in participants’ responses to the measures be caused by this?
THREATS TO EXTERNAL VALIDITY
Nonrepresentative sampling
Are the participants in the research study so unrepresentative of those people who need to be understood? This would preclude generalization of the research results from the former to the latter.
Nonrepresentative research context
Is the context in which the research study was carried out so unrepresentative of contexts where the behavior in question takes place as to preclude generalization of the research results from the former to the latter?
THREATS TO STATISTICAL CONCLUSION VALIDITY
Inappropriate use of statistical techniques
Use of a statistical test lacking sufficient power
Mistaking a trivial effect for support for the research hypothesis