PEP 6305 Measurement in
Health & Physical Education
Topic 1: Science,
Measurement, & Statistics
Section 1.3
Click to go to
back to the previous section (Section 1.2)
Statistical
Inference
Experimental Validity
n
Experiments have two types of validity:
internal validity and external validity.
¨
Experimental validity is not the same as the validity of
measurement, discussed in
Section 1 of this Topic and in Topic 12.
n
Internal validity means that the control of the conditions
in the experiment are sufficient to obtain data that can be appropriately used
to test the null hypothesis.
¨
The better that investigators control conditions, the more accurately they can answer the research question.
¨
If the experiment does not have the proper controls, then effects
observed in the study may be due to causes other than those in which the
investigator is interested. Only by excluding these “other” explanations,
through experimental control, can the observed effects be attributed to a
specific cause.
¨
The textbook describes several things that lower internal validity:
learning effects, lack of a control group, intervening variables, instrument
error, and investigator error.
¨
The key to remember is that if changes in the dependent variable
can be explained by something other than the independent variables being studied
by the investigator, then the study has a “threat to internal validity.”
n
External validity means that the results of the experiment
can be applied to people other than the specific subjects in the study.
¨
This depends on how closely the sample of subjects and the conditions of the study
represent the people and conditions in the “real world.”
¨
For example, if you want to study the effects of a certain diet on the health of
elderly patients in assisted living facilities, you should not use
undergraduates living in college dorms as your subjects! Undergraduates and
college dorms
differ in many important and relevant ways from elderly people and assisted
living facilities. The way diet affects health is altered by these differences.
n
There is a
tradeoff between internal validity and external validity:
¨
Increasing control of the experimental conditions increases
internal validity, but may make the experiment unrealistic (low external
validity).
¨
Highly controlled conditions (high internal validity) may not represent actual conditions in
the world (are
the conditions in your day-to-day life under a “high level of control?”).
¨
But by releasing controls to make conditions more realistic
(higher external validity), internal validity decreases (because control is
decreased). We have less confidence that observations are due to the variables
that we're studying.
¨
This is one reason why a series of experiments are required to
study important problems in science; no single experiment can answer any
question completely.
n
Internal and external validity depend on the logic and structure
of the experiment, which is why
research design is so important.
¨
Once the data
are collected, it is too late to improve internal or external validity.
n
Inference involves making
conclusions for a large group, called a population, from what is observed
in a small portion of the group, called a sample.
¨
A typical inference is making a conclusion about a population
parameter (the true value of a variable in the population) using a
statistic (an
estimate of the true value computed from a sample of the population). Thus, a
statistic is an estimate of a parameter, but the parameter is what we
are really interested in studying.
n
Ideally, subjects in a sample are selected at random from the
population of interest; this is called random sampling. Random
sampling prevents characteristics of the subjects from influencing the results
because the distribution of those characteristics are the same in all groups:
¨
If the subjects are randomly sampled before being randomly assigned to groups…
¨
then the differences in characteristics are also randomly
distributed among the groups…
¨
which means that the distributions of the characteristics averages
out to be approximately equal in all groups--i.e., they essentially become
constants (same value for all groups).
n
However, some groups may have few members relative to the
population, which means that simple random sampling from the whole population
would select very few subjects from the small group. Most of the subjects would
be from the larger group.
¨
This means
that the estimate of study effects in the smaller group is less precise (because
it is based on fewer subjects) than the estimate in the larger group, making
direct comparisons difficult.
n
In this situation, the process of stratified sampling
allows for random sampling from those segments of the population while
maintaining equivalent accuracy of estimates.
¨
Groups of interest are identified in the population.
¨
The same proportion of people in each groups are sampled: [#people sampled]
÷ [#people in group] is the same for all
groups.
¨
Subjects are randomly selected from each group to obtain numbers of subjects to
ensure accurate estimates of study effects.
¨
See the
middle section of this Web page for
more information about stratified sampling.
n
The goal of sampling is to obtain an unbiased sample; bias
means that something is influencing which subjects end up in the sample—so when
bias exists, the sample is not representative of the
population.
¨
If the sample is not representative of the population, we cannot make an
inference about the population using data from the sample.
Misuse of Statistics
n
Statistics can be misleading ONLY
when the audience does know (or is not told) under what
conditions the data were collected and how they were analyzed.
n
You can be part of a better informed audience, and then you will not be
mislead by statistics.
n
When you see “statistics” in research or popular reports or the
media, ask
questions, such as:
¨
Who is the sample supposed to represent? Does it?
¨
Is the sample random or were the subjects selected (hand-picked), possibly
limiting the results?
¨
How large is the sample?
¨
Are the variables well defined?
¨
Under what conditions were the data collected? Were they
controlled? Are the representative of the typical conditions?
¨
How were the data analyzed?
n
Be smart when you are reviewing statistics as evidence.
¨
Statistics can be good evidence, as long as the research design
and data are sound.
¨
Statistics can be misleading if the research design and data are
weak.
n
Do not yourself be guilty of propagating unsubstantiated
statements based on weak or invalid statistical and scientific evidence.
Formative
Evaluation
n
In the study you designed in the Section 1.2, to what larger population
of people do you wish to infer
that the results observed in your study apply?
n
Can an study have high internal validity but low external
validity, and vice-versa? If so, how?
You have reached the end of Topic 1.
Make sure to work through the Formative Evaluation
above and the textbook problems (end of the chapter).
You must complete the review quiz (in the Quizzes
folder on the Blackboard course home page) before you can advance to the next topic.