- Observations are likely our most reliable source of claims.
However, we know even that observations can go wrong. Knowing
the limitations and occasional failures of observations will help
us avoid those failures.
- Observations come in at least two kinds
- Particular occurent observations: a claim
about some particular thing observed right now.
"This table is red."
- Generalizations from observation
Particular and Occurrent Observations
- Particular observations are observations of a single event
or individual thing.
- Occurrent observations are observations made right here,
- Particular and occurrent observations are usually our
most reliable sources of claims.
- However, particular and occurrent claims are susceptible
to a number of problems.
- Our observations can be influenced by our
expectations. (Rosenthal and Fode 1966, 1963; Rosenthal and Lawson 1964).
- We attend to things in our environment, and remember them, if
they are particularly vivid -- for example, if they are emotionally
salient. Similarly, there is some evidence that our observations can
be influenced by our emotions so that we are more likely to
interpret those observations in a way consistent with our emotions.
This a phenomenon called "emotional congruence in perception"
- Particular remembered claims (that is, particular but not
occurrent) can be inaccurate. Memories can form in an inaccurate
way, or can be revised over time inaccurately. This is true even for
vivid, flashbulb memories. ($$$)
- Generally, we must recognize that particular observations are
strong but fallible evidence for a claim.
Controlling for Bias in Particular Occurrent Observations
- One way to avoid potential problems with particular occurrent
observations, particularly a bias arising from expectations, is to
control for the expectations of the observer.
is what is done in double blind experiments. Such experiments are
constructed typically in such a way that the individuals making
observations do not know which observations have the relevant
- For example, suppose that we wanted to test the effects of
substance X on some rats.
- We would construct the experiment so
that half of our rats were fed substance X, and the other half would
be given a placebo.
- The actual feeding of the rats would be done in
such a way that the experimenter who will record the
observations will not know which rats are fed which substance.
We could do this, for example, but having the feeding
done by experimenter A, or by having the food prepared by
experimenter A, when another experimenter B will make
- Experimenter A and B should not talk about the experiment.
Ideally, they should not even meet.
- Experimenter B makes observations not knowing which rats
received which food.
- Ideally, experimenter A (or some other experimenter)
interprets the results of B without knowing which rats are which.
- Here expectation bias cannot directly effect observations
because the observer is not sure which of the things it is observing
have the cause about which she has some expectations.
- Often we need to make general claims based on a finite
number of observations. This kind of reasoning -- generalizing
from some sample of observations to conclude something about
an entire population of events or things -- is sometimes called
- When we generalize from particular to general in this way,
we need to ensure, to the best of our ability, that our sample
observations are representative of the population.
- If our population is not homogenous regarding the feature
we are measuring, we can try to ensure that a sample is
representative by ensuring that it is random.
- A sample is random if every potential observation on our
population had an equally likely chance of being in the sample
- Getting a random sample of observations is not easy.
- Many populations will also have natural variation. If we
are measuring the height of people, for example, there is significant
variation in the heights of individuals. But this means that a small,
even if random, sample has a chance of being quite different than
the mean (average) in the population.
- We try to ensure that variation in the population is controlled
for by getting a sample that is sufficiently large.
- The size of a sufficiently large sample is a function of how
accurate we hope to be, the
size of the overall population, and the expected variability of the
population. That calculation is beyond our goals here, but in
general we should at least be aware that:
- Generally, the larger the sample size, the
more accurate our generalization is likely to be.
- Generalizations have a margin of error
that is a function of the size of the sample, the
size of the population, and the variability in the population.
- The margin of error is the expected likely error in our
generalization, given our sample size and our knowledge of
Common Reasoning Mistakes with Generalizations
- Forgetting that larger samples are more
representative of the population (and that thus
smaller samples are more variable).
- Failing to recognize when samples are not
- Over-valuing anecdotes, and not recognizing
that these represent a small, even irrelevantly small, sample.
Vivid or concrete anecdotes have an especially strong
influence on people's generalizations. Vivid information can
overwhelm uninteresting information in people's perceptions.
- Resisting revision of first generalizations.
Generalizations to Probabilities from Observation
- One form of generalization from observations is the
derivation of a probability from past events. In most
cases this is a simple derivation: if we observe that
n% of our sample has some property P, then assuming our
sample is random and sufficiently large, we assume that
observations in the future will have property P n% of
- Example: we observe in a random sample of oysters
that 1% of them have a pearl. For any given oyster that
we may later encounter, we conclude there is about a 1%
chance that there is a pearl in it.
- Three problems commonly occur for this kind of
generalization; these echo concerns about other forms
of generalizations from observations discussed above.
- Availability. Individuals tend to confuse
the availability of something with its frequency.
That is, if something is available to them, they
may assume that it is more frequent than it actually
is. (This is akin to generalizing from one's own,
- Vividness. As noted above, people tend to
notice and to recall more vivid experiences. This
can lead people to overestimate the probability
of some event.
- Gamblers fallacy. This common mistake
arises when someone confuses the probability of
two or more outcomes, taken as a whole, with the probability of an
single outcome after some other outcomes of that kind have preceeded it.
- An example of the gambler's fallacy. Above, we
assumed that 1% of oysters had pearls. Consider the following two cases:
The odds of finding a pearl in an oyster are 1%, we assumed.
You should then expect it to be very likely to find a pearl in
a sample of 100 of them. However, if you have opened 99 and found
no pearl, the odds of a pearl in the last oyster is 1%. That is,
you must not confuse the case of many many oysters with each single
one. The gambler's fallacy occurs if Jones assumes she is now due
for a pearl, and expects that it is very likely the last oyster
has a pearl.
- Jones has two oysters, and wants to know what the
odds of finding a pearl in each is.
- Jones has 100 oysters, opens 99 of them and finds no pearls
inside, and now wants to know what the odds
of finding a pearl in the 100th oyster are.
- The gamblers fallacy has led many a gambler to poor
reasoning, such as concluding that if they have been losing for
a long time they are due to win.