When you make a claim that is either true or not, there is an hypothesis to be tested, stated as a fact, and a claim of truth or falsity. As in research the truth is presumed to be unknown, there are four possibilities.
| Statistical errors | fact is true | fact is false |
|---|---|---|
| true is claimed | Correct | Type II error |
| false is claimed | Type I error | Correct |
The problem faced by social scientists is that human behavior is conditioned by many factors, which are rarely controllable by the researcher, and even when controllable, there are usually too many to control effectively. We model these uncontrollable factors as randomness, and analyze the data using probability models, that is, statistics.
Because the cause-and-effect relationship is uncertain, there is some probability of an incorrect claim by the researcher, regardless of whether the fact is true or false.
One of Kahneman's letters on replication Another Kahneman letter on replication