Evaluating Extraordinary Stories

WHEN someone tells you an extraordinary story, whether it be about miracles, angels, ghosts, aliens, psychics, worldwide conspiracies or anything else far outside the realm of our ordinary experience, it can be very valuable to ask yourself a simple question: “is this claim testable?”

For example, suppose your friend tells you that he had a dream a few days before 9/11 predicting that the Twin Towers would be destroyed by airplanes. Let’s say for the sake of argument that this friend is by and large a trustworthy and rational person who you consider to be a reliable friend.

Naturally we are inclined to want to believe what other people say, especially those we think of as friends. We tell ourselves things like, “he has no reason to make something like this up”, and give people the benefit of the doubt even when their claims seem downright astounding. But, its important to remind ourselves that we humans are very far from perfect creatures. We all, at times, exaggerate the truth, embellish stories, read into random coincidences, speculate without much evidence, believe falsehoods, misperceive reality, make logical mistakes, see things, misinterpret events, misremember memories, and delude ourselves. And there are some of us who even confuse dreams with reality, mistake internal experiences for external ones, or hallucinate events that never occurred. So how can we be sure that our friend truly had this prophetic dream?

Perhaps our friend is embellishing his story, and really did dream about the Twin Towers, but not that they would be destroyed by planes. Or maybe the details of the dream are accurate, but it actually happened after 9/11 and he is mistaken about when it had occurred. Or possibly, by some remarkable coincidence, he did have a dream where some buildings were destroyed just before 9/11, but his fuzzy memories of it were replaced by false ones after 9/11 occured (which switched the buildings in his dream with the Twin Towers and inserted airplanes that weren’t originally there). On the other hand, there is also always a chance, if only small one, that my friend made the whole thing up just to have a great story to tell.

Now, we ask ourself the simple question, “is his claim testable?” The fact of the matter is that there are many possible explanations for our friends experience, and the vast majority of them are of the perfectly normal (rather than paranormal) variety. There is no way we will ever be able to know precisely what his dream consisted of, and what is misremembered or embellished. We can never rewind history and see into our friends mind to confirm or deny his story, so ultimately his claim is entirely untestable. We can NEVER be certain that what he says is true… at the very best, we could only say that it is probably at least partially true.

More generally, since unverifiable claims of an extraordinary nature such as this one can never be shown to be true or false, they are not much worth worrying over. To believe these stories whole heartedly would be to underestimate the many reasons why we cannot always believe what people tell us, and would be setting ourselves up to believe in falsehood. We know that many extraordinary  claims that people make are not true (like that astronauts never went to the moon, or that the illuminati secretly rule the world, or that the earth was created 6000 years ago, or that we all accidently eat many spiders while we sleep), so a willingness to believe in extraordinary, untestable claims will almost certainly lead us into being misled. On the other hand, there is little that we can be sure of with absolute certainty, so we cannot be 100% sure that the claims are bogus. A safe rule of thumb seems to be that we should never say that an extraordinary untestable claim is anything better than “most likely, at least partially true”.

On the other hand, sometimes people make extraordinary claims that are in fact testable. In these cases, since verification is possible, why should we ever believe what is claimed until we actually put the claim to the test (or, at least, convince ourselves that other reliable people have done the tests for us). Furthermore, if we can’t perform the test ourself, and can’t find anyone else who has performed it, then why not reserve judgement until that time when the test has been done? There is nothing wrong with thinking, “I don’t know if this is true. I’ll wait until their is more information before making a decision.”

Of course, not all tests are created equal. Unfortunately, many experiments and tests that are conducted (especially by those who are inexperienced in the area of testing, or who have a bias towards a certain result) do not properly rule out all possible alternative explanations to a phenomenon with high probability. If a test is flawed, then even after the test is performed we still cannot know what to believe because multiple seemingly valid hypothesis still exist.

To give one example, suppose that a friend tells me that he can predict whether a coin that is flipped will land on heads or tales. This is clearly a claim that is easily testable, so we choose to put it to the test. We tell my friend to flip a coin one hundred times, each time guessing what it will land on. We then record the results and compare the number of guesses he got right to what we would expect would happen if he was merely guessing. If he gets 52 out of 100 right, then he is very likely mistaken about his supposed power, as such a result is likely to occur by chance alone. On the other hand, if he gets 98 right out of a hundred right, then that seems to be compelling evidence of his skills.

Unfortunately, this not a well designed test. Can you see why? It is because even if my friend guesses 98 out of 100 flips right, it does not rule out all (probable) explanations for the results other than the explanation that he gives us. The problem is that since we allowing him to flip the coin himself, he could be employing slight of hand trickery to force the coin to land one way or another. There are undoubtedly many magicians in the world today who can do this without a spectator ever suspecting. So at this point we are forced to go back to the drawing board and redesign our experiment so that the results are genuinely a conclusive test of his powers.

This time, I use a coin that I acquired myself, and that my friend has never had an opportunity to touch (after all, it is possible to replace a coin with a biased one that is heavier on one side, which would improve his chance of impressive results). During each trial I flip the coin myself, and cover it immediately when it lands on my hand. We also videotape the session so that the tape can be reviewed if any trickery or problems with the protocols of the test are suspected. Ideally, experts in testing and sleight of hand would be consulted, to make sure that no other controls on the experiment are needed, and an expert in statistics should he consulted to help with interpretation of the results. The protocol should be completely laid out beforehand and agreed to by both parties in advance, without permitting any changes to be made at the last minute. Finally, when the test is over, I should review the heads and tails that were achieved, to make sure their aren’t any obvious patterns that could indicate cheating or a biased (weighted) coin.

Once this new, carefully controlled test is complete, I can state with a lot of confidence whether my friend’s claims are justified. If he thinks he can get 75% of the guesses correct, but only gets 51%, then he is probably full of shit. Of course, he could be having an off day, but certainly at that point there would be no reason at all to believe that what he says is true. On the other hand, if he claims that he can achieve 75% accuracy and gets 74%, there is a strong reason to believe he does have the powers he claims.

In conclusion, it is usually fairly straight forward to divide extraordinary claims into the two categories of “testable” and “untestable”. If a claim is testable, then why would we ever believe it before it has been put to the test? There is a straightforward way to verify what is being claimed, so there is no need to simply speculate as to whether it is true. On the other hand, with untestable extraordinary claims we will never truly be certain of their legitimacy. Of course, the more extraordinary the claim is, the more skepticism is warranted. Extraordinary events, by definition, almost never happen, so it should be more difficult for us to believe in them! With untestable claims we can ask ourselves, “why should we ever believe strongly in something that we know cannot be proven?” If we unquestioningly believe we are simply setting ourselves up to be tricked by all the falsehood that eventually comes our way.

This entry was posted in Skepticism. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *