The Nature of Anecdotes

Medical guidelines should insists on proof that time-honored medical practices and procedures that cost money and may harm or kill patients are actually effective. This Forum is about how to force organized veterinary medicine to issue Evidence Based Guidelines.

The Nature of Anecdotes

Postby malernee » Sat Sep 11, 2004 6:02 am

The Nature of Anecdotes
by Paul Lee, PT


Introductory remarks

While this article applies in many ways to anecdotes in general, it is especially written in the context of this website, which is the use
of anecdotes to justify the use or promotion of so-Called Alternative Medicine (sCAM) and other unscientific methods and ideas. Even
more relevant in this setting is the use of anecdotes *instead of* objective evidence using the scientific method. While science seeks
to sort cause from coincidence, anecdotes often confuse the issues and even favor coincidences rather than proven causes.

Now just what is an anecdote? Why do we pay special attention to some experiences and tell about them? Are they true or false? Can they be trusted? Why do we selectively ignore other experiences? These are questions of vital importance for people who consider it important to really learn what is fact and truth.

Let's start with a definition:

Anecdote: a usually short narrative of an interesting, amusing, or biographical incident.

The very fact that we tell anecdotes is that we find them interesting and often personal. They are unusual, and are often what we have personally experienced or heard about. We wouldn't notice them or bother telling them if they were the norm, rather than the exception. We use them as reinforcements to shore up our arguments. We use them when we lack proof, IOW, for lack of anything better. We also notice them because they may reaffirm some cherished idea we have:

"Man prefers to believe what he prefers to be true." - Francis Bacon

We also use them because they give things a personal touch. They are part of us. Cold facts don’t touch hearts. But cold facts will continue to exist, whether we believe in them or not. They are not changed by what we believe about them.

Any or all of the above are inherent to the nature of anecdotes. They are by their very nature generally unusable when making conclusions for general application. While it is naturally interesting to us, our personal experience cannot be used to judge what might be applicable to others, since it may well be the exception, rather than the rule. Doing so can lead to the spread of dangerous scaremongering ideas, with disastrous consequences for others, who most likely represent what is typical, and not what is exceptional. We may well be more unique than we realize or will admit. Using ourselves and our personal experiences as a standard for others may well be very unwise.

Untrustworthy Subjectivity

The lack of trustworthiness of anecdotal information is notorious. Anecdotes are prone to the same weaknesses that gossip suffers from: lack of objectivity, exaggeration, distortion through repetition, lying, one-sidedness, etc. Many less than savory motives often come into play. Anecdotes are well-chosen, usually representing exceptional incidents, while the vast majority of experiences that are contrary to them are conveniently forgotten, ignored, or even hidden.

It is an unavoidable fact of human nature, that what is personally experienced is interpreted by previous knowledge and experience. In other words it is filtered, and is not totally objective nor necessarily true. Since this previous knowledge and experience is necessarily limited and has likewise been interpreted, there may well be a tendency for more experience to represent more self-delusion. The problem gets compounded. Self-delusion becomes an ingrained part of the personality.

Some people are good at constructing fantasy worlds, consisting of their own inaccurate understanding of things. But that's not how things should work. The existence of reality is independent of belief, but the perception and experience of reality is influenced by it, so be careful what you believe:

"The mind may find unique ways to express what it conceives of reality, but reality remains unchanged by the mind's
efforts." - David Haas

"Belief and sincerity do not define truth; it exists despite belief and sincerity." - Joshua David Mather

"Reality is that which, when you stop believing in it, doesn't go away." - P.K.D.

"You can do little with persons who are disposed to accept these curious medical superstitions. You have no fulcrum you can
rest upon to lift an error out of such minds as these, often highly endowed with knowledge and talent, sometimes with genius,
but commonly richer in the imaginative than the observing and reasoning faculties." - Oliver Wendell Holmes MD, in remarks
before the graduating class of the Bellevue Hospital Medical College in 1871

Healthy Skepticism

An important function of education is to impart knowledge of the real world, and of how to act in a socially constructive way. While high education cannot guarantee common sense, or prevent belief in false ideas, it does help to expose us to more of what is known and proven to be true and reliable, than if we are not so educated:

"Scholars are trained to scrutinize, to insist on adequate evidence, to ferret out logical inconsistencies and weak arguments. We
are naturally suspicious of claims that go beyond our experience. Scholars are trained skeptics. Our professional motto is 'show
me'. Where's your evidence? If you can't prove it, you shouldn't believe it!...If trust is the natural disposition of childhood, doubt
is our disposition as adults. Academic training cultivates an ethic of suspicion, if not unbelief....we've learned to put every aspect
of life through the fire of critical reflection....[But] the fact we don't know everything doesn't mean we don't know anything."
- Richard Rice, Ph.D.; Spectrum, v. 28:1, pp. 39-40.

"It is the mark of an educated mind to be able to entertain a thought without accepting it." –Aristotle

"Yes, you are correct that the individual patient may believe that xyz worked for them and granted it is difficult to respond to that
belief, but it is more difficult to agree with what has no basis for being correct. It will not take long for that blind agreement to
catch up with you." - Charles Morrow

"Randomized controlled trials appear to annoy human nature--if properly conducted, indeed they should." - K.F. Schultz

To the scientific mind, certain phenomena are readily understood, or at least recognized, and therefore relegated to the place they deserve. Medical personnel with such minds avoid the pitfalls to which minds untrained in critical thinking and scientific observation easily fall prey. They have learned to distinguish the exception from the rule. The following quotes illustrate this point well:

"You also seem to use a lot of "post hoc, ergo propter hoc" "reasoning." Coincidence doesn't imply correlation and correlation
doesn't imply causation. For example, you mention a man who had Ross River virus and got better after he took homeopathic
drugs. What else did he do? Did he call his mother on the phone? Did he take a walk? Did he pet a dog or cat? Did he read a
book? Why shouldn't any of those things be given credit for his recovery? Given that ONE case, there's the SAME "evidence"
that ANYTHING else he did brought about his recovery as for the homeopathic drugs." - Unknown

"And, like most things from chiropractic to surgery, allied health professionals or the law or the public will only tend to see the
"failures", because the successes, by definition, don't tend to present to other providers or in other situations." - Greg Smith, MD

"One of the dangers of any medical practice is making false assumptions based on sampling errors. Often the patients happy with
their treatment return for ongoing or future treatment, and those unhappy go elsewhere. We all are tempted upon reflection to
notice that most or all of our returning patients were helped by us, and that most of those treated by others did less well. Though
this is just a statistical sampling error that does not necessarily reflect the validity and quality of our work, it tempts us to claim
that we have done well by our patients. This is why your claims must be scrutinized in their context, and why anecdotes are
relatively unconvincing even when tantalizing." - Steve Zeitzew, MD

The well-established scientific and logical principle - "The burden of proof is on the claimant" - does not forbid curious scientists from seeking explanations for puzzling, maybe even paradoxical, observations and anecdotes. The greatest value possessed by anecdotes, is that impressive anecdotes can spur to further research.

Anecdotes can be considered neutral, and therefore worthy of being examined by anyone. They are usually about what someone has experienced, even if they may have been interpreted incorrectly. They aren't usually lies. Something was perceived to have happened. It may be that the perception is flawed, but the experience cannot usually be denied. Before doing so, one should be in possession of good evidence that the person is lying and intending to deceive, which can certainly happen. The anti-vaccination activists are a good example of deliberate deception.

The scientific community is well aware of the deceptiveness of human experience. This they have learned from the sometimes bitter experiences which misinterpretation of data have led to. Since many believers in so-Called "Alternative Medicine" (sCAM) do not have this experience in conducting research, and often lack education in these matters, they glibly accept any positive anecdote as proof of effectiveness. This doesn't have anything to do with intelligence, but of knowledge. There are many geniuses who believe some pretty weird things!

Good scientists and advocates of Evidence-Based Medicine (EBM) understand the folly of blindly believing first impressions, and the following quotes reflect the wisdom of this position:

"Science is what we have learned about how to keep from fooling ourselves." - Richard Feynman, physicist

"A habit of basing convictions upon evidence, and of giving to them only that degree of certainty which the evidence warrants,
would, if it became general, cure most of the ills from which this world is suffering." - Bertrand Russell

"The brightest flashes in the world of thought are incomplete until they have been proven to have their counterparts in the world
of fact." - John Tyndall (1820-1893), physicist

"Scientific thinking might be defined as learning to distinguish the exception from the rule. I'd have a hard time entrusting my
health to someone who didn't know the difference." - Stan Polanski

"The most important quality of the scientific method is its capacity to eliminate bias. That's what randomisation, blinding, proper
sampling and so on is about. That's where almost all non-scientific medicine breaks down." - Patrick Bramwell-Wesley

"Proof is not anecdotal. Proof is scientific. Show me the research." - Mark Levine, DC

"Anecdotes are useless precisely because they may point to idiosyncratic responses."

"The plural of anecdote is not data." - Roger Brinner

"The last treatment before "cure" by natural causes (perhaps running it's course) should not be considered cause and effect."

"Of course, the best (worst) part of it is that the alt-med people are convinced that it is all plausible because they are looking for
evidence to bolster their belief rather than build a belief based on evidence. Which is what makes them 'alternative.'" - Greg Hart

The Weaknesses of sCAM

These quotes represent another world view and paradigm than is common in the world of sCAM. An understanding of their own ignorance and fallibility seems to be lacking among believers in sCAM. They lack the humility produced by the humbling effects of great knowledge, which is a characteristic of good scientists. Their belief system seems to consist more of rationalization, circular reasoning, wishful thinking, and speculation, than of experimentation and critical observation. They often exhibit the arrogance of ignorance, even standing in judgment on scientists and MDs by far their superiors in every way.

By definition, sCAM methods are either unproven or disproven, even if they have millions of positive anecdotes and several thousands years of use. Neither age nor popularity can make truth error, nor error truth. That has nothing to do with it. Precisely because anecdotes often point to idiosyncratic responses, they cannot be trusted to represent the truth of a claim. Anecdotes have interest only as a motivation to perform research for the purpose of confirming whether the claims of effectiveness are true or not.

The sCAM field is rampant with wild claims, lack of documentation, and lack of desire to document the claims. Own results and anecdotes are considered sufficient. Single case studies are abundant, and anecdotes are often used as “proof” of effectiveness. This is a very characteristic sign of quackery, and is a big red flag. Where anecdotes are used to sell an idea, beware.

There is also a lack of accountability. While there do exist charlatans of the worst kind, I believe that most practitioners are honest and well-meaning people with good motives. But they should not be exempted from the same accountability all others are subject to.

One of the worst things, besides injuries or the delayment of effective treatment, is the weird ideas people get. They get exposed to a lot of half-baked ideas from practitioners with just enough knowledge to make them sound authoritative, and therefore more dangerous. They also get a large dose of skepticism against EBM, MDs and the governmental agencies that exist to protect the public. But they exercise no skepticism at all towards the practitioners who have no medical education, only homemade theories, and no better evidence than anecdotes. Quite a perversely paradoxical situation!

The Function of Well-Documented Anecdotes

The anecdotes which these shoddy practitioners use are not even clinical anecdotes, which usually have more value, because the person telling them has the background and education to determine their value and meaning. Such well-documented anecdotes can be written up as case studies, which can then be used to stimulate interest. Such interest may then lead to research. But it often requires a number of good quality anecdotes to create such an interest. Poor quality anecdotes can number in the millions, and discerning scientists will hardly raise an eyebrow.

The proper way to deal with anecdotes is to document them and write them up as case histories. A case history is a short description with whatever it was that happened, with a hypothetical explanation for why it might have happened. While the hypothetical conclusion may be based on flawless logic, it nevertheless is based on an unresearched phenomenon, and can topple as easily as a house of cards. Scientists are well aware of this, and wouldn't dream of basing a therapy, or marketing anything, based on such a case story.

Writing case histories is a job for trained scientists and medical personnel. The education of these professionals should have prepared them to analyze such stories and roughly determine if there might be something to them. Such case histories prove nothing in themselves, but they can be so convincing that they can impel researchers to check them out.

Most often it turns out that there are logical errors, sampling errors, or other factors (often unknown at the time) that render the hypothesis invalid. The story stops there, and no more research is done. But sometimes the original story turns out to be the tip of an iceberg, and things of interest are revealed by the research. New knowledge is thus developed. Serendipity does play a part in some important discoveries. (And some people are prone to be serendipitous!)

Consensus Science

sCAM is characterized by "consensus science", which is the idea that if enough people believe in something, it must therefore be true. This is not science, and is a dangerous delusion. "Everyone is entitled to their own opinions. No one is entitled to their own facts." - James Schlesinger. Likewise it is very easy to link cause and effect in practice because of what is seen. This does not make the correlation correct and may mislead patients and practitioners alike.

There is an important difference between the kind of evidence that is acceptable in a court of law, and that which is demanded in a scientific setting. In a court, it is common practice to pile anecdotes (which would be weakly convincing in isolation) on top of each other until they are accepted as a strong probability, and this is then accepted as sufficient evidence.

Scaremongering Causes Mass Hysteria

One of the areas where "consensus science" rules, is when private persons collect lots of sensational anecdotes regarding some vilified substance. The results are used to justify scaremongering, and can create hysteria and mass fear in the general population. Such cases have been termed "mass sociogenic illness" and "mass psychosomatic reaction” (Lancet, July 3, 1999).

Such hysterical reactions occur from time to time, and have been with us all throughout history. Here are some examples: witch hunting in Salem or Europe, fear of contaminated Coca Cola in Belgium, prejudice against gene modified plants, or suspicion directed at Aspartame, Alar, amalgam fillings, vaccines, fluoride, Roundup, UFOs, refined sugar, milk, gluten, you-name-it. Many of these fear campaigns feed on the insecurities of people in the face of new technology and the unknown.

Sometimes they are poorly disguised campaigns that are created and exploited by antagonists of modern medicine and science. These people and groups often have anti-modernistic, anti-capitalistic sympathies, IOW, they are simply anti-establishmentarians. Other times they are simply the well-meaning, but misguided efforts of people who wish to warn against what they perceive as something dangerous.

The habit of collecting such stories can easily undermine the trustworthiness of the collector. Such a habit often says more about them and their views, than it does about the validity of their hypothesis.

This solicitation and collection of anecdotes is very problematic, since there is no certainty that the reported symptoms have anything to do with the feared substance. They may even be caused by fear itself:

"Maybe we should have a warning label to put on warning labels that warn that warning labels have been known to cause adverse
reactions in some people :)...." - Phyllis Buxton

"In developed countries toxic environmental chemicals seem to have replaced alleged evil spirits and witches.

"Mass hysteria can cause rashes of physical illness. Taking that into account can prevent incorrect treatments, misdirected
responses, and public anxiety and can contribute to humaneness in the handling of such problems.” - Swogger, Raso. ... bronx.html

Hypochondriacs respond well to such hysteria. It's right up their alley. But there may well be people with real symptoms caused by other, very real, illnesses, such as MS and cancer. These people then get a wrong diagnosis from the fearmongerers, and don't get proper treatment. It's like putting the wrong man in prison - the real criminal is then left to walk the streets and commit more crimes.

What is lacking with these collections of anecdotes, is the proper method to collect and analyze them. The anecdotes are often called forth by the very act of hearing that such-and-such can cause the symptoms the person has. But since their symptoms can be caused by many other things, they end up climbing on the wrong bandwagon. The very way that they are questioned is often very leading, and can only lead to the desired conclusion: "Yes indeed! You are just another of the many thousands who are being poisoned by such-and-such. It's good we have now registered your case."

The very fact that such a story is registered is then considered to be more proof against the vilified product or substance. These collections of anecdotes often say more about the scaremonger than they say about anything else. All the cases can be traced back to one person, with their scare stories.

The registration of complaints with the FDA can illustrate the problem. When the FDA receives a complaint, it is registered, without any control of whether the complaint is legitimate or not. But the scaremongers then point to the thousands of registered complaints as proof against their favorite hate substance.

This misuse of anecdotes is very damaging to the public. They become conditioned to believe that something is true, just because lots of people make a claim, even when that claim is unfounded and based on speculation, improper harvesting of data, and leading questions.

I’ll let Karen Daskawicz’s words of wisdom close this article:

"I don't think anyone condemns you or anyone else for entertaining a notion about a particular cause and effect. What I and others
think is that it's best to concentrate on ideas that have evidence to back them as opposed to any idea that might possibly be true.
Most wasteful of time and energy is entertaining ideas that have either weak, circumstantial, or anecdotal evidence for but have
solid evidence *against* them."

Paul Lee, PT
October 2002

This article has also been published here (the first paragraph was added later):

Skeptic Report
September 2003


Other related articles by the author:

A Consumer Advocate's Dilemma:
Should CAM be Researched?

Complementary & Alternative Medicine COSTs!!
(British Medical Journal, Jan. 21, 2001)

"Alternative" Physical Therapy?
Article in the Journal of the Danish Physiotherapy Association

Weil's Non Science-Based "Medicine"
(British Medical Journal, Feb. 1, 2001)

Site Admin
Posts: 462
Joined: Wed Aug 13, 2003 5:56 pm

Return to evidence based vet guidelines

Who is online

Users browsing this forum: No registered users and 1 guest