A Primer on Critical Reading

The world is rife with inaccurate scientific claims about vegetarianism. Some of these are made by detractors and can make it difficult to be confident about being vegetarian. Others are made by supporters, creating a lack of credibility that tends to undermine the many valid arguments in favor of the cause. These sources of erroneous information present a serious challenge for those who want to expand the appeal of vegetarianism to the larger community. They also create a challenge for vegetarians who want good information about scientific issues, particularly nutrition and health. While no collection of popular information is flawless, the information about vegetarian diets and other alternative health practices seems particularly vulnerable to misinformation. The following offers some basic advice on resisting that misinformation. A longer version of this paper, with more detail, explanation, and references, is available as a Sci-Veg working paper.[1]

Don't Believe Everything You Read

Wait, don't turn the page yet. This is not as simple as it sounds.

Words get written down in many ways that do not promise accuracy. The word published, which researchers use to mean carefully reviewed by experts but many people think of as simply written down and widely distributed, creates some of this confusion. Anyone can write down almost anything and widely distribute it. Most people know that researchers put a lot of stock in something being published, but they do not realize what the researchers mean by that word.

For something to be turned into a book by a popular-press publishing house, it does not need accuracy so much as a patina of credibility along with the catchiness to sell a lot of books. Strong assertions that overstate the current scientific knowledge are usually a lot catchier than the conservative statements that would be more accurate.

"Veganism protects against colorectal cancer and possibly other cancers," will sell fewer books than "you can save yourself from breast cancer by eating a vegan diet, "even though the latter claim is clearly overstated and may be wholly wrong according to current knowledge. Sadly, in the realms of vegetarianism and alternative health, the slick and substantially inaccurate books probably outsell those that provide solid information by a hundred to one.

Putting a statement in a pamphlet or on the Internet does not even require convincing a publisher that it will sell. I can make any false statement that can be constructed in English and post it to an e-mail list or a nice looking Web page. It would appear there, looking professional, posted by a university professor, and it would still be worth as much as the paper it wasn't printed on. This is not to say that Web pages, pamphlets, popular-press books, or newspaper stories are always suspect--often the material is well-researched, honest, and accurate. But you need to think about where the information came from, who processed it, and what it is being used for. This article cites URLs, but notice they are material I wrote, effectively continuations of what you are already reading. Had I cited non-reviewed Web sites as scientific sources, it would have been a good clue that the information was suspect.

Good secondary research--research of the existing research rather than generation of new data--is the way that most of us learn about most things. A lay reader may misinterpret research publications without such an intermediary. Tentative findings based on a particular population may be published so other researchers can benefit from them, but they often get read and interpreted as having definitively proved something that generalizes to everyone. The key lesson is that no single study is worth very much. Another study may have very different results due to differences in methods or random chance. The popular press typically ignores this in its health and science reporting (and the researchers often do little to help). To make sense of the latest study, someone with an understanding of the research needs to review it in the context of other existing knowledge.

Your default attitude toward scientific claims in popular press sources (especially claims that contradict what you thought to be true) should be suspicion rather than acceptance. I say this as someone who writes a lot of things on the Internet and in non-reviewed forums, expecting people to believe me. This presents the dilemma that I will return to below: At some point, you have to believe some people, but how do you decide whom?

Not As Well Supported As They Look

A second important lesson is to recognize that most health findings are much less certain than reported. Statistical tests have rules about how they can be used. Those rules are seldom obeyed, resulting in gross overstatements of confidence. Most health researchers, let alone journalists or lay readers, do not realize this. Quite frequently, an exciting new discovery is due to random chance. (More technical details appear in the full version of this paper.)

A common error of both researchers and lay readers is to mistake a large sample size--the number of subjects studied--for a good study design. While it is true that a study with 10 subjects can tell you very little about the average person, a study with 1,000 subjects may not tell you much more. Large sample size does nothing to correct problems like measurement error (such as calling someone vegetarian when he or she really is not) and confounders (unmeasured variables that are related to both action and outcome, such as vegetarians being more health-conscious, making it difficult to determine how much of our better health is due to not eating meat and how much is due to other healthful behaviors). These are serious problems and result in almost all health research claims being much less certain than is generally claimed.

Another problem is sometimes called "data mining". If a researcher collects data on hundreds of foods and dozens of health measures, some of them will be related due to random chance. The usual test for statistical significance is that there is only a 5% chance a relationship is due to random chance. This means that even for completely unrelated (or even random) series of numbers, one out of 20 pairs will be significantly related (and quite possibly be published). If you took a group of people and compared 20 measures of health outcomes (like blood pressure, occurrence of cancer, etc.) to the last digit of their phone numbers, there would be, on average, one "significant" relationship. The same principle applies if instead of comparing the health outcomes to phone numbers you compared health outcomes to how much coffee someone drinks or how much seaweed he or she eats.

A less forgivable error, one that is all too frequent in the alternative health literature, is to keep treating someone until their condition improves, and then claim that the intervention--whether it be a raw food diet, eating according to blood type, fasting, taking supplements, or many others things--has cured them.[2] People undergoing interventions are usually selected because they were suffering a bout of health problems. Chances are they are healthier most of the time, and will be so again no matter what they do. If I pick ten people who have colds this week and paint their thumbs blue, chance are that I will succeed in curing most of them within a week. Time heals most ailments, and the physicians and gurus take the credit.

The worst problems of erroneous claims that appear to be well supported, though, are much less technical: authors picking and choosing citations based on the hypothesis they want to support. Someone can pretend to be doing unbiased secondary research when he or she is actually just collecting only the information that supports the claim he or she likes or the position he or she is advocating. Indeed, much of the vegetarian popular press seems to be written by authors who glom onto any statement, of whatever quality, that has positive implications for vegetarianism, without even trying to understand the full body of evidence in the field, evidence that frequently suggests they are wrong.

You might glance at the footnotes of a book and think ,"look at all the journal articles--this book is well researched."  But recall that no single study is worth very much. It might be impressive to see three studies linking dietary fat and cancer in a footnote, until you realize that there are thousands of studies that address this relationship. You can hunt down three of them that say anything. Even if the author reported one hundred such studies in the footnotes, you could not be sure there were not five hundred others that said just the opposite. Health research is a rather inexact business, and good studies can produce different results. Ultimately, the reader is really relying on the author's implicit assertion that he or she understood the full weight of evidence on a topic and represented it accurately.

Who To Believe

Ultimately, there are two paths to determining what to believe, and the first of them is seldom possible. The first path is knowing enough technical details about a topic or a method of inquiry to be able to judge it directly. But none of us can have the required level of expertise in more than a few areas. This leaves only the second path, figuring out who to believe. The preceding sections both ended with the need to believe someone. More than 99% of what each of us knows about science comes from what someone told us, even for those of us who spend most of our lives doing research. No matter how much we would like to find an objective alternative, we usually just have to believe someone. The question of who to believe is tough, but a few general rules can provide a start.

Authors credentials provide some insight. A university affiliation implies some connection with reviewed research. Someone associated with a government agency, a respected non-governmental organization, or a major company has some credibility at stake and was chosen to represent the organization, and so is probably not spouting complete nonsense. Beware if you cannot figure out why someone is claiming any expertise to write something or, as in case of many pamphlets or newsletter articles, you cannot even figure out who wrote it.

Many supposed credentials are not worth much. Many registered dietitians (RDs) have narrow knowledge bases, and in particular, may know little about vegetarianism. Training as an MD does not teach people how to do research and seldom contains much information about nutrition. A PhD is more promising because the essence of the degree is learning how to do and critically read research, both primary and secondary. But there is always a lot of variation, with some of the best and some of the worst information about vegetarianism coming from people with each of these credentials.

An author who makes major new scientific claims in the popular press, never having published the material in peer reviewed articles (a common practice in most of the self-help books about diet and health), is highly suspect. While a peer-reviewed study can never be definitive, a lack of any peer-reviewed study is quite damning. Usually the author knows the material will not stand up to scrutiny or does not care about getting a serious review by the scientific community because he or she would rather sell his ideas than make sure they are right. In some cases, he or she simply does not have any evidence to support his claims. Beware especially if the claims are strongly contrary to conventional wisdom. If the author's only argument is to imply that you would be naive and backward or a dupe of some institutional interest to believe the conventional wisdom rather than his or her exciting new theories, he or she probably does not have much of a case.

These general rules are clearly not enough. Ultimately, you have to learn which network of authors you trust. You will notice that in the vegetarian community, there are groups of authors and organizations who tend to quote and reference each other more often. These indicate networks of trust and mutual respect, and it is easier to recognize the shared credibility of a group than of a single author. To be satisfied with your trust in authors, you will have to draw your own conclusions. If you choose to trust me, I commend to you the authors who published in reviewed journals, clearly have a broad understanding of the published research, and have the scientific understanding to discuss the material. Among the (very) few vegetarian publications having this kind of credibility are those from the American Dietetic Association's Dietetic Practice Group on Vegetarian Nutrition, the Loma Linda University Vegetarian Nutrition & Health Letter, nutrition and food ingredient information from The Vegetarian Resource Group, and a concise general pamphlet, Vegan Outreach's "Why Vegan?"[3] An ongoing discussion with a generally high level of expertise can be found on the Sci-Veg email list.

Conclusion

To find reliable material, the first step is to find material with enough fundamental credibility that you believe that the author is not going to simply make things up. However, due to the author's lack of understanding about research methods, or the goal of persuading you of a certain claim, he or she can still mislead while sticking to the truth. In response, it helps to have a basic understanding of research methods and a few general rules about when an author is credible. But ultimately you will most often have to fall back on deciding who to believe rather than judging all the details.

One final note: People often ask why the researchers who are producing good information and honest analysis cannot just identify the bad information and refute it. The problem is that this is a completely hopeless battle. Even if the experts in vegetarian-related science had the inclination, they would simply be incapable of keeping up with all the pseudo-science and bad information. Thus, whatever the efforts of experts and reviewers, every reader will ultimately face the challenge of critically evaluating much of what he or she reads.