media reports are reasonably accurate vs peer review

Medical guidelines should insists on proof that time-honored medical practices and procedures that cost money and may harm or kill patients are actually effective. This Forum is about how to force organized veterinary medicine to issue Evidence Based Guidelines.

media reports are reasonably accurate vs peer review

Postby malernee » Tue Apr 27, 2004 12:53 pm

CMAJ • April 27, 2004; 170 (9). doi:10.1053/cmaj.1040005.
articles found in:
--------------------------------------------------------------------------------
Commentary
--------------------------------------------------------------------------------
Commentaire

Science reporting to the public: Does the message get twisted?
Celeste Condit
From the Department of Speech Communication, University of Georgia, Athens, Ga.

Lay people get a substantial amount of information about health and related topics from the media. Communicating with the public through the media can be vexing for medical professionals because they lack direct control over the final reporting. It is the reporter's framing of the information and his or her words that reach the public, rather than the scientist's or the clinician's. Moreover, there is a mismatch between the expectations of the scientist and those of the reporter. From the medical researcher's perspective, the news should be a sort of double helix, with media reports correctly matching, letter for letter, the original scientific publication. But this ideal is unachievable because the public cannot understand the language of an increasingly subspecialized scientific enterprise. Even scientists have trouble communicating across subdisciplines. Given that the ideal cannot be realized, what standards should be used for judging reporting about medical science and how well is current reporting meeting those standards?

The standards of accuracy applied in the popular press are set by the need for reporters to translate the precisely honed technical descriptions found in scientific writing into lively and clear summaries, using lay vocabulary. At best, this standard includes fidelity to sources, a balance among and inclusion of different viewpoints, and a translation that conveys some main idea from a study clearly and without the kind of distortions that might encourage inappropriate (or even dangerous) behaviour or unrealistic expectations.

Given these differences in expectations and the lack of personal control, it is not surprising that medical researchers are quick to blame the media for problems with public communication and to assume that media coverage is characterized by inaccuracy. However, the research that Tania Bubela and Timothy Caulfield1 present in this issue (page 1399) suggests that these assumptions are not well founded. Their results indicate that media reports are reasonably accurate, except in specific types of controversial areas, and that cases of inaccuracy may be as much a product of the researcher's overenthusiasm as of error by the reporter. These findings are consonant with related research.

Early social scientific research on genetics reporting tended to assume that the media were "getting it wrong,"2 but it soon became clear that anecdotal and sweeping assessments were subject to observer bias. It was all too natural for critics to notice and reprint examples of egregious reporting, portraying these as typical of the rapidly burgeoning area of genetics reporting. Similarly, these critics applied their own assumptions — usually that any favourable reporting about genetics was undesirable — to condemn all reporting about genetics as bad, simply because much of it contained favourable elements. More recent research has used quantitative measures, paid attention to sampling and generalizability, used standardized and accepted measuring instruments, checked coder reliability and used increasingly sophisticated analytic methods such as the classification and regression tree analysis applied in the study by Bubela and Caulfield.1 Such research highlights that studying media coverage of genetics is as complex as the genetics itself, for communication is not a simple, linear process.

Bubela and Caulfield explicitly address the question of the fidelity of news reports to the scientific reports on which they are based. They find that disagreement between scientific findings and media reporting is rare. It might well be, however, that both journalists and scientists offer an exaggerated vision of the prospects of genetic medicine. The methods used by Bubela and Caulfield would not define such congruent exaggeration as inaccuracy. But, using different methods and a different sample, Mountcastle-Shah and associates3 found a similar level of inaccuracy and noted that exaggeration by the media occurred in only a minority of news reports.

Another possibility is that press coverage could be faulted for being "unbalanced." That is, even though it might not exaggerate wildly or contain blatantly inaccurate statements, it might have a pro-genetics or anti-genetics slant. In fact, an overly optimistic slant has been detected in most studies.4,5,6,7 This lack of balance includes a tendency to quote from scientists more than from other sources4,5,7 and a failure to include topics such as potential risks2 or specific ethical considerations.8

In addition to difficulties in defining the criteria by which press coverage of science should be assessed, there is evidence of differences among specific newspapers and across specific topics, as Bubela and Caulfield's data suggest.1 Differences have also been found between different types of stories, coverage by different media and coverage published at different times: generally, "hard" news reports are more accurate than feature stories,9 print media are more accurate than television,10 and later coverage is more accurate than earlier coverage.5,7

The use of exaggeration or slant (whereby some features are ignored and others are overemphasized) may be motivated by the conflicting responsibilities faced by both scientists and journalists, as a study by Wilcox11 has made clear. She noted that reporters need to gain newspaper space (and ultimately an audience) for their topics, so they are prone to include sensationalistic, absolutist or at least dramatic statements. This drive conflicts with the norms of science journalism, which encourage cautious, detailed, balanced reporting, thus reflecting the norms of the science that is covered. Such conflicts may become evident as inconsistencies in coverage. In the interview setting, these dynamics are often manifested as reporters' efforts to get researchers to state the practical implications of the research, and the subsequent tendency to portray these applications as more general and perhaps more immediate than they may well be.2 Similarly, researchers may be influenced by conflict between their responsibilities to the norms of science and their desire for academic promotion, grants for the maintenance of research staff and laboratories, or simply personal financial gain though patents and royalties.12 They may also be under pressure from commercial sponsors of their research.

What then does the line of research represented by Bubela and Caulfield's study mean for medical researchers trying to communicate with the public about health? First, researchers should assume that what they say in their scientific publications may make its way into the public sphere and that press coverage may treat speculative "discussion" sections as fact. Second, researchers should talk to reporters. The public, who fund research with their hard-earned tax dollars, and patients, who indirectly fund research by purchasing products developed by private industry, have a right to know about that research, and news reporters are a major conduit. But researchers must prepare for such interviews as carefully as they would prepare for a talk at a scientific conference. The researcher should know exactly what she or he intends to communicate to the reporter and should resist the reporter's efforts to gain commentary that is different from what the researcher wishes to communicate. Detailed guidance about what is desired by lay audiences, as agreed upon by scientists and reporters, is provided by Mountcastle-Shah and associates;3 noteworthy in their catalogue are replication status, opinion of outside experts, prevalence of the genotype and phenotypes, and symptoms of disease. It is perfectly reasonable to ask a reporter to send written questions in advance of the interview. When that is not possible, the researcher should at least be prepared to avoid answering the reporter's inevitable question — "What is this good for?" — with an enthusiastic forecast of potential applications. Given the dynamics of science reporting, the reporter will probably feature rosy forecasts if the scientist is willing to offer them, yet such forecasts may all too often come to be seen as broken promises. If it is appropriate to link the research to the development of medical treatments, supplying colourful metaphors that emphasize the distance from the ultimate goal may be a good way to do so. Thus, the researcher might say that this is one baby step on the long journey toward the cure for X or that it is one tiny piece in the giant jigsaw puzzle that might someday enable prevention or treatment of Y. Finally, if a reporter exaggerates or is inaccurate in the final story or broadcast, the researcher should let the reporter's editor know and should tell other researchers about the experience with that particular reporter.

An old communication aphorism has it that one "cannot not communicate." This is as true for medical researchers as it is for others; the corollary is that they must act responsibly in communicating with the media.

ß See related article page 1399


Footnotes

Competing interests: None declared.

Correspondence to: Dr. Celeste Condit, Department of Speech Communication, University of Georgia, 110 Terrell Hall, Athens GA 30602; fax 706 542-3262; ccondit@uga.edu


References


Bubela TM, Caulfield TA. Do the print media "hype" genetic research? A comparison of newspaper stories and peer-reviewed research papers. CMAJ 2004;170(9):1399-407.[Abstract/Free Full Text]
Nelkin D, Lindee S. The DNA mystique: the gene as cultural icon. New York: WH Freeman; 1995.
Mountcastle-Shah E, Tambor E, Bernhardt BA, Geller G, Karaliukas R, Ellison Rodgers J, et al. Assessing mass media reporting of disease-related genetic discoveries. Sci Commun 2003;24(4):458-78.
Conrad P, Weinberg D. Has the gene for alcoholism been discovered three times since 1980? Perspect Soc Probl 1996;8:3-25.
Sheedy KM. Transcending tragedy: a "comic" critique of public discourse about breast cancer genetics, 1987-97 [dissertation]. Athens (GA): University of Georgia; 2000.
Condit CM. What is "public opinion" about genetics? Nat Rev Genet 2001;2:811-5.[Medline]
Ten Eyck TA, Williment M. The national media and things genetic. Sci Commun 2003;25(2):129-52.
Craig D. Ethical language and themes in news coverage of genetic testing. J Mass Commun Q 2000;77:160-74.
Henderson LH, Kitzinger J. The human drama of genetics: "hard" and "soft" media representations of inherited breast cancer. Soc Health Illness 1999;21:560-78.
Condit CM, Ofulue N, Sheedy K. Determinism and mass media portrayals of genetics. Am J Hum Genet 1968;62:979-84.
Wilcox SA. Cultural context and the conventions of science journalism: drama and contradiction in media coverage of biological ideas about sexuality. Crit Stud Media Commun 2003;20(3):225-47.
Mitchell GR. Strategic deception: rhetoric, science and politics in missile defense advocacy. East Lansing (MI): Michigan State University Press; 2000.
malernee
Site Admin
 
Posts: 462
Joined: Wed Aug 13, 2003 5:56 pm

malernee

Postby malernee » Tue Apr 27, 2004 1:01 pm

CMAJ • April 27, 2004; 170 (9). doi:10.1053/cmaj.1040005.
articles found in:
--------------------------------------------------------------------------------
Commentary
--------------------------------------------------------------------------------
Commentaire

Science reporting to the public: Does the message get twisted?
Celeste Condit
From the Department of Speech Communication, University of Georgia, Athens, Ga.

Lay people get a substantial amount of information about health and related topics from the media. Communicating with the public through the media can be vexing for medical professionals because they lack direct control over the final reporting. It is the reporter's framing of the information and his or her words that reach the public, rather than the scientist's or the clinician's. Moreover, there is a mismatch between the expectations of the scientist and those of the reporter. From the medical researcher's perspective, the news should be a sort of double helix, with media reports correctly matching, letter for letter, the original scientific publication. But this ideal is unachievable because the public cannot understand the language of an increasingly subspecialized scientific enterprise. Even scientists have trouble communicating across subdisciplines. Given that the ideal cannot be realized, what standards should be used for judging reporting about medical science and how well is current reporting meeting those standards?

The standards of accuracy applied in the popular press are set by the need for reporters to translate the precisely honed technical descriptions found in scientific writing into lively and clear summaries, using lay vocabulary. At best, this standard includes fidelity to sources, a balance among and inclusion of different viewpoints, and a translation that conveys some main idea from a study clearly and without the kind of distortions that might encourage inappropriate (or even dangerous) behaviour or unrealistic expectations.

Given these differences in expectations and the lack of personal control, it is not surprising that medical researchers are quick to blame the media for problems with public communication and to assume that media coverage is characterized by inaccuracy. However, the research that Tania Bubela and Timothy Caulfield1 present in this issue (page 1399) suggests that these assumptions are not well founded. Their results indicate that media reports are reasonably accurate, except in specific types of controversial areas, and that cases of inaccuracy may be as much a product of the researcher's overenthusiasm as of error by the reporter. These findings are consonant with related research.

Early social scientific research on genetics reporting tended to assume that the media were "getting it wrong,"2 but it soon became clear that anecdotal and sweeping assessments were subject to observer bias. It was all too natural for critics to notice and reprint examples of egregious reporting, portraying these as typical of the rapidly burgeoning area of genetics reporting. Similarly, these critics applied their own assumptions — usually that any favourable reporting about genetics was undesirable — to condemn all reporting about genetics as bad, simply because much of it contained favourable elements. More recent research has used quantitative measures, paid attention to sampling and generalizability, used standardized and accepted measuring instruments, checked coder reliability and used increasingly sophisticated analytic methods such as the classification and regression tree analysis applied in the study by Bubela and Caulfield.1 Such research highlights that studying media coverage of genetics is as complex as the genetics itself, for communication is not a simple, linear process.

Bubela and Caulfield explicitly address the question of the fidelity of news reports to the scientific reports on which they are based. They find that disagreement between scientific findings and media reporting is rare. It might well be, however, that both journalists and scientists offer an exaggerated vision of the prospects of genetic medicine. The methods used by Bubela and Caulfield would not define such congruent exaggeration as inaccuracy. But, using different methods and a different sample, Mountcastle-Shah and associates3 found a similar level of inaccuracy and noted that exaggeration by the media occurred in only a minority of news reports.

Another possibility is that press coverage could be faulted for being "unbalanced." That is, even though it might not exaggerate wildly or contain blatantly inaccurate statements, it might have a pro-genetics or anti-genetics slant. In fact, an overly optimistic slant has been detected in most studies.4,5,6,7 This lack of balance includes a tendency to quote from scientists more than from other sources4,5,7 and a failure to include topics such as potential risks2 or specific ethical considerations.8

In addition to difficulties in defining the criteria by which press coverage of science should be assessed, there is evidence of differences among specific newspapers and across specific topics, as Bubela and Caulfield's data suggest.1 Differences have also been found between different types of stories, coverage by different media and coverage published at different times: generally, "hard" news reports are more accurate than feature stories,9 print media are more accurate than television,10 and later coverage is more accurate than earlier coverage.5,7

The use of exaggeration or slant (whereby some features are ignored and others are overemphasized) may be motivated by the conflicting responsibilities faced by both scientists and journalists, as a study by Wilcox11 has made clear. She noted that reporters need to gain newspaper space (and ultimately an audience) for their topics, so they are prone to include sensationalistic, absolutist or at least dramatic statements. This drive conflicts with the norms of science journalism, which encourage cautious, detailed, balanced reporting, thus reflecting the norms of the science that is covered. Such conflicts may become evident as inconsistencies in coverage. In the interview setting, these dynamics are often manifested as reporters' efforts to get researchers to state the practical implications of the research, and the subsequent tendency to portray these applications as more general and perhaps more immediate than they may well be.2 Similarly, researchers may be influenced by conflict between their responsibilities to the norms of science and their desire for academic promotion, grants for the maintenance of research staff and laboratories, or simply personal financial gain though patents and royalties.12 They may also be under pressure from commercial sponsors of their research.

What then does the line of research represented by Bubela and Caulfield's study mean for medical researchers trying to communicate with the public about health? First, researchers should assume that what they say in their scientific publications may make its way into the public sphere and that press coverage may treat speculative "discussion" sections as fact. Second, researchers should talk to reporters. The public, who fund research with their hard-earned tax dollars, and patients, who indirectly fund research by purchasing products developed by private industry, have a right to know about that research, and news reporters are a major conduit. But researchers must prepare for such interviews as carefully as they would prepare for a talk at a scientific conference. The researcher should know exactly what she or he intends to communicate to the reporter and should resist the reporter's efforts to gain commentary that is different from what the researcher wishes to communicate. Detailed guidance about what is desired by lay audiences, as agreed upon by scientists and reporters, is provided by Mountcastle-Shah and associates;3 noteworthy in their catalogue are replication status, opinion of outside experts, prevalence of the genotype and phenotypes, and symptoms of disease. It is perfectly reasonable to ask a reporter to send written questions in advance of the interview. When that is not possible, the researcher should at least be prepared to avoid answering the reporter's inevitable question — "What is this good for?" — with an enthusiastic forecast of potential applications. Given the dynamics of science reporting, the reporter will probably feature rosy forecasts if the scientist is willing to offer them, yet such forecasts may all too often come to be seen as broken promises. If it is appropriate to link the research to the development of medical treatments, supplying colourful metaphors that emphasize the distance from the ultimate goal may be a good way to do so. Thus, the researcher might say that this is one baby step on the long journey toward the cure for X or that it is one tiny piece in the giant jigsaw puzzle that might someday enable prevention or treatment of Y. Finally, if a reporter exaggerates or is inaccurate in the final story or broadcast, the researcher should let the reporter's editor know and should tell other researchers about the experience with that particular reporter.

An old communication aphorism has it that one "cannot not communicate." This is as true for medical researchers as it is for others; the corollary is that they must act responsibly in communicating with the media.

ß See related article page 1399


Footnotes

Competing interests: None declared.

Correspondence to: Dr. Celeste Condit, Department of Speech Communication, University of Georgia, 110 Terrell Hall, Athens GA 30602; fax 706 542-3262; ccondit@uga.edu


References


Bubela TM, Caulfield TA. Do the print media "hype" genetic research? A comparison of newspaper stories and peer-reviewed research papers. CMAJ 2004;170(9):1399-407.[Abstract/Free Full Text]
Nelkin D, Lindee S. The DNA mystique: the gene as cultural icon. New York: WH Freeman; 1995.
Mountcastle-Shah E, Tambor E, Bernhardt BA, Geller G, Karaliukas R, Ellison Rodgers J, et al. Assessing mass media reporting of disease-related genetic discoveries. Sci Commun 2003;24(4):458-78.
Conrad P, Weinberg D. Has the gene for alcoholism been discovered three times since 1980? Perspect Soc Probl 1996;8:3-25.
Sheedy KM. Transcending tragedy: a "comic" critique of public discourse about breast cancer genetics, 1987-97 [dissertation]. Athens (GA): University of Georgia; 2000.
Condit CM. What is "public opinion" about genetics? Nat Rev Genet 2001;2:811-5.[Medline]
Ten Eyck TA, Williment M. The national media and things genetic. Sci Commun 2003;25(2):129-52.
Craig D. Ethical language and themes in news coverage of genetic testing. J Mass Commun Q 2000;77:160-74.
Henderson LH, Kitzinger J. The human drama of genetics: "hard" and "soft" media representations of inherited breast cancer. Soc Health Illness 1999;21:560-78.
Condit CM, Ofulue N, Sheedy K. Determinism and mass media portrayals of genetics. Am J Hum Genet 1968;62:979-84.
Wilcox SA. Cultural context and the conventions of science journalism: drama and contradiction in media coverage of biological ideas about sexuality. Crit Stud Media Commun 2003;20(3):225-47.
Mitchell GR. Strategic deception: rhetoric, science and politics in missile defense advocacy. East Lansing (MI): Michigan State University Press; 2000.

CMAJ • April 27, 2004; 170 (9). doi:10.1053/cmaj.1030762.
© 2004 Canadian Medical Association or its licensors
Abstract of this Article - Résumé de cet article
PDF Version of this Article
- Tiré à part (PDF) de cet article
Email this article to a friend
eLetters: Submit a response to this article
- Lettres électroniques : répondre à cet article
Related articles in eCMAJ
Similar articles found in:
eCMAJ
This Article has been cited by - Cet article a été cité dans:
other online articles - autres articles en direct
Search Medline for articles by
- Chercher dans Medline des articles de:
Bubela, T. M. || Caulfield, T. A.
Alert me when - Veuillez m'avertir si:
new articles cite this article
- de nouveaux articles citent cet article

Download to Citation Manager
- Télécharger à un logiciel de gestion des citations

Collections under which this article appears
- Collections dans lesquelles cet article paraît:
Research & Publication ethics
Other medical informatics

--------------------------------------------------------------------------------
Research
--------------------------------------------------------------------------------
Recherche

Do the print media "hype" genetic research? A comparison of newspaper stories and peer-reviewed research papers
Tania M. Bubela and Timothy A. Caulfield

Abstract
Top
Abstract
Methods
Results
Interpretation
Appendix 1
Appendix 2
References


Background: The public gets most of its information about genetic research from the media. It has been suggested that media representations may involve exaggeration, called "genohype." To examine the accuracy and nature of media coverage of genetic research, we reviewed the reporting of single-gene discoveries and associated technologies in major daily newspapers in Canada, the United States, Great Britain and Australia.

Methods: We used neutral search terms to identify articles about gene discoveries and associated technologies hosted on the Dow Jones Interactive and Canadian NewsDisk databases from January 1995 to June 2001. We compared the contents, claims and conclusions of the scientific journal article with those of the associated newspaper article. Coders subjectively assigned the newspaper articles to 1 of 3 categories: moderately to highly exaggerated claims, slightly exaggerated claims or no exaggerated claims. We used classification tree software to identify the variables that contributed to the assignment of each newspaper article to 1 of the 3 categories: attention structure (positioning in the newspaper and length of the article), authorship, research topic, source of information other than the scientific paper, type and likelihood of risks and benefits, discussion of controversy, valuation tone (positive or negative), framing (e.g., description of research, celebration of progress, report of economic prospects or ethical perspective), technical accuracy (either omissions or errors that changed the description of the methods or interpretation of the results) and use of metaphors.

Results: We examined 627 newspaper articles reporting on 111 papers published in 24 scientific and medical journals. Only 11% of the newspaper articles were categorized as having moderately to highly exaggerated claims; the majority were categorized as having no claims (63%) or slightly exaggerated claims (26%). The classification analysis ranked the reporting of risks as the most important variable in determining the categorization of newspaper articles. Only 15% of the newspaper articles and 5% of the scientific journal articles discussed costs or risks, whereas 97% of the newspaper articles and 98% of the scientific journal articles discussed the likelihood of benefits of the research.

Interpretation: Our data suggest that the majority of newspaper articles accurately convey the results of and reflect the claims made in scientific journal articles. Our study also highlights an overemphasis on benefits and under-representation of risks in both scientific and newspaper articles. The cause and nature of this trend is uncertain.



--------------------------------------------------------------------------------
The public gets most of its information about genetics from television, radio, magazines and newspapers.1,2,3,4 It has been suggested that media representations of genetics are inaccurate or exaggerated and lead to a phenomenon that has been called "genohype," 5,6,7 the "hyped" portrayal of both the benefits and risks associated with genetic research and the application of genetic technologies. It has also been suggested that this sensationalization of genetics may have an adverse impact on the public's ability to participate in policy discussions1,8 and on the utilization of genetic services because it creates inflated perceptions of the value of, for example, specific genetic tests.5,9
Several studies and commentaries have suggested various degrees and sources of reporting inaccuracies. For example, institutional press releases may be incomplete (with risks and limits being underplayed),10 media stories about medications often include inadequate or incomplete information about risks and costs,11 and abstracts from scientific meetings often receive a significant amount of media attention even though the validity and importance of the research has not "been established in the scientific community." 12 Some have speculated about the general weaknesses of the popular media's coverage of medical breakthroughs13,14 and their preference for positive over negative results.15

Surprisingly few systematic studies have examined the accuracy of media reporting in the context of genetics, most focusing on coverage of a single issue, such as sexual orientation16,17 or the discovery of susceptibility to breast and prostate cancer.18,19 The available data are instructive but not definitive.14,16,17,18,19,20

We examined broadsheet newspaper coverage of genetic research in Canada, the United States, Great Britain and Australia, countries with a scientific community heavily involved in genetic research. We attempted to measure media "hype" and the factors that contribute to inaccuracy and exaggeration. In addition, we attempted to gain insight into the possible source (researchers, scientific journals or the media) and nature of any "hype."


Methods
Top
Abstract
Methods
Results
Interpretation
Appendix 1
Appendix 2
References


The search strategy identified gene discovery stories in the print media. We used neutral search terms — ("gene" or "genes" or "genetics") and "discovery" and ("scientist" or "scientific") and "research" — to identify suitable newspaper articles hosted on the Dow Jones Interactive and Canadian NewsDisk databases from January 1995 to June 2001. These databases offered the most complete collections of Canadian newspapers and included stories published before and after 2 landmark genetic events, the cloning of Dolly the sheep in 1997 and the completion of mapping of the human genome in 1999. Both of these events were highly publicized and probably increased the public's interest in news coverage of genetics. We excluded articles on reproductive cloning because cloning was not the focus of this study and because the large number of newspaper articles on this subject had the potential to skew the results of the study. Although newspapers represent just one of many relevant media, we chose to study newspaper coverage because existing databases facilitated the comprehensive collection and analysis of relevant articles.

We then identified all newspaper stories that had as their source a paper in a peer-reviewed scientific journal. We used specific search terms to locate the scientific papers in MEDLINE or PubMed and then to identify all associated print media articles in the newspaper databases. The specific search terms included the name of the journal, the name of one of the researchers, the name of the institution where the research had been conducted, and the name of the gene, disease or genetic technology. We limited our study to all articles appearing in 26 newspapers from 4 countries. The broadsheet newspapers selected from the United States, Great Britain and Australia were chosen because they were national in distribution, had a large circulation, were of high quality, printed articles from one of the major newswire services and were roughly equivalent to the Canadian newspapers. We included most major Canadian local and national papers to ensure regional representation and thus captured most articles on gene discoveries and related technologies that were printed in that country. If an article appeared in more than one newspaper, it was coded only once; in such cases, priority was given to articles published in 1 of the 2 national newspapers, the National Post and the Globe and Mail.

We developed a coding frame, based on a previously published European media study,21 to compare the content of the scientific paper with the newspaper articles it generated. The coding frame consisted of a series of questions with standardized categorical responses on identifying information for the article; attention structure (positioning in the newspaper and length of the article); authorship; content; source of information other than the scientific paper; assessment of risk, benefits or controversy; valuation tone; framing; accuracy; and use of metaphors (see Appendix 1). All questions except those on framing, authorship and attention structure were then repeated for the scientific paper.

The coders, all of whom had scientific backgrounds, were asked to subjectively assess both the technical accuracy of the newspaper article and whether the claims made in the newspaper article reflected those in the scientific journal article. For the latter, the coders assigned each newspaper article to 1 of 3 categories: moderately to highly exaggerated claims, slightly exaggerated claims and no exaggerated claims. We assumed that there were no technical errors in the scientific journal articles. A newspaper article was considered not to have been exaggerated if its claims had first been made in the corresponding scientific journal article.

We conducted a pilot study with 2 independent coders to test the coding frame. For the study itself, we used 3 independent coders who were not familiar with the project and who had a strong background in genetics and biotechnology. We held 2 moderated meetings early in the study so that the coders could discuss their interpretation of the coding frame.

To assess intercoder reliability, we asked each of the 3 coders to assess the same random selection of 84 (13% of the total) newspaper articles and their associated scientific papers. We calculated intraclass correlation coefficients (model 2)22 for questions in the coding frame with mutually exclusive response. In all cases the coefficient was greater than 0.75 (Table 1), which indicates good agreement.22




View this table:
[in this window]
[in a new window]
Table 1.





We performed a classification tree analysis, using CART classification and regression tree software, version 4.0,23,24,25 to determine which variables from the coding frame contributed to the assignment of a newspaper article to 1 of the 3 categories of claims. This nonparametric statistical analysis has been used extensively for clinical risk assessment.25,26,27,28 The CART software uses specific criteria to split the data into 2 groups with greater homogeneity than the original group.24,25 Each group is then split into increasingly homogeneous groups until stopping criteria are reached and a set of terminal groups is produced. The variables are then ranked according to their importance as "splitting variables" (i.e., the relative contribution of each variable in assignment of the newspaper articles to the 3 categories).


Results
Top
Abstract
Methods
Results
Interpretation
Appendix 1
Appendix 2
References


We examined 627 newspaper articles reporting on 111 papers from 24 scientific and medical journals. The most commonly cited scientific journals were Science (196 newspaper articles [31%]), Nature (120 [19%]), Nature Genetics (101 [16%]) and Cell (100 [16%]). Two hundred and seventy-six (44%) of the articles appeared in the A section of the newspaper, and 142 (23%) appeared on the first page.

Twenty percent (127) of the newspaper articles were written by clearly identifiable, in-house specialist science and technology or medical journalists, whereas 22% (138) were identified as being from a newswire service; the latter were almost exclusively brief announcements of a gene discovery with little or no analysis.

In 95% of the newspaper articles (591/625), the main source of information cited was the scientific paper or its authors (see Appendix 1). This is not surprising, given our search strategy. Only 2 newspaper articles cited press releases as their source. Public sector research scientists (those perceived to be associated with a public sector employer, such as a university) were interviewed or cited in 90% of the newspaper articles (563/626), and private sector scientists in 5% (33/626). Opinions outside the research community or biotechnology sector were sought in only 8% (50/626) of newspaper articles, and these opinions were considered the main "voice" in only 2% (14/626) of the newspaper articles. Ninety percent of the newspaper articles (564/627) were framed as either a description of the basic research or as a celebration of progress or new research developments (Appendix 1). Most of the articles (483 [77%]) were not framed as a controversy; only 14% (85) were framed as a balanced controversy and 9% (59) as an imbalanced controversy.

The majority of newspaper articles were categorized as having no exaggerated claims (393 [63%]) or slightly exaggerated claims (165 [26%]). Only 11% (69) of the newspaper articles were categorized as having moderately to highly exaggerated claims (Table 1). Similarly, 82% (513) of the articles had no significant technical or scientific errors, and 18% (112) had 1 to 3 such errors. Only 2 of the newspaper articles contained more than 3 significant errors. Canadian newspapers, both in general and in comparison with those in other countries, had low levels of exaggerated claims in their reporting (Fig. 1).





View larger version (23K):
[in this window]
[in a new window]
Fig. 1: Degree of exaggeration of claims in newspaper articles about genetics published in Canada, the United States, the United Kingdom and Australia, relative to their sources (scientific journal articles). The n values represent the number of articles from each newspaper.





Forty-six (41%) of the 111 scientific papers reported research on human genetics, 40 (36%) used a nonhuman model, and 25 (23%) used both (Table 1). However, 87% (97/111) of the scientific papers and 98% (612/627) of the newspaper articles extrapolated the results to humans, regardless of whether the research model was human.

Beginning with the entire sample, CART identified the variable that best split the data into 2 subgroups with greater homogeneity than the whole and then repeated the process to create a classification tree (see Appendix 2). The relative contribution of the variables (their importance) as primary splitting variables was used in determining the category of exaggerated claims for the newspaper articles (Table 2). The misclassification cost for the learn data indicated a good fit of the model to the data: 0.25 for moderately to highly exaggerated claims, 0.24 for slightly exaggerated claims and 0.13 for no exaggerated claims.




View this table:
[in this window]
[in a new window]
Table 2.





The most important variable in determining the categorization of newspaper articles was the likelihood of risks or costs being reported in the newspaper article, and the fourth most important variable was whether risks or costs were reported in the scientific paper (Table 2). Appendix 1 shows why mention of risk may be an important variable. Only 15% of the newspaper articles and just 5% of the scientific journal articles discussed costs or risks. Of the 94 newspaper articles that reported risks, 67 (71%) quantified the risk as very unlikely to somewhat likely, and 25 (27%) quantified the risk as very likely to already present. In the 6 scientific journal articles that mentioned risk, the assessment of risk ranged from very unlikely to very likely. In contrast, 97% of the newspaper articles and 98% of the scientific journal articles discussed the likelihood of benefits arising from the research (Appendix 1); the type and likelihood of benefit were therefore identified as less important in the CART analysis (Table 2). About 70% of the newspaper and scientific journal articles quantified these benefits as very likely or certain.

Also important were the themes of the scientific paper and the associated newspaper articles (Table 2). Newspaper articles on behavioural genetics (e.g., genes for sexual orientation, alcoholism, mental illness or criminality), genetically modified organisms (e.g., "glow-in-the-dark" rhesus monkeys or genetically modified killer pox virus), longevity (e.g., clock genes) or reproductive technologies (e.g., ooplasmic transplantation) were more likely to include moderately to highly exaggerated claims. Newspaper articles on diseases such as obesity were also more likely to include exaggerated claims, whereas articles on life-threatening and prevalent diseases such as cancer, stroke and heart disease did not include exaggerated claims (data not shown).

Our data may also indicate a more subtle form of media hype, in terms of what research newspapers choose to cover. Behavioural genetics and neurogenetics were the subject of 16% of the newspaper articles. A search of PubMed on May 30, 2003, with the term "genetics" yielded 1 175 855 hits, and searches with the terms "behavioural genetics" and "neurogenetics" yielded a total of 3587 hits (less than 1% of the hits for "genetics"). Although the latter is probably an underrepresentation of primary research papers in these fields and the former an overrepresentation of all research in genetics, the magnitude of the difference suggests that behavioural genetics is receiving a disproportionately large amount of coverage in the media.


Interpretation
Top
Abstract
Methods
Results
Interpretation
Appendix 1
Appendix 2
References


Our data suggest that most newspaper articles accurately convey the results of and reflect the claims made in scientific journal articles. This high degree of conformity is consistent with other research results. For instance, Wilkes and Kravitz29 found that most first authors (as listed in the byline) had a generally positive experience with the media, and 86% rated coverage of their scientific studies as accurate; only 3% called the coverage inaccurate. This level of accuracy corresponds well with our finding that most newspaper articles had no exaggerated claims (63%) or only slightly exaggerated claims (26%). However, the media do seem to overemphasize particular topics, such as behavioural genetics. The high profile of these types of stories may be one reason for the perception that newspaper stories are often hyped. In other words, although we found that only 11% of the newspaper articles had moderately or highly exaggerated claims, these few stories might have a significant impact on public perceptions. As noted by Ransohoff and Ransohoff,8 "While most scientific writing is done well, the current level of exaggeration, even if infrequent, may discredit good reporting."

In our study the 2 most important variables in determining the category of exaggeration were the reporting of risks and costs and the topic of the research. Our study also highlights an underreporting of risks and an overemphasis on benefits. A small proportion of the newspaper articles (15%) and an even smaller proportion of the scientific journal articles (5%) dealt with risks. Again, this finding is consistent with existing data. For example, Moynihan and associates11 found that only 15% of 207 news stories on drugs used for disease prevention presented both relative and absolute benefits. Similarly, in a study of Canadian newspaper coverage of new prescription drugs, all articles mentioned some benefit, but the majority made no mention of side effects or harms.30 Our study corroborates concern about the lack of reporting of risk, in both the scientific literature and newspaper coverage. In the rare instances that risk is reported in either type of article, there is a greater likelihood of exaggeration. In contrast, benefits are emphasized in both scientific and newspaper articles. This trend may contribute to a general hyping of genetic research, potentially inflating the expectations of the general public and special interest groups such as patient groups and investors.

Our study had a number of limitations. First, we surveyed only the print media and limited our analysis to the more respected newspaper publications, which do not necessarily have circulation levels as high as some tabloids. Second, our study was limited to newspaper stories that were directly related to peer-reviewed articles and published abstracts. As a result, stories flowing from other sources, such as abstracts from scientific meetings, which might never be published,12 may be underrepresented. Also, all of our coders had strong scientific backgrounds, which undoubtedly brought a certain perspective to the interpretation of both the newspaper stories and the scientific articles.

Nevertheless, our data indicate that journalists may not always be the primary source of exaggerated claims. Although more research is required to confirm the nature and cause of this trend (e.g., investigating the editorial practices of the top journals and analyzing other media such as television and the Internet), a reasonable interpretation is that the media, scientific journals and the scientific community at large may be inadvertent "complicit collaborators" in the subtle hyping of science stories.8 Our data also raise interesting questions about how stories are selected and edited by the top scientific journals (e.g., the possibility of selection bias) and how the research community may "sell" science to the public, to scientific and lay publications, and to research funders.31,32,33,34

ß See related article page 1415


Appendix 1
Top
Abstract
Methods
Results
Interpretation
Appendix 1
Appendix 2
References







View this table:
[in this window]
[in a new window]
Appendix 1.





Appendix 2
Top
Abstract
Methods
Results
Interpretation
Appendix 1
Appendix 2
References






View this table:
[in this window]
[in a new window]
Appendix 2: Classification tree showing the classification of newspaper articles into 1 of 3 categories: class 1 = moderately to highly exaggerated claims (relative to scientific article); class 2 = slightly exaggerated claims, class 3 = no exaggerated claims.
The tree shown is the smallest tree within 1 standard error of the "%relative cost"% (i.e., misclassification rate) of the smallest tree determined by 10-fold cross-validation. Twoing for ranked data was the splitting method.24,25 The following paragraphs describe how the classification software functions (for an overview of the software used in this analysis, see www.salford-systems.com/products-cart.html) and details the left-most partitions of the tree.

The construction of classification and regression trees (CARTs) has become a common method for building statistical models from simple data. The results can be represented as binary trees, permitting simple graphic presentation, even when many variables are under consideration.

At each branching point or node of the decision tree is a binary question or statement (with a Yes or No answer) about some feature of the data set. The terminal groups or "%leaves"% of the tree represent the best split of the data, according to the "%learn"% or "%training"% data set. A terminal group may be a single member of some class (as in this analysis), a probability density function or a predicted mean value for a continuous variable. The basic CART algorithm is given a set of samples and instructed to find some variable that splits the data so as to maximize the differences (and minimize the similarity or "%impurity"%) of the 2 partitions. This splitting of the data into groups is applied recursively until some stopping criterion is reached, such as a minimum number of samples in an individual partition (or branch of the tree).

Because our purpose was to discriminate between 3 categories of newspaper articles (those with moderately to highly exaggerated claims, those with slightly exaggerated claims and those with no exaggerated claims), we used the classification tree analysis to determine the contribution of the descriptor variables listed in Table 2 to the assignment of the respective newspaper articles to the 3 categories.

The interpretation of this tree is straightforward. The left-hand partitions of the tree are described here in some detail, and the other branches can be interpreted in a similar fashion. The relative contribution of each variable to the overall shape of the tree is presented in Table 2 (as the variable importance score). At the top of the tree, all of the data are best divided into 2 groups according to the likelihood of risks or costs being reported in the scientific paper. The next node (branching point) to the left splits the data on the basis of the likelihood of risks or costs being reported in the newspaper article, and the same criterion applies to the third node. Year of publication is the variable that splits the remaining data into 2 penultimate groups. At the terminal split, the final groups (or "%leaves"% of the tree) are determined by the primary theme reported in the newspaper article. Node 1 (n = 29) corresponds to cases with slightly exaggerated claims in the newspaper article, whereas node 2 (n = 14) corresponds to cases with no exaggerated claims. Node 2 is "%pure,"% in that all cases were assigned to class 3 (100%); however, node 1 is "%mixed"% because both class 2 (48.3%) and class 3 (51.7%) are represented. Further splitting would resolve these apparent impurities, but the CART pruning and stopping rules that we employed determined that this was the most appropriate terminal group







Footnotes

This article has been peer reviewed.

Contributors: Both authors contributed to the study conception and design and the data analysis and interpretation. Both drafted and revised the article and approved the final version.

Acknowledgements: We thank Edna Einsiedel and Usher Fleising for study design; Tim Karels for statistical advice; David Hik for statistical advice and editorial comments; Jordan Ward, Christine Caulfield and Qusair Mohammedbai for coding; and Nina Hawkins at the Health Law Institute for administrative assistance.

This study was supported by Genome Prairie, the Social Sciences and Humanities Research Council of Canada, the Alberta Heritage Foundation for Medical Research and the Stem Cell Network.

Competing interests: None declared.

Correspondence to: Timothy A. Caulfield, Health Law Institute, Law Centre, University of Alberta, 91 University Campus NW, Edmonton AB T6G 2H5; fax 780 492-9575; tcaulfld@law.ualberta.ca



References
Top
Abstract
Methods
Results
Interpretation
Appendix 1
Appendix 2
References




Geller G, Bernhardt BA, Holtzman NA. The media and public reaction to genetic research. JAMA 2002;287:773.[Free Full Text]
National Science Foundation, Division of Science Resources Statistics. Science and engineering indicators 2002. Arlington (VA): The Foundation; 2002 Apr. Available: www.nsf.gov/sbe/srs/seind02/pdfstart.htm (accessed 2003 Dec 3).
Science and the public: a review of science communication and public attitudes to science in Britain. London (UK): Wellcome Trust; Office of Science Technology; 2000. Available: www.wellcome.ac.uk/en/images/sciencepublic_3391.pdf (accessed 2004 Mar 26).
Biotechnology Australia. Biotechnology public awareness survey: final report. Sydney: Millward Brown Australia; 2001. Available: www.biotechnology.gov.au/library/conten ... esults.pdf (accessed 2004 Feb 20).
Caulfield T. Underwhelmed: hyperbole, regulatory policy and the genetic revolution. McGill Law J 2000;45:437-60.[Medline]
Holtzman NA. Are genetic tests adequately regulated? [editorial]. Science 1999; 286:409.[Free Full Text]
Fleising U. In search of genohype: a content analysis of biotechnology company documents. New Genet Soc 2001;20:239-54.
Ransohoff DF, Ransohoff RM. Sensationalism in the media: when scientists and journalists may be complicit collaborators. Eff Clin Pract 2001;4:185-8.[Medline]
Melzer D, Zimmern R. Genetics and medicalisation [editorial]. BMJ 2002; 324: 863-4.[Free Full Text]
Woloshin S, Schwartz LM. Press releases: translating research into news. JAMA 2002;287:2856-8.[Abstract/Free Full Text]
Moynihan R, Bero CL, Ross-Degnan D, Henry D, Lee K, Watkins J, et al. Coverage by the news media of the benefits and risks of medications. N Engl J Med 2000;342:1645-50.[Abstract/Free Full Text]
Schwartz L, Woloshin S, Baczek L. Media coverage of scientific meetings. JAMA 2002;287:2859-63.[Abstract/Free Full Text]
Sibbison J. Covering medical breakthroughs. Columbia J Rev 1988:36-9.
Loo LK, Byrne JM, Hardin SB, Castro D, Fisher FP. Reporting medical information: Does the lay press get it right? [abstract]. J Gen Intern Med 1998; 13 (Suppl 1):60.[Medline]
Koren G, Klein N. Bias against negative studies in newspaper reports of medical research. JAMA 1991;266:1824-6.[Abstract]
Conrad P, Markens S. Constructing the "gay gene" in the news: optimism and skepticism in the US and British press. Health 2001;5:373-400.
Petersen A. The portrayal of research into genetic-based differences of sex and sexual orientation: a study of "popular" science journals, 1980 to 1997. J Commun Inq 1999;23:163-82.
Bernhardt BA, Geller G, Tambor E, Mountcastle-Shah E, Mull JG, Holtzman NA. Analysis of media reports on the discovery of breast and prostrate cancer susceptibility [abstract 268]. Am J Hum Genet 2000;67(Suppl 2):62.
Henderson L, Kitzinger J. The human drama of genetics: "hard" and "soft" media representations of inherited breast cancer. Sociol Health Illness 1999; 21: 560-78.
Condit C, Ofulue N, Sheedy KM. Determinism and mass-media portrayals of genetics. Am J Hum Genet 1998;62:979-84.[Medline]
Gaskell G, Bauer MW, editors. Biotechnology 1996–2000: the years of controversy. London: Science Museum; 2001.
Portney LG, Watkins MP. Foundations of clinical research: applications to practice. 2nd ed. Upper Saddle River (NJ): Prentice-Hall; 2000.
Steinberg D, Colla P. CART — classification and regression trees. San Diego: Salford Systems; 1997.
Breiman L, Friedman JH, Olshen RA, Stone CJ. Classification and regression trees. Belmont (CA): Wadsworth International; 1984.
CART product overview [on-line]. San Diego: Salford Systems. Available: www.salford-systems.com/products-cart.html (accessed 2004 Mar 3).
Begg CB. Statistical methods in medical diagnosis. Crit Rev Med Inform 1986;1:1-22.[Medline]
Steadman HJE, Silver J, Monahan PS, Appelbaum PC, Robbins EP, Mulvey T, et al. A classification tree approach to the development of actuarial violence risk assessment tools. Law Hum Behav 2000;24:83-100.[Medline]
Huland H. Radical prostatectomy: opinions and issues. Eur Urol 2001;39:3-9.[Medline]
Wilkes MS, Kravitz RL. Medical researchers and the media: attitudes toward public dissemination of research. JAMA 1992;268:999-1003.[Abstract]
Cassels A, Hughes MA, Cole C, Mintzes B, Lexchin J, McCormack JP. Drugs in the news: an analysis of Canadian newspaper coverage of new prescription drugs. CMAJ 2003;168(9):1133-7.[Abstract/Free Full Text]
Nelkin D. Selling science: how the press covers science and technology. New York: Freeman; 1995.
Lawrence PA. The politics of publication. Nature 2003;422:259-61.[Medline]
Media studies for scientists [editorial]. Nature 2002;416:461.
Condit C. What is "public opinion" about genetics? Nature Rev Genet 2001;2:811-5.[Medline]
malernee
Site Admin
 
Posts: 462
Joined: Wed Aug 13, 2003 5:56 pm


Return to evidence based vet guidelines

Who is online

Users browsing this forum: No registered users and 6 guests