Track your comments!
[x]


When you register, comments on your articles and replies to your comments appear here. Register Now!

Sign in to your account
[x]

Not a Scientific Blogging member yet?

Register Now for a Free Scientificblogging.com Account

  • Customize your profile with pictures, banner, a blogroll and more.
  • Leave comments on articles, add other members to your friend lists, chat with people on the site.
  • Write blog posts that can be seen by hundreds of thousands of readers.

It's free and it only takes a minute!

Already a Scientific Blogging member?

Sign In Now

Fake Banner
By Gerhard Adam | October 26th 2009 01:08 PM | 9 comments | Print | E-mail | Track Comments
In light of recent discussions about vaccines and scientific consensus it is sobering to consider that fully one-third of medical studies turn out to be wrong.

"A study in 2005, published in the Journal of the American Medical Association, found that one-third of all medical studies turn out to be wrong."
http://www.livescience.com/health/091026-medical-research-fraud-errors.html


In addition, many remedies of non-scientific origin may be vindicated despite earlier claims from doctors.
"In 2007, scientists showed that honey works better than cough medicines in soothing children's coughs.


"A new review of studies finds 29 percent of cancer research published in high-profile journals had disclosed a conflict of interest.
While it's a good thing that the conflicts were disclosed, the review also found conflicts affect the research outcomes."

"The findings add to a mountain of evidence suggesting you should be skeptical of health and medical advice."

http://www.livescience.com/health/090511-medical-studies-conflict.html

While many may argue that these disclosures are important since they were revealed.  In addition one can hardly argue that all scientific research is flawed in this fashion.  Similarly, this doesn't represent a loop hole where every quack remedy suddenly acquires legitimacy.

In particular, the problem is more insidious because it undermines the very credibility that scientists and doctors rely on when attempting to deal with public health issues.  It is little wonder that counter claims regarding medical practice gain such traction when so much research can be questioned based on conflicts of interest.

One can certainly understand that sometimes errors can occur and peer review can also ensure errors or faulty research can be identified and addressed.  However, that process only works within the scientific community.  The far greater problem is how such events are viewed by the public which is expected to take advice and recommendations seriously.

Until this issue is addressed by the scientific peer community, one shouldn't be surprised when the general public views scientific pronouncements as debatable and subject to alternate interpretations.


Comments

Becky Jungbauer's picture
fully one-third of medical studies turn out to be wrong

Actually, the JAMA article doesn't say that. What it does say is that "controversy and uncertainty ensue when the results of clinical research on the effectiveness of interventions are subsequently contradicted. Controversies are most prominent when high-impact research is involved." This is no surprise. The authors also note that "controversies are most common with highly cited non-randomized studies, but even the most highly cited randomized trials may be challenged and refuted over time, especially small ones." Again, no surprise. That's why you do multiple trials - to make sure the effect you saw is correct. And you should read non-randomized trials, and other certain types of trials, with a grain of salt anyway. I feel like a statement like the one above hurts science more than it helps - people think, well, since scientists have a 33% chance of being wrong, why should I trust what I read? That's not true - we're constantly refining, and it's not black and white. Besides, the numbers can always be massaged. If you take wrong to mean contradicted, then in actuality only 15.5% of studies were contradicted by subsequent studies (7 of 45). Another 15.5% found that the effects were actually weaker than the first study suggested, which isn't wrong, just clarified. But even then, only 14 of 45 studies had different outcomes, great or small, from original studies, and we don't even know whether the subsequent studies were just more rigorous and therefore found more true effects, or were better designed, or what. That's 31% of studies, which means 69% of studies are not "wrong," no matter how you define it.


Gerhard Adam's picture
I understand, but as I said, the problem isn't the studies, but rather the feeling that is conveyed to the public by such reports.  In effect, whether the number is small or large, it suggests (right or wrong) that much that is reported with certainty may not be, while others which are more vague may be accurate.

In short, the issue isn't the science, but rather how it is presented to the public, especially when it is to be used to influence opinion and engage public policy.

jtwitten's picture
Leaving aside that discussing this issue without an in depth discussion of the scientific method, peer review, and that the "one-third wrong" is not randomly distributed through the literature threatens to increase misunderstanding, it seems that your point is best made by the 2006 and 2008 studies addressing how science is reported to the public, including not-disclosing conflicts of interest and not identifying results as unpublished (therefore, not peer-reviewed).
A study in 2006
found that out of 175 stories in the popular media that discussed
unpublished research, only two noted that the research was unpublished.

-http://www.livescience.com/health/090511-medical-studies-conflict.html

Gerhard Adam's picture
Leaving aside that discussing this issue without an in depth discussion
of the scientific method, peer review, and that the "one-third wrong"
is not randomly distributed through the literature threatens to
increase misunderstanding...

When it comes to public perception there is no issue about the scientific method, peer review, or the distribution of errors.  The point is simply that the public is being lead to believe that the information may be unreliable which promotes the incidences of seeking alternative choices. 

When this is coupled with personal experiences suggesting less than scrupulous care, the situation is primed to make people feel that there is no credibility behind authoritarian pronouncements.  In addition, the incidences at Love Canal and in Hinkley, CA (made famous by the Erin Brockovich movie), suggest that when monied interests are involved, then the public is expendable.

I realize that it isn't the responsibility of scientists, specifically, to engage in such social issues, but in every one of these cases, there were plenty of scientists that were willing to compromise the scientific method and peer review to advance the cause of corporations.  As I've said before, it doesn't matter whether this is right or wrong, but it can hardly be argued that the public doesn't have any basis for being suspicious. 

At this stage, people have been conditioned to discount almost anyone making claims when corporate money is involved.  You can see the controversy surrounding global climate change.  How much more significant is it when someone or their health may be specifically involved?

Even personally, I'm more than willing to be educated in areas where I lack expertise and to pursue the science.  However, I'm also skeptical about information that comes from big finance.  It isn't that I think corporations are intrinsically evil, but I'm not naive enough to believe that scientists are any different from others in wanting to preserve their jobs and career paths. 

Gerhard Adam's picture
I would make one other point; as far as I can tell the JAMA article requires a subscription.  This means that from the perspective of keeping the public informed about the state of research, it might as well not exist.

Therefore, unless I'm mistaken the only information that will be passed to the public is the 33% number.

Becky Jungbauer's picture
True. Which means we need reporters that are actually educated in science. The old guard has for too long fought back, saying all you have to do is ask the right questions, you don't need a degree. But how do you know the right questions if you don't know the subject?

Gerhard Adam's picture
I agree completely.  I would also add that the media needs to get over the issue of ratings or trying to draw in the largest audience.  While I can appreciate the business aspect of their operation, it compromises the reliability of the information they present.

The news reporting needs to be accurate and not assembled to offend the least number of people.  All too often, it is clear that what is presented is intended for broad appeal or even to generate controversy rather than providing useful information.  In this regard, the media is one of the primary offenders in compromising credibility.

kerrjac's picture
Keep in mind that the costs of being wrong are very variable across disciplines / research projects. Too often in science I think people are afraid of being 'wrong' without realizing just how low the stakes are. Aside from highly applied research like drug trials, I'd like to see scientists take more chances and really put themselves out there. Worst case scenario, you're wrong; best case, you're right.

In life that's how I often try to approach things, and so long as you keep the stakes in mind&your ego out of it, it seems to work pretty OK. Creativity, imagination, ideas like many of the best things in life are free, and their upside potential often - but not always - dwarfs their down-side.

Like Marty's girlfriend Jennifer in Back to the Future said to him, "Marty, one rejection isn't the end of the world."

Gerhard Adam's picture
I agree, that being wrong can be useful in itself.  One of the problems though is that science isn't simply about acquiring knowledge, or even discovering how nature works.  Science is a career which carries a whole lot of excess baggage which can undermine the quest for information and knowledge.

This creates the problem that funding may be limited as well as career opportunities if one is perceived as always being on the wrong side of experimental results.

Add a comment

The content of this field is kept private and will not be shown publicly.
  • Allowed HTML tags: <sup> <sub> <a> <em> <strong> <center> <cite> <code> <TH><ul> <ol> <li> <dl> <dt> <dd> <img> <br> <p> <blockquote> <strike> <object> <param> <embed> <del> <pre> <b> <i> <table> <tbody> <div> <tr> <td> <h1> <h2> <h3> <h4> <h5> <h6> <hr> <iframe>
  • Lines and paragraphs break automatically.
  • Web page addresses and e-mail addresses turn into links automatically.
CAPTCHA
If you register, you will never be bothered to prove you are human again. And you get a real editor toolbar to use instead of this HTML thing that wards off spam bots.