BBC Radio 4 - Saving Science from the Scientists

[semente]

I listened to episode 2 this morning while driving...

 

 

Saving Science from the Scientists

 

Episode 1

 

Is science quite as scientific as it's supposed to be?

After years of covering science in the news, Alok Jha began to wonder whether science is as rigorous as it should be, and in this two-part series, he will try to find out.

Many of us might be forgiven for assuming that the pursuit of scientific knowledge is a precise and controlled process, one that involves detailed experiments, careful analysis, peer review and demonstrable evidence. But what if it's not as simple as that?

Scientists are human beings after all, so what if they are prone to the same weaknesses, failings and uncertainties as everyone else? And what would that mean for their findings?

Alok delves into dodgy data, questionable practices and genuine ambiguity to ask if human decision making is impeding scientific progress, and if anything can be done about it.

Along the way he hears from academics who think almost all science is wrong, scientists who think the system is in crisis and those who say error and uncertainty are actually an integral part of science's creative process. He'll also talk to a former professor caught out after going to the ultimate extreme - faking his data - to find out what drives someone to betray their entire field.

 

BBC Radio 4 - Saving Science from the Scientists, Episode 1

 

 

Episode 2

 

In the second and concluding part of this series, Alok looks at the practices and cultures undermining the integrity of scientific research.

Are scientists being pushed into shortcuts and unethical behaviour by the competitiveness of their field?

 

BBC Radio 4 - Saving Science from the Scientists, Episode 2

[petaluma]

Alok delves into dodgy data, questionable practices and genuine ambiguity to ask if human decision making is impeding scientific progress, and if anything can be done about it.

 

He'll also talk to a former professor caught out after going to the ultimate extreme - faking his data - to find out what drives someone to betray their entire field.

 

 

Semente, I think we could make a strong case for the human decision process negatively impacting ALL human activities..... :-)

 

Also on the question of destruction of a field\industry\economy thru dishonest means. I remember watching Alan Greenspan testifying at the congressional hearings post the worldwide 2008 economic melt down. I was absolutely stunned when he stated that he completely missed the boat on how bad the melt down was going to be due to the fact he could not understand how a group of individuals would act in a manner that would be detrimental to themselves, their businesses, and the industry as a whole. This guy was the freaking Chairman of the Federal Reserve, some would argue one of the most powerful institutions on the planet, and he did not understand one of the most fundamental elements of the human condition. We are all doomed! :-)

[wgscott]

At the best, the guy spoke in Talmudic riddles. However, he was far from stupid, and knew exactly what was going on. There was never a crisis when the homeless population exploded, US manufacturing was systematically dismantled and shipped overseas, the gap between the richest 1% and everyone else increased exponentially, and the housing market shut out the middle class and then foreclosed on those who managed to get a foot in the door. It only became a crisis when Wall Street speculators got burned by their own systematic fraud. But he lacked the temerity, even then, to state the obvious, but just pretended to be a confused senile goofball. An honest version of Greenspan would have called for wide-spread prosecutions of those who created the mess. Instead, a very few received slaps on the wrist.

[plissken]

Overall science should be a self correcting endeavor. When you publish you open it up to independent verification.

[wgscott]

Overall science should be a self correcting endeavor. When you publish you open it up to independent verification.

 

Only if sharing the data or samples doesn't destroy it.

[plissken]

Only if sharing the data or samples doesn't destroy it.

 

Nothings perfect.

[wgscott]

But, seriously, folks ...

 

Plissken is exactly right. In fact, if Science didn't work like this, how would we know if error or fraud had occurred in the first place? The alternative, where everyone gets to be right about everything, means we would have no way to distinguish between knowledge and bogus information. Scientists are almost always wrong on some level, usually for completely innocent reasons. But this is why critical analysis and testing results under the most stringent, unforgiving conditions possible ultimately leads us to higher ground.

[plissken]

But, seriously, folks ...

 

Plissken is exactly right.

 

Thank you, Thank you, hold the applause... ;-)

[petaluma]

An honest version of Greenspan would have called for wide-spread prosecutions of those who created the mess. Instead, a very few received slaps on the wrist.

 

Bill, absolutely agree. But if I was a betting man I'd lay better odds on discovery of a unicorn before any upstanding member of that particular community see a day in the slammer. :-)

 

Hope things are going well for you. Working my hours up on the bike now that the weather is changing. Man I hate using the trainer in the winter.

 

Ciao

 

David

[wgscott]

I got my wheel back from Chris King and am ready to roll.

[Jud]

Two interesting and possibly connected phenomena:

 

- A surprising number of "breakthrough" studies are gradually "disproved" as more data comes in.

 

- There was a recent discussion about the degree to which statistics are misunderstood in the sciences, both social and hard, including prominently such things as the meaning of p-values.

 

Will try to find references if anyone wants.

[wgscott]

http://www.nytimes.com/2015/08/28/science/many-social-science-findings-not-as-strong-as-claimed-study-says.html

[esldude]

Two interesting and possibly connected phenomena:

 

- A surprising number of "breakthrough" studies are gradually "disproved" as more data comes in.

 

- There was a recent discussion about the degree to which statistics are misunderstood in the sciences, both social and hard, including prominently such things as the meaning of p-values.

 

Will try to find references if anyone wants.

 

Seems most such problems are in psychology and medical fields. Also there are a few people in these fields calling for 3 sigma (.27%)thresholds rather than 2 sigma (5%). Once you discover the more obvious issues with 2 sigma controls you get too many false positives to keep it from seeming like chaos.

 

I believe it was Shewhart and Demming in simple practical terms found you need 3 sigma statistical control for consistent quality in manufacturing. When manufacturing tried using 2 sigma controls quality was worse and you were always chasing your tail due to various issues. Going to 3 sigma brought things from chaos to control in terms of manufacturing quality. At least in some situations once you do have genuine control of what is going on it then becomes possible to get even tighter control of genuine results.

 

Which is why I have said at times these blind audio tests need to use 3 sigma thresholds. Of course I simply get blamed for moving the goalposts.

[jabbr]

3 sigma statistical control for manufacturing is an entirely different animal that medical research. Suppose you personally have, say, liver cancer and are told that no current treatment has a prayer of working. What odds would you take that a new treatment has a chance to cure?

[jabbr]

Also suppose you personally want to fund a study to find a cure for your child, and say you have exactly $1,000,000 to fund the study -- what would you want to do if told that the cost of funding a 3 sigma study was $1,000,000,000 ?

[esldude]

Also suppose you personally want to fund a study to find a cure for your child, and say you have exactly $1,000,000 to fund the study -- what would you want to do if told that the cost of funding a 3 sigma study was $1,000,000,000 ?

 

The cost difference is not that much. 3 sigma requires about double the samples. The advantages are that once research has solid results you can move on rather than revisiting matters. Slowly at first then faster solid progress accrues. Down the line you may progress to areas two sigma will never lead.

[esldude]

3 sigma statistical control for manufacturing is an entirely different animal that medical research. Suppose you personally have, say, liver cancer and are told that no current treatment has a prayer of working. What odds would you take that a new treatment has a chance to cure?

 

That is an emotional argument. I have a relative in that situation right now. I am sending this from the hospital, and it is no basis upon which to make such determinations.

[petaluma]

That is an emotional argument. I have a relative in that situation right now. I am sending this from the hospital, and it is no basis upon which to make such determinations.

 

When you are dying and you have exhausted all "mainstream options" sigma standards don't mean squat. You grasp at potential options and hope for a miracle. I have a relative in this situation as you do, my mother, and you will consider experimental unproven options. Sigma levels are worthless in late stage 4 or terminal diagnosis situations. Of course its F*#ing emotional.

[esldude]

When you are dying and you have exhausted all "mainstream options" sigma standards don't mean squat. You grasp at potential options and hope for a miracle. I have a relative in this situation as you do, my mother, and you will consider experimental unproven options. Sigma levels are worthless in late stage 4 or terminal diagnosis situations. Of course its F*#ing emotional.

 

Then perhaps you agree this is not the conditions optimal for deciding the best way to do science.

[petaluma]

Then perhaps you agree this is not the conditions optimal for deciding the best way to do science.

 

Dennis, I agree completely but the statement by jabbr was regarding last case options. Continue conducting research using 3 levels of Sigma, but allow patients with less then optimal potential outcomes to be test subjects on non Sigma 3 validated options that might have potential however remote. Those that show any potential move into the more scrutinized and peer validated research.

[jabbr]

In any case there is no arbitrary one right sigma. The scientific community broadly accepts p < 0.05 as being significant. That's how it is. You are free as an individual to limit yourself to treatments having the most definitive results. Better get reading though.

[jabbr]

That is an emotional argument. I have a relative in that situation right now. I am sending this from the hospital, and it is no basis upon which to make such determinations.

 

But people make these decisions under these circumstances every minute of the day and all across the country.

[esldude]

But people make these decisions under these circumstances every minute of the day and all across the country.

 

No they don't. Not involving whether or not a treatment is considered sound by looking at p values. Your doctor does that and you either trust the doctor's judgement or not. If medical research starts getting more solid results with more chances of knowing the outcome your doctor's choices will be clearer and/or more reliable.

 

If you become desperate, and eschew doctors for other magical or alternative medicine even less so is one deciding based upon research.

 

If your intent was that possible treatments that only meet 2 sigma or less and are yet to be fully accepted be available to terminal patients then fine. Rarely is that the case with the whole infrastructure of modern medicine. Until treatment has at least gotten close to passing the general medical muster it isn't available.

[esldude]

In any case there is no arbitrary one right sigma. The scientific community broadly accepts p < 0.05 as being significant. That's how it is. You are free as an individual to limit yourself to treatments having the most definitive results. Better get reading though.

 

The modern medical and some other communities accept p .05. That isn't applicable for the whole of science.

[esldude]

Some writings by those in the medical research field. The need to tighten up standards for research isn't some hair brained idea I cooked up on my own. Nor that doing so would in the long run be beneficial.

 

http://arxiv.org/pdf/1407.5296.pdf

 

PLOS Medicine: Why Most Published Research Findings Are False

 

http://www.dcscience.net/Johnson-PNAS-2013-statistical-standards.pdf

 

And a short comment about it from a good scientist though not from the medical research field.

Stiffening the Standards of Scientific Research