Jump to content
IGNORED

How the Cognitive Biases of Researchers Affect Research Results


Jud

Recommended Posts

One never knows, do one? - Fats Waller

The fairest thing we can experience is the mysterious. It is the fundamental emotion which stands at the cradle of true art and true science. - Einstein

Computer, Audirvana -> optical Ethernet to Fitlet3 -> Fibbr Alpha Optical USB -> iFi NEO iDSD DAC -> Apollon Audio 1ET400A Mini (Purifi based) -> Vandersteen 3A Signature.

Link to comment
3 minutes ago, Iving said:

 

Why is it excellent Jud. What did you get from it such that you thought it fitting here?

 

What we'd ideally like to do in getting at the objective truth about audio is to be able to find relevant scientific research. This presentation gives us some of the reasons our job isn't done then. We have to be cautious, even skeptical, in evaluating what we find, perhaps most especially if it agrees with the ideas we already have.

 

It gets at some of the reasons behind what has been called the "reproducibility crisis" (subsequent researchers being unable to reproduce results), some that should be evident, some a little more esoteric. And it reminds us about confirmation bias, which affects not only researchers, but we who "consume" research.

One never knows, do one? - Fats Waller

The fairest thing we can experience is the mysterious. It is the fundamental emotion which stands at the cradle of true art and true science. - Einstein

Computer, Audirvana -> optical Ethernet to Fitlet3 -> Fibbr Alpha Optical USB -> iFi NEO iDSD DAC -> Apollon Audio 1ET400A Mini (Purifi based) -> Vandersteen 3A Signature.

Link to comment
2 hours ago, Iving said:

 

Thanks Jud,

Like Dorothy, I have a background in psychology research. Although I've been out of it for a while, I recognise very much the culture in which she presents. I understand what she's getting at as an insider so to speak. [In fact I just mention in passing - as a "small world" interest - that I was an undergraduate somewhere else with a now-Professor colleague of Dorothy's at the same institution as hers; i.e., St. John's, Oxford.]

I watched part of the presentation, but had to see to something and got diverted. I will try to watch the remainder later.

There's no doubt in my own mind (a very small corner of the universe) that - generally speaking - scientific research is just like the spiritual life: you tend to find what you're looking for sooner or later. What drives or motivates individual researchers has to be a matter of potentially endless consideration. As in life at large, some people's motives are "purer" (for a given pursuit) than others.

Just about all published research is funded somewhere along the line. Researchers have salaries and pensions - careers and reputation at stake. I dare say even Dorothy is imbued in her own niche "cognitive schemata". Perhaps she recognises that - I haven't watched enough of her presentation to see.

[That said - I was trained very much in an "honest" research culture. Empirical research is all about developing convincing evidence. If ever you were discovered to have fallen short, that would be it - lights out.]

She wants to see "cleaner" research I dare say. That has to be a good thing. Perhaps it is impossibly idealistic as a goal. A sceptic might suspect that she is promoting ring-fenced resources ("money anyway") for Universities which already have a good research reputation - such as her own!

imo the most compelling research will be un-sponsored - except from the heart of the competent researcher - and by "competent" I mean a researcher with first hand experience of "living" the research context germane (as well as familiar with the necessary approach and methods etc).

There are plenty of folk like that here at AS.

There's some great threads already in "Objective-Fi" pointing to this area.

 

In general I agree, though let's face it, a lot of research requires funding. In that case the best we can do is try to insulate the researcher from funders' biases.

One never knows, do one? - Fats Waller

The fairest thing we can experience is the mysterious. It is the fundamental emotion which stands at the cradle of true art and true science. - Einstein

Computer, Audirvana -> optical Ethernet to Fitlet3 -> Fibbr Alpha Optical USB -> iFi NEO iDSD DAC -> Apollon Audio 1ET400A Mini (Purifi based) -> Vandersteen 3A Signature.

Link to comment
2 hours ago, jabbr said:

 

Hmm ... perhaps expecting "Science" to give an objective truth is akin to expect Alan Dershowitz to tell you what he really thinks 🤣

 

Science doesn't use terms like "truth" rather "theory" and "model".

 

And that's the truth! ;)

 

2 hours ago, jabbr said:

Science is not looking into questions regarding the "sound" of power cables. (don't confuse marketing speak for science)

 

 

No, though it doesn't prevent research from being tangentially relevant (e.g., there is research on changes in electrical properties through cryogenic treatment that may be at least tangentially relevant to claims about cryoing cables). And of course actual quantum theory research shows claims about various "special" quantum properties in audio equipment are BS.

 

2 hours ago, jabbr said:

Show me an unbiased scientist and I'll show you a scientist in a coma.

 

Of course. Best we can do is try. The interesting bits of the presentation and slides have to do with details regarding how scientists work bias into their research results, consciously or unconsciously. Some of the stuff on "p-hacking" was enlightening to me.

 

2 hours ago, jabbr said:

Kuhn always comes to mind. Have you read?

 

No, but have read about. Here: https://blogs.scientificamerican.com/cross-check/what-thomas-kuhn-really-thought-about-scientific-truth/

One never knows, do one? - Fats Waller

The fairest thing we can experience is the mysterious. It is the fundamental emotion which stands at the cradle of true art and true science. - Einstein

Computer, Audirvana -> optical Ethernet to Fitlet3 -> Fibbr Alpha Optical USB -> iFi NEO iDSD DAC -> Apollon Audio 1ET400A Mini (Purifi based) -> Vandersteen 3A Signature.

Link to comment
9 minutes ago, Jud said:

 

You really really should read the book, its right up your alley ...

 

 

Quote

Kuhn said. Just because modern physics has spawned computers, nuclear power and CD players, he suggested, does not mean it is truer, in an absolute sense, than Aristotle's physics. Similarly, Kuhn denied that science is constantly approaching the truth. At the end of Structure he asserted that science, like life on earth, does not evolve toward anything but only away from something.

 

 

Custom room treatments for headphone users.

Link to comment
1 minute ago, jabbr said:

 

You really really should read the book, its right up your alley ...

 

 

From what I've read (and your quote seems representative), he's a bit too "philosophical" and a little less "science-y" than I'd like. Sure I think science is more than fallible, a pursuit of multiple fallible people. But it's also undeniable we have many things today that operate, and operate accurately, based on physics very, very different than Aristotle's. To me that makes it "truer."

One never knows, do one? - Fats Waller

The fairest thing we can experience is the mysterious. It is the fundamental emotion which stands at the cradle of true art and true science. - Einstein

Computer, Audirvana -> optical Ethernet to Fitlet3 -> Fibbr Alpha Optical USB -> iFi NEO iDSD DAC -> Apollon Audio 1ET400A Mini (Purifi based) -> Vandersteen 3A Signature.

Link to comment
15 minutes ago, Jud said:

No, though it doesn't prevent research from being tangentially relevant (e.g., there is research on changes in electrical properties through cryogenic treatment that may be at least tangentially relevant to claims about cryoing cables).

 

Isaac Newton spent the later years of his life working on this!

Custom room treatments for headphone users.

Link to comment
Just now, Jud said:

 

From what I've read (and your quote seems representative), he's a bit too "philosophical" and a little less "science-y" than I'd like. Sure I think science is more than fallible, a pursuit of multiple fallible people. But it's also undeniable we have many things today that operate, and operate accurately, based on physics very, very different than Aristotle's. To me that makes it "truer."

 

It makes it more useful !! Read the book 🙂

Alternatively Quine's Two Dogma's, and "American Pragmatism" https://www.philosophytalk.org/shows/american-pragmatism#:~:text=

Custom room treatments for headphone users.

Link to comment
7 minutes ago, jabbr said:

 

It makes it more useful !! Read the book 🙂

Alternatively Quine's Two Dogma's, and "American Pragmatism" https://www.philosophytalk.org/shows/american-pragmatism#:~:text=

 

I certainly find that which is true more useful than that which isn't. 😉

One never knows, do one? - Fats Waller

The fairest thing we can experience is the mysterious. It is the fundamental emotion which stands at the cradle of true art and true science. - Einstein

Computer, Audirvana -> optical Ethernet to Fitlet3 -> Fibbr Alpha Optical USB -> iFi NEO iDSD DAC -> Apollon Audio 1ET400A Mini (Purifi based) -> Vandersteen 3A Signature.

Link to comment
31 minutes ago, Jud said:

 

I certainly find that which is true more useful than that which isn't. 😉

 

Explaining a bit further: I call myself an atheist rather than an agnostic. Though I can't of course have absolute knowledge, I feel for practical purposes my level of certainty is good enough. By the same token, I treat relativity and quantum mechanics as "true," though knowing there is some conflict between them, because for practical purposes I can depend on them (for example, GPS and nanoscale electronics work). Can "truth" on this level change? Sure, when the Ptolemaic system just got too darn complicated and Kepler's explained things a lot more simply; same when Einstein extended Newton's explanations. Is the knowledge that the "truth" can change somewhat Kuhnian? Sure. But just for practical purposes and conciseness, I call it truth anyway.

One never knows, do one? - Fats Waller

The fairest thing we can experience is the mysterious. It is the fundamental emotion which stands at the cradle of true art and true science. - Einstein

Computer, Audirvana -> optical Ethernet to Fitlet3 -> Fibbr Alpha Optical USB -> iFi NEO iDSD DAC -> Apollon Audio 1ET400A Mini (Purifi based) -> Vandersteen 3A Signature.

Link to comment
10 hours ago, Jud said:

Some of the stuff on "p-hacking" was enlightening to me.

 

I call it "P Peeping" probably because it reminds me of my other great interest, photography, and the analogy of pixel peeping. Not sure it's a great analogy but it conjures up images of drilling down or "dredging" into the data as a form of selective reporting.

 

There are pressures obviously in published journals to produce statistically significant results. After all, who wants to hear about insignificant results?

 

The issue is that it creates an inflated effect size (inflation bias) and increased false positives.

 

The only good news is that it tends to 'all come out in the wash' in meta-analyses. Clever statisticians can also identify P-hacking.

 

 

Sound Minds Mind Sound

 

 

Link to comment
3 hours ago, Audiophile Neuroscience said:

 

I call it "P Peeping" probably because it reminds me of my other great interest, photography, and the analogy of pixel peeping. Not sure it's a great analogy but it conjures up images of drilling down or "dredging" into the data as a form of selective reporting.

 

There are pressures obviously in published journals to produce statistically significant results. After all, who wants to hear about insignificant results?

 

The issue is that it creates an inflated effect size (inflation bias) and increased false positives.

 

The only good news is that it tends to 'all come out in the wash' in meta-analyses. Clever statisticians can also identify P-hacking.

 

 

 

That it happens of course I was aware of, but these methods by which one can so easily and dramatically increase the chances of a statistically significant result, I was not.

One never knows, do one? - Fats Waller

The fairest thing we can experience is the mysterious. It is the fundamental emotion which stands at the cradle of true art and true science. - Einstein

Computer, Audirvana -> optical Ethernet to Fitlet3 -> Fibbr Alpha Optical USB -> iFi NEO iDSD DAC -> Apollon Audio 1ET400A Mini (Purifi based) -> Vandersteen 3A Signature.

Link to comment
15 hours ago, Jud said:

The interesting bits of the presentation and slides have to do with details regarding how scientists work bias into their research results, consciously or unconsciously. Some of the stuff on "p-hacking" was enlightening to me.


We delved into physics of relativity & quantum mechanics. Physics does use statistics but the whole p > .01 should not be seen as a fundamental characteristic of Science, but perhaps more “Medical Science” which may or may not be Science! So yeah there’s a lot of twisted stuff where folks are trying to get a result in a drug effect, and you do DBTs and t-tests and multivariate analyses... that’s NOT hard Science. Not just cognitive bias but real financial bias where your company can make Billions if the p value fits,
 

If you look into the science of electronics there’s statistics related to quantum effects, likewise particle detection in the LHC etc, but people by and large aren't tweaking p values 😳

Custom room treatments for headphone users.

Link to comment

@Jud mentioned cryo treatments for cables etc ... suppose someone wants to analyze this scientifically 

 

I would want to see a measurement of some type demonstrating an electrical difference before I became interested...

 

Take gravity: sure we could do a DBT proving that apples travel in roughly a parabolic orbit ... wait! That’s not how it was done ... now we have graviton detectors ...

 

https://www.nature.com/articles/s41598-019-50098-z

 

Science is advanced not by DBT but rather by the development of new measurement techniques. That’s my own bias 😂

Custom room treatments for headphone users.

Link to comment
5 minutes ago, jabbr said:

@Jud mentioned cryo treatments for cables etc ... suppose someone wants to analyze this scientifically 

 

I would want to see a measurement of some type demonstrating an electrical difference before I became interested...

 

Take gravity: sure we could do a DBT proving that apples travel in roughly a parabolic orbit ... wait! That’s not how it was done ... now we have graviton detectors ...

 

Science is advanced not by DBT but rather by the development of new measurement techniques. That’s my own bias 😂

 

Suppose an experiment

very rigorous DBX

demonstrated at p<0.001

1,000 Ss could tell the difference between cryo-treated vs. non-cryo-treated where there was no other difference between the cables

then we'd know "beyond all reasonable doubt"

that there was a SQ difference (for better or worse)

 

we just wouldn't know what it was about cryo-treatment that accounted for that diff

 

this approach could be regarded as "top down" vs. "bottom up"

 

both are as "hard" as each other - "hard" being defined as p<0.001 plus rigour of Method

 

If we'd insisted on measuring an electrical (or other) difference first - then omg we could have cut off our nose to spite our face ..

 

?

Link to comment
43 minutes ago, jabbr said:

@Jud mentioned cryo treatments for cables etc ... suppose someone wants to analyze this scientifically 

 

I would want to see a measurement of some type demonstrating an electrical difference before I became interested...

 

Take gravity: sure we could do a DBT proving that apples travel in roughly a parabolic orbit ... wait! That’s not how it was done ... now we have graviton detectors ...

 

https://www.nature.com/articles/s41598-019-50098-z

 

Science is advanced not by DBT but rather by the development of new measurement techniques. That’s my own bias 😂

Cryo treatments will improve thermal conductivity, thermal noise...   

What the impact on audio cables, I dont'know.

 

 

P1091/1092

Kalsi20110.pdf

 


Link to comment
17 minutes ago, Iving said:

both are as "hard" as each other - "hard" being defined as p<0.001 plus rigour of Method

 

I’m going to disagree as my definition of soft science is a reliance on p values. There’s good soft science when rigorously done. Hard science, to my definition, relies on measurements and theory. 
 

DBX in the audio world is a pipe dream — hard to really do, at best it’s single blinded but folks like to through the terms out. That’s ok this is a hobby, not a new cancer treatment.
 

17 minutes ago, Iving said:

 

If we'd insisted on measuring an electrical (or other) difference first - then omg we could have cut off our nose to spite our face ..

 

That’s pretty much how electronics engineering works. People learn theory, design a schematic, perhaps run a sim, then build circuits, make measurements, listen, tweak circuits, make measurements, listen ... no DBT involved!

Custom room treatments for headphone users.

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now



×
×
  • Create New...