Does BIAS affect audio test results?

[pkane2001]

15 hours ago, Audiophile Neuroscience said:

There is no doubt that bias affects audio tests. The real debate is on what follows: how much, when, does it matter, and how to deal with it if it does matter.

 

I welcome the debate, but it cannot be based solely on opinions and conjectures. Please provide objective evidence to support any claims you make, or feel free to make them in another thread.

 

There's enough evidence cited in this thread that biases lead to significant errors in subjective testing. This has been quantified to be anywhere between 20% and 40% in perceived sound preference change due to just the expectation bias, and that's only one of over a dozen biases that have already been covered.

 

Everything posted here is meant to bring out the possible errors in testing and mitigation strategies that have been shown to work in research and experiments. If you have specific errors to add to the list, such as what you and Chris seem to be claiming about DBTs, for example, then please post objective evidence documenting them.

 

[pkane2001]

One important error source (and a potential source of bias) in testing audio equipment is.... your own hearing! 
 

Offered with no explanation, an app to conduct a hearing test with your own audio system: Earful. The app is still under development, but has the basic functionality I intended. Requires Windows and works with ASIO, WASAPI, and Direct Sound drivers. Earful was developed primarily for my own use, but upon request, I decided to make it available to everyone. Have fun, and share your results, but be careful about setting volume control(s) on your system to avoid nasty surprises.

 

A beta-test thread already exists on ASR, but if there's enough interest/follow-up here, I'll start another thread dedicated to it. Remember that this is intended for entertainment purposes and not appropriate for medical use or diagnosis.

 

[John Dyson]

2 hours ago, pkane2001 said:

One important error source (and a potential source of bias) in testing audio equipment is.... your own hearing! 
 

Offered with no explanation, an app to conduct a hearing test with your own audio system: Earful. The app is still under development, but has the basic functionality I intended. Requires Windows and works with ASIO, WASAPI, and Direct Sound drivers. Earful was developed primarily for my own use, but upon request, I decided to make it available to everyone. Have fun, and share your results, but be careful about setting volume control(s) on your system to avoid nasty surprises.

 

A beta-test thread already exists on ASR, but if there's enough interest/follow-up here, I'll start another thread dedicated to it. Remember that this is intended for entertainment purposes and not appropriate for medical use or diagnosis.

 

I know that I am 'imposing' here by mentioning this -- I don't know your program yet, but intend to look at it soon...   However, I'd love to see a critical/accurate audio listening memory test.   That is, bring the golden ears (or me, if I am wrong) back to reality about precision listening and how long precise hearing can be remembered.

 

My own limit is somewhere like 7-12 (maybe 15 seconds.)   I am not talking about 'sounds good' hearing, but I mean hearing every little nit and being able to detect new and old nits.   This kind of hearing test would not include becoming aware of audio defects and then zeroing  in.  THAT kind of test would create a bias towards 'perfect recollection'.  It would also be good to create a test with something like 'immediate' comparison, 3 seconds delay between comparisons, 6, 9, 12, etc...  FIgure out how long the precision memory lasts...  I doubt that very many people can really reach 30 seconds.   There IS a thing that manifests as perfect or near-perfect pitch (I used to have a great memory for audio frequencies), but NOT remembering complex audio presentation.

 

I mean, a test that is UP-TO approx 20 seconds long (or longer), that has specific defects buried in confusion material (like music for distraction), and get some stats on when people hear false differences (that is my flaw) or make claims about perfect copies when they aren't...

 

My guess is that false differences will happen more often, but these results would be intriguging to me, and might educate the 'golden ears' who probably DO have really accurate hearing, but would help moderate claims about long term precision and memory.

 

(I have been playing whack a mole, and chasing down rabbit holes because of subjective testing where there is no objective measurement possible -- I HATE subjective comparisons, and learning the hate them more -- I keep on making mistakes, and I am NOT that bad at doing comparisons!!!)  However, someitmes subjective measurements must be done, but certainly don't do so because I enjoy dealing with the errors!!!

 

John

 

[pkane2001]

2 minutes ago, John Dyson said:

I know that I am 'imposing' here by mentioning this -- I don't know your program yet, but intend to look at it soon...   However, I'd love to see a critical/accurate audio listening memory test.   That is, bring the golden ears (or me, if I am wrong) back to reality about precision listening and how long precise hearing can be remembered.

 

My own limit is somewhere like 7-12 (maybe 15 seconds.)   I am not talking about 'sounds good' hearing, but I mean hearing every little nit and being able to detect new and old nits.   This kind of hearing test would not include becoming aware of audio defects and then zeroing  in.  THAT kind of test would create a bias towards 'perfect recollection'.  It would also be good to create a test with something like 'immediate' comparison, 3 seconds delay between comparisons, 6, 9, 12, etc...  FIgure out how long the precision memory lasts...  I doubt that very many people can really reach 30 seconds.   There IS a thing that manifests as perfect or near-perfect pitch (I used to have a great memory for audio frequencies), but NOT remembering complex audio presentation.

 

I mean, a test that is UP-TO approx 20 seconds long (or longer), that has specific defects buried in confusion material (like music for distraction), and get some stats on when people hear false differences (that is my flaw) or make claims about perfect copies when they aren't...

 

My guess is that false differences will happen more often, but these results would be intriguging to me, and might educate the 'golden ears' who probably DO have really accurate hearing, but would help moderate claims about long term precision and memory.

 

(I have been playing whack a mole, and chasing down rabbit holes because of subjective testing where there is no objective measurement possible -- I HATE subjective comparisons, and learning the hate them more -- I keep on making mistakes, and I am NOT that bad at doing comparisons!!!)  However, someitmes subjective measurements must be done, but certainly don't do so because I enjoy dealing with the errors!!!

 

John

 

 

Hi John,

 

Earful app is designed to test hearing thresholds, but an audio memory test is a great idea, and something I've not seen as a test-it-yourself app. Echoic memory is claimed to be around 10 seconds, but it would be great to be able to prove it to yourself with testing. I can imagine a few such tests, some including tonal and amplitude differences, frequency response, and some with audible distortions, compression and such. I'll do some digging to see if I can find existing references to such tests.

[pkane2001]

A subjective test using biased-controlled MUSHRA (MUltiple Stimuli with Hidden Reference and Anchor) methodology. This one was designed to check for which of a few audio impairments results in a greater impact on the subjective audio quality: spatial presentation or reduced frequency/tonal accuracy. Long story short, frequency response limitations were found to be much more significant than the 5.1 spatial representation, aka, sound stage:

 

Slawomir K. Zieliński, et al, Comparison of Quality Degradation Effects Caused by Limitation of Bandwidth and by Down-mix Algorithms in Consumer Multichannel Audio Delivery Systems, Institute of Sound Recording, University of Surrey, UK

 

http://epubs.surrey.ac.uk/543/1/fulltext.pdf