Jump to content
IGNORED

The walking dead and "blind listening"


Recommended Posts

It's puzzled me for years why some always try to discredit an audio review by calling it invalid if it wasn't a

"blind listening" test.

What's so hard for you subjective listeners to understand?

It's very simple,

Without blind listening confirmation any review will all ways be just that persons opinion.

You can protest all you like and tell the world what "golden ears" you have, but any review has no more validity than the next guys without proof of blind tests and measurements.

"The gullibility of audiophiles is what astonishes me the most, even after all these years. How is it possible, how did it ever happen, that they trust fairy-tale purveyors and mystic gurus more than reliable sources of scientific information?"

Peter Aczel - The Audio Critic

no-mqa-sm.jpg

Link to comment
You can protest all you like and tell the world what "golden ears" you have, but any review has no more validity than the next guys without proof of blind tests and measurements.

 

Very few have the ability to organise properly conducted and scientifically verifiable UNBIASED Blind Tests.

Properly organised Blind Tests take a lot of organising and correctly setting up, and are rarely performed due to the time and cost constraints involved.

Many of those who keep insisting on Blind Tests couldn't even organise a "Chook Raffle" in a Pub !!!

 

How a Digital Audio file sounds, or a Digital Video file looks, is governed to a large extent by the Power Supply area. All that Identical Checksums gives is the possibility of REGENERATING the file to close to that of the original file.

PROFILE UPDATED 13-11-2020

Link to comment

Many of those who keep insisting on Blind Tests couldn't even organise a "Chook Raffle" in a Pub !!!

 

Cheap insults add no validity to your position.

"The gullibility of audiophiles is what astonishes me the most, even after all these years. How is it possible, how did it ever happen, that they trust fairy-tale purveyors and mystic gurus more than reliable sources of scientific information?"

Peter Aczel - The Audio Critic

no-mqa-sm.jpg

Link to comment
Very few have the ability to organise properly conducted and scientifically verifiable UNBIASED Blind Tests.

Properly organised Blind Tests take a lot of organising and correctly setting up, and are rarely performed due to the time and cost constraints involved.

Many of those who keep insisting on Blind Tests couldn't even organise a "Chook Raffle" in a Pub !!!

 

I understand that a blind test may take a bit longer to perform but what are the "cost constraints" involved?

Sometimes it's like someone took a knife, baby
Edgy and dull and cut a six inch valley
Through the middle of my skull

Link to comment
I understand that a blind test may take a bit longer to perform but what are the "cost constraints" involved?

 

Suitably high quality amplification and speakers, optimised room acoustics, highest quality source material ,( NOT via flawed USB either) and optimised seating for all participants, (no big heads/upper bodies in the way) and to be scientifically valid, numerous repeats without generating listener fatigue which will result in false negatives.

The switching would need to be seamless, preferably using well implemented remote controlled relay switching instead of a piss poor test like using Foobar 2K which is far from "The Gold Standard" as a software player.

If the S/W route was used you would probably need to use something like Miska's HQ player.

Obviously, it would need a lot more care in setting up than at a typical HiFi show with tiny rooms etc. and your typical home listening room is far from suitable.

 

How a Digital Audio file sounds, or a Digital Video file looks, is governed to a large extent by the Power Supply area. All that Identical Checksums gives is the possibility of REGENERATING the file to close to that of the original file.

PROFILE UPDATED 13-11-2020

Link to comment
Suitably high quality amplification and speakers, optimised room acoustics, highest quality source material ,( NOT via flawed USB either) and optimised seating for all participants, (no big heads/upper bodies in the way) and to be scientifically valid, numerous repeats without generating listener fatigue which will result in false negatives.

The switching would need to be seamless, preferably using well implemented remote controlled relay switching instead of a piss poor test like using Foobar 2K which is far from "The Gold Standard" as a software player.

If the S/W route was used you would probably need to use something like Miska's HQ player.

Obviously, it would need a lot more care in setting up than at a typical HiFi show with tiny rooms etc. and your typical home listening room is far from suitable.

 

 

Other than the switching, everything you mention would seem to apply to sighted tests as well.

Sometimes it's like someone took a knife, baby
Edgy and dull and cut a six inch valley
Through the middle of my skull

Link to comment
Other than the switching, everything you mention would seem to apply to sighted tests as well.

 

But we aren't talking about sighted tests here. Even at our regular Sydney listening sessions, there is so much gear powered up to avoid warm up problems, that some of us aren't even aware which actual component was being used until told later.

( I usually don't know , as after the Spinal op. last year, I remain seated while they fiddle around behind the equipment, changing interconnects etc.)

 

How a Digital Audio file sounds, or a Digital Video file looks, is governed to a large extent by the Power Supply area. All that Identical Checksums gives is the possibility of REGENERATING the file to close to that of the original file.

PROFILE UPDATED 13-11-2020

Link to comment
But we aren't talking about sighted tests here. Even at our regular Sydney listening sessions, there is so much gear powered up to avoid warm up problems, that some of us aren't even aware which actual component was being used until told later.

( I usually don't know , as after the Spinal op. last year, I remain seated while they fiddle around behind the equipment, changing interconnects etc.)

 

 

Sounds like you are already doing blind testing. May not be "scientifically verifiable" but good enough for 99.9% of the folks out there.

Sometimes it's like someone took a knife, baby
Edgy and dull and cut a six inch valley
Through the middle of my skull

Link to comment
I understand that a blind test may take a bit longer to perform but what are the "cost constraints" involved?

 

I believe the "cost constraints" are the time inputs from various people needed for a properly conducted blind test

 

I saw this description of blind testing from ThorstenL on DIYaudio once & took note of it:

First - Statistics, understand that the underlying analysis method is statistics. The issue is that we try to determine if the test result was likely due to chance or an actual difference being present.

 

Such statistical analysis is subject to two possible errors, one is a "Type A" error, where we conclude in error that a difference is present when the test results were caused by chance and a type B error where we conclude that no difference exists when in fact one is present.

 

Les Levental presented the required math to work out the likelihood of both errors. The upshot is simple, with small datasets (the N I have been referring to) and a high significance level (such as commonly demanded by the ABX/DBT Mafia) will result in a very large risk of Type B errors.

 

IN PRINCIPLE one may perform a much more advanced statistical analysis that gives more meaningful results, but the ABX/DBT Mafia in their never ending quest to rid audio from the Bogey Man and Voodoo refuse to engage in such practices, as this would stop returning reliably "null" results from their tests, which invariably feature very small N and high significance.

 

So much for the statistics. It means if you want a test that will not reliably gloss over existing differences and you want to put one over on the ABX/DBT Mafia you require a very large N. If you do the test with very few participants and sensibly low numbers of trials you are playing "their" game and you will return "null" results.

 

So much on statistics.

 

Second, human perception. It is fickle and deceptive.

 

NB1 - Test situations create stress (remember when you did you any final exams).

 

Any stress mitigates against perception sensitivity. As we cannot avoid the knowledge of a "test situation" we need to reduce stress.

 

Long test sequences show increasing stress. Usually five sequential presentations in one go tend to be an observed limit, before attention flags off and stress rises and sensitivity goes to hell.

 

So make sure to have multiple breaks, preferably activity filled that can re-set stress & attention levels, between presentations. As you are going to have very few participants the best you can do is run multiple tests with the same participants. While in my view it does not really adds to confidence levels, it is acceptable to the ABX/DBT Mafia and hence you may as well employ it, to maximise your chances.

 

NB2 - beware of expectation bias.

 

Expectation bias is the result of having expectations about the outcome while undertaking the test. I once did a blind test at an audio society meeting, comparing my own modifications of a Marantz CD-67 to a stock machine. I knew my mods were better. I knew my machine would win.

 

(Note, Sy's measurements would have pegged them "not different", Note 2 - Yes it is the CD-67 from the TNT-Audio modification article)

 

Well, once the "blinds" were down I could NOT hear any difference. I literally could not. But over 70% of those present (a group of maybe a little under 20, mostly seated very sub-ideally) could hear differences with good reliability (4/5) and expressed a preference for the modified unit.

 

Intrigued by my own failure I did another test, in which I told the participants we would test mains cable sound differences. The group was very small four plus me, so statistical significance is low, we may treat this as "anecdotal evidence".

 

Most had no particular opinion on this, all cables were cheap to make DIY Mains cables and should they prove effective, anyone could build their own. One participant was a EE with BBC background very vocal cable sceptic. I professed openly, that I had not listened either and I had no idea if there was a difference or not, but would like to find out.

 

What I really did was to reverse the polarity of one speaker in the stereo set. Everyone heard the differences correctly 10/10, except out "cable sceptic" WHO SCORED RANDOM! Due to the small N no real significance, but one may assert that in this case expectation bias was sufficient to mask something as gross as wiring one speaker out of polarity.

 

third - Calibrate your tests.

 

Just as I ask that any measurements are accompanied by demonstrations of the limits the equipment can resolve and as proof that the tests are competently implemented, any listening test should be calibrated with "known audibly different" stimulae, to ensure sufficient sensitivity to at least distinguish known audible phenomenae.

 

I suggest as a minimum polarity reversal of one channel, 1dB level difference and 0.3dB level difference, I personally feel that at times adding both channel polarity reversal helps to screen out "clothears", but this is not generally acknowledged as "audible", consider this as one to do for "extra credits" only.

 

In fact, the routine lack of any such calibration in the ABX/DBT Mafia's testing is one of my strongest arguments (together with "awfully bad statistics") towards the need to simply disregard their extensive set of published "null" results as evidence.

 

Fit the fourth - let's get down to the dirty work

 

I would suggest that you should try for an N no lower than 100, This allows you to select a significance level that has about equal risks of type A & B errors. With 20 participants and 5 trials each this requirement is satisfied.

 

As we would like some "calibration" and would like to remove "clothears" but keep "goldenears" you probably will need to start with a larger number of participants and not to be rude, you should not dismiss the "clothears" early.

 

So perhaps 50 Participants with the 20 who score highest in the preliminary calibration tests having their actual tests included in the final analysis (you should not cherry-pick people who score highly in the final real tests, but excluding participants that show a low sensitivity to known audible stimulae is acceptable).

 

Now, you can of course use much smaller numbers and all that, but not only do you find that the significance (that is the general applicability of your test results) is very low, but equally the risk of type B errors (erroneously returning null results) becomes unacceptably large.

 

If following all that you do not feel much like undertaking a "controlled listening test", I cannot blame you.

Link to comment
I believe the "cost constraints" are the time inputs from various people needed for a properly conducted blind test

 

I saw this description of blind testing from ThorstenL on DIYaudio once & took note of it:

 

Thanks for sharing this. Makes a lot of sense.

Sometimes it's like someone took a knife, baby
Edgy and dull and cut a six inch valley
Through the middle of my skull

Link to comment
Thanks for sharing this. Makes a lot of sense.

 

I think there's even more to it than this, involving some specifics regarding the science of human hearing. Been reading academic papers and hope to write an article eventually. But Real Life is busy and it's proving very hard to find the time, so no one hold should be holding his/her breath in anticipation.

One never knows, do one? - Fats Waller

The fairest thing we can experience is the mysterious. It is the fundamental emotion which stands at the cradle of true art and true science. - Einstein

Computer, Audirvana -> optical Ethernet to Fitlet3 -> Fibbr Alpha Optical USB -> iFi NEO iDSD DAC -> Apollon Audio 1ET400A Mini (Purifi based) -> Vandersteen 3A Signature.

Link to comment

"If following all that you do not feel much like undertaking a "controlled listening test", I cannot blame you."

 

 

Then you fall right back to any review done without these controls is strictly an opinion with no proven scientific validity.

You can't avoid scientific confirmation and call your proclaimed results fact. It just don't work that way.

"The gullibility of audiophiles is what astonishes me the most, even after all these years. How is it possible, how did it ever happen, that they trust fairy-tale purveyors and mystic gurus more than reliable sources of scientific information?"

Peter Aczel - The Audio Critic

no-mqa-sm.jpg

Link to comment
Suitably high quality amplification and speakers, optimised room acoustics, highest quality source material ,( NOT via flawed USB either) and optimised seating for all participants, (no big heads/upper bodies in the way) and to be scientifically valid, numerous repeats without generating listener fatigue which will result in false negatives.

The switching would need to be seamless, preferably using well implemented remote controlled relay switching instead of a piss poor test like using Foobar 2K which is far from "The Gold Standard" as a software player.

If the S/W route was used you would probably need to use something like Miska's HQ player.

Obviously, it would need a lot more care in setting up than at a typical HiFi show with tiny rooms etc. and your typical home listening room is far from suitable.

 

all participatants? i would never participate in a listening test unless i was smack dab in the sweet spot at all times

Link to comment
"...Then you fall right back to any review done without these controls is strictly an opinion with no proven scientific validity.

You can't avoid scientific confirmation and call your proclaimed results fact. It just don't work that way.

 

That is not what an audio review is! I have never read an audio equipment review that claimed to have any proven scientific validity in regards to the sound the reviewer heard in their system. The only thing scientific are any measurements they made, the rest is the opinion of the reviewer.

 

If a reviewer recommends a component in a review they might say it might be worth your audition but never go so far as to try to tell you how it will sound to you. If they did I would be screaming bloody hell. So….

I have dementia. I save all my posts in a text file I call Forums.  I do a search in that file to find out what I said or did in the past.

 

I still love music.

 

Teresa

Link to comment
What's so hard for you subjective listeners to understand?

It's very simple,

Without blind listening confirmation any review will all ways be just that persons opinion.

You can protest all you like and tell the world what "golden ears" you have, but any review has no more validity than the next guys without proof of blind tests and measurements.

 

You are way too gullible IMHO. Even with blind listening confirmation it is still only be that individual reviewer's opinion since we all hear differently, have different listening rooms, etc. One person cannot listen for another person, that's impossible!

 

I am skeptical of all reviews, specifications and tests, I have to hear the thing with my ear/brain system, with my audio system, in my room, with a wide variety of music over many weeks before I can determine if I like it or not.

 

This can be done blind to calm any fears you might have if you have someone willing to assist you. However, avoid quick A-B'ing as your brain will defeat you. I explained this in more detail in Post 26

 

Blind or sighted AB or ABX testing does not work as it fails to reveal anything expect very, very large differences. The biggest problems being:

 

  • Cognitive bias - your brain will fill in missing information thus making both sound the same on repeated switching.

  • Listener Fatigue - switch back and forth too much and both will sound like crap.

To conclude: there is no replacing long term listening with a iron-clad money-back guarantee. A/B’ing doesn’t work with any of the five human senses as our brain tends to equalize things. Comparing anything audio is one of the hardest and time consuming things I have ever done, I abhor it! I prefer listening to music for pleasure, and only compare when I absolutely have to.

I have dementia. I save all my posts in a text file I call Forums.  I do a search in that file to find out what I said or did in the past.

 

I still love music.

 

Teresa

Link to comment

Blind or sighted AB or ABX testing does not work as it fails to reveal anything expect very, very large differences. The biggest problems being:

 

  • Cognitive bias - your brain will fill in missing information thus making both sound the same on repeated switching.

  • Listener Fatigue - switch back and forth too much and both will sound like crap.

 

There is solid peer-reviewed scientific literature supporting what you say here.

One never knows, do one? - Fats Waller

The fairest thing we can experience is the mysterious. It is the fundamental emotion which stands at the cradle of true art and true science. - Einstein

Computer, Audirvana -> optical Ethernet to Fitlet3 -> Fibbr Alpha Optical USB -> iFi NEO iDSD DAC -> Apollon Audio 1ET400A Mini (Purifi based) -> Vandersteen 3A Signature.

Link to comment

LOL...

 

I believe it's also called "having a life".

 

;-)

 

I have the answer...invite them over for a beverage of their choice and have a listen to their favorite tracks. Then go to their place and have a listen with your favorite beverage and your favorite tracks. Then go together to a concert and discuss the music over your favorite beverages.

 

It's called friendship...

 

John

Source:

*Aurender N100 (no internal disk : LAN optically isolated via FMC with *LPS) > DIY 5cm USB link (5v rail removed / ground lift switch - split for *LPS) > Intona Industrial (injected *LPS / internally shielded with copper tape) > DIY 5cm USB link (5v rail removed / ground lift switch) > W4S Recovery (*LPS) > DIY 2cm USB adaptor (5v rail removed / ground lift switch) > *Auralic VEGA (EXACT : balanced)

 

Control:

*Jeff Rowland CAPRI S2 (balanced)

 

Playback:

2 x Revel B15a subs (balanced) > ATC SCM 50 ASL (balanced - 80Hz HPF from subs)

 

Misc:

*Via Power Inspired AG1500 AC Regenerator

LPS: 3 x Swagman Lab Audiophile Signature Edition (W4S, Intona & FMC)

Storage: QNAP TS-253Pro 2x 3Tb, 8Gb RAM

Cables: DIY heavy gauge solid silver (balanced)

Mains: dedicated distribution board with 5 x 2 socket ring mains, all mains cables: Mark Grant Black Series DSP 2.5 Dual Screen

Link to comment
"If following all that you do not feel much like undertaking a "controlled listening test", I cannot blame you."

 

 

Then you fall right back to any review done without these controls is strictly an opinion with no proven scientific validity.

You can't avoid scientific confirmation and call your proclaimed results fact. It just don't work that way.

This is a silly, unscientific attitude - you want to accept a finding just because it has a smell of science about it but when it's pointed out what you need to do to make it truly scientific you balk at the requirements?

 

Results are either valid & rigorous or not. Both sighted & pseudo-scientific "blind" tests are neither.

 

Live with it!

Link to comment
Then you fall right back to any review done without these controls is strictly an opinion with no proven scientific validity.

You can't avoid scientific confirmation and call your proclaimed results fact. It just don't work that way.

 

"None so deaf as those that will not hear. None so blind as those that will not see"

Link to comment
I think there's even more to it than this, involving some specifics regarding the science of human hearing. Been reading academic papers and hope to write an article eventually. But Real Life is busy and it's proving very hard to find the time, so no one hold should be holding his/her breath in anticipation.

Hey, just give us an executive summary or links, please - very interested in this, Jud.

Link to comment

I have no possibility of auditioning even a small portion of the components I'm interested in. Reviews help me narrow down the components I decide to make a special effort to hear. If I'm also familiar with how someone writes and what they like, I can be pretty sure I will like/prefer components they prefer. At the moment, there are even a couple of reviewers who have the same DAC and speakers as me, so I give extra weight to their reviews.

 

So measurements are welcome, but I'm more interested in hearing what these people think of various components. I've heard lots of devices with great measurements that I personally didn't like (others did).

Main listening (small home office):

Main setup: Surge protector +>Isol-8 Mini sub Axis Power Strip/Isolation>QuietPC Low Noise Server>Roon (Audiolense DRC)>Stack Audio Link II>Kii Control>Kii Three (on their own electric circuit) >GIK Room Treatments.

Secondary Path: Server with Audiolense RC>RPi4 or analog>Cayin iDAC6 MKII (tube mode) (XLR)>Kii Three .

Bedroom: SBTouch to Cambridge Soundworks Desktop Setup.
Living Room/Kitchen: Ropieee (RPi3b+ with touchscreen) + Schiit Modi3E to a pair of Morel Hogtalare. 

All absolute statements about audio are false :)

Link to comment
I have no possibility of auditioning even a small portion of the components I'm interested in. Reviews help me narrow down the components I decide to make a special effort to hear. If I'm also familiar with how someone writes and what they like, I can be pretty sure I will like/prefer components they prefer. At the moment, there are even a couple of reviewers who have the same DAC and speakers as me, so I give extra weight to their reviews.

 

So measurements are welcome, but I'm more interested in hearing what these people think of various components. I've heard lots of devices with great measurements that I personally didn't like (others did).

Indeed & this is the sensible approach to evaluating what works for you - a personal evaluation in your own playback system.

Anybody who decides (or ignores) on a purchase of audio equipment based solely on audio reviewers (or "blind tests") is operating on the basis of a belief system founded on very shaky premises

Link to comment
What's so hard for you subjective listeners to understand?

It's very simple,

Without blind listening confirmation any review will all ways be just that persons opinion.

You can protest all you like and tell the world what "golden ears" you have, but any review has no more validity than the next guys without proof of blind tests and measurements.

 

Actually, you make a perfectly valid point.

 

The question back to you is, so what's wrong with taking the opinion of someone who is both talented and skilled in listening to audio equipment?

 

In fact, those are exactly what you want to gather, opinions. Then listen and form your own.

 

Would you say that it makes more sense to buy a piece of kit based purely on the technical specifications, without listening to it? Or does it make more sense to buy a piece of kit based upon a few expert opinions, again without listening to the gear first?

 

Me? I want the specs first, then the opinions, then a listening trial of my own. Perhaps two or three trials even.

 

-Paul

Anyone who considers protocol unimportant has never dealt with a cat DAC.

Robert A. Heinlein

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now



×
×
  • Create New...