Jump to content
IGNORED

The flaws of blind listening tests


Recommended Posts

11 hours ago, Blackmorec said:

Expectation sure...expectation bias, not necessarily.  One doesn’t automatically lead to the other. You can expect an outcome, perform a test and find your expectation isn’t fulfilled. Surely that happens often in hi-if. I’ve tried various footers under my amps but in each case preferred the amp sitting directly on the rack. If I hadn’t expected the footers to make an improvement I wouldn’t have tried them. They didn’t so I didn’t buy them, because my initial expectation was unfulfilled. 

I suspect (in fact I KNOW), that if one has just spent heavy pocket lettuce on a pair of Nordost (or equivalent) interconnects, one is going to hear a difference between those new cables and the old ones whether that difference exists or not. That's Expectational Bias. Usually the new, expensive component will sound better than the old one (that's our egos getting involved, and again, whether it's really better or not), because that's human nature. We need a mechanism to remove that type of self delusion from the process, and so far DBTs of one style or another are the only sure-fire way of doing that as far as I know.

George

Link to comment
4 hours ago, Jud said:

 

Although for this particular topic, the background, or at least curiosity and desire for education, should be in the human auditory system and associated areas.

Actually'flaws in blind listening tests' include not using statistics thereby getting noisy results.  For example,  is also helpful to know  that the environment can also cause things like transducers to vary in characteristics.  The various aspects of human hearing ARE important, but there are lots of really interesting areas in that field which are not important.  There are other fields of study which will also help to get the best possible results.   I wouldn't spend all of my time resource on  human 'hearing', but also study the areas of the appropriate fields which will help to get/process good end results.  BTW -- even the general idea of 'collecting data about human hearing' isn't sufficent to test the listening characteristics of audio equipment (but cannot hurt to already know as much as possible.)

 

When reading papers (studying for a work purpose), sometimes a certain focus is needed.  Much of the time, we EEs, DSP, and software people (I am all three) cannot be basic experts in all of the areas where we must work, so for the sake of efficiency, studying appropriate  areas with a focus on the needed information (basically 'getting to the point') is critical.

In this instance, trying to determine and fix the flaws in listening tests, understanding the techniques needed to remove as much bias as possible, while STRUCTURING the test is just as important as knowing the loudness curves or (for example) problems with hearing in old age (and that problem IS causing me troubles -- with rather frustrating changes in my hearing from day to day!!!)   Also -- appropr iate for the N hemisphere right now --  the big change in humidity in homes due to outside air temperature can signifcantly change the characterstics of certain kinds of transducers.  Also, home brew equipment is likely to be 'eccentric' relative to lots of factors (power supply variations/etc), so making sure to consider those factors might also be important.   There are very many (external) things that can mistakenly help to cause more error than really need be.  Lots of strange external factors can creep in -- then the results can have more error than need be.

 

I don't think that focusing primarily on the characteristics of human hearing is best allocation of resources, but some background in hearing, stats, engineering, and attempt to avoid personal bias (yes -- I know -- really cannot eliminate personal bias, but we can try) will make it possible to get usable AND CONVINCING results. 

 

BTW -- I don't basically disagree with you that it is INFINITELY more useful to do a bit of research before giving personal opinons.  If I hadn't done the 'data collection and presentation' thing many times, I probably wouldn't have written much about this.

 

John

Link to comment
3 hours ago, Kal Rubinson said:

OK.  So, how about a specific, even if particular, example?

 

Well, on this forum I've laid out the steps on the first rig that came together, 30 years ago. And pointed out the thread, on my blog, that shows the steps I'm going through with the current NAD combo. Is this what you're thinking of?

 

If I was starting from scratch, right now, with money to "get the job done fast", one option I would seriously consider are the Dutch & Dutch active monitors that are the current flavour of the month - they appear to get most things right in raw form; it's highly likely not much extra would be needed to extract the necessary integrity.

Link to comment
1 hour ago, Blackmorec said:

Then, I’ve encountered many situations where adding an extremely well reviewed item simply hasn’t brought the hoped for results.  This is often to do with the other components you’re ‘mixing’ it with. 

 

Sorry my post is OT but just pointing out the common practice among audiophiles. 

 

I don’t inderstand why you have to mix in the first place. There are many manufacturer who make all the equipment and yet audiophiles prefer to mix them with other brands. Somehow, the manufacturer who designed the preamp and amplifier missed the magical combination. Strange. 

Link to comment
7 hours ago, John Dyson said:

Actually'flaws in blind listening tests' include not using statistics thereby getting noisy results.  For example,  is also helpful to know  that the environment can also cause things like transducers to vary in characteristics.  The various aspects of human hearing ARE important, but there are lots of really interesting areas in that field which are not important.  There are other fields of study which will also help to get the best possible results.   I wouldn't spend all of my time resource on  human 'hearing', but also study the areas of the appropriate fields which will help to get/process good end results.  BTW -- even the general idea of 'collecting data about human hearing' isn't sufficent to test the listening characteristics of audio equipment (but cannot hurt to already know as much as possible.)

 

When reading papers (studying for a work purpose), sometimes a certain focus is needed.  Much of the time, we EEs, DSP, and software people (I am all three) cannot be basic experts in all of the areas where we must work, so for the sake of efficiency, studying appropriate  areas with a focus on the needed information (basically 'getting to the point') is critical.

In this instance, trying to determine and fix the flaws in listening tests, understanding the techniques needed to remove as much bias as possible, while STRUCTURING the test is just as important as knowing the loudness curves or (for example) problems with hearing in old age (and that problem IS causing me troubles -- with rather frustrating changes in my hearing from day to day!!!)   Also -- appropr iate for the N hemisphere right now --  the big change in humidity in homes due to outside air temperature can signifcantly change the characterstics of certain kinds of transducers.  Also, home brew equipment is likely to be 'eccentric' relative to lots of factors (power supply variations/etc), so making sure to consider those factors might also be important.   There are very many (external) things that can mistakenly help to cause more error than really need be.  Lots of strange external factors can creep in -- then the results can have more error than need be.

 

I don't think that focusing primarily on the characteristics of human hearing is best allocation of resources, but some background in hearing, stats, engineering, and attempt to avoid personal bias (yes -- I know -- really cannot eliminate personal bias, but we can try) will make it possible to get usable AND CONVINCING results. 

 

BTW -- I don't basically disagree with you that it is INFINITELY more useful to do a bit of research before giving personal opinons.  If I hadn't done the 'data collection and presentation' thing many times, I probably wouldn't have written much about this.

 

John

 

Very nice, thank you for this. I was aware of a lot of it, but nice to see it collected.

 

The one thing I'll add is that being aware of some quirks of ear/brain operation can help to understand which aspects of test setup may be surprisingly important or surprisingly unimportant.

One never knows, do one? - Fats Waller

The fairest thing we can experience is the mysterious. It is the fundamental emotion which stands at the cradle of true art and true science. - Einstein

Computer, Audirvana -> optical Ethernet to Fitlet3 -> Fibbr Alpha Optical USB -> iFi NEO iDSD DAC -> Apollon Audio 1ET400A Mini (Purifi based) -> Vandersteen 3A Signature.

Link to comment
11 minutes ago, Blackmorec said:

Manufacturers making all components is very common at the consumer level...Denon, Sony, Panasonic, Technics etc. But rarely the case at the specialist audiophile level....Magico, Rockport,  YG Acoustics, Tannoy, Constellation, Boulder, BAT, Devialet, Innuos, Nordost,  Synergistic Research, Shunyata, Entreq etc etc. 

 

...but i have also seen some mixing up despite having all separates. Recent visit to all Traingle Art Master Reference setup but speakers from different manufacturer. IMO, the Triangle Art speakers were the best of last years AV show that I attended. 

 

And you also have suggestion from experts that best combination should tube preamps and SS amps. Some mix to have their preferred flavour. 

 

Once, i replaced my Classe preamp with Supratek. Yes...that tube sound which was....,nice. 

Link to comment
3 hours ago, Blackmorec said:

In all cases, my expectations were set by the glowing reviews but in all cases my ears told me that in my system those enhancements weren’t getting me closer to my goal of greater musical enjoyment.

 

I agree with this 100%. I spent several weeks at the budget end of the HiFi market building my computer-sourced HiFi system through trial and error. Highly-rated components (e.g. a DAC) caused me no end of grief with noise, as did many cables. Some components didn't produce the sound that emotionally involved me. After trialling many different components, cables, speakers, I settled upon a set of components that worked well together in my 2.1 system. In fact, I even had to get my IEC power cable replaced by the manufacturer of my subwoofer to get it 'dark' enough for my ears, one which has a mysterious black box in the middle of the cable. Imagine that, replacing an IEC power cable!!!

 

I enjoy the music from this system for around 5 hours a day when I'm working from home.

 

Back in the 80s/90s it took me around 10 years to build up a decent HiFi system through 2-3 upgrades of components and cables every year, all through trial and error.

 

That's why it's a hobby for me - it's the same as cycling, photography, etc. All are wallet-emptying but thoroughly enjoyable and absorbing. It's a top-down approach for me - start off with a sound in your mind that you are seeking and then hook together the tech that delivers that sound.

Link to comment
7 hours ago, fas42 said:

 

Well, on this forum I've laid out the steps on the first rig that came together, 30 years ago. And pointed out the thread, on my blog, that shows the steps I'm going through with the current NAD combo. Is this what you're thinking of?

Possibly.  Do you have a link to those?

Kal Rubinson

Senior Contributing Editor, Stereophile

 

Link to comment
13 hours ago, Allan F said:

 

Another "pablum" generality completely devoid of any substantive content. Why don't you tell us how great it is to breathe air? Bye Bye.

I agree with the idea that it is frustrating to actually communicate when someone is trying to express themselves  without the engineering/math/scientific expertise.  Sometimes a provincial language emerges that is used to communicate amongst the non technical individuals.  I don't know the answer to the matter other than 1) a semi-permanent division between those who really understand and those who THINK that they understand  OR 2) an attempt to communicate and inform/educate those who are coming up to speed. (Maybe a combo of the two choices -- there might be more choices, just giving examples.)

 

I do have a somewhat 'soft spot' for those who are missing some technical background - being a very flexible and quick learning engineer, then I sometimes have a LOT to learn in specialty fields.  Sometimes I am clumsy when communicating with the experts in a specialty.

Even though I am a fairly competent DSP/EE/and software engineer/developer, there are areas where I have limited knowledge.  Just happens is that I am currently learning a little bit about REAL professional mastering/recording, etc.   I am NOT an expert in the field at all, but my software project is a tool that such individuals might need (probably the first fully functional software DolbyA decoder that REALLY sounds like a DolbyA.)   Just because I wrote the software, developed the rather sophisticated algorithms to emulate rather eccentric hardware design, doesn't mean that I really know how to professionally use the device :-).

Just trying to explain -- I do have toleration for those with a bit of pseudo-knowledge, as long as those people are willing to learn something new.  In fields other than my own real expertise and experience, I might seem to be full of 'pseudo-knowhow' also!!!

However -- I do not like to deal with people who have lots of pseudo-knowledge and NOT have an open mind to learn.

 

Just my opinion....

John

Link to comment

I think what you get are people with a good knowledge about measurements and measuring who often have strong opinions based on what they know.  And you get people who do not have the skill/education in electronic test and measurement,  but who may have strong opinions based on what they hear, which may by the way may be accurate, sensitive and quite well refined without suffering from expectational bias, like a skilled wine taster, who is not a chemist but can still correctly differentiate wines and their ‘attributes’ . 

Link to comment
23 minutes ago, Blackmorec said:

I think what you get are people with a good knowledge about measurements and measuring who often have strong opinions based on what they know.  And you get people who do not have the skill/education in electronic test and measurement,  but who may have strong opinions based on what they hear, which may by the way may be accurate, sensitive and quite well refined without suffering from expectational bias, like a skilled wine taster, who is not a chemist but can still correctly differentiate wines and their ‘attributes’ . 

 

I'm sure skilled wine tasters would make similar arguments with regards to flaws inherent to blind testing.

 

https://www.theguardian.com/lifeandstyle/2013/jun/23/wine-tasting-junk-science-analysis

 

 

Link to comment
29 minutes ago, Blackmorec said:

like a skilled wine taster, who is not a chemist but can still correctly differentiate wines and their ‘attributes’ . 

 

Or like a flat-earther, who can tell that the earth is flat by just looking. All measurements of curvature are wrong, it's plainly obvious to anyone only willing to open their eyes.

 

Link to comment
5 hours ago, John Dyson said:

I agree with the idea that it is frustrating to actually communicate when someone is trying to express themselves  without the engineering/math/scientific expertise.  Sometimes a provincial language emerges that is used to communicate amongst the non technical individuals.

 

John,

I appreciate your opinion. However, IMO the issue in this instance is neither a question of difficulty of expression nor a lack of expertise. Rather, It has to do with repeating the same nonsense about good and bad recordings, including the notion that tweaking playback equipment can convert bad recordings into good ones, absolutely convinced that he is right. For example:

Quote

"Bad [recordings]" are just recordings put down with no concern for the 'sensitivities' of the playback equipment. "Good recordings" are the Politically Correct ones, they have been painstakingly doctored so that they offend no-one, no matter how 'bad' the status of their playback rig is.  And the ones that become very, very boring when one has a competent system - like gulping down large slabs of strained baby food."

 

Thank God for Bad Recordings! They have given me immense pleasure for the years, because they take no prisoners; all the excitement of intensely felt music making is bound up in them - may there be no "Auntie" music in heaven!.

 

I will avoid further frustration by simply ignoring his posts. :)

"Relax, it's only hi-fi. There's never been a hi-fi emergency." - Roy Hall

"Not everything that can be counted counts, and not everything that counts can be counted." - William Bruce Cameron

 

Link to comment

No matter what you do, or how many disciplines you are trained in, or cross-trained in, "there [will always be] areas [related to sound reproduction] where [you will] have limited knowledge."

 

As examples: consider that "hearing" requires understanding the basics of acoustics, neuro-physiology, AND cognitive psychology.

 

equipment design can require digital electronics, analog electronics, AND vibration analysis.

Link to comment
2 hours ago, pkane2001 said:

 

Or like a flat-earther, who can tell that the earth is flat by just looking. All measurements of curvature are wrong, it's plainly obvious to anyone only willing to open their eyes.

 

Surely you mean 'only willing to close their eyes'?

System (i): Stack Audio Link > 2Qute+MCRU psu; Gyrodec/SME V/Hana SL/EAT E-Glo Petit/Magnum Dynalab FT101A) > PrimaLuna Evo 100 amp > Klipsch RP-600M/REL T5x subs

System (ii): Allo USB Signature > Bel Canto uLink+AQVOX psu > Chord Hugo > APPJ EL34 > Tandy LX5/REL Tzero v3 subs

System (iii) KEF LS50W/KEF R400b subs

 

Link to comment
19 minutes ago, pkane2001 said:

 

flat-earther = trust your open eyes :)

 

Maybe ! I couldn't help finding your comment 'you only need to open your eyes' at odds with defending blind tests.

System (i): Stack Audio Link > 2Qute+MCRU psu; Gyrodec/SME V/Hana SL/EAT E-Glo Petit/Magnum Dynalab FT101A) > PrimaLuna Evo 100 amp > Klipsch RP-600M/REL T5x subs

System (ii): Allo USB Signature > Bel Canto uLink+AQVOX psu > Chord Hugo > APPJ EL34 > Tandy LX5/REL Tzero v3 subs

System (iii) KEF LS50W/KEF R400b subs

 

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now



×
×
  • Create New...