The flaws of blind listening tests

[Blackmorec]

Speak to a guru and he will tell you that human consciousness has many different levels, which are possible to experience through meditation. Listening also has several different levels.  Listen while not really concentrating and what you hear is the homogination of all the voices and instruments. Concentrate and depending on what you concentrate on, you may hear the tune and timbre of an individual instrument or if you concentrate on the music, you’ll hear all the instruments and their interplay, with your attention jumping from instrument to instrument as each catches your attention. 

The problems with blind listening are several fold. Let’s say you are evaluating a new cable.  You listen to the original cable and then to what may be the original or new cable, the test being whether you can actually hear a difference. Here’s the problem. Your baseline comprises the sum of the original music, the changes made to it by the recording process and the changes made to it by the replay system.  You now listen to the second playing. What you are hearing is the music, the changes made to it by the recording, the changes made by the replay system and the changes, if any made by removing the old cable and replacing it with the new cable. Now that’s already pretty complex but combine that with the way we listen by concentrating on different things within the music and its no surprise that confusion reigns in blind testing. There are simply far too many variables. The evaluated subject matter (music) is overly complex and the test process is subject to emotions and the flighty, spotlighting nature of human concentration. 

In my experience the best way I’ve found to test anything related to audio is human voice and spoken word.  We are very practiced at hearing voices (the joke’s too obvious to be funny),  very attuned to minor changes in pitch, cadence, emphasis, tone etc. Get a good recording of human voice and familiarise yourself with all its characteristics, then use that in blind testing. You’ll find that even quite subtle differences are easy to hear because there is only 1 thing to concentrate on and you are very skilled at analysing human voice. You’ll also notice that while human voice is centred around a fairly narrow frequency range, changes to response at frequency extremes are still clearly heard  (chestiness, foundation, body, sparkle, mouth sounds, sibilance, presence etc). 

 

I have just spent several months tuning and Improving my network streaming supply which involved adding components like PSUs and cables then waiting for them to run in. By far the most useful tool in evaluating progress was spoken word. Coloration was particularly easy to hear, added treble (sibilance), bass emphasis (chestiness)  and it was easy to pinpoint when the system had fully run in (a natural sounding voice with no apparent colouration or anomalies). Obviously your source has to be reliable and not subject to its own instabilities. 

[Blackmorec]

2 hours ago, SoundAndMotion said:

If you had named this thread "Pitfalls and considerations for listening tests", I would praise your ideas as important and relevant. What you call "concentrating on", I would call "attention", and it is a very important consideration for test design. But what makes your ideas only relevant to "blind" tests? How, exactly does "blinding" a test harm it? And why are they "flaws", if they are taken into account in the design of the test?
 

I fully accept and agree with your comments.....this isn’t just about blind testing, its really about any listening tests, so good point. Again ‘concentrating on’ and ‘paying attention to’ are exactly the concept I was referring to. 

The reason I picked on blind testing was because it rarely reaches the conclusion you’d expect i.e being able to clearly pick out what seem like obvious differences in performance.  The reason you can reliably pick out the differences sighted is very likely because its sighted. As soon as the test is carried out blinded, the difference apparently disappears.  The point I was making but admittedly didn’t emphasise is that the whole test is flawed.....sighted or blind, because of the reasons I gave. However if the tests are carried out using human voice, the differences become much easier to identify, sighted or unsighted, because you’ve removed a great deal of complexity of the source material, focused on something we’re already absolutely expert at and removed the possibility of wandering or varying focus of attention 

[Blackmorec]

1 minute ago, Shadders said:

So, no expectation bias then.........

Expectation sure...expectation bias, not necessarily.  One doesn’t automatically lead to the other. You can expect an outcome, perform a test and find your expectation isn’t fulfilled. Surely that happens often in hi-if. I’ve tried various footers under my amps but in each case preferred the amp sitting directly on the rack. If I hadn’t expected the footers to make an improvement I wouldn’t have tried them. They didn’t so I didn’t buy them, because my initial expectation was unfulfilled. 

[Blackmorec]

1 hour ago, fas42 said:

The difficulty with listening tests is using them in the "expectation" that one can use them to find whether one component, or change is "better" than another. Doomed to failure, in my book - what matters is if the system is working to a certain standard or not - and you do what is necessary to achieve that standard. Which may entail anything and everything between, very slightly altering the position of a cable somewhere; to ejecting, discarding the whole chain of components and starting afresh - the latter because you've realised that there is a fundamental limitation in the whole setup, and nothing one can reasonably do will overcome that.

 

You're not wandering around a range of hills, looking for a pretty spot to take some photos; there is a mountain peak in the middle, and you keep your eyes firmly glued on that point, and the landscape between you and that height; and you steadily and unerringly make your way closer and closer, higher and higher - until you are firmly footed on that peak, and can turn in any direction, and see "forever" ...

Very poetic, however when you climb your mountain peak, what you see are the next peaks, higher and more magnificent than the one you’re standing on. When you achieve a system that ‘disappears’ leaving just a big, beautiful soundscape in space...no identifiable sources like speakers....when you’re there, you’ve achieved an important milestone, a system that can trick your hearing and brain into constructing a solid 3 dimensional sound stage. But that’s just a stage along the way. Get there and there are still many improvements that can be made.  Greater naturalness, dynamics, speed, rhythmic drive, beauty, communication of the musical message, emotional responses, listener involvement to name just a few areas that can continue to be improved.

[Blackmorec]

6 hours ago, gmgraves said:

I suspect (in fact I KNOW), that if one has just spent heavy pocket lettuce on a pair of Nordost (or equivalent) interconnects, one is going to hear a difference between those new cables and the old ones whether that difference exists or not. That's Expectational Bias. Usually the new, expensive component will sound better than the old one (that's our egos getting involved, and again, whether it's really better or not), because that's human nature. We need a mechanism to remove that type of self delusion from the process, and so far DBTs of one style or another are the only sure-fire way of doing that as far as I know.

Well that’s a really interesting example that I disagree with about 95%, leaving a little room for a possible grain of truth😊

Firstly, most audiophiles wouldn’t be buying the cables in the first place without a thorough evaluation to ensure they’re a significant step in the right direction. 

Then, I’ve encountered many situations where adding an extremely well reviewed item simply hasn’t brought the hoped for results.  This is often to do with the other components you’re ‘mixing’ it with. 

Example, I use a Finite Element Pagode Master Reference stand. It was designed by a university professor who presumably used university facilities to measure all resonances and design anti-resonance devices which are built into each shelf.  Despite trying several expensive and well reviewed footers and platforms I have yet to find anything that performs better than the plain FEPR shelf.  Many of the devices brought useful additions to the sound but all lost a degree of naturalness in the process. In all cases, my expectations were set by the glowing reviews but in all cases my ears told me that in my system those enhancements weren’t getting me closer to my goal of greater musical enjoyment. 

Personally I have one overriding goal...I get immense pleasure from listening to music.....anything that disturbs that pleasure is rejected whereas anything that enhanced that pleasure is added (I only listen to things I’m prepared to buy so nothing outside my budget). My ego has nothing to do with this. Satisfying and stimulating my musical pleasure centre is my only motivator. 

If there’s any emotion involved its what I’d call PPP......Post Purchase Paranoia.....usually triggered by an item going through burn-in. 

 

[Blackmorec]

25 minutes ago, STC said:

 

Sorry my post is OT but just pointing out the common practice among audiophiles. 

 

I don’t inderstand why you have to mix in the first place. There are many manufacturer who make all the equipment and yet audiophiles prefer to mix them with other brands. Somehow, the manufacturer who designed the preamp and amplifier missed the magical combination. Strange. 

Manufacturers making all components is very common at the consumer level...Denon, Sony, Panasonic, Technics etc. But rarely the case at the specialist audiophile level....Magico, Rockport,  YG Acoustics, Tannoy, Constellation, Boulder, BAT, Devialet, Innuos, Nordost,  Synergistic Research, Shunyata, Entreq etc etc. 

[Blackmorec]

I think what you get are people with a good knowledge about measurements and measuring who often have strong opinions based on what they know.  And you get people who do not have the skill/education in electronic test and measurement,  but who may have strong opinions based on what they hear, which may by the way may be accurate, sensitive and quite well refined without suffering from expectational bias, like a skilled wine taster, who is not a chemist but can still correctly differentiate wines and their ‘attributes’ .