Jump to content
IGNORED

The flaws of blind listening tests


Recommended Posts

An old timer might say something akin to:"It speaks to me.". There are few things us humans are as adept at as hearing a voice.

Forrest:

Win10 i9 9900KS/GTX1060 HQPlayer4>Win10 NAA

DSD>Pavel's DSC2.6>Bent Audio TAP>

Parasound JC1>"Naked" Quad ESL63/Tannoy PS350B subs<100Hz

Link to comment

By FAR, the biggest variation in my system (very small, limited -- only for technical reasons -- not for casual listening) is my hearing/ears.  The variation is based upon BOTH hearing and my ears.  Hearing is dependent on mental state and mental fatigue, and ears have lots of variable imperfections also.

Controlling the audio listening experiments is incredibly tedious -- because hearing doesn't just vary in the way that something sounds better or worse, but can sound better or worse in MANY ways.

Even though we attempt to measure the experimental results of my project -- sometimes listening is the only way to determine success/failure (quality.)  It is the variable of human hearing that is so difficult to control.

Even simple A/B can result in confusing results.   IMO, getting proper/maximally correct results requires statistical measurement because there are so darned many variables.  If someone believes that there are aspects of their systems that might have variability, then those issues can also be mitigated by statistical measurement.  (The only places that I believe might be a little bit variable are the transducers and whatever electronics poorly designed/drifting.  Sometimes EMI/RF and other enviornmental issues can also cause problems.)   A good example of 'electronics' issues confusing the stats might be homebrew designs that don't properly account for environmental changes.  (Back when I worked at Bell Labs, we had to design electronics that worked over enormous temperature/humdity ranges -- and the required design discipline is much wider ranging than doing a simple room temperature design.)   A minor example that often happens in the meat of audio design might be power amplifier biasing -- but professional quality designs would account for the entire circuit working at specification over VERY MINIMALLY 0 to 50C, and hopefully  -10 to 70C.   Such criteria helps with actual operation in the extended environments, and gives additional margin for reliability.

Link to comment
2 hours ago, SoundAndMotion said:

If you had named this thread "Pitfalls and considerations for listening tests", I would praise your ideas as important and relevant. What you call "concentrating on", I would call "attention", and it is a very important consideration for test design. But what makes your ideas only relevant to "blind" tests? How, exactly does "blinding" a test harm it? And why are they "flaws", if they are taken into account in the design of the test?
 

I fully accept and agree with your comments.....this isn’t just about blind testing, its really about any listening tests, so good point. Again ‘concentrating on’ and ‘paying attention to’ are exactly the concept I was referring to. 

The reason I picked on blind testing was because it rarely reaches the conclusion you’d expect i.e being able to clearly pick out what seem like obvious differences in performance.  The reason you can reliably pick out the differences sighted is very likely because its sighted. As soon as the test is carried out blinded, the difference apparently disappears.  The point I was making but admittedly didn’t emphasise is that the whole test is flawed.....sighted or blind, because of the reasons I gave. However if the tests are carried out using human voice, the differences become much easier to identify, sighted or unsighted, because you’ve removed a great deal of complexity of the source material, focused on something we’re already absolutely expert at and removed the possibility of wandering or varying focus of attention 

Link to comment

The difficulty with listening tests is using them in the "expectation" that one can use them to find whether one component, or change is "better" than another. Doomed to failure, in my book - what matters is if the system is working to a certain standard or not - and you do what is necessary to achieve that standard. Which may entail anything and everything between, very slightly altering the position of a cable somewhere; to ejecting, discarding the whole chain of components and starting afresh - the latter because you've realised that there is a fundamental limitation in the whole setup, and nothing one can reasonably do will overcome that.

 

You're not wandering around a range of hills, looking for a pretty spot to take some photos; there is a mountain peak in the middle, and you keep your eyes firmly glued on that point, and the landscape between you and that height; and you steadily and unerringly make your way closer and closer, higher and higher - until you are firmly footed on that peak, and can turn in any direction, and see "forever" ...

Link to comment

Why some are disturbed by what I say is because I don't talk in the jargon of conventional audiophile beliefs - I'm not "part of the pack". Well, I used to go down that particular road ... but then realised what's possible - to repeat Yet Another Generality :), it revolves around Subtracting Badness, not, Adding Goodness. Since this hobby is obsessed with the latter, "because it's fun!" - and my way involves sitting down and doing lots and lots of fiddly experiments and explorations, the fun factor won't be there - for many. However, it's the way for getting results, with "cheap" gear - and hence an excellent value for money route for those who may be that way inclined ...

Link to comment
7 hours ago, Jud said:

In this discussion, it is not necessary to reinvent the wheel. There are a wealth of academic peer reviewed papers available on the relevant topics.  I personally am aware of dozens, and I'm sure there are at least hundreds if not thousands.  So instead of everyone opining, we would all likely learn more if people brought up relevant papers and discussed them.

I VERY MUCH agree with your sentiments, seems that there is sometimes too much navel contemplation in similar situations, but as long as the learning progresses, and people don't get tied up in useless opinons, I believe that the exercise can be beneficial.

I do worry that there might start being too much polarization like in the 'stair step' and 'jitter' religions.  Not everyone who develops an opinon has the math/engineering/signal processing background for the opinion to have meaning in the real world. (This probably comes from a distrust of 'experts' -- in some cases, rightfully so!!!) It takes lots of discipline from EVERYONE to make the discussion progress and come to coherent conclusion without prematurely developing opinions.

It would be nice if someone actually reads a few of the papers (with references) give a synopsis and point the discussion to those papers (hopefully available.)   Probably wont' happen, and the navel contemplation will ensue  :-).

 

John

Link to comment
33 minutes ago, Allan F said:

Contrary to your view, IMO, listening tests can be very valuable in determining whether a particular change improves the overall sound quality of a system.

 

"Improve" means something very different from the the way the word "better" is usually bandied around in audiophile circles - so, I indeed use listening to determine whether I have "improved" the integrity of the playback chain.

Link to comment
39 minutes ago, John Dyson said:

Not everyone who develops an opinon has the math/engineering/signal processing background for the opinion to have meaning in the real world.

 

Although for this particular topic, the background, or at least curiosity and desire for education, should be in the human auditory system and associated areas.

One never knows, do one? - Fats Waller

The fairest thing we can experience is the mysterious. It is the fundamental emotion which stands at the cradle of true art and true science. - Einstein

Computer, Audirvana -> optical Ethernet to Fitlet3 -> Fibbr Alpha Optical USB -> iFi NEO iDSD DAC -> Apollon Audio 1ET400A Mini (Purifi based) -> Vandersteen 3A Signature.

Link to comment
1 hour ago, fas42 said:

The difficulty with listening tests is using them in the "expectation" that one can use them to find whether one component, or change is "better" than another. Doomed to failure, in my book - what matters is if the system is working to a certain standard or not - and you do what is necessary to achieve that standard. Which may entail anything and everything between, very slightly altering the position of a cable somewhere; to ejecting, discarding the whole chain of components and starting afresh - the latter because you've realised that there is a fundamental limitation in the whole setup, and nothing one can reasonably do will overcome that.

 

You're not wandering around a range of hills, looking for a pretty spot to take some photos; there is a mountain peak in the middle, and you keep your eyes firmly glued on that point, and the landscape between you and that height; and you steadily and unerringly make your way closer and closer, higher and higher - until you are firmly footed on that peak, and can turn in any direction, and see "forever" ...

Very poetic, however when you climb your mountain peak, what you see are the next peaks, higher and more magnificent than the one you’re standing on. When you achieve a system that ‘disappears’ leaving just a big, beautiful soundscape in space...no identifiable sources like speakers....when you’re there, you’ve achieved an important milestone, a system that can trick your hearing and brain into constructing a solid 3 dimensional sound stage. But that’s just a stage along the way. Get there and there are still many improvements that can be made.  Greater naturalness, dynamics, speed, rhythmic drive, beauty, communication of the musical message, emotional responses, listener involvement to name just a few areas that can continue to be improved.

Link to comment
6 minutes ago, mansr said:

"For every expert, there is an equal and opposite expert." 

There you go again, quoting Newton's third law of internet forums!

"Relax, it's only hi-fi. There's never been a hi-fi emergency." - Roy Hall

"Not everything that can be counted counts, and not everything that counts can be counted." - William Bruce Cameron

 

Link to comment
8 hours ago, Blackmorec said:

Surely that happens often in hi-if. I’ve tried various footers under my amps but in each case preferred the amp sitting directly on the rack. If I hadn’t expected the footers to make an improvement I wouldn’t have tried them. They didn’t so I didn’t buy them, because my initial expectation was unfulfilled. 

 Have you ever tried putting a CD player or something like an Oppo media player directly on a carpet instead of directly in a cabinet to see if it sounds different ?

 I wouldn't expect a normal amplifier which doesn't have sensitive components like Xtal Oscillators to benefit much from using footers, however your preference for sitting the amplifier directly in the rack suggests that something is happening .

I presume that the amplifier doesn't use Vacuum Tubes which can sometimes be sensitive to vibration ?

 

How a Digital Audio file sounds, or a Digital Video file looks, is governed to a large extent by the Power Supply area. All that Identical Checksums gives is the possibility of REGENERATING the file to close to that of the original file.

PROFILE UPDATED 13-11-2020

Link to comment
1 hour ago, Blackmorec said:

Very poetic, however when you climb your mountain peak, what you see are the next peaks, higher and more magnificent than the one you’re standing on. When you achieve a system that ‘disappears’ leaving just a big, beautiful soundscape in space...no identifiable sources like speakers....when you’re there, you’ve achieved an important milestone, a system that can trick your hearing and brain into constructing a solid 3 dimensional sound stage. But that’s just a stage along the way. Get there and there are still many improvements that can be made.  Greater naturalness, dynamics, speed, rhythmic drive, beauty, communication of the musical message, emotional responses, listener involvement to name just a few areas that can continue to be improved.

 

True. Improvements beyond that point can always be made, and I've noted that rigs that use premium components are able, now and again, to bring out some of those aspects, in some recordings. However, I normally can try some of my more 'difficult' CDs on such a setup, and they will totally botch the presentation - so, the path is to, firstly, get at least some recordings to present convincingly; secondly, work through all the 'harder' recordings so that they eventually come good; and then, thirdly, advance along all the avenues you mentioned.

 

For a variety of reasons I have gone down the road of working with what many would call marginal gear - so I've made my job harder in one sense; the pluses are that I've learnt more about the whole shebang, and I have no fears with wrecking what I'm working with - I'm game to try anything, to find out what's going on.

Link to comment
1 hour ago, Allan F said:

 

No, not when you have achieved the state of audio nirvana that he claims to have achieved! :)

 

I've come across other rigs, rarely, that get it right - the point is to understand what needs to be done, each time, to push whatever happens to be in front of you to a plateau of presentation quality.

Link to comment
48 minutes ago, sandyk said:

 Have you ever tried putting a CD player or something like an Oppo media player directly on a carpet instead of directly in a cabinet to see if it sounds different ?

 

The business of precisely how to mount each component, and how thoroughly to stabilise the cabinet, by mass loading or locking in some cradle, is a whole field of investigation in itself. Yes, it shouldn't matter, and a well engineered and implement component should show no variation - but, ...

Link to comment
55 minutes ago, Allan F said:

 

Another "pablum" generality completely devoid of any substantive content. Why don't you tell us how great it is to breathe air? Bye Bye.

 

You appear to have a need for a precise recipe. However, the recipe will be different for every individual, depending upon their particular circumstances. And unless one knows those circumstances, some sort of generalised recipe is not going to be very useful. Time wasting, in fact.

 

However, if you were to start from scratch then there's a chance ... :).

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now



×
×
  • Create New...