The flaws of blind listening tests

[Jud]

In this discussion, it is not necessary to reinvent the wheel. There are a wealth of academic peer reviewed papers available on the relevant topics.  I personally am aware of dozens, and I'm sure there are at least hundreds if not thousands.  So instead of everyone opining, we would all likely learn more if people brought up relevant papers and discussed them.

[Jud]

39 minutes ago, John Dyson said:

Not everyone who develops an opinon has the math/engineering/signal processing background for the opinion to have meaning in the real world.

 

Although for this particular topic, the background, or at least curiosity and desire for education, should be in the human auditory system and associated areas.

[Jud]

56 minutes ago, gmgraves said:

I suspect (in fact I KNOW), that if one has just spent heavy pocket lettuce on a pair of Nordost (or equivalent) interconnects, one is going to hear a difference between those new cables and the old ones whether that difference exists or not. That's Expectational Bias. Usually the new, expensive component will sound better than the old one (that's our egos getting involved, and again, whether it's really better or not), because that's human nature. We need a mechanism to remove that type of self delusion from the process, and so far DBTs of one style or another are the only sure-fire way of doing that as far as I know.

 

And which expectation bias is operating when I like something cheaper and less sleek-looking better?

 

Expectation bias is certainly ubiquitous, but when you can use it to explain absolutely everything, it doesn't get very deep.

[Jud]

7 hours ago, John Dyson said:

Actually'flaws in blind listening tests' include not using statistics thereby getting noisy results.  For example,  is also helpful to know  that the environment can also cause things like transducers to vary in characteristics.  The various aspects of human hearing ARE important, but there are lots of really interesting areas in that field which are not important.  There are other fields of study which will also help to get the best possible results.   I wouldn't spend all of my time resource on  human 'hearing', but also study the areas of the appropriate fields which will help to get/process good end results.  BTW -- even the general idea of 'collecting data about human hearing' isn't sufficent to test the listening characteristics of audio equipment (but cannot hurt to already know as much as possible.)

 

When reading papers (studying for a work purpose), sometimes a certain focus is needed.  Much of the time, we EEs, DSP, and software people (I am all three) cannot be basic experts in all of the areas where we must work, so for the sake of efficiency, studying appropriate  areas with a focus on the needed information (basically 'getting to the point') is critical.

In this instance, trying to determine and fix the flaws in listening tests, understanding the techniques needed to remove as much bias as possible, while STRUCTURING the test is just as important as knowing the loudness curves or (for example) problems with hearing in old age (and that problem IS causing me troubles -- with rather frustrating changes in my hearing from day to day!!!)   Also -- appropr iate for the N hemisphere right now --  the big change in humidity in homes due to outside air temperature can signifcantly change the characterstics of certain kinds of transducers.  Also, home brew equipment is likely to be 'eccentric' relative to lots of factors (power supply variations/etc), so making sure to consider those factors might also be important.   There are very many (external) things that can mistakenly help to cause more error than really need be.  Lots of strange external factors can creep in -- then the results can have more error than need be.

 

I don't think that focusing primarily on the characteristics of human hearing is best allocation of resources, but some background in hearing, stats, engineering, and attempt to avoid personal bias (yes -- I know -- really cannot eliminate personal bias, but we can try) will make it possible to get usable AND CONVINCING results. 

 

BTW -- I don't basically disagree with you that it is INFINITELY more useful to do a bit of research before giving personal opinons.  If I hadn't done the 'data collection and presentation' thing many times, I probably wouldn't have written much about this.

 

John

 

Very nice, thank you for this. I was aware of a lot of it, but nice to see it collected.

 

The one thing I'll add is that being aware of some quirks of ear/brain operation can help to understand which aspects of test setup may be surprisingly important or surprisingly unimportant.

[Jud]

3 hours ago, Sonicularity said:

 

I'm sure skilled wine tasters would make similar arguments with regards to flaws inherent to blind testing.

 

https://www.theguardian.com/lifeandstyle/2013/jun/23/wine-tasting-junk-science-analysis

 

 

 

There are similar blind tests where carbon fiber violins beat out Stradivariuses.  But if you look a little deeper, you find these listening tests aren't controlled for loudness, and the louder violins win, which is no surprise.

 

That's what I'd like to do with listening tests for audio (at least the loudness-controlled ones) - get a little deeper and see *why* the results are as they are.

[Jud]

7 minutes ago, John Dyson said:

You gotta realize that the 'psychoacoustical' issues are also able to be expressed in technical terms.  All of this isn't rocket science nowadays -- it might have been 50-70yrs ago, but not today.

 

I feel, at least from my reading, that there's a bit of a gap between the leading psychoacoustic research and the understanding of psychoacoustics by audio electronics engineers.  But I'd be happy to see where I'm wrong.

 

One other area where I feel we're not quite all the way there yet is modeling the behavior of full audio systems made up of individual audio components and connections, analogously to the way we model audio circuits made up of individual circuit components and connections.

[Jud]

@John Dyson and @March Audio: For example, are you familiar with this academic research that is quite relevant to how one ought to set up a listening test (presumably blinded)?: http://deutsch.ucsd.edu/psychology/pages.php?i=209  A quote from this page:

 

Quote

Most people find the pitch memory judgment much easier when spoken numbers rather than tones are played during the interval between the test tones. This contrasts remarkably with the difficulty that is experienced when extra tones are played during this interval, even though the extra tones can be ignored. So we can conclude that the pitch of a tone is held in a specialized memory store, and that interference takes place between pitches inside this store. Other materials - such as spoken numbers - do not enter the store, so they produce much less interference with memory for pitch.

 

The conditions that caused the worst recall for music in the experiment were playing a tone, playing intervening tones, then trying to remember the initial tone.  Sounds a great deal like sequential A/B listening comparisons, doesn't it?

 

Are you familiar with work by the same researcher and many others that shows we hear surprisingly differently from each other, affected by both nature and nurture?  http://deutsch.ucsd.edu/psychology/pages.php?i=201  A quote from this page:

 

Quote

 

How do we explain these striking perceptual discrepancies? In the case of stereo illusions such as the Octave Illusion, the Scale Illusion, the Chromatic Illusion, the Cambiata Illusion, and the Glissando Illusion, disagreements tend to arise between righthanders and lefthanders, indicating that they reflect variations in brain organization. In contrast, the way the Tritone Paradox is perceived varies with the geographical region in which listener grew up, so differences here are related to the languages or dialects to which people are exposed.

 

Another theme that runs through these illusions concerns relationships between music and speech. During much of the twentieth century researchers believed that these two forms of communication are governed by different brain mechanisms; however, evidence has recently accumulated that music and speech are subserved by overlapping neural circuitries. Several of the illusions described here, particularly the Speech-to-Song Illusion and the Tritone Paradox, provide strong evidence that mechanisms underlying music and speech are intertwined.

 

Other illusions, such as Phantom Words, and Mysterious Melody, as well as the stereo illusions referred to above, point to the strong influence of our knowledge, beliefs, and expectations on how we perceive speech and music. The importance of these influences on perception has been illustrated by many examples in vision, and the audio illusions presented here reflect the same influences in the auditory realm.

 

The illusions described here lead us to wonder what other curiosities of music perception might exist that have not yet been discovered. But using the principles that generate these illusions, we can now produce music that sounds radically different from one listener to another, and even from one audience to another.  [Emphasis added.]

 

 

There is other research on such things as what factors are most important in distinguishing instruments from each other.  Seminal work on this dates back to the 1960s.  Capturing the initial inharmonic attack of an instrument has been found to be extremely important.  When designing digital filters, is due emphasis placed on this capability?

 

So: (1) The usual method of testing, including blinded testing, makes use of a procedure shown to be bad for musical memory.  (2) When we talk to each other about what we are hearing, we cannot even be sure, based on our genes and upbringing, whether we are hearing the same thing in the same way.  (3) I personally haven't read engineers discuss the work on criticality of initial inharmonic attack to recognition of musical sounds and instruments in talking about design of such things as filters for DACs, or amps, but perhaps it goes on and I'm not privy to it.

 

And these are just for starters.    Yes, I agree the way forward is through engineering.  However, I feel that perhaps we still have a way to go to get to an ultimate understanding of what makes things sound "real" to each of us as individuals, and how to achieve that semblance of reality.

[Jud]

2 hours ago, John Dyson said:

3 hours ago, Jud said:

The real problem that you are talking about has to do with the environment and transducers, less so the digital/analog signal processing.

 

Actually, I was thinking more about things like ground and noise currents in actual systems in people's homes.  Unlike SPICE for circuit components, we don't have good software for modeling system topology: Whether you'll get less noise by connecting your DAC, amp, preamp, etc., with cables that have electrical characteristics A, or cables that have electrical characteristics B; with the power cords plugged in with arrangement Y or arrangement Z; etc.

[Jud]

1 hour ago, John Dyson said:

Ground noise is the biggest bugaboo by far.

 

What I've seen on the forums isn't so much the notion of jitter propagating, but of (1) ground noise getting into clocking circuitry and causing jitter; and/or (2) poor signal integrity causing input circuitry in the DAC to be electrically noisy, resulting in electrical noise getting into the clock circuitry and causing jitter.  Whether and how much these matter I don't have the technical knowledge to say.

[Jud]

On 1/13/2019 at 2:20 AM, John Dyson said:

Firstly, there is a demand for a superior quality DolbyA decoder for historical archives.  Much material is DolbyA encoded, and the normal DolbyA HW splats distortion all over the place (the well known 'softening' of the sound.)

Additionally, and more important to the consumer, a lot of that 'NOT QUITE RIGHT'  sounding digital material is actually DolbyA encoded -- as a consumer, you just dont' realize it.  So, if you got that horrible sounding ABBA, Carpenters (even from HDtracks) or whomever recording from before the 1990s, and want a master tape quality copy -- just use my DolbyA decoder.  IT IS A REAL ISSUE.

Refer to my repository -- the files with 'DAencoded' are DolbyA encoded, and files with 'DHDA', 'mastered' or whatever like that have been passed through my decoder.  THE PROBLEM OF LEAKED-INTO-CONSUMER-SPACE  DOLBYA IS REAL.  The sound of DolbyA tends to be fairly subtle on compressed/limited material -- but without decoding, the quality is inferior.

 

Repository:

https://spaces.hightail.com/space/yDG3L339Rn

 

Note that I dont' expect or intend to make any money on the project -- however, well known gov't historical institutions and grammy award winning engineers have shown interest in our project.  The formal name is DHNRDS, and there is a very primitive WWW site (not quite ready for prime time yet) for the project.  (I don't control the PR side of the project -- my recording engineer friend IS in control of the marketing/informational side...  I am just the author/owner of the software -- my engineer friend and myself own the project together. He has full usage/distribution rights to the software so that he can make decisions withiout asking me.)  He has been a WONDEROUS help in keeping me straight on what is needed...

 

john

 

Have you read any of the discussions surrounding Dolby processing (don't know what letter) and Steely Dan's "Katy Lied"?  Supposedly the sound quality of the master was particularly stellar until someone ran it through Dolby processing.

 

Eidt: Looks like my memory was wrong, it was dbx noise reduction that at least partially/initially messed things up.