Jump to content
IGNORED

The flaws of blind listening tests


Recommended Posts

1 hour ago, STC said:

 

If you are slipping in and out then it got nothing to do with engineering (or the magical tweak as you call them). The beauty of engineering is that when you slip out you just revert to the previous state. Not very difficult and will not take 30 years to do that. The stuff you are describing is more appropriate under human psychology forum discussion. 

 

It is engineering - an audio system has to be in a very good state of tune - think of a violin or guitar here; where the audience can immediately pick up the bum notes; the instrument has lost tune, because of aggressive handling or change of humidity, say. A "well engineered" musical instrument would have an internal mechanism which registers the loss of tune, and 'recalibrates', on the fly - perfect tune, always :). However, the musical playing crowd are not interested in that level of sophistication - they make do with manual fiddling ... which is how people like me are currently having to deal with the "bum notes" of audio rigs, ^_^.

 

Of course the engineering should be in place, to ensure a high standard at all times. But the raw state of the components is not adequate, currently - and part of my shtick is to explore that "bleeding edge" where a setup needs extra optimising, to make it fully stable and robust(!) while operating at peak SQ.

Link to comment
2 hours ago, John Dyson said:
3 hours ago, Jud said:

The real problem that you are talking about has to do with the environment and transducers, less so the digital/analog signal processing.

 

Actually, I was thinking more about things like ground and noise currents in actual systems in people's homes.  Unlike SPICE for circuit components, we don't have good software for modeling system topology: Whether you'll get less noise by connecting your DAC, amp, preamp, etc., with cables that have electrical characteristics A, or cables that have electrical characteristics B; with the power cords plugged in with arrangement Y or arrangement Z; etc.

One never knows, do one? - Fats Waller

The fairest thing we can experience is the mysterious. It is the fundamental emotion which stands at the cradle of true art and true science. - Einstein

Computer, Audirvana -> optical Ethernet to Fitlet3 -> Fibbr Alpha Optical USB -> iFi NEO iDSD DAC -> Apollon Audio 1ET400A Mini (Purifi based) -> Vandersteen 3A Signature.

Link to comment
25 minutes ago, Jud said:

@John Dyson and @March Audio: For example, are you familiar with this academic research that is quite relevant to how one ought to set up a listening test (presumably blinded)?: http://deutsch.ucsd.edu/psychology/pages.php?i=209  A quote from this page:

 

 

The conditions that caused the worst recall for music in the experiment were playing a tone, playing intervening tones, then trying to remember the initial tone.  Sounds a great deal like sequential A/B listening comparisons, doesn't it?

 

Are you familiar with work by the same researcher and many others that shows we hear surprisingly differently from each other, affected by both nature and nurture?  http://deutsch.ucsd.edu/psychology/pages.php?i=201  A quote from this page:

 

 

There is other research on such things as what factors are most important in distinguishing instruments from each other.  Seminal work on this dates back to the 1960s.  Capturing the initial inharmonic attack of an instrument has been found to be extremely important.  When designing digital filters, is due emphasis placed on this capability?

 

So: (1) The usual method of testing, including blinded testing, makes use of a procedure shown to be bad for musical memory.  (2) When we talk to each other about what we are hearing, we cannot even be sure, based on our genes and upbringing, whether we are hearing the same thing in the same way.  (3) I personally haven't read engineers discuss the work on criticality of initial inharmonic attack to recognition of musical sounds and instruments in talking about design of such things as filters for DACs, or amps, but perhaps it goes on and I'm not privy to it.

 

And these are just for starters.  :)  Yes, I agree the way forward is through engineering.  However, I feel that perhaps we still have a way to go to get to an ultimate understanding of what makes things sound "real" to each of us as individuals, and how to achieve that semblance of reality.

 

Lately, I am inclined to believe that DBT may not be that reliable for some types of comparisons.  

 

Recently, made a binaural recording of another system and comapred with mine. When listened through headphones, the preferenace was system A and when listened through loudspeakers the preferance was system B.  Although, it could be explained but this illustrated how some blindtests conducted by different system may give a different results. 

 

However, the important point to note here is this was about preference and difference. 

 

There are also situations where our hearing mechanism adapt to filter out certain sound which cannot be reliably distinguished under blind tests. This Is actually related to how human hearing sometimes takes about twenty minutes to normalize. It was an interesting discovery where the sequence of the blind test determines the outcome. 

Link to comment
5 hours ago, STC said:

The stuff you are describing is more appropriate under human psychology forum discussion. 

 

Um, the psychologists might send him packing to the science forums, or even worse to audio forums where science, psychology, and engineering are misrepresented in equal measure ! We show no favoritism 🤣

Sound Minds Mind Sound

 

 

Link to comment
11 hours ago, March Audio said:

So what if the wine taster refuses to accept what the engineer tells them?

I was at a wine tasting last night. The notes for one of the wines mentioned linearity. I wonder if even order distortion tastes better than odd.

Link to comment
21 hours ago, March Audio said:

 

So what if the wine taster refuses to accept what the engineer tells them? 

 

For example, "we don't know everything yet" or “there's a mechanism unknown to science going on that you can't measure which explains my subjective response". 

I agree with your concerns on that matter.  Maturity is important, and controlling ones own ego is also helpful.  i learned a long time ago in my field -- I am so damned good at what I do, until I realize that I am not so good :-).   I finally gave up on trying to be 'right' all of the time, and as a reflex action, step back and try to solve the REAL problem (whatever that might be), ignoring my ego.  My ego sucks, and I know it :-).

 

Now, my being/intention is based on solving the problem, helping to make someone feel better rather than just simply winning the war about who is right.

This maturity (and competency)  problem DOES exist in both the consumer & in the engineering communities.  However, it seems (just my impression) that as some consumers becomes more entrenched into their hobby -- some become more ego-bound and narrow minded.  A SUCCESSFUL (read: SUCCESSFUL, not just experienced) engineer will tend to become less and less ego-bound, and try to solve problems for people.

Maturity is critical, and open, honest and trusting communication is important.  The REAL problem for ALL of us are the snake-oil salespeople!!!   These money grabbers will dyseducate honest but vulnerable people with their profitable song+dance.

We engineers (who are really honest, and TRY to be helpful) can only offer help, and TRY to be kind.

 

Sometimes it can be VERY HURTFUL and DIFFICULT when some self assured and miseducated person (often by snake oil people) makes a misguided claim about the honest person's integrity or competency.  All we can do is to try to keep on doing the best that we can.  IT IS GOOD TO LISTEN, but sometimes gotta filter out the noise.

 

Anecdote:  I have been told over and over again that doing a DolbyA decoder in software is impossible, and that I am an incompent liar of some kind.  However, I and my collaborators have existence proof otherwise.  I am not trying to TELL someone that they need the results of my project, as I'll never make any money on it (far far from recouping the 2000+ person hours on the admittedly seemingly  impossible project.)  However, in some people's eyes, because of their dyseducation -- I am the 'bad' guy.

 

I have given up on trying to win little arguments about right and wrong -- gotta just do the best/right thing to help.

 

John

Link to comment
2 hours ago, John Dyson said:

  I am not trying to TELL someone that they need the results of my project, as I'll never make any money on it (far far from recouping the 2000+ person hours on the admittedly seemingly  impossible project.) 

 There does not appear to much point to your software unless it's for your own personal use, where you have identified quite a few CDs in your collection that were sloppily mastered for release , AND you play regularly.

 However, if there was made available a list of many of the sloppily mastered CDs that could be a different matter, with some people prepared to pay for the software.

However, being able to compile a list like that sounds a bit like " Mission Impossible"  too . ;)

 

How a Digital Audio file sounds, or a Digital Video file looks, is governed to a large extent by the Power Supply area. All that Identical Checksums gives is the possibility of REGENERATING the file to close to that of the original file.

PROFILE UPDATED 13-11-2020

Link to comment
7 hours ago, sandyk said:

 There does not appear to much point to your software unless it's for your own personal use, where you have identified quite a few CDs in your collection that were sloppily mastered for release , AND you play regularly.

 However, if there was made available a list of many of the sloppily mastered CDs that could be a different matter, with some people prepared to pay for the software.

However, being able to compile a list like that sounds a bit like " Mission Impossible"  too . ;)

Firstly, there is a demand for a superior quality DolbyA decoder for historical archives.  Much material is DolbyA encoded, and the normal DolbyA HW splats distortion all over the place (the well known 'softening' of the sound.)

Additionally, and more important to the consumer, a lot of that 'NOT QUITE RIGHT'  sounding digital material is actually DolbyA encoded -- as a consumer, you just dont' realize it.  So, if you got that horrible sounding ABBA, Carpenters (even from HDtracks) or whomever recording from before the 1990s, and want a master tape quality copy -- just use my DolbyA decoder.  IT IS A REAL ISSUE.

Refer to my repository -- the files with 'DAencoded' are DolbyA encoded, and files with 'DHDA', 'mastered' or whatever like that have been passed through my decoder.  THE PROBLEM OF LEAKED-INTO-CONSUMER-SPACE  DOLBYA IS REAL.  The sound of DolbyA tends to be fairly subtle on compressed/limited material -- but without decoding, the quality is inferior.

 

Repository:

https://spaces.hightail.com/space/yDG3L339Rn

 

Note that I dont' expect or intend to make any money on the project -- however, well known gov't historical institutions and grammy award winning engineers have shown interest in our project.  The formal name is DHNRDS, and there is a very primitive WWW site (not quite ready for prime time yet) for the project.  (I don't control the PR side of the project -- my recording engineer friend IS in control of the marketing/informational side...  I am just the author/owner of the software -- my engineer friend and myself own the project together. He has full usage/distribution rights to the software so that he can make decisions withiout asking me.)  He has been a WONDEROUS help in keeping me straight on what is needed...

 

john

Link to comment

 

On 1/10/2019 at 3:48 PM, Shadders said:

Hi,

It could be that the product (footers) were raved about, you did not get the expected nirvanna change in sound, so were disappointed.

 

In any case - as per Jud has said, maybe discuss papers on the subject. I don't have the will to discuss this, as it can be very convoluted and tiring. No offence.

 

Regards,

Shadders.

 

Magazine reviews and webforums produce expectation bias. There's no escaping it.

"Science draws the wave, poetry fills it with water" Teixeira de Pascoaes

 

HQPlayer Desktop / Mac mini → Intona 7054 → RME ADI-2 DAC FS (DSD256)

Link to comment
On 1/11/2019 at 10:57 PM, Jud said:

 

Actually, I was thinking more about things like ground and noise currents in actual systems in people's homes.  Unlike SPICE for circuit components, we don't have good software for modeling system topology: Whether you'll get less noise by connecting your DAC, amp, preamp, etc., with cables that have electrical characteristics A, or cables that have electrical characteristics B; with the power cords plugged in with arrangement Y or arrangement Z; etc.

There is software that is helpful for circuit topology -- but as you say, it isn't good for the application.  Such software is very often used by RF/microwave engineers - I have used such software.   Some free stuff is also available.   The best solution for normal audio work is a set of common knowlege about making sure that voltages don't develop because of inductance/resistance and ground currents.  This is one reason for the old mention of 'running all grounds to a single point'.  In some cases, that might not be best, but such noise is almost always because of  'current*impedance' developing evil voltages.

This syndrome of developing errant voltages is probably 90% of the misguided thinking that somehow clock jitter propagates through circuitry that is  reclocked/resynchronized.  The counterargument that I give about the 'jitter' issue is that internet jitter can be on the order of milliseconds or seconds, how can someone listen to music with so many long delays?  Answer:  buffering/resynchronization - and that happens in any reasonably competent recent design nowadays.

CDjitter does NOT propagate to the audio output unless there is a ground current causing voltage disturbance or equivalent analog issue.  Jitter due to clock edges on clock D flip-flip type schemes  (the usual explanation) can have some effect, but not usually in a competent design with good quality componentry.   Ground noise is the biggest bugaboo by far.

 

John

Link to comment
1 hour ago, John Dyson said:

Ground noise is the biggest bugaboo by far.

 

What I've seen on the forums isn't so much the notion of jitter propagating, but of (1) ground noise getting into clocking circuitry and causing jitter; and/or (2) poor signal integrity causing input circuitry in the DAC to be electrically noisy, resulting in electrical noise getting into the clock circuitry and causing jitter.  Whether and how much these matter I don't have the technical knowledge to say.

One never knows, do one? - Fats Waller

The fairest thing we can experience is the mysterious. It is the fundamental emotion which stands at the cradle of true art and true science. - Einstein

Computer, Audirvana -> optical Ethernet to Fitlet3 -> Fibbr Alpha Optical USB -> iFi NEO iDSD DAC -> Apollon Audio 1ET400A Mini (Purifi based) -> Vandersteen 3A Signature.

Link to comment
2 hours ago, Jud said:

 

What I've seen on the forums isn't so much the notion of jitter propagating, but of (1) ground noise getting into clocking circuitry and causing jitter; and/or (2) poor signal integrity causing input circuitry in the DAC to be electrically noisy, resulting in electrical noise getting into the clock circuitry and causing jitter.  Whether and how much these matter I don't have the technical knowledge to say.

You are hitting 'the nail on the head.'  When I mentioned 'ground noise' -- that is most often the cause of many of the effects that you are mentioning.  Being more complete -- not only ground noise, but also other kinds of conducted noise coming from currents, radiated noise effects and capacitance (which are kind of two different things -- radiation is a 'bigger' process than just capcitance or inductive coupling.)  (probably a few more basic sources that I didn't think of off the top of my head.)  (Just a note -- simple capcitive & inductive coupling fall off very quickly in their effect, while radiation goes MUCH further.)

 

Very very very small amount of the 'jittering' phenomenon (various kinds of analog noise more generally) can maybe caused a LITTLE BIT by timing variations on a clocked flipflop, but there are so many bigger sources (which you also describe the specifics) that cause troubles.  When I hear about a new 'jitter free' this or that, I cringe -- hopefully the vendor is secretly doing the right thing -- basically ignoring jitter, and simply fix the noise problems!!!


John

Link to comment
On 1/13/2019 at 2:20 AM, John Dyson said:

Firstly, there is a demand for a superior quality DolbyA decoder for historical archives.  Much material is DolbyA encoded, and the normal DolbyA HW splats distortion all over the place (the well known 'softening' of the sound.)

Additionally, and more important to the consumer, a lot of that 'NOT QUITE RIGHT'  sounding digital material is actually DolbyA encoded -- as a consumer, you just dont' realize it.  So, if you got that horrible sounding ABBA, Carpenters (even from HDtracks) or whomever recording from before the 1990s, and want a master tape quality copy -- just use my DolbyA decoder.  IT IS A REAL ISSUE.

Refer to my repository -- the files with 'DAencoded' are DolbyA encoded, and files with 'DHDA', 'mastered' or whatever like that have been passed through my decoder.  THE PROBLEM OF LEAKED-INTO-CONSUMER-SPACE  DOLBYA IS REAL.  The sound of DolbyA tends to be fairly subtle on compressed/limited material -- but without decoding, the quality is inferior.

 

Repository:

https://spaces.hightail.com/space/yDG3L339Rn

 

Note that I dont' expect or intend to make any money on the project -- however, well known gov't historical institutions and grammy award winning engineers have shown interest in our project.  The formal name is DHNRDS, and there is a very primitive WWW site (not quite ready for prime time yet) for the project.  (I don't control the PR side of the project -- my recording engineer friend IS in control of the marketing/informational side...  I am just the author/owner of the software -- my engineer friend and myself own the project together. He has full usage/distribution rights to the software so that he can make decisions withiout asking me.)  He has been a WONDEROUS help in keeping me straight on what is needed...

 

john

 

Have you read any of the discussions surrounding Dolby processing (don't know what letter) and Steely Dan's "Katy Lied"?  Supposedly the sound quality of the master was particularly stellar until someone ran it through Dolby processing.

 

Eidt: Looks like my memory was wrong, it was dbx noise reduction that at least partially/initially messed things up.

One never knows, do one? - Fats Waller

The fairest thing we can experience is the mysterious. It is the fundamental emotion which stands at the cradle of true art and true science. - Einstein

Computer, Audirvana -> optical Ethernet to Fitlet3 -> Fibbr Alpha Optical USB -> iFi NEO iDSD DAC -> Apollon Audio 1ET400A Mini (Purifi based) -> Vandersteen 3A Signature.

Link to comment
20 minutes ago, Jud said:

 

Have you read any of the discussions surrounding Dolby processing (don't know what letter) and Steely Dan's "Katy Lied"?  Supposedly the sound quality of the master was particularly stellar until someone ran it through Dolby processing.

 

Eidt: Looks like my memory was wrong, it was dbx noise reduction that at least partially/initially messed things up.

Both DolbyA and DBX I have their evils.   A lot of recording engineers really hate DolbyA because it creates distortion.  DBX is worse when it comes to noise modulation.  I am trying to mitigate some of the DolbyA evils.

 

John

Link to comment
2 hours ago, John Dyson said:

Both DolbyA and DBX I have their evils.   A lot of recording engineers really hate DolbyA because it creates distortion.  DBX is worse when it comes to noise modulation.  I am trying to mitigate some of the DolbyA evils.

 

John

 

Judging by the marked improvement with " ABBA-DayBeforeYouCame-DHDA" with a great deal of success !

 

How a Digital Audio file sounds, or a Digital Video file looks, is governed to a large extent by the Power Supply area. All that Identical Checksums gives is the possibility of REGENERATING the file to close to that of the original file.

PROFILE UPDATED 13-11-2020

Link to comment
53 minutes ago, sandyk said:

 

Judging by the marked improvement with " ABBA-DayBeforeYouCame-DHDA" with a great deal of success !

Thank you!!!  My project partner and myself have already made some more major improvements since those demos.  It is my position and driving motivation that ANY distortion of any kind is not acceptable.  My ego isn't in the 'program', my ego is entirely in the results and making my project partner and the (eventual) users happy with the results.

 

Without having a real recording engineer working with me -- and several others commenting on the various issues, the decoder would have been a total piece of cr*p.  I know DSP and programming, but the real goal is the audio -- and it aint easy!!!  I have been told over and over again that a good software DolbyA decoder is impossible -- and I think that we have an existence proof that claim is wrong :-).

 

We do have a VERY NEW VERY LIMITED web site -- barely getting started, but it is DHNRDS.com.  Serious questions should go there -- however, I can always answer questions unofficially (not as a project spokesperson.)  There is some seriously innovative DSP code in the decoder (especially avoiding the worst of the modulation sidebands when doing the fast gain control -- interesting and tricky subject!!!)  My project partner is the official interface, but again -- I can talk tech talk (e.g. describe how to make really good compressors/expanders.)

 

IN FACT -- I think that the latest play version of my 3 band RMS compressor (source include) resides on the same WWW site as the demos.  It works pretty well -- might even fool people if it is working or not (it is very subtle), and uses the same kind of RMS calculations as DBX and THATcorp (but further optimized in dynamic attack/decay, etc.)  File name: simplecompMB-V1.1B-23dec2018.zip.  I don't maintain it, but if someone needs help, I am happy to make suggestions on how to use it (there are some postings on Hydrogenaudio for simple use) and how to modify it.

 

Take care, and thanks again!!

John

Link to comment
  • 2 weeks later...
On 1/10/2019 at 7:48 PM, Jud said:

 

And which expectation bias is operating when I like something cheaper and less sleek-looking better?

 

Expectation bias is certainly ubiquitous, but when you can use it to explain absolutely everything, it doesn't get very deep.

I can't explain it, but then, I'm no psychoacoustician. Who knows what subconscious factors are at work in any individual at any one time over any given issue? I mean, there are Americans who still think that Trump is doing a good job as POTUS!

George

Link to comment
9 minutes ago, gmgraves said:

I can't explain it, but then, I'm no psychoacoustician. Who knows what subconscious factors are at work in any individual at any one time over any given issue? I mean, there are Americans who still think that Trump is doing a good job as POTUS!

Well being precise -- excellent at being an executive, even as President, but not the expected type of statesman, uses poor restraint  when communicating, and irritates the embedded parts of the government.  Is that good or bad?   In some ways, very good -- in other ways, not so good.  We have a system, the election spoke the will of the electorate per the Constitution, and lets get on with it.  It is better to do what is good for the country, not change ones stance because of hate (e.g. some people wanted the needed layer of security before Trump, and now don't want it because of Trump.)  Too much hate...  Trump might have problems, but the hate trumps Trump.

Link to comment
On 1/11/2019 at 1:24 AM, Blackmorec said:

Firstly, most audiophiles wouldn’t be buying the cables in the first place without a thorough evaluation to ensure they’re a significant step in the right direction. 

Actually, I think this statement is patently false. Most audiophiles do not evaluate cables before they buy. Sure there are a few brick and mortar stores who will let a customer take a few different cables home for evaluation (as long as there is a credit card with a high enough ceiling to cover the cost (and that can be a LOT of money) in case said cables don't come back) But most people buy on the Internet these days and very few will loan cables for evaluation. Sure you can buy them and send them back for a refund if you don't like them, but if you do that more than a couple of times, the vendor will get wise. 

Of course, this reply pre-supposes that there is an actual difference between the cables, and not some placebo effect, which, with cables that are, in fact, merely just conductors and not "boutique filters" (talking interconnects here, not speaker cables)it would be a placebo effect. 

George

Link to comment
On 1/14/2019 at 10:18 AM, John Dyson said:

Both DolbyA and DBX I have their evils.   A lot of recording engineers really hate DolbyA because it creates distortion.  DBX is worse when it comes to noise modulation.  I am trying to mitigate some of the DolbyA evils.

 

John

I used to have  a Dolby-A unit that I used for all my analog recordings. I recorded a pro symphony orchestra for 5 seasons using it as well as various jazz venues (including the San Francisco Jazz Festival) for the NPR "Jazz Alive" series. Aaron Copland, Hubert Laws, Dizzy Gillespie, and Stepan Grappelli are among the luminaries that I recorded. I used 15 ips (38cm/sec) and half Track stereo using Dolby-A for all of these. I'm pretty sensitive to distortion and what I found is that if the operator hasn't been scrupulous with matching the levels with Dolby-A on both the record and playback cycles, it will introduce distortion. My opinion is that this is fairly straightforward to do, but it is time consuming and I can see many technicians getting sloppy when they have to adjust separate Dolby units for as many as 48 tracks!. I'm going to venture a guess and say that this is probably where the notion of Dolby-A (and possibly DBX with which I have no experience) introducing distortion came from. I was only recording in two channel, and therefore took the time to make sure that my R-to-R tape decks were properly and meticulously set up and paired to my Dolby-A unit. 

Gotta say, when digital recording came along, I was happy to put those laborious chores behind me!

George

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now



×
×
  • Create New...