Should blind testing discussion be banned on CA? POLL

[esldude]

I just noticed that DBT thread has been deleted. Pity it wasn't just closed as I thought there was some useful information on it & I feel the message about DBTs lack of error rate for false negatives is a very important one - needs to be aired & more people made aware of it.

 

Fact of the matter is blind tests can be useful as one of many personal checks on our auditory perception - it's the pre-eminence that is attached to it, by some, that is grating.

 

I do agree that, on audio forums, the only real problem is the use of DBTs as a way of derailing a thread & it doesn't have to be aggressively done to be annoying, just being done insistently is enough.

 

DBT's are really a last resort. You can post measurements, explain processes, describe signal flow and go over why some sonic perception seems unlikely (in some cases impossible), and people will insist it is heard anyway. Complaints you measure the wrong thing, not everything is known etc. etc. soon follow. So if you know nothing else about how something could occur you say, "well sh*T, we don't have to know why or anything, if you can show you hear this unlikely effect blind then it is for real, we just will have to figure out why later". So it isn't so much a pre-eminence as a last resort. And really, if no one can determine why you are hearing something you claim to hear and you can't demonstrate you hear it then what is one to do? How do you then pursue the effect other than chaotically?

[christopher3393]

"Makes me wonder why it is so damn hard to tell 'ideas and beliefs' apart from 'forum behaviors' ?

It's not rocket science, is it ?"

 

Good question! I wonder if, on this forum, the 2 are quite tangled. I also recall the previous poll and remember that the main problem some have with dbt is the manner in which it is brought up.

 

For me the problem is not with the scientific language used, by any side, but with the rhetorical language when it is polemical. Remember that Polemos was the greek god of war. Now sometimes confrontational language can be helpful. There can be productively confrontational conversations if they are truly civil. But it must be clear by now that this is volatile stuff. It requires a good deal of tact, I think, to pull it off without having it blow up in our collective faces.

 

The thing I could do without is some of the snark that offers a generalized ridicule of audiophiles, sometimes masked as humor, sometimes not. Some of you seem to revel in this stuff. It is like it never gets old for you! It is now very old for me.

[Daudio]

DBT's are really a last resort. You can post measurements, explain processes, describe signal flow and go over why some sonic perception seems unlikely (in some cases impossible), and people will insist it is heard anyway. Complaints you measure the wrong thing, not everything is known etc. etc. soon follow. So if you know nothing else about how something could occur you say, "well sh*T, we don't have to know why or anything, if you can show you hear this unlikely effect blind then it is for real, we just will have to figure out why later". So it isn't so much a pre-eminence as a last resort. And really, if no one can determine why you are hearing something you claim to hear and you can't demonstrate you hear it then what is one to do? How do you then pursue the effect other than chaotically?

 

WTF ?

 

Dennis, you have hijacked your own thread ! It is not about "how cool DBT's are and the competition is crap", it is about the acceptance/not of banning a specific thread subject. Says so in the title !

 

So why are you trolling us with this off topic post about "my side is better then yours" ? Are you trying to influence the poll results ?

[alfe]

WTF ?

 

Dennis, you have hijacked your own thread ! It is not about "how cool DBT's are and the competition is crap", it is about the acceptance/not of banning a specific thread subject. Says so in the title !

 

So why are you trolling us with this off topic post about "my side is better then yours" ? Are you trying to influence the poll results ?

 

Nearly changed my vote after reading Dennis post:)

[christopher3393]

deleted

[Jud]

And really, if no one can determine why you are hearing something you claim to hear and you can't demonstrate you hear it then what is one to do?

 

Let it go?

[4est]

+1

Let it go?

[Paul R]

First reeftanks. Now bikes?

 

"We are such stuff as dreams are made on, rounded with a little sleep".

 

Besides, horses have hoofs, not wheels. Or is it hooves?

[Paul R]

Let it go?

 

=+1!

[esldude]

WTF ?

 

Dennis, you have hijacked your own thread ! It is not about "how cool DBT's are and the competition is crap", it is about the acceptance/not of banning a specific thread subject. Says so in the title !

 

So why are you trolling us with this off topic post about "my side is better then yours" ? Are you trying to influence the poll results ?

 

 

I clearly communicated myself poorly if this was your reading of it.

 

DBT's ultimately are about understanding something. In my opinion they aren't preferred. And in use they are as I described the last resort.

 

One can waste much time chasing phantoms. If something can be measured you may be onto something. If not, a DBT is firstly a last resort step to understand if something is happening. It might even be wrong. But as I said, if you are person wishing to understand something where can you go after that?

 

If understanding is not of any concern, and just posting impressions and rolling along on faith they are all right, then no problems. Or if getting listening impressions, making assumptions and seeing if it pans out or not work then fine. But it is usually at best very inefficient.

 

Finally I am not sure why discussing the relationship between DBT and other audiophile topics is something out of place in a poll on whether it should be disuccesed. Heck maybe I need to change my vote.

[kumakuma]

Yes. DBTs are the third rail of audiophile forums.

[Superdad]

Is it less hubris to have and abide by multiple pieces of evidence or to say I heard it and you should not dispute what I heard while offering no other evidence in support? An abundance of evidence will lead to confidence in someone's ideas. A single bit of evidence not corrobated makes one touchy if they have nothing else in support beyond other anecdotes from other people.

 

Except that people demanding more evidence (DBTs or measurements) are never satisfied in the least. Why is it that multiple anecdotes of people around the world experiencing the same thing is not considered as valid evidence that something audible is going on?

 

Dennis, do you not trust ANY anecdotal reporting? If you read a dozen reviews on Yelp where the people said the food and service at a restaurant was terrible and made them sick, will you demand photos, doctor's notes and lab reports before you believe that they had a bad experience? Of it a dozen people report that an item they bought on Amazon was cheaply made crap, will you discount that without documented proof?

 

And in audio, if dozens of people say that HQ Player's Poly-sinc family of filters are the best sounding of the bunch, what kind of "proof" has to be published in order for you to give that some credence?

 

I know that you have had past experiences where you felt your ears deceived you, but I hope that you have not lost all trust in those very fine instruments you were born with. I truly believe it is ultimately far more relevant to use a combination of solid bench engineering and sighted listening to determine if a product or design decision is more or less musically accurate. Measurement parameters well correlated enough to what we hear don't yet exist, and while someone is welcome to administer a blind test to me (my system, my room, my music, and my choice of start/stop/switch), I have never in 35 years felt one was necessary to validate my choices. Either I hear a difference or I don't.

 

But when lots of people praise a product, device, or piece of s/w and describe differences they heard, then there IS something going on that warrants further investigation. Not that I am saying I will take the word of the crowd and believe that the DUT is an improvement and not a euphonic embellishment, but I will at least believe that they heard something repeatable.

 

Ciao,

--Alex C.

[mmerrill99]

Nearly changed my vote after reading Dennis post:)

Yea, how do I get back into the poll to reverse my vote - I'm convinced by Dennis - it's just that I don't know what I'm convinced of

or As L Cohen says:

I know that I’m forgiven,

But I don’t know how I know

I don’t trust my inner feelings –

Inner feelings come and go.

[sandyk]

What's funny is that the vote on that one was pretty evenly split. BTW, unlike this one, that poll was a public one (in that you could see the member names after voting).

 

AlexC

You still can.

Click on the results number shown for each option.

[mmerrill99]

Exactly, Alex C. - it's the wisdom of the crowd at play - independent impressions from enough people all stating the same, more or less, characteristics apply, is ample evidence for me to personally investigate further.

 

I'm all for tests which could be relied upon to "validate" listening but I find the majority of audio DBTs have unknown sensitivity & it's inappropriate to call such a test, a test of "last resort"

 

From Wiki:

Sensitivity and specificity are statistical measures of the performance of a binary classificationtest, also known in statistics as classification function. Sensitivity (also called the true positive rate, or the recall rate in some fields) measures the proportion of actual positives which are correctly identified as such (e.g., the percentage of sick people who are correctly identified as having the condition), and is complementary to the false negative rate. Specificity (sometimes called the true negative rate) measures the proportion of negatives which are correctly identified as such (e.g., the percentage of healthy people who are correctly identified as not having the condition), and is complementary to thefalse positive rate.A perfect predictor would be described as 100% sensitive (e.g., all sick are identified as sick) and 100% specific (e.g., all healthy are not identified as sick); however, theoretically any predictor will possess a minimum error bound known as the Bayes error rate.

For any test, there is usually a trade-off between the measures. For instance, in an airport security setting in which one is testing for potential threats to safety, scanners may be set to trigger on low-risk items like belt buckles and keys (low specificity), in order to reduce the risk of missing objects that do pose a threat to the aircraft and those aboard (high sensitivity). This trade-off can be represented graphically as a receiver operating characteristic curve

[Allan F]

Is it less hubris to have and abide by multiple pieces of evidence or to say I heard it and you should not dispute what I heard while offering no other evidence in support? An abundance of evidence will lead to confidence in someone's ideas. A single bit of evidence not corrobated [sic] makes one touchy if they have nothing else in support beyond other anecdotes from other people.

 

Rather self-serving, wouldn't you say, Dennis? Lack of corroboration may make you "touchy", but I hardly think that describes the reaction of most people. As far as I can see, multiple reports by different individuals that they hear the same thing, absent proof in the form of "scientific evidence", are treated with the same disdain as a single anecdotal claim by those who dispute such reports.

[sandyk]

Except that people demanding more evidence are never satisfied in the least. Why is it that multiple anecdotes of people around the world experiencing the same thing is not considered as valid evidence that something audible is going on?

 

Dennis, do you not trust ANY anecdotal reporting? If you read a dozen reviews on Yelp where the people said the food and service at a restaurant was terrible and made them sick, will you demand photos, doctor's notes and lab reports before you believe that they had a bad experience? Of it a dozen people report that an item they bought on Amazon was cheaply made crap, will you discount that without documented proof?

 

And in audio, if dozens of people say that HQ Player's Poly-sinc family of filters are the best sounding of the bunch, what kind of "proof" has to be published in order for you to give that some credence?

 

I know that you have had past experiences where you felt your ears deceived you, but I hope that you have not lost all trust in those very fine instruments you were born with. I truly believe it is ultimately far more relevant to use a combination of solid bench engineering and sighted listening to determine if a product or design decision is more or less musically accurate. Measurement parameters well correlated enough to what we hear don't yet exist, and while someone is welcome to administer a blind test to me (my system, my room, my music, and my choice of start/stop/switch), I have never in 35 years felt one was necessary to validate my choices. Either I hear a difference or I don't.

 

But when lots of people praise a product, device, or piece of s/w and describe differences they heard, then there IS something going on that warrants further investigation. Not that I am saying I will take the word of the crowd and believe that the DUT is an improvement and not a euphonic embellishment, but I will at least believe that they heard something repeatable.

 

Ciao,

--Alex C.

 

+1

Also, why is it that Alex C has offered many times to demonstrate the things that he reports, to those that live within a reasonable travelling distance from him, as well as enjoying his hospitality, but very few, if any, appear to have taken him up on his generous offer ? It's also a chance to make new friends and discover new recorded music.

[Daudio]

DBT's ultimately are ...

...

...

blah, blah, DBT, blah, bah,

...

...

I am not sure why discussing the relationship between DBT and other audiophile topics is something out of place in a poll on whether it should be disuccesed. Heck maybe I need to change my vote.

 

I guess it's because pushing more propaganda is not the wisest course of action, if you don't want it banned

[esldude]

Except that people demanding more evidence (DBTs or measurements) are never satisfied in the least. Why is it that multiple anecdotes of people around the world experiencing the same thing is not considered as valid evidence that something audible is going on?

 

I don't see that so much. I see people finding no reason for a sound difference. People claim there is one. They do a blind test and find nothing. People still claim one and say since they hear one the test is wrong.

 

 

Dennis, do you not trust ANY anecdotal reporting? If you read a dozen reviews on Yelp where the people said the food and service at a restaurant was terrible and made them sick, will you demand photos, doctor's notes and lab reports before you believe that they had a bad experience? Of it a dozen people report that an item they bought on Amazon was cheaply made crap, will you discount that without documented proof?

 

All depends on the circumstances. Someone says they got a product and it died no I don't need a test. Someone says wire A sounds so much better than wire B, yeah I want some more evidence. People say service was poor at a restaurant I can believe that. Such isn't exactly on the margins of perception are they. Hundreds in an area say they saw strangle lights and agree on the time and color of the lights. I can believe it. People say they saw aliens, even thousands of them, I'll be looking for confirmation beyond the anecdote thank you.

 

And in audio, if dozens of people say that HQ Player's Poly-sinc family of filters are the best sounding of the bunch, what kind of "proof" has to be published in order for you to give that some credence?

 

I would like to see some more evidence. As this is the kind of thing people report which is marginal and often turns out not to be so.

 

 

....... I have never in 35 years felt one was necessary to validate my choices. Either I hear a difference or I don't.

 

Yes, but that isn't validation is it. And what would you think about an ABX where you didn't hear a difference, but could reliably choose while under blind listening conditions?

 

But when lots of people praise a product, device, or piece of s/w and describe differences they heard, then there IS something going on that warrants further investigation. Not that I am saying I will take the word of the crowd and believe that the DUT is an improvement and not a euphonic embellishment, but I will at least believe that they heard something repeatable.

 

Well sometimes they do and sometimes they don't. Usually there is some definable reason upon deeper investigation. If you have trouble finding one and the effect is relativly small, it is a good time to be a step or two more skeptical.

[sandyk]

and while someone is welcome to administer a blind test to me (my system, my room, my music, and my choice of start/stop/switch), I have never in 35 years felt one was necessary to validate my choices. Either I hear a difference or I don't.

Alex C.

Do you have a spare guest room that Dennis could stay in overnight to help make a long distance visit worth his while ?

Otherwise we will be still having the same discussions in 2020 !

[Daudio]

I don't see that so much...

All depends on the circumstances...

I would like to see some more evidence...

Yes, but that isn't validation is it...

Well sometimes they do and sometimes they don't...

 

 

Weasel words all. Sad to see you still hiding behind the same old dogma

 

Just because you discovered that your ears fail you sometimes, does NOT mean that everyone else's ears fail them ALL the time. Surely you can see that ?

 

And the DBT's that we are realisticly talking about here, have been proven to be quite imperfect. Like ears ! Knocked off their little 'Perfection' pedestal, you'd have to admit.

 

I don't get this all or nothing business. That is, it has to be all DBT, or all Ears. Why can't it be some of both ? I find that a balance is usually the more effective and elegant solution.

[esldude]

Weasel words all. Sad to see you still hiding behind the same old dogma

 

Just because you discovered that your ears fail you sometimes, does NOT mean that everyone else's ears fail them ALL the time. Surely you can see that ?

 

And the DBT's that we are realisticly talking about here, have been proven to be quite imperfect. Like ears ! Knocked off their little 'Perfection' pedestal, you'd have to admit.

 

I don't get this all or nothing business. That is, it has to be all DBT, or all Ears. Why can't it be some of both ? I find that a balance is usually the more effective and elegant solution.

 

Sorry, assuming you are human, your ears can fail you too.

 

I never said it had to be all or nothing. Stop putting words in my mouth. It can be both. When the two disagree which one are you going to think is right? Well it will depend on the circumstances. Maybe weasel words to you, but more like someone who knows it is rarely black or white.

[Skeptic]

Except that people demanding more evidence (DBTs or measurements) are never satisfied in the least. Why is it that multiple anecdotes of people around the world experiencing the same thing is not considered as valid evidence that something audible is going on?

Because there are too many variables outside of a properly conducted test which may skew the results in favor of one format.

 

I don't deny that some people are able to identify the difference between a high-res track and a cd-quality track in their own personal system when doing sighted listening.

 

Are they still able to do it with unsighted testing when someone else controls the test? (not DBT, but just asking a friend/family member to try changing the tracks)

Are they still able to do it when the CD-quality track is properly upsampled to the same rate as the high-res one?

Or upsampling both to the highest rate that the DAC supports? (e.g. 16/44 and 24/88 to 32/352)

What if the high-res file is downsampled to 24/44, or converted to 16/88?

What if the 16/44 track was not correctly dithered, or the conversions listed above were not correctly dithered?

 

Is something else in their setup "giving away" the fact that the high-res track is the one playing and that's what is being identified, rather than actual audible information in the file?

E.g. a specific resonance in the room, a change in distortion characteristics from the speaker (perhaps causing the tweeter to ring), a very slight click at the start of DSD playback or a slight difference in the delay when switching sample rates etc? It may not be a conscious thing, but something which allows the tracks to be identified.

 

Is there a ground loop or other grounding issue in their audio setup which is not resulting in a noticeable hum over the background audio level, but transmitting noise over the USB connection between the PC and the DAC, thus showing differences between USB cables, which would not exist if this issue were fixed? (rather than spending thousands on a USB cable to treat the symptoms instead of than the cause)

 

The noise which is being transferred over the cable may even be linked to CPU usage or disk access, and that changes the character of the sound when comparing different lossless/uncompressed file types for example. Again, they may be identifiable in your setup due to some highly specific quirk, but you should look to address the cause, not the symptoms of that issue.

 

If you read a dozen reviews on Yelp where the people said the food and service at a restaurant was terrible and made them sick, will you demand photos, doctor's notes and lab reports before you believe that they had a bad experience? Of it a dozen people report that an item they bought on Amazon was cheaply made crap, will you discount that without documented proof?

Is that item costing thousands, if not tens of thousands of dollars, details of which seem to fly in the face of generally accepted science? (i.e. the limits of human hearing)

 

 

Now I will say that I don't necessarily agree that blind testing is the be-all and end-all.

I did my own testing a while back comparing a few headphone amplifiers/DACs, and when they were all hooked up to the switch box (which I changed at random) I was unable to identify any of them.

And yet when I did a "sighted" test by removing the switch box and swapping the connections myself, differences between them were quite clear to me.

Now perhaps the switch box was the issue there, or something else, but when I did further testing it was obvious that there were real differences between them. With one of them you could play a tone in one channel, listen to the other side, and still be able to hear the tone, while the better amp/DAC was completely silent - as it should have been.

But when playing music and having everything connected to a switch box, I couldn't identify one from the other in this test.

 

So I generally lean towards the side of DBT being useful to confirm a positive result (e.g. there is a difference between X and Y) but would say that a negative DBT result could mean that more investigation/testing is warranted, if people are still reporting that there is a difference to be heard.

 

But that does not make those people right either. It could very well be something else in their setup which lets them identify X from Y, rather than what they believe the cause to be.

[sandyk]

But when playing music and having everything connected to a switch box, I couldn't identify one from the other in this test.

With a typical switch box there will be a small degradation due to using 2 cables and an extra set of RCA connectors (plugs and sockets). There is also likely to be a reduction in channel separation (added capacitance or inductive coupling) due to the switching used, PCB tracks etc. and possibly a reduction in S/N.

When using the switch box, are you still using the original interconnect PLUS an additional interconnect, which means more total capacitance seen by the source device ?

Is there a ground loop or other grounding issue in their audio setup which is not resulting in a noticeable hum over the background audio level, but transmitting noise over the USB connection between the PC and the DAC, thus showing differences between USB cables, which would not exist if this issue were fixed? (rather than spending thousands on a USB cable to treat the symptoms instead of than the cause)

I have covered that very real issue quite a few times in posts over several years.

[audiventory]

With a typical switch box there will be a small degradation due to using 2 cables and an extra set of RCA connectors (plugs and sockets). There is also likely to be a reduction in channel separation (added capacitance or inductive coupling) due to the switching used, PCB tracks etc. and possibly a reduction in S/N.

When using the switch box, are you still using the original interconnect PLUS an additional interconnect, which means more total capacitance seen by the source device ?

 

If switch box accuratelly manufactured is has very very small influence to sound. But bad manufactured perfectly catch low frequency signal 50...60 Hz by power network.

 

If even supposed some degradation of the switchbox, we don't compare "with vs. without switchbox".

 

Switchbox must have less influence to sound than we suppose for compared feature usually 3 ... 10 times.

 

Possible this value different in different countries, more details must be available in local metrological standards.

Example:

We compare two signals with different levels (listen or not loudness difference 3 dB).

Switchbox must have diference between inputs less 3 dB/10=0.3 dB.

 

When we read results of DBT we must consider, as example, channel separation (stereo crosstalk) of switchbox when double blind test comparing quality of stereo pictures by two amps. Possible with other switchbox we get other results.

 

Therefore before DBT created methodics where noted all used equipment.

 

Other words, result each DBT is not final true. Only true for used conditions (equipment, methodics, people,...).