Controversy of ABX testing

[Mahesh]

I remember reading somewhere that the validity of ABX testing, or blind testing, is controversial in audiophile circles. I've been searching quickly for more information on this controversy, but could not find any relevant links. Could anyone provide a short resume of what the dispute is, or provide some links to discussions or articles on the subject?

 

Thanks,

Mahesh

 

[BobH]

stick - ABX testing audiophile - into google. The first page of hits should be enough!

 

ABX testing is the mechanism by which claims can be made for high-end audio products without fear of contradiction.

 

In brief, the only way to prove that an audible difference does/does not exist is via an ABX test. If the test proves your point then it is a good test. If the test disproves your point then the testing procedure is declared flawed and the results invalid. Neat! Head(ache)s I win, tails you lose.

 

[Mahesh]

Ooops, I guess I could have put a little more effort into my searching... ;-)

 

Thanks!

 

Mahesh

 

[BobH]

You'll still wish you hadn't!

 

[Mahesh]

How come?

 

Mahesh

 

[BobH]

Think - Barry's excellent effort in the current 'A first test....' thread, everything ever written about jitter, Mac or PC, cables, usb or firewire - that'll do!

 

In other words, it's only words!

 

The shortest threads on this and any other audiophile forum you care to visit, will be about music. The longest and most heated threads will be about stuff we can't prove. When it is proven, the argument will be about the validity of the proof, or the proof of the validity - anything as long as it leaves the question open.

 

Enter the infamous ABX! Scientists generally accept correctly applied ABX testing as satisfactory evidence. Audiophiles, you will find, are somewhat harder to please!

 

[Mahesh]

Bob,

 

Thanks for enlightening me. I must confess that I'm sort of attracted to those heated discussions. I see it as some sort of intellectual exercise to throw my own posts in there as good as I can. I don't care so much about who "wins" the discussions, as long as (at least some of) the arguments are of some quality.

 

I'm looking forward to taking on the ABX heat... ;-)

 

Mahesh

 

[wgscott]

I just read The Ten Biggest Lies in Audio, link originally posted by Josh when I asked about bi-wiring.

 

I'm a scientist and am amazed that this (ABX or double-blind testing) is even controversial.

 

[Shadorne]

Well even science can become corrupted when money is at stake and ethical issues are widespread these days. Some claims are most obviously beyond rationality. Usually claims are based on a grain of truth and then claims are simply blown out of all proportion. Like the panic being spread over supposed man-made global warming or the panic over the recent volcanic eruption where aircraft were grounded for 1000's of miles. There is a grain of truth in these things but the over-reaction is unjustified, much like the hyperbole over bits of wire between components or speakers. Of course, ABX testing would show that none of the outlandish wire claims are justified.

 

Pear Anjou cables for $8,000 anyone? Michael Fremer says they sound absolutely amazing...

 

[wgscott]

Double-blind trials were developed essentially to address this problem. When testing the efficacy of a highly profitable drug, for example, this eliminates those potential sources of bias. In fact, it is the ONLY recognized way to do so. Sure, some scientists make claims to the contrary, like with the claim that vaccines cause autism, but they are only taken seriously by conspiracy theorists and ex-playboy models.

 

[Mahesh]

wgscott,

 

Thanks for the link! I fell for this paragraph:

 

"The tweako cultists will tell you that ABX tests are completely invalid. Everybody knows that a Krell sounds better than a Pioneer, so if they are indistinguishable from each other in an ABX test, then the ABX method is all wet—that’s their logic. Everybody knows that Joe is taller than Mike, so if they both measure exactly 5 feet 11.25 inches, then there is something wrong with the Stanley tape measure, right?"

 

I also have a scientific background, and try to navigate my way through the hifi jungle the best I can. Many times it feels almost like attending a seminar on alternative medicine...

 

Have you seen this article: http://sound.westhost.com/cables.htm

 

I would also recommend James Randi's dispute with the high-end audio business and Stereophile in particular, quite entertaining to read:

 

http://www.randi.org/site/index.php/component/search/stereophile.html?ordering=&searchphrase=all

 

Mahesh

 

[cfmsp]

 

"Of course, ABX testing would show that none of the outlandish wire claims are justified."

 

 

I would respectfully submit that if you're never heard the difference that a cable can make in a system, then you don't need audiophile gear - any gear will do.

 

The issue with ABX testing controversy is, those who hear differences (in cables, for instance) could care less about 'proving' it to anyone, for the most part. The feeling being, I"m not gonna do their (i.e. non-believers) work for them. If they don't believe, that's their issue.

 

In audiophile circles, ABX testing is almost exclusively recommended by non-believers (of a tweak, cable difference, etc.) to discredit the beliefs of those who don't give a shit if the non-believers believe them or not.

 

In fact, ABX testing can never prove that a phenomena does not exist, only that it didn't test as a positive difference in the particular instance of testing.

 

IOW, those who don't believe (in a tweak) can't use ABX to test their own hypothesis. I.e., that they perform the test on themselves and don't detect a difference does NOT mean that there is no difference.

 

 

The fact that any single ABX test is NOT incontrovertible evidence is another factor in it's lack of use.

 

Non-believers question whether someone got 'lucky' when they pass a test (that they expected to prove out negatively), and believers point to the fact that any 'negative' ABX test does not universally apply it only means that a difference could not be detected by the testers in a particular instance.

 

 

All that said,

This is not to say that (I believe) all differences reported are real - far from it IN MY OPINION!

 

This is just to point out the nature of the controversy that seems to have resulted in a stalemate within the audiophile-dom.

 

 

 

YMMV, and probably will.

 

clay

 

 

 

 

 

 

 

 

 

[K Shep]

of great information regarding ABX.

 

http://seanolive.blogspot.com/2009/04/dishonesty-of-sighted-audio-product.html

 

[Mahesh]

clay,

 

If someone claims to be able to distinguish cable A from cable B, and fails when put through a (properly executed) ABX test with the cables, we have shown that his or her claim does not hold. No? Of course, it goes the other way as well, if the person passes the ABX test, he/she can truly claim his or her abilities.

 

I don't think anyone has claimed that a single ABX test proves that it is universally impossible to hear a difference, but it certainly does hold as evidence for the particular case.

 

I must add that, personally, I think it is important to be open to all sorts of phenomenon. In my opinion, I think James Randi, for instance, maybe goes a little bit over the top in his quest for convincing the world that Science and Logic are the only ways to see the world. Opening a nice bottle of wine, putting on my favorite music (on a hifi system with or without biwiring ;-), and sitting next to the person I love makes me feel like forgetting about science and logic, and just enjoy life.

 

Regards,

Mahesh

 

[wgscott]

There are parallel arguments one encounters when discussing religion.

 

Most people who believe in God will acknowledge that there is no controlled experiment that can either prove or disprove the existence of their favored deity. They then conclude that the non-believer has no more or less of a claim to legitimacy than they do, experimental tests cannot favor one hypothesis (the deity's existence) over another (the deity's non-existence) and so the two beliefs must therefore be somehow logically equivalent.

 

This is a bit of a fallacy. The null hypothesis is the simplest explanation. eg: there is no deity, there is no physical way anything but inductance, resistance and capacitance can influence the quality of audio transmissions through speaker wire, etc. The contrary claims are more involved, eg: a deity exists, so there must be some phenomenon that is super-natural and un-accounted for if this deity can have any influence on the world, or there must be some law of physics violated or incorrectly applied or not fully understood when describing audio transmission through speaker wire. Both are, as Carl Sagan would have said, extraordinary claims that require extraordinary burdens of proof. Their logical negations, on the other hand, are very simple claims.

 

Put slightly differently, one must be able to describe under what conditions he would consider his claim to be refuted. For example, I think my $0.31 optical toslink cable is just as good sounding as $50 ones. Maybe it is, and maybe it isn't. If a double-blind test reveals a listener can tell the difference, I will be willing to admit my assertion is wrong. But since I cannot fathom why the streams of 0s and 1s would be any different, my hypothesis that all optical cables that work all sound the same is the null hypothesis, and the burden of proof should rest on the shoulders of those who claim otherwise (and presumably want either to sell you a more expensive cable or at least feel justified in their own purchase of a more expensive cable).

 

[cfmsp]

 

"If someone claims to be able to distinguish cable A from cable B, and fails when put through a (properly executed) ABX test with the cables, we have shown that his or her claim does not hold. No?"

 

Actually, no. You'd have only shown that the difference could not be demonstrated under the ABX test conditions!

 

Mahesh, you perhaps missed a distinction I made above. Those who claim to hear differences but who have no interest in proving to those who 'don't hear differences' are not interested in putting themselves through an ABX test.

 

There are various, and probably valid reasons, for not doing so.

 

One of the reasons is that ABX testing does not lend itself well to distinguishing differences that are not easily heard via instantaneous back & forth comparison testing. There are others, as well.

 

So, we're back to the quandary I mentioned above, where each side believes the other's objections are basically invalid.

 

clay

 

 

 

 

 

[Mahesh]

wgscott,

 

"[...] The contrary claims are more involved, eg: a deity exists, so there must be some phenomenon that is super-natural and un-accounted for if this deity can have any influence on the world, or there must be some law of physics violated or incorrectly applied or not fully understood when describing audio transmission through speaker wire. Both are, as Carl Sagan would have said, extraordinary claims that require extraordinary burdens of proof. Their logical negations, on the other hand, are very simple claims."

 

I agree. But I see no point in a believer in God being obliged to prove that God exists. (I mean, if he or she wanted to, that's ok, but I see no legitimacy in that the non-believer can demand that from him or her.) As long as you use phrases like "I believe ...", "I think ...", "I can hear ...", it doesn't come with an obligation to prove anything. Along the same lines, I see no legitimacy in requiring from an audiophile to take an ABX test in order to prove what he or she hears (or not).

 

But the moment you are marketing a product that you charge money for, or if you're an audio reviewer influencing the way audiophiles spend their money, then any extraordinary claim should be backed by scientific arguments and evidence. Unfortunately, we all know that is not the case.

 

As a side note, we must not forget that some of the most important breakthroughs in science have come from people that went against what the majority considered to be true. Therefore, I think it is important to keep an open mind to claims that on first thought sounds unreasonable. That being said, all paradigm shifts in science have come about due to the new radical theory being backed by irrefutable evidence.

 

Mahesh

 

[cfmsp]

"But since I cannot fathom why the streams of 0s and 1s would be any different, my hypothesis that all optical cables that work all sound the same is the null hypothesis, and the burden of proof should rest on the shoulders of those who claim otherwise "

 

 

This null hypothesis would put you into the camp I referred to as 'non-believers'.

 

 

Those who believe in a particular hypothesis (such as cables make a difference) start with the opposite null hypothesis based on their own individual experience of having heard (or believing that they have heard) a difference.

 

They would respond to non-believers with - "Right Back Atcha!" That is to say, we believe we can hear a difference, therefore "the burden of proof should rest on the shoulders of those who claim otherwise".

 

In the absence of belief in the validity of ABX testing (e,g,, the potential that an ABX test will produce false negatives), these folks (i.e. the believers) are not willing to undergo the most practical testing that the non-believers can use to refute the believer's claims.

 

and therein lies the problem. Let me repeat, the believers could care less if the non-believers are convinced.

 

clay

 

 

 

[Mahesh]

clay,

 

I don't think our opinions are that different, see my reply to wgscott above. :-)

 

I still stand by my point, though, that claiming something to be universally true requires to be backed by scientific arguments and evidence. And don't forget that the very principles of physics that let you enjoy your hifi system are based on the same fundamental logical reasoning that makes the ABX test scientifically valid. ;-)

 

Mahesh

 

[cfmsp]

 

"the very principles of physics [...] are based on the same fundamental logical reasoning", which is to say, it's all based on mathematical probabilities!

 

;0

 

 

clay

 

 

PS, wgscott, I forgot to say (in the event it wasn't obvious) that I agree with your use of "parallel arguments" to describe the situation.

 

[Mahesh]

clay,

 

Mathematical probability is the way we measure the degree of certainty of the outcome of the ABX test. The test design is based on logical reasoning, just like all scientific research methods...

 

Mahesh :-)

 

[Audio_ELF]

clay ... I know in a lot of things we do seam to be on a similar wavelength but I would like to expand on your statement here a little.

 

clay said... "Those who claim to hear differences but who have no interest in proving to those who 'don't hear differences' are not interested in putting themselves through an ABX test. :)"

 

I have no issue with people who are not interested in proving to others that differences are real ... but on forums (this one included) there are a lot of claims maid which are stated in a very militant way, and when anyone disagrees with them the original claim is repeated repeatedly without any shred of evidence except for listening in non-blind situations. I'm happy people have faith in their claims and their equipment, etc. but when you challenge them they cannot offer any explanation or scientific testing - having completed some ABX testing could offer at least a modicum of scientifically accepted proof that though it can't be directly measured (or easily explained) there is some difference in the sound output.

 

So yes, if people feel that £8,000 Nordost Valhalla cables make their Cambridge Audio DACMagic sound better than a dCS Scarlatti stack (to them) fine. But if they are unwilling to do any testing to prove those claims ... to my mind the skeptics are entitled to adopt a "put up or shut up" attitude.

 

Eloise

 

P.S. for any ABX testing to be scientific, the situation in which the test was done must be explained and open to scrutiny and it should be possible to repeat the test with other testers, etc. If people refuse to explain what equipment was used (often questioning the relevance of this to the results) then that can automatically give doubt to the validity of such testing (talking from a scientific point of view).

 

[cfmsp]

 

 

we're saying the same thing, Mahesh.

 

My point was that the so-called laws of physics are based on mathematical probabilities, as are any results from ABX tests (as you note).

 

 

 

clay

 

[wgscott]

Outside of the quantum regime (which we can safely ignore for our discussions), the laws of physics are not at all based on mathematical probabilities.

 

[cfmsp]

 

"I'm happy people have faith in their claims and their equipment, etc. but when you challenge them they cannot offer any explanation or scientific testing - having completed some ABX testing could offer at least a modicum of scientifically accepted proof that though it can't be directly measured (or easily explained) there is some difference in the sound output."

 

Agreed. I wasn't suggesting that people believe someone else's claim without questioning. I was simply indicating the nature of the face-off, in particular that those who are not inclined to use the scientific method could care less about employing same to appease those that do.

 

In my mind, ideas presented here (and elsewhere) without scientific claim can be laid out on a continuum, which might start with generally accepted, move on to plausible, and then through unlikely to preposterous.

 

Probably many of us have heard/read claims we might consider preposterous, no matter how open-minded.

 

Where one's views are along the continuum (for that particular idea) probably has a correlation to whether they are likely to ask for evidence in the face of repeated, militant (to use your words) claims.

 

The so-called non-believers (or skeptics, if you prefer) are at one end of the continuum, and thereby their threshold for expecting evidence in the face of claims is much lower.

 

As for me, I tend to hang out in the plausible area of the continuum. It there is some credible notion to support the idea, I would not be inclined to question the claims of others very much. Indeed, I am more likely to look to help push the notion forward. If we want to understand computer audio, my thinking goes, we need to work together to understand things on the edge of our current knowledge.

 

I might appear to hang out further towards the preposterous end of the continuum (than plausible), but this is due to my belief that others should be able to share opinions without being continually shouted down by the (former) militant non-believers here (you know who they were!)

 

It's likely that this latter belief was shaped by my very own first post experience here, in which someone responded to my sharing of an experience by saying that what I experienced was simply not possible (because engineers in his employ told him so).

 

clay