Controversy of ABX testing

[TimDH]

Agreed that ABX testing can never tell us what to like. But it can tell us what most people can discriminate. There is a lot of similarity in audio to wine making/enjoyment. You get better at it with practice and some people can develop really great palates/ears. You can decide to enjoy modestly priced wine or really expensive stuff. Wine is actually much more subjective- I don't know that there is a master Bordeaux the way we have master audio files.

 

Now where would we be without the famous blind French tasting trials? The French connaisseurs KNEW that American wines weren't nearly as good. Until their own blind tests ranked some of the California Cabernets as among the best in the world.

 

People have every right to believe what they want. But when people vehemently talk about "night and day" differences with whatever component they want to (anybody every heard this in audio land?) then they are in fact making a claim that someone else will also hear and benefit from it too. That brings us out of pure subjectivity into the world of ABX and/or measurement testing.

 

[cfmsp]

 

"It is generally NOT possible to hear differences on well designed and well matched gear with adequate cabling over reasonable distances. (excluding ridiculous cables with filters or faulty inadequate designs, or extreme cable lengths)"

 

"The above statement, while really an opinion, is actually stated as if it is a fact."

 

 

Barrows,

You make a good point. In my opinion, the only issue with the statement is that it was presented as if fact.

 

If we assume, as I think we should, that any statement of opinion is supported by the weight of one's own experience, the comment is much more benign, reading as it would:

 

"[in my experience,] It is generally NOT possible [for me] to hear differences on well designed and well match gear...."

 

 

clay

 

 

 

[Janomatic]

Hi everybody,

this discussion has been around for a long time in the audiophile community. What many people fail to mention though is that even differences that are objectively measurable are not neccessarily detectable in an ABX-Test. Especially tests in which one would need to identify relatively small differences seem to be rather difficult to pass.

Have a look at this very simple ABX-Test which will take about three to five minutes of your time:

www.sieveking-sound.de/abx

It was designed to emulate the process of taking a test in which gaps do appear as you would imagine it to happen when exchanging components in a Hifi system. The test seems to be very humbling for many individuals because the users set their identification threshold themselves and then often find out that the thing they were so sure about before suddenly becomes very difficult to identify in an ABX-Environment.

 

Have fun doing the test. It is not a matter of life and death but is supposed to give some perspective.

 

Jan

 

[Shadorne]

Your "experience" is not. There are no scientific measurements or tests which have demonstrated meaningful or significant audible benefits of special super expensive wires over adequate wires. (as I mentioned most equipment is so bad or some conditions are so extreme (long runs) that you can get audible results but it ain't the fault of the wires)

 

There is a lot of science on wires such that engineers and physicists understand well the requirements at audio frequencies for IC's and speaker cables.

 

[The Computer Audiophile]

Hi Shadorne - Please take this in the spirit in which it's intended. It's time to put up or shut up :~) Please notice the smily face as this is not intended to be an attack against you. Rather the use of an old phrase came to mind as I write this.

 

 

"My opinion is supported by science. Your "experience" is not."

 

"There are no scientific measurements or tests which have demonstrated meaningful or significant audible benefits of special super expensive wires over adequate wires."

 

Please direct me to any scientific studies that empirically prove "adequate" wires don't have a "meaningful" or scientific audible sonic difference. If I've mixed up the words and made this impossible please forgive me. But the point should be clear. Meaningful is not a scientific term. Significant is a meaningless term because it's subjective. Any difference at all no matter how large or small is a difference. Anyway, Please direct me to any scientific study that uses scientific methods approved by the general scientific community that concludes anything about audio wires. Honestly I am interested.

 

 

 

 

"...most equipment is so bad or some conditions are so extreme (long runs) that you can get audible results but it ain't the fault of the wires"

 

Please direct me to these scientific studies.

 

 

 

By the way this link has some great quotes about the use of facts. A good one I like is from Albert Einstein. "If the facts don't fit the theory, change the facts." or an anonymous quote, "My mind is made up ... don't confuse me with the fact"

 

http://www.bored.com/findquotes/cate_479_Facts.html

 

 

[Shadorne]

Sorry Chris but the science is overwhelming. The burden of proof is to show that appropriate (but standard) wires are simply not good enough and that significant audible performance improvements can be gained through special wires (often extremely high priced). I know about "skin effect" and the tiny roll off that you can get above 12 to 15 Khz - I do understand the science and I am well supported by acoustic science when I say that these wire effects are really "negligible" relative to room issues and speaker distortion/response.

 

It is interesting when a moderator takes sides on an issue and your "put up or shut up" came across loud and clear - I didn't have to read between any lines there. I understand that your interest could very likely be related to advertising and sales conducted through your site...

 

I expect you aware that most high end systems have more invested in cables than the actual speaker transducers (the parts that convert electrical signal to sound) - lots of money to be made selling wires on the unsuspecting.

 

 

 

 

 

[The Computer Audiophile]

Hi Shadorne - There is no need to be sorry. I'm simply asking for scientific data from those who are adamant about science proving their point of view. That's all. There is nothing in my post that suggests I am taking sides. Asking for more information from someone who explicitly says "the science is overwhelming" is not taking sides. Rather it's prudent for a moderator to ask people to show some evidence when they claim it is there. I think it would be great if you could scientifically back up what you are saying. I believe that type of empirical evidence would be fabulous.

 

 

"The burden of proof is to show that appropriate (but standard) wires are simply not good enough and that significant audible performance improvements can be gained through special wires (often extremely high priced)."

 

I don't think people on either side of this discussion can decide where the burden of proof resides. Those who think cables do matter will say the other side has the burden to prove they do not and vice versa. If there is any scientific data that suggests anything related to audio cables I would love to post it here on Computer Audiophile.

 

 

"I understand that your interest could very likely be related to advertising and sales conducted through your site..."

 

There are no sales conducted through my site. Yes I accept advertisements, but that is the only way support such a site. The fact my site is open for anyone in the world to leave a comment is very important. If I ever said anything out of bounds I would certainly hear about it here on the site. Transparency is the key and I intend to keep things transparent around here. Not to be confused with the Transparent Cable Company :~)

 

By the way, the smily face and following comments in my original post were meant to show my good intentions and good spirited form of discussion. If it didn't come through that way maybe it will now that I've clarified it again.

 

 

[barrows]

"There are no scientific measurements or tests which have demonstrated meaningful or significant audible benefits of special super expensive wires over adequate wires. (as I mentioned most equipment is so bad or some conditions are so extreme (long runs) that you can get audible results but it ain't the fault of the wires) "

 

The above statement is actually incorrect. Nordost and The Chord Company are spearheading new ways of scientifically measuring the performance of various cables, using actual music signals (instead of single frequency sine waves). While this methodology is in its infancy, the results are quite fascinating. I suspect that you can investigate this further by going to Nordost or The Chord Company's websites to learn more, they should be able to point you in the right direction. I learned of this new testing methodology in a symposium put on at RMAF this past fall, jointly sponsored by Nordost and The Chord Company. Note that this testing is jointly sponsored by companies that actually compete with each other.

While I personally do not need any "science" to validate my opinion on the subject, I do find it interesting thta these companies are developing the science to actually measure the differences that many audiophiles already hear.

Additionally, just because "science" may not, at a given time, be able to describe a difference, does not mean the difference does not exist, it only means the "science" has not been developed enough to describe the difference.

 

[Part-Time Audiophile]

Seems like this thread is likely to go stale before long, which is sad. This is terribly interesting stuff, well, okay, at least to me.

 

I have to say this: ABX testing is scientifically sound as a methodology. It just is. The problem everyone is having isn't with the epistemic status of ABX testing -- or at least, it shouldn't be. The problem is with the implementation of the test, that is, with the study's design. Most scientific studies are not new, nor groundbreaking. It's all in teh refinement! And this problem is actually where most scientists spend their career -- identifying issues with existing designs, proposing remedies and perhaps re-running the study with the corrections in place.

 

Cable comparisons are very easy to do. All you need is a good cable switcher, like the one made by Manley. This allows different sources to play through the same rig. Or the same source (or two identical sources) to play with different wires through the same rig. This allows instantaneous path switching -- and allows very simple involvement with a second party (they can do the blind switching for you). Is it perfect? No -- but that brings me to another point.

 

Barrows made some comments earlier about the difficulties found in many ABX tests that I think are overstated -- all the sameness isn't required to hear a difference in many cases. Further, it shouldn't be necessary to hear ANY difference. If it is -- then we have something I'm now calling (and I want all credit for this, you bastids) "The Threshold Problem".

 

The Threshold Problem works like this: even if there is a difference on offer, whether positive or negative, if it isn't reliably perceptible, that is, its sonic contribution falls below the threshold of perception, it might as well not exist. This neatly captures all of the variances with system matching, subtlety of a given change, or even training. The complaint that the differences are so subtle it can only be heard with specialty gear or by specialty ears, or under specialty circumstances, is met with another indifferent shrug. If the differences on offer are so small, or so minor, who cares?

 

This is fairly crucial. One, because most reviews and reviewers are exercises in hyperbole. The differences between any two things, especially at the high-end, tend to be hair-splitting -- but written up as monumental, obvious, or catastrophic -- it's good copy. Two, while we humans are discrimination machines and pattern-matching geniuses, there is always a healthy built-in margin of error. 9 out of 10 audiophiles cannot discriminate between a 1dB difference in two output streams. Measurable. Real. Imperceptible.

 

I have a third comment here, at least wrt cables. Cables do vary. If you're looking for measurements, check Steve Nugent's page. Of course, this is for long cables, but still -- clear, obvious evidence that cables can and do in fact sound different. The question open is why. I've posted this elsewhere, but the only variables on offer are resistance, capacitance and inductance. Some suggest that the crucial variable varies depending upon where the cable uses: capacitance for interconnects and inductance for speaker cables. If this is true -- and electrically, this is all we have to play with -- then varying the capacitance could and probably should change the sound. Two cables with the same capacitance, then, ought to sound the same.

 

Which brings me to my own Cable Company experience. Over the Christmas holiday, I spent way too much money on loaners from them -- I got Cardas GR, Audience Au24e, PAD Aqueous Anniversary, and half a dozen others. All were 1 meter. All were RCA. And all were interchangeable -- they were audibly indistinguishable. If there was a difference on offer, that contribution fell afoul of the Threshold Problem. My current cables? Blue Jeans.

 

That, however, did not happen when I swapped out DACs. The differences between the BADA, Ayre, PS Audio and Lavry DA-11 were all tiny. Is it permitted to say that? Yes, the BADA was the best -- but the margin? Yes, over the Threshold -- but the margin was not wide.

 

I stand by yet another thread in which I submitted that the contributions of DACs are not huge when compared to the overall sound of the whole. Cables are even less so. But, yes, some are over that Threshold -- but many are not. And ABX testing -- of some sort -- may be the only (less subjective) way to learn that.

 

[cfmsp]

Shadorne posts:

 

"My opinion is supported by science

 

Your "experience" is not. There are no scientific measurements or tests which have demonstrated meaningful or significant audible benefits of special super expensive wires over adequate wires."

 

Actually Shadorne, the opinion which you stated (and with which I disagreed) is NOT supported by science.

 

You said:

 

"It is generally NOT possible to hear differences on well designed and well matched gear with adequate cabling over reasonable distances..." [where "Adequate can usually be found for less than $100 (for a set of cables with connectors)."]

 

To support your statement you would need science that proves it is NOT possible to hear differences on well designed and well matched gear [etc, etc.].

 

NOTE: I do realize that you inserted 'generally' above, but doing so takes the oomph right out of your statement, as it would also say, in effect, that it IS possible. IOW, as Chris says above, with the 'generally' present, your statement is basically meaningless. Since your other comments here seem rather more absolutist, I'll assume that you don't mean generally, and that your position is that it is NOT possible.

 

If I'm wrong, and you're saying that it is possible, then we can move beyond this as we'll be in complete agreement that it is possible to discern differences between cables on well designed gear.

 

 

You've also just stated:

 

"There are no scientific measurements or tests which have demonstrated meaningful or significant audible benefits of special super expensive wires over adequate wires."

 

That these tests might not exist doesn't mean differences can't be detected. As Barrows said: "it only means the "science" has not been developed enough to describe the difference."

 

More importantly, for science to support your opinion that differences can not be detected (with well designed gear, et al), you would need tests - many, many tests - that prove that it's not possible - of which there are NONE!

 

 

Somewhere earlier in the thread you said:

 

"Usually claims are based on a grain of truth and then claims are simply blown out of all proportion."

 

It seems to me that these words apply to your claims here. You're taking results of a test (which you've failed to provide here even after Chris asked to see them) that failed to detect a difference, and then blowing your claim all out of proportion to what it says. Effectively, you're assuming that the results are binary - that is to say, if you find a test result that couldn't prove a difference, then you claim that it proves that there cannot be a difference.

 

enjoy,

clay

 

 

 

 

 

 

[The Computer Audiophile]

Nice post Scot. I am very interested in this subject as well.

 

In my mind it seems like a longitudinal study is required.

 

It can take extended listening periods for me to discern a difference between two components. If I A/B/X components by switching inputs on a preamp and do this back and forth the sounds begin to blend together sometimes. However, if I listen to my Alpha DAC for a few days, then throw in the new Weiss DAC202 I can immediately notice a difference. I follow this with extended listening of the DAC202 before trying another component or the Alpha DAC again. The extended listening allows me to identify what I'm hearing and make absolutely sure I am hearing what I think I am hearing.

 

Bringing this to cables it would be cool to test people who listened to a certain cable(s) for a long period of time, then make a switch or no switch etc...

 

 

 

[cfmsp]

 

Scot,

 

I can agree with you that often times the magnitude of differences reported (by others) seems much greater than how I would describe them based on my experience with a similar change.

 

I assume that my hearing is average, at best, amongst audiophiles, and that i'll never hear some of the really far out stuff. Indeed, I hope I don't.

 

clay

 

PS, the most significant improvement I've experienced this year was AC conditioning/tuning gear.

 

 

 

 

 

[barrows]

Agreed, this discussion is interesting.

 

"If the differences on offer are so small, or so minor, who cares?"

 

I am going to propose something here that will be controversial, as there may be an aspect in what I am trying to say that appears metaphsyical or spiritual in nature-I do not expect everyone to agree!

 

To me, music, at its best, is a magical, transformational, experience. I want my audio system to be able to take me to that transformative place, this is a place where time stands still, I am unaware of my body and surroundings, and my consciousness is with the music.

 

In order to get to this experience, I will make changes in my system-the changes may result in differences that in normal consciousness are "so small,..., who cares", but it is my suggestion that it just may be differences this small that make the difference between a transcendental listening experience, and one that is less extraordinary.

 

I am not saying that the above theory is correct, just suggesting that it is a possibility. This theory may explain why differences which are so small as to be seemingly almost imperceptable, may actually be very important.

 

[firedog]

Took the test a few times. My threshold for seeing a difference in the 2 colors was 2 or 3. However, when doing the A/B part I found that my recollection of what the 2 original were colors were wasn't good. And it deteriorated once I had looked at 2-3 test samples. The samples became confused in my mind with what I had originally looked at.

 

If I pumped the threshold difference up to the highest level (well above my true thereshold) I did better (above 50%), but it was still difficult.

 

I think this shows why an ABX test is hard to pass and not necessarily useful in the context of evaluating audio. It's why I noted earlier (like Chris) that in my own listening I tend to use a component for a long time, switch it out, and then see if I hear a difference.

 

[Part-Time Audiophile]

Again, I think the problem you're having is in the study design, and does not have any impact on the theory that that design draws on. And the fact that you, as a subject, do not score well on a particular design does not in any way invalidate the design nor say anything interesting about you. As far as you know, the test was designed to induce a 10% pass rate. Anything higher indicates that you have "Golden Ears"! This is the detail you're not getting by simply taking the test.

 

It's trivial to design a test in such a way as to analyze gross vs fine discrimination -- especially visually, when it is color you're using as your variable. That this is harder or easier with the other 4 senses is irrelevant to this point -- it comes down to the design, not the methodology nor the theory that that methodology rests upon.

 

As for ABX testing of audio components, I have to say that I also do the long-test. I listen a couple hours a day for several days, then make a change and take notes. Repeat as necessary. The farther away in time I get from the original, the less reliable my observations, but sometimes, the changes are so great, that isn't an issue. Likewise, sometimes they're so great, that a long, immediate baseline familiarity with either isn't required to discriminate. I have to submit that this was the case with DACs and transports -- and not the case with cables. YMMV, but I guarantee (statistically) that it isn't a system thing. Chances are your system won't pick up those differences. Unless you have horns with silly sensitivities and ultra-low watt amps. Then everything you do and anything you change will be shouted out in bold-face, italics, and underscores. Which is a great reason to not get horns. ;-)

 

[Part-Time Audiophile]

I knew it!

 

There is a gestalt synergy thing, I suppose, where the whole is worth more than the sum of the parts. In philosophy, we call this an "emergent property". I am not dismissing this as real, but I do submit that this -- at least in the realm of psychology and perception -- might be more a factor of the subject than the object. But the end result is the same -- a perception of superiority, and hopefully, a concomitant increase in enjoyment, so what's the harm?

 

After all, I, too, shoot for transcendence. Else, why bother?

 

[Part-Time Audiophile]

Clay, I've had two big changes this year. New transport (Legato) with new USB cable (SRT) and a full Alan Maher power distribution system. Both beat the Threshold Problem, but the latter hasn't stabilized yet, I don't think. But when the stars align, the view is pretty breathtaking. ;-)

 

[barrows]

I absolutely agree that there is a subjective component to my listening experiences. One's state of mind/body certainly affects one's ability to experience the music. Distinguishing the subjective component from the objective can be very difficult (and, perhaps, is often impossible); I think that we all have to accept this: the listening experience is subjective in nature.

Back to ABX and how it relates. In my opinion, the subjective nature of listening is a very good example of how irrelevant ABX testing can be vis a vis long term listening, as long term listening will test the component under evaluation in many different subjective states, resulting in an "average" of sorts where the long term differences perceived can be attributed to the component rather than to the subjective state of the listener.

 

 

[Shadorne]

Fred E. Davis, Effects of Cable, Loudspeaker, and Amplifier Interactions June 1991, AES

 

This old paper illustrates the kind of minuscule differences when using a huge variety of cables - he concluded that amplifier behavior dwarfed anything the cables did.

 

I am not sure why I bother though. Since you can take any 1st or 2nd year electrical engineering text book and you will find that audio cables (analog ones) or wires are treated as lumped elements. Simple calculations show that the slight frequency response variations are really negligible - as much as one-third of a decibel difference may be expected at 15 Khz when using extremely different speaker cables. These differences are swamped by speaker and amplifier behavior - the main culprits are the components and not the cables. Badly designed and badly matched gear will simply give you problems - whatever cables you use. The poor quality output stage in most pre-amps will also give problems with IC's as capacitance rises with longer runs. And the use of RCA IC's is simply asking for trouble - serious audiophiles will only use balanced XLR.

 

[Part-Time Audiophile]

"And the use of RCA IC's is simply asking for trouble - serious audiophiles will only use balanced XLR."

 

This is specious. And, BTW, contradictory. If cables are irrelevant, than connectors on cables are less so. QED.

 

[davidR]

Shadorne, what is your reference playback system?

 

[Shadorne]

I suggest you google Bill Whitlock (Jensen Tranformers) and read up about ground loops and issues with unbalanced RCA connections.

 

[The Computer Audiophile]

Are there any scientific ABX tests available since this thread is also about ABX?

 

 

 

[Part-Time Audiophile]

Sounds like poorly designed gear to me. With my properly designed and well implemented Plinius system, ground loops can be removed at several places in the rig without having to resort to a bandaid cable solution. Snap!

 

[ciamara]

@Shadorne

 

Balanced cables and connectors use three conductors (hot, cold, ground) and operate at +4dB. Unbalanced/single-ended use two (hot, cold) and run at -10dB. Balanced cables are used in most professional applications, and because of the design, they can be run over very long distances without accumulating noise, hum, etc. We generally favor balanced cables, but not in every situation. It can sometimes make sense to connect the unbalanced outputs of a preamp or active crossover to the balanced inputs of the power amplifiers (with an appropriately configured cable, of course). Why would one do this? When gain leveling a system, this offers a nice way to attenuate the signal, and if done well, to a pleasing effect. Moreover, there are several very high end preamplifiers and amplifiers that only use unbalanced inputs and outputs.

 

RE: ABX Testing ....

 

This is always a tough one. Last fall, we looked at ways of doing a DAC shootout, and it invited quite a bit of debate here on CA. Some felt that fast switching between precisely tuned comparison arms was the way to go, whereas others did not. What we found was that switching too quickly was simply too confusing. As unscientific as it may sound, we feel the best way to evaluate and compare different pieces of equipment is to spend time with each one. Listen to one for a while, then switch to the other, and then switch back ... and then maybe once more. We find it easier to detect the removal of something pleasing than the addition of it. Once you get used to something you like, taking it away is discomforting. Of course, the music better be good, because I for one don't fancy listening for that long unless I like what I'm hearing