Controversy of ABX testing

[Mahesh]

clay,

 

I'll give you that, any empirical knowledge is true with a certain degree of probability. Hence, we cannot be 100% true that the "so-called laws of physics" hold. But they have given us moon landings and free Internet porn; if that's not proof of concept, I don't know what is... ;-)

 

But if we were to find out one day that Ohm's law doesn't hold, after all, does that imply that the research methods we used to derive it are flawed?

 

Mahesh

 

[barrows]

OK, I am going to weigh in once here to state my opinion and experiences for consideration in this discussion. My experience is based on working for a high end audio company where I listen tested various components/systems/parts for sonic performance evaluation as a small part of my job, as well listening to my own system and various "upgrades".

To my mind there are two main problems with "scientific" ABX style testing:

1. It is very difficult to setup a valid ABX test, as all parameters must be exactly the same for the test to have any hope of working. Examples: testing two DACs, the input and source must be identical (same input cable, same source, etc), the positon of the DAC must be identical in regard to proximity to other components (it must be on the same shelf on the rack), one must use the same power cable and analog output cables, and the output must be connected to the same input on the preamp (yes, different inputs will sound different). Both DACs will need to be well broken in, and warmed up equally (preferably powered up for 24 hours prior to the test) and power should not be interupted to either component under test during the course of the test. Considering these factors one can imagine how difficult it is to set up a good test of this type, and how quick back and forth switching woudl be very difficult.

2. (and this is my main reason for not giving a lot of value to ABX testing)-ABX testing introduces a stress factor, that is in opposition to how we really enjoy music. The stress produced by being under pressure to hear differences can be very distracting from actually hearing those differences. In my experience I often tested products for work in my home system (despite the fact that the reference system at work featured a dedicated listening room with better speakers than I have at home) because it was much easier to relax at home, away from the pressures of the work environment.

I prefer longer term listening to determine the character of given components, using music that one knows well. Big differences are easy to spot in quick ABX style testing, but the small differences (which can be very meaningful for listening pleasure/emjoyment) are very hard to discern in quick testing. Finally, one must accept that the enjoyment of listening to music in our homes is subjective in nature, and absolute "scientific" differences in sound do not really matter when we are getting down to the small differences (like the difference between a Berkeley Alpha DAC and a PS Audio PerfectWave DAC); we listen for musical enjoyment, and we should test in a way that is similar to how we listen.

 

[wgscott]

So, is it then just impossible to test whether the $0.33 cable is worse than the $100 cable?

 

[Mahesh]

wgscott,

 

"So, is it then just impossible to test whether the $0.33 cable is worse than the $100 cable?"

 

I guess you could just compare them in your system... Or do you mean a scientific test?

 

Mahesh

 

[Mahesh]

barrows,

 

Thanks for providing your justified criticism of using ABX in audio, that was just what I was looking for.

 

Mahesh

 

[wgscott]

I would have to spend $100 to test it myself. What would be nice is if the people who are going to make a profit on my purchase provided the evidence to justify the claim that the more expensive cable sounds better (or perhaps more objectively, transmits data more faithfully).

 

Put slightly differently, I don't want to spend $100 on a cable if it isn't any better than my $0.33 cable, but if it does make a significant difference, I probably do want to spend that much, given that I don't want it to be the limiting factor in my system.

 

[Mahesh]

wgscott,

 

Many dealers can provide products on home loan, so you can try them out in your system before you buy. Whether it can be proved that the $100 cable is better, I'm not so sure. I think it is highly subjective what "significant difference" means in this context.

 

One problem with the spdif signal is that the timing information is encoded in the audio data stream. So, even if you are able to transfer all bits perfectly, you might still have a problem if the physical properties of the cable somehow interferes with signal timing. However, I find it highly unlikely that a short stretch of toslink cable made to specifications will be able to tamper that much with the signal.

 

Mahesh

 

[cfmsp]

Agreed with Mahesh, having 30 day return privileges is 'de riguer' for cable purchases.

 

Another alternative is to buy them used (and therefore already broken-in). If you don't like them, you can turn them around for what you paid for them (or near that).

 

 

In either instance, one can run one's own ABX testing, 'in-house', so to speak.

 

 

SIDE NOTE: I've often wondered what percentage of the people who suggest that others must prove their claims (with ABX testing) actually use ABX testing to prove they can hear differences (before opining on line) with respect to their own purchases/upgrades, EVEN when they hear a clear/obvious difference ? I'm gonna guess less than 10%. ;0

 

 

 

clay

 

[wgscott]

If I think cable X sounds better than cable Y, and state this as an opinion, it is assumed the subjective qualifiers are implicit. (It sounds better to me.)

 

If someone selling cable X for $100 claims it is better than cable Y which sells for $0.33, it is not unreasonable to suggest that some sort of empirical evidence be supplied, whether this be double-blind listening tests to show a sample of people really can distinguish the difference, or some sort of read-out that shows greater fidelity in data/signal transmission, or whatever.

 

Similarly, I can claim my Toyota is safer than a Ford Explorer on some car forum, and people think I am just stating my opinion. But if Toyota wants to make that claim, then it is incumbent upon THEM to back it up with some empirical data, not up to the individual purchaser to buy one of each and subject it to crash tests.

 

[Shadorne]

The tin ear argument:

 

"I would respectfully submit that if you're never heard the difference that a cable can make in a system, then you don't need audiophile gear - any gear will do. :)"

 

The technical response is:

 

"I would respectfully submit that if you hear a big difference that a cable can make in your system, then you don't need your audiophile gear - but you for sure need some help selecting some properly designed and properly matched equipment!"

 

Well designed equipment will not care about your wires provided they are adequate. Adequate can usually be found for less than $100 (for a set of cables with connectors). Despite what cable companies will tell you regarding the "science" of their designs.

 

Finicky, unreliable and poorly designed equipment will be influenced by everything from humidity to room temperature to your fridge compressor starting in your kitchen to utility power variations during the day. A system like this may never sound the same twice anyway...ABX or no ABX, $100 or $10,000 cables.

 

[Mahesh]

wgscott,

 

"If someone selling cable X for $100 claims it is better than cable Y which sells for $0.33, it is not unreasonable to suggest that some sort of empirical evidence be supplied, whether this be double-blind listening tests to show a sample of people really can distinguish the difference, or some sort of read-out that shows greater fidelity in data/signal transmission, or whatever."

 

Agreed. But I don't think audio cable manufacturers are the first to demonstrate questionable marketing ethics. If we want to have hifi as a hobby, I think we'll just have to get used to hearing claims like that, and then decide for ourselves whether we believe it or not.

 

Citing The Ten Biggest Lies in Audio:

 

"[...] it’s not really the audio industry that should be blamed but our crazy consumer culture coupled with the widespread acceptance of voodoo science. The audio industry, specifically the high-end sector, is merely responding to the prevailing climate. In the end, every culture gets exactly what it deserves."

 

Mahesh

 

[Mahesh]

clay,

 

"In either instance, one can run one's own ABX testing, 'in-house', so to speak."

 

I think doing an ABX test on your own is quite difficult. In order to have a proper experiment, you are not supposed to know which of A or B is the unknown for each repetition. Besides, I don't think an ABX comparator is on the top of every audiophile's wish list... ;-)

 

Regards,

Mahesh

 

[cfmsp]

Shadorne says

 

"the technical argument is:"

 

LOL! that's very funny. One opinion is as good another.

 

 

 

BTW, if you're gonna disparage someone with epithets like "tin ear", you ought to at least read what was said.

 

I said "I would respectfully submit that if you're never heard the difference that a cable can make in a system, then you don't need audiophile gear - any gear will do."

 

If you want to disagree with what I said (which apparently you do since you labeled my response as "the tin ear argument"), the opposite of what I said would be "it's not possible to hear a difference between cables."

 

I only posited that if you can't hear the difference that a cable can make, any gear will do. Good cables / bad cables can each effect the sound, but if you're incapable of hearing any differences, ANY GEAR WILL DO!

 

You've mangled what I said in your enthusiasm to be disagreeable. Either that, or you are stating that it's not possible to hear a difference between cables. Which is it?

 

 

clay

 

 

 

FWIW, despite that I did NOT say it in my post (and that you therefore jumped to a conclusion) I do believe that improvements can be had via use of cables that cost above $100.

 

I'll pass along your comments about improper design (of my gear) to Gordon Rankin, Nelson Pass, Don Garber, and BJ Buchalter (designer of the world class ULN-8 / LIO-8 Metric Halo units).

 

 

 

 

 

[cfmsp]

 

Mahesh,

 

 

agreed and understood, but then again, I'm not the one recommending that people do ABX testing to support their opinions - therefore I'm not the (potential) hypocrite.

 

 

clay

 

 

[Shadorne]

Clay,

 

It is possible to hear differences in cables when using some types of equipment because some equipment is affected by almost anything as it is not well designed or well built or well matched or it is faulty. A lot of equipment suffers from ground loops - for example.

 

It is generally NOT possible to hear differences on well designed and well matched gear with adequate cabling over reasonable distances. (excluding ridiculous cables with filters or faulty inadequate designs, or extreme cable lengths)

 

Of course, people like to think they can hear differences but when they perform an ABX test on a properly setup system then they soon discover that this is a placebo effect (after spending a lot of money on a new wire people are naturally pre-disposed to hear an improvement - just as they often report that they feel better after taking a placebo pill)

 

 

[cfmsp]

 

Shadorne,

 

You state: "It is generally NOT possible to hear differences on well designed and well matched gear..."

 

I respect your (and almost anyone's) opinion, but this (and the rest of your post) is soooo different from my real world experience and the (real world) experience of many, many people whose opinions I respect, it seems there's no point in further discussion here.

 

 

enjoy yourself,

clay

 

 

 

 

[Shadorne]

No worries. The placebo effect is most definitely part of the "real world experience" for many, many respected opinions.

 

[cfmsp]

oh, Shadorne,

 

Now you're disrespecting my opinion and that of my friends (some here on CA and some who left because they tired of wading through posts by people like yourself).

 

oh well.

 

bless you.

 

As I said earlier,

enjoy yourself,

clay

 

 

 

 

 

[The Computer Audiophile]

There's no room here for disrespect. As Clay said it drives people away from CA and from this wonderful hobby. I really couldn't care less if people use coat hangers for speaker wire or cryogenically treated Nordost Odin.

 

I encourage the discussion here as long as it's respectful.

 

Here is a link to an article written by Robert Harley about blind testing. I think it's pretty interesting just as I think articles from the Audio Critic are interesting.

 

http://www.avguide.com/forums/blind-listening-tests-are-flawed-editorial

 

[barrows]

"It is generally NOT possible to hear differences on well designed and well matched gear with adequate cabling over reasonable distances. (excluding ridiculous cables with filters or faulty inadequate designs, or extreme cable lengths)"

 

The above statement, while really an opinion, is actually stated as if it is a fact. In my (considerable) experience, the above statement is entirely incorrect, and I respectfully disagree entirely with it. It is my experience, that the sonic differences between cables are easily discernable to anyone with a decent system, who bothers to take the time to listen for themselves. The good news is, that it is generally quite easy for a listener to test the sonic differences between various cables with little risk: most good dealers will lend expensive cables for home auditions, and if ones local dealer is unable/unwilling to allow home auditions, The Cable Company maintains an extensive "lending library" of cables for customers to test in their systems.

Further, I would put forward that it is unfortunate to suggest that ones system/components are poorly "designed/matched" if sonic differences are noted with cable swaps/upgrades. In my opinion, statements like this are a disservice to audiophiles looking to learn, and to improve the sonic performance of their systems. I would encourage any audiophile who is looking to optomize the performance of their system to try out some different cabling in their system, and to believe what they hear, rather than what they read on the internet.

 

[Mahesh]

Chris,

 

Thanks for the link to the Harley editorial. That's the type of information I was looking for.

 

Mahesh

 

[firedog]

Mahesh-

I have a pretty scientific - statistics oriented bent, but I'm also doubtful about the efficacy of AB testing in audio.

 

Why? The Robert Harley article touches some of my concerns. In general what we are testing here is perception, not the equipment. This is problematic for several reasons:

 

1) Different people hear differently. Not as a result of physical differences in their ears/brains, but because listening is a learned perception: you can improve your ability to hear things over time and with practice. Example: a trained professional violinist (say Yitzhak Perlman) can easily tell a great violin like a Strad from another very good violin; you and I probably can't.

 

In the same way, I can hear all sorts of things in recorded music playback that most of my friends can't. I'm not imagining those things, it's just that my friends are used to listening to mp3s on iPods and haven't developed the skills to hear what I hear. Some of them think 128k mp3s sound "the same" (not just as good for their purposes, but actually "the same") as a well recorded CD played back on a good system. Since we all know the amount of audio information in such and mp3 file is much reduced, they are clearly "wrong", but they can't perceive the difference.

 

 

2) The brain has "musical memory". If you switch back and forth between 2 sounds, your memory of how it sounded before isn't erased and influences what you hear. Also, if you are used to hearing a certain track reproduced a certain way, you tend to "expect" it to sound a certain way, and your brain can "fill in" what it thinks you "SHOULD" hear.

 

So I don't do usually do "quick" back and forth A/B testing to compare products. Usually I use a new product (say an amp) for a week, and then switch back in my "original" to see how it sounds. Sometimes I even do this again: Original item>new item>original>new.

 

I think I get a better idea of how the 2 products compare this way.

 

Of course there can be instances where a new product is so superior to the one it replaces that the difference is immediately obvious.

 

3)You have to check very carefully the methodology of double blind testing with audio. Read the fine print: what equipment was used, what music, and what listeners.

 

There's a test result floating around on the net "proving" that there "isn't any audible difference" between higher and lower resolution music files, b/c listeners cant tell them apart in listening tests. But read the fine print. The testers let listeners supply their own PC's, sound cards, and speakers. So clearly there could have been cheapo PC components involved that couldn't accurately reproduce the differences in the files in a way that was even audible.

 

In addition, the authors of the study themselves noted that in spite of their conclusion, one subgroup in the test could reliably differentiate between the high and low resolution files: the group members identifying themselves as "audiophiles". In other words, listeners with a trained ear who knew what they were listening for could differentiate between the files, but "average" listeners couldn't (see point #1).

 

So what the test actually proved was that average listeners on indifferent equipment couldn't differentiate between files of low and higher resolution, not that "there isn't any audible difference" between between the 2 types of file. "Audible" is always "audible - to whom?"

 

We don't know enough about how and what sounds (or sonic related phemomena) humans perceive to make sweeping conclusions that something "isn't audible". Humans clearly can hear (or perceive, or have hearing perception altered) by all sorts of factors in sound that aren't easily measureable as 20 to 20K wavelengths. Twenty years ago audiophiles who claimed CD music reproduction "didn't sound right" were scoffed at by engineers who could prove that CD reproduction was "perfect". Today no one makes that claim.

 

Robert Harley is right when he points out that testing differences in audio reproduction isn't the same as testing the efficacy of drugs. Those who insist that it is are guilty of oversimplification.

 

 

[Audio_ELF]

I think at the end of the day, ABX testing is a tool which can be used as part of scientific methodology to test whether certain changes in a system make a difference. Just like blind (without ABX) and even sighted testing also makes a difference. I've always thought there is a lot of relevance to the fact that HiFi Choice (UK monthly magazine) when doing their group tests use a combination of sighted and unsighted.

 

If you start with the assumption that there IS a difference between (for example) cables, then an ABX test is irrelevant. On the other hand, I think we also have to accept there is is an element of placebo and psycoaccoustic elements in differences in systems. Blind and ABX testing in these circumstances can be useful (as part of an overall testing process).

 

As I was writing this, I did realise that I was going to type "testing and demo process" but then realised that ABX testing has little, or no use in a demonstration situation - here what is important is being relaxed and that you enjoy the system rather than which is necessarily technically best. It's the difference between the science behind the HiFi, and the Art of the music.

 

Anyway, as I say, ABX testing is a tool which can be used to ascertain if there is a real difference between two systems. Having said that, an old adage comes to mind: if all you have in your toolbox is a hammer, every problem looks like a nail.

 

Eloise

 

[gcorley]

I agree that ABX, blind and double-blind testing all have both limitations and useful applications. It has, although, always seemed a little worrisome that the traditional hifi community has an almost phobic reaction to the very idea of objective testing. Much is at play!

 

On a related issue: every wonder why your local dealer spends so much time leading into a listening session? The really good ones even tell you what you will hear!

 

[Shadorne]

CD format is indeed near perfect for playback requirements in a home system. It has had some issues - jitter etc. however CD format is really good enough and issues have been resolved although CD loudness wars is problem...