Controversy of ABX testing

[ciamara]

Regarding cables ... I have to say that good cables do make a difference. This is an area of audio in which there is a lot of snake-oil, not to mention over-priced gimmicks. But good cables are important. We spent the last several months testing cables from a number of different manufacturers, and we settled on a small handful of manufacturers that focus on quality rather than marketing and sales incentives. We also identified a number of characteristics in cables that lend themselves well to certain applications, and less so to others. For example, we like silver in darker sounding systems - the increased conductivity improves efficiency and can brighten the overall tone. Conversely, we favor copper when working with a highly revealing horn-loaded system (particularly horn-loaded mid/high and high frequency ranges). We also find that changing all interconnects in a system at once provide a more obvious change than swapping a single pair (in our tests, we worked with 4-way and 5-way active sound systems).

 

In the end, it really comes down to each and every component chosen for a system and how wise those choices are. Cables are components, just like the power amplifiers, drivers, crossover circuits (passive or active), and of course, the room. Get most things right and you get good sound. Get it all right, and the results are truly stunning.

 

[tonmeister86]

 

 

THE cable manufacturers and retailers DO care when scientists debunk expensive cables because they stand to lose the + 1000 points of margin they make when the cables are sold to unsuspecting audiophiles.

 

Expensive audio cables fall under the category of "audio jewelry". A Rolex watch may not keep better time than a Timex but they make you feel more confident and good about yourself and your status in life - Expensive cables serve much the same purpose.I have no problem with that as long as people admit it.

 

Personally, I'd rather save my money on expensive cables and put it towards purchase of good music or wine. That is something that makes me feel good, and its psychophysical effects are easily measurable w/o needing a double-blind ABX test to validate it.

 

 

[Shadorne]

The core of the setup consists of soffit mounted ATC SCM 100A + standalone ATC SCM 0.1/15 sub driven by a Benchmark DAC1 + rather ordinary XLR interconnects from a local music store.

 

Possibly influenced by many years of studying and working in engineering and physics (not in audio related - I have no affiliation with the audio industry at all) but Sean Olive and the work done by Harman is the kind of stuff I eat up - great company with great products. Glad to see they pull no punches with respect to the audible benefits of extremely high priced wires.

 

[barrows]

curious... Shadorne, perhaps you should pick up some nice product from Harman's Mark Levinson brand. Sorry Sean, no disrespect meant, I could not resist.

In any case, I have little experience with truly expensive cables (I mean the five figure ones) so I cannot comment on how much performance improvement they might offer versus the three and four figure cables that I have the most experience with. On the other hand, I consider cables approaching and beyond four figures expensive, and I also note significant performance improvement (a huge difference actually) of these cables versus run of the mill mass market stuff, and I do prefer balanced components/connections in my system.

Shadorne: I agree with your comments re: frequency response variations in cables. A simple, reasonably designed cable can achieve flat frequency response within the audio band pretty easily. But, I do not think the improvements offered by some of the better cables have anyting to do with frequency response within the audio band. Timing problems are another story though. BTW, things like dialectric effects, skin effects, and eddy current effects are well understood by RF engineers-most of these engineers, if not audiophiles, believe these effects are not relevant for analog audio signals, but audiophiles, and audiophile engineers know better.

In any case, as noted previously, it is easy and low risk for listeners to try some different cables in their systems and decide for themselves, and I encourage people to experiment if they are interested in getting the most out of their systems.

 

[cfmsp]

Sean, How 'expensive' is expensive?

 

It occurs to me that attacking (manufacturers of) expensive cable without naming names is basically a SMEAR campaign against all cable manufacturers. Exactly, what 'expensive' cables have been proven NOT to be detectable as different/improvements from garden variety cables?

 

What 'expensive' cables have you tested? What 'expensive' cables have been tested by others? Seems to me that without testing ALL the expensive cables you denigrate with your comments, you're just generating FUD (aka Fear, Uncertainty, & Doubt) without any data to back it up.

 

Sean, it's not even clear what $$$ amount you're referring to as 'expensive'.

 

Shadorne at least did that for us - he claims that anything over $100 is a waste of money.

 

Here's an idea for you - have you considered trying to prove that there are NO heretofore unmeasurable characteristics of cables (& their interactions) that might possibly be responsible for differences in sound heard by those that spend 'real' money?

 

 

"THE cable manufacturers and retailers DO care when scientists debunk expensive cables because they stand to lose the + 1000 points of margin they make when the cables are sold to unsuspecting audiophiles."

 

For all your talk about science debunking expensive cables, one would think that there is incontrovertible evidence that cables could not possibly improve the quality of the reproduced sound.

 

Show us the tests which prove - note, it would take MANY, MANY tests to do so - that no one is capable (ie. that it's not possible) of detecting improvements in sound quality with the use of cables above entry level price points!

 

 

"Expensive cables serve much the same purpose.I have no problem with that as long as people admit it."

 

AB/X tests can prove almost nothing other than about the (listening abilities of the) subjects undergoing a particular test for a particular set of gear. I've got no problem with people squawking about AB/X testing so long as they admit that.

 

 

 

 

enjoy

clay

 

 

 

[Part-Time Audiophile]

I, for one, believe that spending more that $100 on cables is a waste of money and I for one, have actually compared a $33 cable (Blue Jeans) to $2000 cables (PAD AA and half a dozen others) and as a result, am using the former.

 

Tell you what -- if you have an expensive set of cables, take the Socrates Challenge. Go to BlueJeans.com and buy yourself a replacement set of BJ Cables. Then, send me your expensive cables so I can sell them. You'll feel good about spending a boatload for a strand of wire and after I collect all that dough, I'll get to feel just like an exotic cable vendor! A classic win-win, if you ask me.

 

[Part-Time Audiophile]

"Here's an idea for you - have you considered trying to prove that there are NO heretofore unmeasurable characteristics of cables (& their interactions) that might possibly be responsible for differences in sound heard by those that spend 'real' money?"

 

We can't first diss all the tests that do this very thing and then follow this up with a demand for the same such. The fact of the matter (and yes, I mean that as it is written) is that this has been done -- via ABX testing. The issue many have with this is not that the methodology is flawed -- it isn't -- but rather that the study was flawed -- which is debatable.

 

Valin in a response to comments made on Harley's editorial in the link above is very interesting. His rejection was modeled on an ad hominem attack, but did very deftly draw a distinction between ABX testing and double-blind trials used in drug research. The problem with his analysis is that there is a certain verisimilitude (Google is your friend) between the two approaches that is belied by his dismissive tone. There is, no doubt, a study that can be created that would address his issues rather handily. This would still, perhaps uncomfortably, draw heavily on the blind-experiment paradigm that both the ABX tests and drug trials used.

 

The issue at hand is, IMO, very interesting. I call it the "Established Narrative" problem. It goes like this: when data are discovered that threaten an established narrative, that data are ignored, ridiculed, and dismissed. Why? Because paradigm shifts are a PITA, and are generally avoided like the plague that they are. Harley and Valin both argue backwards -- there is obviously a difference, therefore any test that fail to confirm this is suspect, and quite possibly the efforts of intellectual terrorists. This is, prima facie, absurd. But quite understandable. We have quite a bit of it on this forum too, and a healthy dose on display here in this thread.

 

The problem with the ABX testing and the informal listen-for-a-while-and-then-swap approach is that there is very little in place to eliminate basic psychological compensation mechanisms. This is different from the placebo effect, but its related. The fact is that we can habituate to just about any stimulus, and further, our sensory memory is terrible. Proper testing design accounts for this. Armchair hypothesizing by contrast does not.

 

[barrows]

I am glad that you have found cables that work well for you. It does bother me a little that you seem to not respect the listening experiences of others though. The good news is that the scientific, measurements, that Nordost/The Chord Company are developing do objectively show differences between cables which confirm the listening experiences many audiophiles are having with expensive cables.

Anyone who chooses to disregard the listening impressions of experienced audiophiles regarding cable performance owes it to themselves to familiarize themselves with these new measurement techniques being developed by Nordost/The Chord Company.

 

[cfmsp]

 

"We can't first diss all the tests that do this very thing and then follow this up with a demand for the same such."

 

Please take these comments in the spirit with which they are intended! Don't you just love that comment - totally leaving the determination of the intended 'spirit' up to the reader!

 

First of all, who is 'we'? I'm not taking sides here - I have my own opinion and that's all that matters to me. I"m not an AB/X tester, nor do I believe that everyone needs expensive cables. Nor am I trying to convince anyone that my opinion is absolutely correct.

 

As for the comment 'diss the test' and then 'demand for the same such' - I'm NOT (the one) making any claims here, I'm only asking that those who DO (make claims) show some evidence (using their own preferred method of proof) that its NOT possible for those of us who claim to hear a difference to do so.

 

I'm not disclaiming the AB/X methodology per se, although I am aware of the "fatal flaw" in it's use to support the sort of opinions that many seem to believe that it supports. The problem is, it simply can't prove the non-existence of differences.

 

"but rather that the study was flawed -- which is debatable."

 

You'd need a multitude of studies to prove that it's not possible - which is the claim made by both Shadorne and Sean, as far as I understand it. You can't take one group (or even a handful) of listeners and one set of cables in one sitting with one set of gear and then extrapolate the results across all listeners and all cables and the entire industry. I didn't study physics, but I did study math - and enough so that I don't base my beliefs on "lies, damn lies & statistics" (aka probabilities) in the face on contrary personal experience. ;0

 

 

" It goes like this: when data are discovered that threaten an established narrative, that data are ignored, ridiculed, and dismissed. Why? Because paradigm shifts are a PITA, and are generally avoided like the plague that they are."

 

RIght back atcha! Actually, the threatened, established narrative is the one belonging to the folks that would use AB/X to prove that they are right against new beliefs. so yes, I would agree with you, that new information is being ignored, ridiculed and dismissed. This is no different than what happened in years gone by - remember when people thought all amplifiers sounded the same?

 

For the record, I'm NOT a supporter of expensive cable manufacturers. As you will no doubt recall Scot, recently I made similar arguments (as Shadorne) with respect to USB cables - i.e. if expensive USB cables made a HUGE difference in my system I would investigate the causes of such.

 

 

cheers,

clay

 

 

 

 

 

 

 

 

 

[Shadorne]

Shadorne, perhaps you should pick up some nice product from Harman's Mark Levinson brand. Sorry Sean, no disrespect meant, I could not resist.

 

Harman have bought many classy companies. They have assembled some world class engineering - I do not necessarily see this as a drawback unless you like to own boutique stuff. In college I enjoyed a speaker that Dr Floyd Toole helped test back in his days at Canada's NRC acoustic labs. Dr Toole's measurements and ABX testing were hugely influential in the progress of loudspeaker design. Dr Toole helped further industry understanding of the importance of dispersion in loud speakers, among many other things. Dr Toole is VP of Acoustical Engineering at Harman. The consolidation in the industry is just a reality given cheap OEM products from overseas. Hopefully Harman will be able to preserve and continue to offer some of the fine products that they have acquired - better this then seeing the demise of respected brands due to insolvency.

 

[barrows]

I was just taking a little dig at ya Shad, considering my opinion that their ML brand is quite in the "overpriced audiophile jewelry" category.

Honestly, I have nothing against Harman-and I especially applaud their efforts to market high end gear to affluent folks who may not otherwise know that high end audio gear even exists. The industry as a whole should support this kind of marketing effort, as it is a shame that such a small segment of the US population even knows that the "high end" exists.

As to consolidation, I view that as a shame, and hope that the entire industry is not forced to go this way, as generally it tends to result in homogenization rather than innovation.

 

[Part-Time Audiophile]

" Please take these comments in the spirit with which they are intended! "

 

Oh, no worries -- and don't get me wrong. I'm not wedded to any of this. I do just love a good argument. Especially one fraught with epistemic angst. Heady stuff!

 

" First of all, who is 'we'? I'm not taking sides here - I have my own opinion and that's all that matters to me. I"m not an AB/X tester, nor do I believe that everyone needs expensive cables. Nor am I trying to convince anyone that my opinion is absolutely correct."

 

There is a very pervasive meme in American culture that says everyone has a right to their own opinion. I find this to be terrifically problematic as well as intellectually dishonest. Despite what momma told you, you are never entitled to believe a falsehood -- and anyone offering one, whether it is couched in social pleasantries or caveats, is not only guilty of bad form but is enabling the perpetuation of a deceit (or conceit). Your opinions, once loosed, are not yours. They become statements and statement must be subject to scrutiny. If it turns out that the beliefs that underwrite them are indeed false, then it is our duty as sentient creatures to reject those beliefs, dearly held (or not,) though they may be.

 

" As for the comment 'diss the test' and then 'demand for the same such' - I'm NOT (the one) making any claims here, I'm only asking that those who DO (make claims) show some evidence (using their own preferred method of proof) that its NOT possible for those of us who claim to hear a difference to do so."

 

I think this has already been alluded to? The most strongly worded version: you don't, because you can't, because you're deluding yourself. More gently, there are quite a few generally well understood psychological phenomenon that have been studied rather exhaustively that can be applied here. But while I am not suggesting that this is the case, I do believe that the point is rather hard to evade.

 

As for me, I've already said that cables can sound different, and I've already identified a particular variable in cable manufacture that could perhaps lead to that audible variation: capacitance in interconnects and inductance in speaker wire. That said, I have posited that, in most installations and with the most common cable lengths (under 2m), that variance will be lost to the Threshold Problem. To underscore, that last is an untested theory proposed by yours truly. However, my admittedly anecdotal experiences does seem to give credence to the theory (and in part, sprang from it).

 

" I'm not disclaiming the AB/X methodology per se, although I am aware of the "fatal flaw" in it's use to support the sort of opinions that many seem to believe that it supports. The problem is, it simply can't prove the non-existence of differences."

 

Shad may have claimed this; I didn't, but the difference between our positions nets out to pretty much the same thing. Interestingly, a scientific test cannot prove something doesn't exist. I say interestingly because this problem is actually a streamlined version of Descartes' problem, and as you probably know, there are some serious problems with his cogito ergo sum solution to it. That said, we don't have to regress to the audiophile equivalent of the Turtle (because, really, "its Turtles all the way down"), but simply apply Occam's Razor at this point -- I have no reliable evidence for it's existence, therefore, parsimony dictates that it probably does not. QED.

 

"You can't take one group (or even a handful) of listeners and one set of cables in one sitting with one set of gear and then extrapolate the results across all listeners and all cables and the entire industry. I didn't study physics, but I did study math - and enough so that I don't base my beliefs on "lies, damn lies & statistics" (aka probabilities) in the face on contrary personal experience. ;0 "

 

Well, actually ... that is precisely what you need to do. Extrapolate. Statistical evidence is evidence. It just needs some serious scrutiny. And if you can explain away your outliers, well ....

 

" " It goes like this: when data are discovered that threaten an established narrative, that data are ignored, ridiculed, and dismissed. Why? Because paradigm shifts are a PITA, and are generally avoided like the plague that they are.""

 

" RIght back atcha! "

 

LMAO. Touche!

 

" For the record, I'm NOT a supporter of expensive cable manufacturers. As you will no doubt recall Scot, recently I made similar arguments (as Shadorne) with respect to USB cables - i.e. if expensive USB cables made a HUGE difference in my system I would investigate the causes of such."

 

Clay, you can deny it all you like, but everyone knows you're a huge fan of uber expensive USB cables. ;-)

 

Interestingly, there are good mechanical reasons why those USB cables do in fact sound better (separation of power FROM signal) that do not translate to interconnects (power AS signal) or speaker wire, where the differences between offerings are all electrical.

 

Ahh, this is great stuff. Nothing like a little Philosophy of Science with my afternoon coffee. Makes me want to crank the stereo up ....

 

[Part-Time Audiophile]

"It does bother me a little that you seem to not respect the listening experiences of others though."

 

This, I think, requires an apology, and Barrows, you are right to point it out. Mea culpa; I'm sorry because I truly don't mean any disrespect.

 

That said, you're still wrong. :-p (Couldn't resist, sorry, sorry!).

 

I did hear something about the new Nordost measurement system. I'd love to hear more about it as they move things along. Seems like it's definitely something to stay abreast of.

 

[Moses]

Clay, are you saying:

'(a) There are differences between cable A and cable B, and

(b) I (Clay) can only hear those differences when I know which particular brand I'm listening to.'?

 

I suppose this is *not* what you mean, but I would love to know your answer to that. My guess is that the problems people have are not with blind testing as such, but with particular A-B methodologies (for example, those that require forced decisions after a short period of listening, put subjects under stress, and so on. It *looks* as though your own reservation has to do not with blind testing as such, or even with A-B methodologies, but the idea that a random sample of listeners could reflect your own perceptions, is that right? In which case ABX testing with a population of one would be just fine. But these are all different questions). Harley's article referenced above gives a very interesting example, but it doesn't tell against blind testing at all.

 

Great discussion!

John

 

[cfmsp]

 

 

Noce post, Scot, I shot my wad on another post just now - or I'd have more time to respond.

 

You said:

"Well, actually ... that is precisely what you need to do. Extrapolate. Statistical evidence is evidence. It just needs some serious scrutiny. And if you can explain away your outliers, well ...."

 

I'm also a fan of Ockham, but...

Regarding explaining away the outliers, seems to me the outliers out-number the statistical evidence (offered by AB/X tests) by multiple orders of magnitude. ;0

 

enjoy

clay

 

 

 

 

 

 

 

[cfmsp]

 

 

"...but with particular A-B methodologies (for example, those that require forced decisions after a short period of listening, put subjects under stress, and so on. It *looks* as though your own reservation has to do not with blind testing as such, or even with A-B methodologies, but the idea that a random sample of listeners could reflect your own perceptions, is that right?"

 

You've got the jist of my concerns John, more the latter than the former, as you surmise, but more specifically taking the random sample of listeners and projecting (or rather extrapolating, as Scot says) the results out across the entire known universe of listeners & cables (in this instance).

 

Overall my concern has more to do with the tenor of overall conversations here, as opposed to my own opinion.

 

Regarding the 'tenor' comment, one of the things that makes this forum great is that people can freely talk about things that are controversial amongst people who respect each other and can have intelligent discussions with. I only jumped in here to disagree with those that claim their knowledge as some sort of absolute.

 

enjoy,

clay

 

 

 

 

[wgscott]

Take the Socrates Challenge

I, for one, believe that spending more that $100 on cables is a waste of money and I for one, have actually compared a $33 cable (Blue Jeans) to $2000 cables (PAD AA and half a dozen others) and as a result, am using the former.

 

Tell you what -- if you have an expensive set of cables, take the Socrates Challenge. Go to BlueJeans.com and buy yourself a replacement set of BJ Cables.

 

I did. But BJ Cables for me was the expensive option. I compared their 12 gage wire to a $20 spool of 16 gage speaker wire I bought at Sears. I cannot really claim to hear a difference.

 

[tonmeister86]

Do you have a link to their new measurement techniques? Do the measurements correlate with results from controlled listening tests?

Thanks

 

[tonmeister86]

Clay,

 

Expensive is a relative term. If a cable costing $500 has the same measurable electrical properties as a $50 cable with no audible difference -- then to me, it's too expensive. But for a millionaire, $500 is chump change for a cable.

 

 

This is not a smear campaign at all. If exotic cable manufacturers make advertising claims about the superior sound and technical performance of their cable then they should be able to back it up with data that can be verified: specifications, double-blind listening test results,etc. If they can't then, I'm highly suspicious, particularly since the claims often challenge current accepted scientific understanding/knowledge in electroacoustics and human perception of sound.

 

As someone already pointed out above: it shouldn't be left up to me or other scientists to prove there are no audible differences in cables. It should be the responsibility of the cable manufacturer to prove that there IS an audible difference if they are making such claims. After all, it is much easier to prove there is an audible difference than the reverse.

 

I'm still waiting. (for some proof)

 

 

 

 

[firedog]

I also use Blue Jeans cables as well as some others (all under $100). I do hear some differences between cables.

 

I haven't been convinced that expensive (in my definition over $100, and especially those that cost many times that per meter) are worth the cost.

 

I like using Blue Jeans for comparison, as I know they are well made. That's important, b/c many big box store, off the shelf cables aren't well made (example: RCA plugs not attached properly) and that's one of the reasons they don't sound good. The first step in the process is ensuring that the cables are basically well made, b/c otherwise we don't know if the more expensive cable is intrinsically better, or it's better simply b/c it's constructed properly.

 

[Gregor Samsa]

It's an irony of almost breathtaking proportions for someone from Harman to come to this forum and excoriate other audio manufacturers for bilking "unsuspecting audiophiles" for "high profit margins" by selling "audio jewelry" which adds no sonic merit.

 

Mr. Olive can you please direct us to the double-blind, multi-center studies which establish the sonic superiority of your Lexicon Blu-Ray over the identical (except for the case) Oppo? Refereed journals, of course. If it's not a trade secret, what margin do you obtain for charging 7 times what Oppo does?

 

I did take your advice in one respect. I bought an Oppo, and sent it to Modwright, who actually did something to make it sound better. I still came out about $2000 ahead, which I spent on 2007 Southern Rhones.

 

[Part-Time Audiophile]

"I did. But BJ Cables for me was the expensive option. I compared their 12 gage wire to a $20 spool of 16 gage speaker wire I bought at Sears. I cannot really claim to hear a difference."

 

That's awesome!

 

Interestingly, Blue Jeans will tell you the same thing -- for speaker cables, lamp wire is great stuff. Interconnects are a bit different, it seems, as the capacitance number is actually important. LOL.

 

[barrows]

"As someone already pointed out above: it shouldn't be left up to me or other scientists to prove there are no audible differences in cables. It should be the responsibility of the cable manufacturer to prove that there IS an audible difference if they are making such claims. After all, it is much easier to prove there is an audible difference than the reverse."

 

Actually, I disagree with the above statement. I generally expect a manufacturer to be relatively transparent about the features, specifications, and circuit implementations of their designs, and to describe why they do things a certain way, and what benefits may be expected (without expecting them to completely divulge proprietary technology), but actual testing should generally be done by third parties: reviewers, and end users. I admit it can be hard to trust "testing" done by some manufacturers, especially if the results are accompanied by language obviously written by the marketing dept. rather than engineers or actual listening test subjects; I would always recommend that one tests things in their own system, with their own music and ear/brain.

On the other hand, the new testing methodology being developed by Nordost and The Chord Company appears to be quite sophisticated and expensive to implement, and no reviewer or end user would be capable of doing it. (They have enlisted the help of mathematicians associated with the HMS Navy, sonar/submariner engineers).

I have not been able to find any info on the web yet that details this testing methodology-it was in an introductory state at RMAF last fall when presented there, but I do recall that it is mentioned in Nordost's latest print ads. I will do some more searching, and if I can find anything I will post it here.

The results (which were somewhat preliminary, and presented as such at RMAF) did correlate with generally accepted listening preferences as presented at the time.

 

[Shadorne]

Some of you may be or may not be surprised to learn that "the case" costs a LOT of money - the front face plate is often the most expensive individual item in electronic components. Just as the veneer or piano black gloss on a speaker costs a fortune - again often the most expensive item going into the overall cost of the speaker.

 

So it is fair to say that there is a jewelery aspect to expensive items from high end manufacturers.

 

Some audiophiles like to display their speaker cables prominently - sometimes lifted off the floor using the kind of ceramic isolators used in the power transmission industry. The aesthetic effect is rather extreme - like something out of a Frankenstein movie but the message to others is "I will spare no expense when it comes to audio". Provided the owner has not been duped into thinking that they "hear" these extra aesthetic/superficial qualities from over sized cables that look like a boa, components with stunning fascias, speakers that look like a Steinway...etc.. then it is all OK. I draw the line however at hyperbole over technical claims - especially with passive items such as a bit of wire.

 

 

[Part-Time Audiophile]

"Mr. Olive can you please direct us to the double-blind, multi-center studies which establish the sonic superiority of your Lexicon Blu-Ray over the identical (except for the case) Oppo? Refereed journals, of course. If it's not a trade secret, what margin do you obtain for charging 7 times what Oppo does?"

 

That's a bit harsh, even if the irony is a bit thick. In Lexicon's defense, they do add something that Oppo doesn't -- a linear PSU if nothing else.

 

That said, I don't think it odd that the "cable situation" raises the ire of an audio-gear manufacturer. After all, there's a ton o' money in that sub-industry -- money which could possibly be better spent on gear. ;-)

 

I've asked several times if there is such a difference on offer from, say, power cables, why don't manufacturers include them? Why would they be willing to let chance affect the sonic performance of their products? Interesting, I think.