Controversy of ABX testing

[Larry B]

Forrest, never underestimate the abilty of humans to food/deceive themselves.

 

[4est]

IME, usually when people are hell bent on fooling themselves there is a strong motive. I do not see one at play here. At least not one strong enough for me to waste a lot of time and money over besides the obvious "I want better sound.". I understand that trap, and have suffered from going backwards and pissing money away. I'd have to fool myself into fooling myself. Those odds are just too small for me to put much stock in.

 

[Paul R]

Did you notice the writer assumed psychological bias? The key word being "assumed."

 

Try it along these lines - do you do blind tests when you are choosing between two new vehicle models? Or do you drive them and then base which one you like better based on comfort, color, size, performance, and dozens of other factors you can only judge when you are actually driving the beastie?

 

If both cars were available at the same time, they blindfolded you, and took you for a ride in each one, your judgement on which one you like better might change.

 

Of course, a purchase decision involves other factors, price, reliability, warranty, etc. But if your blind choice was in conflict, which one would you choose? Assuming all else is equal? Would you automatically assume all the factors that made a difference were purely psychological?

 

I would hope not.

 

-Paul

 

 

[Larry B]

Paul:

 

"If both cars were available at the same time, they blindfolded you, and took you for a ride in each one, your judgement on which one you like better might change."

 

Precisely.

 

 

Indeed, many factors go into making a choice, as I discussed this in an article a number of years ago: http://www.dagogo.com/View-Article.asp?hArticle=455. I have a friend who bought his preamp because it has a red top plate; it's his money and if a red plate makes him happy, then it was a wise choice.

 

But the key issue here is that in the case of audio, some (in fact, many) people discount the influence of those other factors and assume that they are completely unbiased listeners. I daresay this is rarely if ever the case.

 

Larry

 

[wakibaki]

'do you do blind tests when you are choosing between two new vehicle models? Or do you drive them and then base which one you like better based on comfort, color, size, performance, and dozens of other factors you can only judge when you are actually driving the beastie?'

 

No Paul, what I do is read the specifications.

 

Specifications I can depend upon being kept honest by the manufacturers competitors and the government trading standards bodies.

 

I dropped out of my first (chemistry) degree 45 years ago to play the guitar. Tune in, turn on, drop out. Some 20 years later I went back to do a degree in satellite communications. I can happily laugh at anyone who thinks I am soulless, particularly when it often turns out that they can't play a note.

 

Unfortunately the standards imposed in the auto industry do not obtain in the audio business, in part due to the efforts of individuals such as yourself.

 

Only yesterday on the TV I saw a representative of the tobacco industry (a non-smoker) rejoicing in the fact that cigarette sales in the Phillippines are unrestricted. She justified this by saying that while tests in other parts of the world showed that tobacco caused cancer, none of those tests had been done in the Phillippines.

 

Therein lies the problem with an over-facile use of language.

 

The one person you don't want to end up kidding is yourself.

 

w

 

[Paul R]

You buy a car based only on specifications? Good luck with that mate. I would not do that.

 

 

-Paul

 

[Paul R]

But the key issue here is that in the case of audio, some (in fact, many) people discount the influence of those other factors and assume that they are completely unbiased listeners. I daresay this is rarely if ever the case.

 

Larry, if you ever find a person like that, let me know. I have never met one. In fact audiophiles tend to be filled with more self-doubt that most other groups of people. It;s one reason they get so defensive.

 

I think that very few if any audiophiles assume they are unbiased listeners. Indeed, most audiophiles want to listen to gear for an extended period before they make any decisions.

 

-Paul

 

 

[Part-Time Audiophile]

I wish it were this simple! Measure the same, sound the same. Wouldn't it be grand?

 

The problem is that nothing ever does. Everything sounds different. The argument then goes, "well, maybe we're measuring the wrong things" or "maybe we don't know what we should be measuring".

 

Whatever. If someone could actually tell me what measurements actually matter (bonus points for when they matter!), that'd be great! I'd love to know.

 

Here's a complicating factor: there is no consensus on what "good" actually is. It's different for, well, everybody. And yet we argue as if we were all using the words the same way, or meaning the same thing by them. Yeah, about that ....

 

The fact that we can communicate at all is something of a miracle.

 

 

[Guest Caner]

" In fact audiophiles tend to be filled with more self-doubt that most other groups of people. It;s one reason they get so defensive"

 

Isn't this another reason to bring standards for better comparison? What do you think? I am new to this and as I mentioned before I am totally neutral to cable topic, since I do not know enough.

 

And I trust my ears, yes, but at least to narrow down on my choices I have to read reviews published; like those on Stereophile (and/or other web sites). Than I try correlate the reviewer's impressions with the measurments they publish. This is how I come up with an initial impression about the gear. That's why I enjoy Stereophile a lot I suppose, as John Atkinson do extensive measurments on the gear that they test.

 

It would have been excellent though if I could add some blind testing data to my armementarium as well. Sadly there is no industry wide standards in place.

 

I happen to know that Revel does extensive bilended testing for their speakers (which to me is very comforting). Possibly due to lack of standardization, this valuable information never becomes a public domain though.

 

(Disclaimer: I have no affliation with ANY group in audio industry, I am non conflicted and have no financial interests about this matter)

 

So in essence while we do not make our decisions based solely on numbers, it would have been great to have enough objective data as a valuable tool (and I understand methodolgy is an issue here)

 

Lastly to answer Chris's question on my intention to post my impressions on this topic: I belive evidence based practice makes a positive difference in all the venues of our lives; extending to audio domain I believe, will contribute greatly to this beautiful hobby.

 

Caner

 

[4est]

Really though if you stop to think about it, if ABX testing really worked, it would be used more. People such as myself have spent a lot of time thinking, working and testing stuff over the years. If I found the results from the published ABX tests akin to my own, I'd be all about it. Why not! The sad fact is that they just are not accurate. There may not be tons of truth in the reviews, but there is no "conspiracy" either. Just people muddling about trying to make gear for a living.

 

If/when they develop testing and measurements that indicate the sonic merits of a piece, it will be widely adopted quickly out of economic necessity from consumers. None of us are fools or fool hardy, even if were fool ourselves occasionally. Until then, I will stick with the best measuring devices I have- my ears and my brain. They are road tested- even though they may not be perfect.

 

[Larry B]

Paul, anyone who "believes his ears" (or any expression with the same meaning) is ignoring the effects of bias. Whether they do so consciously or subconsciously is a separate discussion.

 

Regards,

 

Larry

 

[Jud]

Larry, certainly non-blinded testing necessarily involves the potential for listener bias. On the other hand, "living with" components over extended periods, which is rather difficult to do in a blinded test environment, has the potential to reveal aspects of component sound that A/B/X testing does not.

 

One might analogize this to the comparison between multiple choice tests with blind grading, and essay tests graded by teachers who have long familiarity with the students writing the essays. Each has the potential to reveal different aspects of student performance.

 

I have also mentioned in other threads the potential for A/B/X testing to present an environment sufficiently "clinical" or stressful to be inimical to the musical enjoyment so important to evaluation of components by consumers who may own them for very long periods. (I've owned my turntable for over 25 years, my speakers or the previous iteration of the same model for over 20, DAC, interconnects and speaker cables for 18 or more, and so on.)

 

Finally, there is also the maintenance of proper humility essential to scientific inquiry, that is, to admit that we don't know and can't yet measure all that goes into the 'sound' of various components. Many of us remember well when jitter first began to be spoken of as a sonically significant specification for digital equipment, or how mp3 files were supposed, according to tests of what people could hear, to be sonically indistinguishable from lossless source material.

 

This is all to say, while those who 'trust their ears' must admit to the potential for bias, those who 'haven't seen the numbers' must be careful that due skepticism doesn't prematurely turn into the bias of disbelief prior to all the evidence (and there are many kinds of scientifically useful evidence) being in.

 

[Larry B]

Jud, I agree completely.

 

Despite what you might have conclude from my posts, I neither particpate in DBTs, nor encourage others to do so. Where I do however differ from some is in my recognition of the effects of bias - not just in lip service, but in a deep and true belief.

 

Best regards,

 

Larry

 

[Paul R]

No Larry - that would be applicable only to fools, of which there is a very short supply around here - actually a non-existent supply. Well, except for a few airheads from that Hydrogen Audio play school.

 

Obviously there are a lot of folks around here with strong opinions. Some folks claim quickly that opinions are not "scientific" and any such opinions should be ignored because they do not fit into an arbitrary and too tightly defined, if comfortable, set of simple rules.

 

Of course, that is just their opinion. And a very wrongheaded one at that.

 

-Paul

 

 

[Larry B]

Paul, this is getting too contentious, so I will politely bow out.

 

larry

 

[Paul R]

It does get a bit tiresom when people assume thier point of view is automatically the correct one. Even if by some chance it is me doing it. Just get me started on networks. Nothing is as annoying as some coming out with a snearing "Well! That's your opinion!"

 

Of course it is opinion. It does not matter to some folks if it is a very good oPinion, backed up with long experience and a ton of compelling evidence. If it does not fit into their agenda, well!! We get a little of that around here, because as I said, we suffer from an extreme shortage of fools. Thank goodnes!!

 

Stick around, it will lighten up and get fun again soon. Promose!

 

 

 

[Larry B]

Fear not, I'm not going far.

 

[lightminer]

For those who like reading about this, I have a similar fun thread over on Audiogon:

 

http://forum.audiogon.com/cgi-bin/fr.pl?cspkr&1217550410

 

There is a guy, Richard Clark, who said he'd give 10k or something like that to anyone who could ABX the difference between any 2 amps.

 

Don't know if you guys have discussed his situation as I haven't read the whole thread above, but it is very interesting read.

 

Short version: 1) For the most part he was right. (He did mandate that you had to operate them within their watt rating and ohm rating, and he used a frequency changer thingy so that they each had the same freq response.) 2) For some speakers, such as maggies, the watt and ohm rating is king so eliminating that makes a test that is not worth testing, and 3) the one musical source that seemed to produce difference the most were vocals with 10 or more singers (choirs).

 

[lightminer]

Oh and one overall comment about science versus non-science and all that (as I am a trained scientist).

 

Science is king as it proceeds, but it isn't always fast. People can come to conclusions before science can use its processes to determine how and why. In fact, that is happening a lot right now in drug discovery - there is such a rush to get drugs to market many of them are not understood as to how or why they work, but if people get the result they want the drug goes to market way before the metabolic or cellular pathways have been all worked out. This is pragmatism.

 

 

I believe as I said that science is king, but at any point in time, science is only so far along. You can only know 10 - 30 years later if something is really really true (maybe 100 years?), as people discover more and more aspects of what is going on as time goes on. Geez, they only were able to figure out how bees fly 5 - 10 years or so ago.

 

So my personal take on the whole thing is to fundamentally use the lense of science to look at life through, but be open to the fact that science is a process and human beings through their conscious and subconscious minds can sometimes be a few decades ahead of what is fully understood.

 

So, moderacy and humility are good . But, as it goes, science will always win over time.

 

We will, one day, know for sure 100% if those darn super-expensive Bybee Quantum thingies actually do anything, for example. But I don't know if that day is today if we want to say we are really really really sure.

 

(Is a scientist 100% sure of anything? I'd say no. Since Einstein added relativity to Newtonian mechanics, that meant ultimately that another scientific revolution could come along and further alter basic equations. Sure, Einstiens equations only affect, for exmaple, the Newtonian approximation of the speed of say, a falling baseball from a building by 0.00000000001% - they really only come into play as you get closer to extremes, but that is what I'm talking about - 100% for a scientist is a sort of, well, a sacred thing. The next part of this issue is do we need to be 100% certain? What is the percent certainty required for action? 90%? 80% Once you know that, then you have a criterion for action, and for real scientists it is not 100%. Not if you are familiar with scientific revolution (i.e., read Kuhn) and/or the process of making a scientific "law" that the rest of humanity takes for granted as 100% (large preponderance of lack of disproof, and mild existence of proof) by a bunch of old white guys with beards in Europe. Well, I guess these days in Europe and America, and few Canadians and Aussies thrown in for good measure.)

 

[lightminer]

Speaking of relativity, when I took that physics class we had one open-ended assignment with some guidelines and I decided to calculate how much longer or shorter a person who lived on top of a 100-story building would live compared to someone on the 1st story. He would live a longer or shorter time-span because on average he is traveling faster as he is farther from the center of the earth's core as it spins, (for this crowd - think outside of record travels faster mph than inside) so he is traveling closer to the speed of light, and thus relativity comes into play.

 

Okay - so who knows the jitter level on your DAC? That will give you some reference for this number. The guy will live approximately 10 micro-seconds longer than the other person. In jitter-terms, that is a lot of jitter. In life terms, it won't help much.

 

[Mark Powell]

'Blind' testing is a simple, straightforward method that has been used in many scientific fields for a hundred yesrs or more.

 

The small and expensive hifi manufacturers hate the idea that the 'public' or even a panel of 'experts' who know what to listen for often cannot tell the difference (or find the cheaper one better) between (say) 40000 dollars worth of Audio Research amplifier and a mass market Sony, even when very 'high quality' (ie expensive) sources and speakers are used.

 

Why cannot they tell? - BECAUSE THERE IS NO DIFFERENCE AND THE TEST PROVES IT.

 

Sometimes of course, a difference is observed. EQUALLY THAT PROVES THAT THERE IS A DIFFERENCE.

 

This is why the expensive 'hifi' business is thought to be a complete joke by engineers in other fields who have more expertise and experience in the field that all of the people in the hifi business added together.

 

Try starting with cables.

 

[Mark Powell]

Deleted

 

[Mark Powell]

All observations about convincing each other (or not) and the contribution of magazines, and so on are totally missing the point.

 

If I came to your house (any of you) with a whole bunch of equipment, both mass market and expensive, and put you, personally, not a 'listening panel' or anyone else, through a blind test you would find that you eventually preferred one of my pieces than the one you have. That is inevitable if I came with a large number of pieces (it need not be an infinite number!).

 

It might well turn out that it is a mass market piece that you prefer. Your opinion. Not anyone trying to 'convince' you.

 

That is why all these tiny and expensive 'hifi' manufacturers don't like blind testing.

 

I am not prejudiced against the 'specialist' providers (except the cable charlatans). I am off to listen to a McIntosh 275 and an Audio Research amp next week. Will probably buy one or the other, without every having listened to a mass market Cambridge or Sony. Don't have to, my 'customer satisfaction' will be higher because all you guys have told me that I will be on the right track if I spend 5000 dollars plus. I am happy with that.

 

[Mark Powell]

'Good' equipment must, by definition, reproduce the sound on the recording as perfectly as possible. No 'added character' no 'sounding warm' or whatever. Just similar to the recording.

 

Thus the closer we get to the 'High End' the more the various brands of equipment should sound similar to each other.

 

Does that make sense?

 

From that I conclude that 'blind' testing, or any other form of testing, should not show up any differences. Because there should not be any.

 

 

[Sri]

When people without a financial interest in showing the hi end sacred cows (interconnects, hi res, you know the list) are true their ABX testing shows they are pretty much all snake oil.

 

The people with the marketing and research budgets, the people selling the sacred cows have no interest in funding work that shows they are selling snake oil.