Controversy of ABX testing

[4est]

We aren't talking about truth, belief or justifications, but observances. How they are different, better/worse or the reasons therein would be a different matter. In fact, if I understand things, the burden of proof would be to prove they are the same as it stands to reason that different parts constitute a different whole.

 

[Larry B]

Scot:

 

Every few weeks I correct someone who has incorrectly used "begging the question." As you might imagine, I am very popular.

 

 

Larry

 

[AudioCynic]

 

http://en.wikipedia.org/wiki/Blind_experiment

 

How did you miss that one?

 

[Larry B]

was blinded by science.

 

[Alan Sircom]

...is used when the reviewer thinks a product at a given sticker price performs like one costing considerably more. The reviewer can do this if they know the product and its peer group. If a reviewer thinks this product is better than others in its peer group, this should be mentioned.

 

A bad reviewer can only judge the value of a product by the size of their own bank balance. If you think something is good value because it's affordable to you as a reviewer, fine, but you also have to determine the value of products that might cost a trivial amount to you, or that cost more money than you earn in a decade. If you cannot do that, you shouldn't be a reviewer.

 

Personally, I don't use the word 'bargain' in audio, but not for reasons you might expect. I find the term cheap. Bargains are the discount frozen peas sold two days before they are thrown in the dumpster. Bargains are the cheap, uncomfortable couch that lasts a month longer than its warranty. Audio is much better than that.

 

 

 

 

 

 

[antipodes_audio]

I don't buy this demand for proof thing.

 

Different people have their BS meter calibrated in different places, either based on their gullibility/wonder/adventurousness, or by different levels of experimentation experience, or by denial/belligerence. When someone suggests say cables make a difference, then for some, their BS meter goes off and they insist on standards of proof they will likely never get. For others, either due to their gullibility or their experience, the notion that two cables could sound difference is old knowledge. For others, it is a reason to borrow some and develop their own opinion.

 

This demand for proof thing seems to me to be a deflection tool for someone in denial, or someone that just wants an argument from an intellectual high-ground.

 

I was once skeptical about say power cables. I am not now. I no longer require anyone else to provide me with proof. In fact I never did. But if someone tells me that painting my roof pink will improve the bass of all systems in the house, my BS meter goes off and I won't be buying any paint.

 

How do you buy a car? Do you look at the specs, check what other people have to say about it and its competitors, and drive one? I do. When your neighbour suggests you buy X, do you insist he provide you with scientific proof of something, or do you add it to the weight of other opinions based on your respect for his opinion?

 

I am not saying that there aren't a lot of claims that are more to do with gullibility, placebo and marketing hype than they have to do with reality. But I don't think you can fool more than a very few of the people all of the time. If my BS meter goes off I look for weight of opinion on the matter and if thousands of posts, including opinions I respect, reported painting the roof pink was beneficial, would I wait for the world's learned scientists to study the matter? No, I would think about it giving it a go, maybe painting a small patch over my music room. After all, having an open mind to things I can't explain has paid off before

 

[James1776]

The 'M' word, as in 'moderately' priced, and for the same reason...I picture a couple of audio reviewers, vintage wine in hand, surrounded by $50K in gear (for lesser folk but $20K in gear for them) speaking of how moderately priced is the latest $40 bottle of wine being tasted, not to mention the $2000 cartridge for their $15K vinyl front end.

 

The attitude reflects an alarming 'let them eat cake' attitude on the part of a lot of the industry.

 

[Part-Time Audiophile]

"We aren't talking about truth, belief or justifications, but observances."

 

It's funny that you say that because I think this is one of the places we all get tripped up. Epistemology, properly, is concerned with the nature of knowledge. Science, one could argue (as I have), is a branch of epistemology, one that is concerned with theory testing. From Wikipedia (Entry: Theory):

 

... in modern science the term "theory", or "scientific theory" is generally understood to refer to a proposed explanation of empirical phenomena, made in a way consistent with scientific method. Such theories are preferably described in such a way that any scientist in the field is in a position to understand, verify, and challenge (or "falsify") it.

 

While I could quibble on the use of the word "explain" (as opposed to "describe"), the point is knowledge. It's acquisition, it's extent, the process by which we come to it. All epistemic questions. Interestingly, all scientific ones too, if philosophical in nature.

 

Why this relates has to do with the way we all (ought) to arrive at knowledge. And whether or not the claims that are made have interesting epistemic properties like justification, or more strongly, warrant, or even referents.

 

More specifically, the question at hand is one of justification. There are lots of different kinds of justification and methodologies that can be deployed to support our claims. Judicious use of the Scientific Method is only one. An interesting question might be, is it the proper one. I believe that it is sufficient, but not necessary -- but I'm not convinced I can successfully provide justification for that belief, much less, provide a clear case for having warrant.

 

[Alan Sircom]

You are the editor of a high-end audio magazine. Your publisher has just placed a blanket ban on long-term loans and accommodation purchases. Around the same time, you are offered the 'scoop' on the first review of an important, but very expensive, high-end product. The review is contingent upon the product being placed in a system of a similarly high grade.

 

The object of the exercise is not to get fired for ignoring the publisher's edict, while also not ending up printing a magazine with a 'scoop'-shaped hole in the contents list.

 

How would you do it? What would you do to give the public what it wants, without doing something you would consider getting your hands dirty?

 

Sorry, but saying "I don't know, but I don't like it" doesn't count. I don't like the fact that beetroot exists in the world - it should be extinct, IMO - but that won't make the thing go away. If you think "that's your job", well... now's your chance to show how you would do it better. If you are going to simply pontificate about the evils of long-term loans and accommodation prices, but offer no solution to the problem, you are part of the problem. Put simply, put up or shut up!

 

 

I'll check back in a day or so and see how you got on. I'll then provide some comments about your ideas and plans.

 

 

 

[gmgraves]

"I completely get it now."

 

Do you? I don't think so. You said "We know...". Now, if you'd have said "I know...", or more accurately, "I believe..." there would be less room for error.

 

"Gee whiz, IME those are more important than the chips used."

 

Actually I don't think so. Some systems use a complete chip-set solution, in which case all of those systems comprised of that chip set will sound more-or-less the same. Most chip sets have a "sound" and they differ subtly because of it. But none of them sound "bad" now that the days of brick-wall analog filters are behind us. When I say that "I'd hate to live on the difference." I mean that the difference simply isn't significantly important.

 

I was recently invited to join-in a DBT between a bunch of DACs held by a local hi-end recording studio. They were upgrading their monitoring/playback system. Some of the DACs tested were cheap ($500-$1K) some were expensive ($1K-$5K), and some were very expensive ($5K+). It was fairly easy to reliably detect the difference between them, but honestly, from my notes, I didn't find that the $15K MSB Platinum DAC IV was any better sounding than the $2K Antelope Zodiac DAC. Different? Yes, somewhat. Better? Not really. Which was correct? I can't say as I had no idea what the "correct" DAC sounds like. I did note that I thought that the Zodiac had a little smoother treble region than did the MSB, but was that because the Zodiac was more proper, or was it because the MSB resolved the top end that was "really there" more accurately? Who knows? Needless to say I didn't hear any DAC that day, which was so different from the others, that their character and differences from their competition wouldn't fade into insignificance after a few days of living with them. In the end, it came down (for me) to features and I left, still happy with the up-sampling, dual-differential, 24-bit 192 KHz DAC in my new Harman-Hardon HK990 Integrated amplifier.

 

[Alan Sircom]

I can't afford vintage wine. I'm too busy spending money on audio equipment just to try to stay in the game.

 

[AudioCynic]

You've just described the policy of the New York Times. The answer is simple. Every manufacture has a demo setup, they need it for trade shows, they need it for day to day sales activities. The ship the demo setup to the reviewer with a prepaid return label. In a couple of weeks he sends it back. Easy.

 

[Part-Time Audiophile]

Actually, I agree with you. I think that owning a piece of gear makes it very hard to discriminate amongst all manner of biases, DBT and placebo and all that notwithstanding. Pro reviews, especially for expensive gear, ought to be done with demo units.

 

On the other hand, I see a small problem.

 

The problem is all that "other gear". At some point, a reviewer is going to have to buy something other than what is being reviewed. That is, if he's reviewing speakers, it might help to have an amp or two. Where did that amp come from? Was it reviewed? Did the reviewer purchase it after the review, or before? What if, during a review, it became apparent (to the reviewer at least) that the reviewed component was superior to his standard? Should the reviewer then abstain from purchase for fear of being accused of bias or corruption?

 

I'm thinking that we're probably over thinking this.

 

[Alan Sircom]

2/10 for the answer

 

1. If the company has a demo system, that set-up is a best-case scenario for the product. It's going to be the most optimized system to highlight the performance of the device under test. Is it good because the product is good, or because of the rest of the system? You've just described a review that you would reject for lacking any frame of reference

 

2. Most manufacturers don't have a demo set-up. They might have a reference system that the designer uses, but that is rarely made up of new products and would not be considered relevant to the review process. Most of the other equipment they borrow (or even rent) for shows. Unless the manufacturers used get a mention as part of the review, they won't play ball. So, instead of making a more honest review schedule, the review has to roll out a series of 'thank you' statements, like a bad Oscar acceptance speech, to a bunch of companies you have no control over

 

3. Can a manufacturer guarantee the other manufacturers will get their part of the deal in at the same time? From experience, it's like herding cats; just trying to get a three-make system arrive within the same calendar month is something close to a minor miracle - one will be stuck with another magazine, one will be out of stock, one will arrive late because the courier left it in the warehouse overnight. We run up to six months in advance just to assume we get everything needed for a 13 review magazine in time, and we still end up scrabbling round for a couple of products in the last few weeks

 

4. If the magazine demands its own reference products to maintain a consistent frame of references, but was not the subject of a long-term loan, someone would have to cover the costs of a product being repeatedly sent between the reviewer and the manufacturer. Who would cover this cost? In essence, there would still be a long-term loan, just the storage of the loan product would be at the manufacturer

 

 

So... sorry. Nice try. Doesn't work.

 

[iamimdoc]

Again, for reviewers/manufacturers why not just have fair and open disclosure and let the readers decide as to what to believe, accept, ignore or be jazzed about.

 

Let the reviewer say, in the "list of kit" (as I think it is said in the UK) section: I bought the Hippo DAC at 50% off retail or I will be able to use the Jazzifier Amp for 2 years at no cost".

 

Or (better)

 

I used this for 3 months and sent it back to the manufacturer.

 

[Paul R]

Every professional in every field builds up the best, the very best set of tools he can acquire. Hobbyists tend to the do the same thing, but only as far as disposable income goes.

 

Example: I have sold off 90% of my personal collection of computers, and yet, not counting the old Macs and PCs laying around, I have better computers in this house than most under $1M businesses own. That's my profession.

 

Also, were I to pursue a writing career in the audio field, I guarantee I would acquire much higher end audio gear than I currently own. But I expect I would do so as ethically as I do everything else. By that I mean I might take advantage of "accomodation" pricing if available, but not on something I planned on reviewing in the near future. Ethical behavior. It's a pro-survival activity in any professional endeavor.

 

Something I had already reviewed, I might ask for a discount on to purchase, but you can bet it would be documented, and cleared with whomever was paying my paycheck before it happened. If I was freelance, I would make a pubic disclosure or something.

 

Now, on the subject you mentioned, when the review was only going to go ahead only if put into a high end system, I would have told them "Sure thing, I understand your concern. Our policy is that I determine what systems are suitable, and I will certainly be aware of this when I select the reviewer and system. We will include the review system details and send you a pre-publication copy of the review for you to comment on."

 

There might be something wrong with deal if the product owner doesn't buy off on a deal like that. In which case, you might just have a scoop of a different flavor anyway...

 

-Paul

 

 

 

[thesurfingalien]

Hi Alan,

 

I have been able to acquire both tracks (took a bit of time), but I am not sure if those are exactly the ones you used. Could there be more instances where both conductors have conducted the same orchestras on the same pieces?

 

The reason I am asking is that I can ABX the differences with both versions within seconds after it starts. I tried 24 times and got it right 24 times (result posted below). I mean... Really, it´s that simple! As you can see the whole test took less than 7 minutes.

 

If I am indeed ABX'ing the exact same versions you used, the only reason I can think of why your test-subject(s) failed is that they, due to the fact you played the whole movement every time (if I remember correctly), they simply forgot which one was which. Let's face it, one pass lasts roughly 24 minutes...

 

As far as I know, the "official" ABX testing-protocol does not require a whole track to be played. One can decide for how long or short one wants / needs to make the choice.

 

Any thoughts yourself?

 

Regards,

Peter

 

EDIT: My wife did not have time to do the test.

 

foo_abx 1.3.4 report

foobar2000 v1.1.5

2011/07/07 15:15:33

 

File A: C:Documents and SettingsPeterMy DocumentsMy MusicTESTSymphonie No. 7 A-Dur Op.92 2. Allegretto, von Karajan.wav

File B: C:Documents and SettingsPeterMy DocumentsMy MusicTESTSymphonie Nr. 7 A-Dur op.92 2. Allegretto. Kleiber.wav

 

15:15:33 : Test started.

15:16:04 : 01/01 50.0%

15:16:19 : 02/02 25.0%

15:16:37 : 03/03 12.5%

15:16:56 : 04/04 6.3%

15:17:10 : 05/05 3.1%

15:17:22 : 06/06 1.6%

15:17:36 : 07/07 0.8%

15:17:58 : 08/08 0.4%

15:18:14 : 09/09 0.2%

15:18:28 : 10/10 0.1%

15:18:41 : 11/11 0.0%

15:18:55 : 12/12 0.0%

15:19:06 : 13/13 0.0%

15:19:19 : 14/14 0.0%

15:19:33 : 15/15 0.0%

15:20:05 : 16/16 0.0%

15:20:16 : 17/17 0.0%

15:20:32 : 18/18 0.0%

15:20:42 : 19/19 0.0%

15:20:55 : 20/20 0.0%

15:21:08 : 21/21 0.0%

15:21:17 : 22/22 0.0%

15:21:26 : 23/23 0.0%

15:21:41 : 24/24 0.0%

15:21:56 : Test finished.

 

 

 

[Alan Sircom]

The main point of the exercise was to compare movement with movement, not inter-track. So yes, each test does take about half an hour to conduct, and there need to be a lot of gaps between each test, and preferably quite a few different pieces of music to compare (or a large group of test subjects).

 

The reason for the test in this way is simple. People claim a preference of one over the other (and as you have shown the two versions are very different under close examination), and yet, if they listen to the two movements under these test conditions as complete entities, their ability to differentiate the two falls away. The standard response to this is to put it down to our poor memory for sounds, for good evidentiary reasons.

 

What I'm interested in exploring is whether that really is the case, or if there is some other process at work. My next plan is to repeat the same whole-movement test (this time using a larger sample), in ABX, triangle and constant-reference duo/trio form.

 

As you can imagine, given the length of the test in each case, this is not an overnight test.

 

 

 

[Alan Sircom]

Several issue arise from your statements:

 

1. "I bought the Hippo DAC at 50% off retail"... congratulations, you have just put Hippo (and the company's agents) out of business, because you stating that has set everyone reading it their discount target. "Well the price is listed at $300 but I'm not paying a penny above $160, because you sold it to the reviewer for $150!"

 

2. Is there the potential for a two-tier ranking of reviews, that those who loan are marked out as being a lower-class reviewer than those who buy?

 

3. What checks and balances are there for "I own it" fanboydom? In my experience, the worst reviewers are those who make a point of wearing their pride of ownership like a badge of honor and who tear down any product that challenges products they own because they fear they might damage their own equipment's resale value.

 

4. Is "I used it for three months but the review was delayed by the editor adjusting the mix of the issue and delayed again because the distributor asked if it could be put back because they didn't have enough stock in place, I returned it but it was returned to me because the distributor was out when it was delivered it a day later than the courier said and then spent the next seventeen months languishing in my paid storage until the manufacturer finally appointed a new distributor who now has a prospective client waiting for the product so I ended up holding on to the product for about two years" too much information? That's the usual story for about one in four products I review.

 

[Part-Time Audiophile]

Looks like somebody has gotten singed in the past:

 

http://audiophilereview.com/cables/getting-shun-mooked.html

 

[barrows]

makes many good points regarding the reviewing process. I have no problem with accommodation purchasing, and would also point out that inter-industry discounts like this are the norm in every field I have worked in.

Additionally, I would even prefer that a professional reviewer have unlimited access to any and all equipment free of charge, as actually making purchases limits gear selection, and could even create a bias

Note: accommodation purchasing does not create any preference, or present a conflict of interest, as accommodation pricing is equally available from all manufacturers.

Those here who seem to have a problem with it seem to have a hidden problem, perhaps they are envious of the reviewers position and access to high end gear? How many reading this would really give up their current career to become a professional audio reviewer, be honest. Personally, I would not want to be a reviewer, as reviewing is a difficult and relatively thankless task requiring a great deal of work, and relatively little compensation.

 

[4est]

Well said barrows!

 

[wakibaki]

Because it's necessary if the subject is not to be swamped by a mass of contributions from the ill-qualified and unthinking.

 

Opinions, as they say, are like 4ssholes - everybody got one. Nowhere is this more true than in audio.

 

'I've got ears, therefore I'm entitled to an opinion.' Maybe so.

 

A few years ago I worked on the design of a basestation for 3G phones. There was no argument about the Air Interface, the PA predistortion, the board-to-board communication, whose FPGAs to use. No. The design work was completed, to all intents and purposes, until it was held up at the end for nearly 2 weeks by a disagreement about whether the front panel should have an LED indicator to show that it was functioning, because every swinging d1ck had an opinion. Even the security guards at the front gate wanted to stick their oar in. They new better than to offer their opinions on the modulation scheme though.

 

I would bet that not 1 in 10 of the contributors, probably not 1 in 100 know what a common emitter amplifier is, to say nothing of designing one. How many know the difference between a direct and an enharmonic modulation?

 

Having worked both as a design engineer and as a musician, I have some appreciation of the difficulties faced by a designer when trying to satisfy a commisioning client who can't specify when the design will be complete but will 'know it when they hear it'. Is that in march time or three four?

 

Never, ever, accept a commission to design a piece of software without an agreed design specification.

 

If those of us who appreciate the necessity for some form of truly objective testing as opposed to the unsupported opinions of self-appointed experts simply stop trying to drag such threads back in the direction of rationality we will have taken a step down the slippery slope of decline in the quality of audio products. The principle factor preventing this is, however, the common sense of the wider buying public (and the [huge] companies who meet their wants) who have already tacitly recognised 'hi-end' audio as a scam impacting principally the pretentious and those with more money than sense.

 

Oh, and as for trying to prevent other people wasting their money, what's wrong with that as an objective? Better than trying to drag others into the same quicksand you've got yourself into.

 

@sandyk

 

Thanks Alex, I'm still in remission and fit enough to walk the dogs every day, which is about as much as you can ask for :-) Still playing guitar and designing electronics too.

 

[AudioCynic]

There's an expression, people lie, markets don't. The market understands that there are aspect to high end that are simply absurd. I'll go a step further. There's a large body of affluent, music loving consumers who are who would buy better than mass market systems. The wild claims made by reviewers and enthusiast about the benefits of multi thousand dollars speaker cables and 200 dollar electric outlets and all the other high margin garbage just undermines their confidence in every review they read and every recommendation they encounter. Consumers don't like to be robbed, and the wild claims make them think that every claim for every piece of equipment is fraudulent.

 

 

[Part-Time Audiophile]

Sorry, Cynic. Your market analogy only holds if that cable market is in some kind of trouble. Interestingly, it isn't. There are new cable makers every year and more and more astonishingly priced offerings from these new companies -- as well as their more stodgy competitors. Using your market analogy, then, seems to support not only the entrepreneurship but also the fact that the products not only find their consumers but that there is *real* value in them.