Two alternative forced choice testing

[esldude]

http://en.wikipedia.org/wiki/Two-alternative_forced_choice

 

ABX testing is a regular and contentious topic on audiophile forums.

 

I think an alternative blind testing methodology might find more acceptance. I have in mind two alternative forced choice testing (2AFC).

 

ABX testing presents stimulus A, then B, and then X an unknown. The test subject then picks whether X was like A or B.

 

In 2AFC testing, the subject is presented with stimulus A and then B which is known in each case to be different in some way. The subject then selects which has more of some quality under investigation. A simple version would be testing for loudness discrimination. Stimulus A and B are presented and you choose which is louder. If the two stimuli differ by 10 dB you would expect everyone to pick the loudest reliably. Down around a difference of 1 dB or a bit less the results will look more like guesswork and you can figure out what levels are discernibly different. The same test can be used to determine if some difference is discernible at all even if it isn't something clearly measured as discernible.

 

You could for instance test USB cables. If the signal is played through two cables then that is the difference. Unlike ABX testing where you are simply trying to select same or different one would need to decide what is being tested. If people have said one USB cable has more bass then you could ask for that. Have subjects pick which stimulus has more bass. One could get even more nebulous and pick say which has a more spacious quality.

 

Now maybe all the measurements electrically say both USB cables present the same measured bass. You would expect test subjects to pick either of the cables correctly about half the time. On the other hand, whether it is due to measured bass response differences or some other psychoacoustic effects, if listener subjects are able to reliably perceive more subjective bass over one USB cable vs. another then something perceivable is really going on.

 

This method is less statistically efficient than ABX testing. If for no other reason than you might need multiple tests. For bass, for treble, for space, for enjoyment or any other thing you might think of discerning. Even if subjects vary on their preferences, say one likes more bass than another, both will be able to reliably discern a real difference.

 

I think such directed perception for testing is much closer to how audiophiles listen when comparing equipment. You listen for certain qualities not for a blanket difference. Yes, you may switch something and see if it is different, but in a few seconds I think most audiophiles have locked onto something they think they hear different and then proceed to listen for it. So listening for a particular quality seems to me more like the style of listening usually done. As a common complaint is ABX testing interferes as it differs so much from how normal listening is carried out I think 2AFC is a blind testing method that is closer to normal audiophile listening. Perhaps it would get better results.

 

As for my opinion, background, motivations etc. I am not a scientist by profession. Just a hobbyist audiophile. I do definitely think I hear some things that I supposedly shouldn't. The feelings that go along with those perceptions are too strong to ignore. When friends seem to hear the same kind of effects in the same ways it only strengthens those feelings. When those are publicized as being similar to our perceptions by others we don't even know then it seems credible. But clearly placebo effects are common to all humans, and I am sure it applies to me. I also observe others who are claiming to hear things they shouldn't be able to or things that in fact aren't happening, as well as being something I don't hear though I try. So I don't believe all claims of audible artifacts. This includes some of my own claims. I have learned over the years I have fooled myself before and will again. Though we humans must live with our feelings it also is clear no matter those feelings, no matter how many agree with them, they can still be wrong in all reality.

 

It seems many are pursuing illusory perception at least some of the time. Often at great expense. This also leads to solving phantom problems. Computer audio seems to be sharpening this contentious situation more so than ever. CA potentially gives audiophiles a golden age of high fidelity. Digital files reproduced in bit perfect fashion, amenable to correction for higher than ever before fidelity in playback, with more convenience than ever. This stuff should be very nearly perfect in audible terms. At least the source shouldn't be an issue just everything that follows the source. Doesn't mean all recordings are good, nor that my preferences are the same as everyone else. But if we can playback files at such high fidelity all that is left is our preferences under our own control. Then a much more useful allocation of our resources is investigating those preferences to our enjoyment rather than suffering effects of expensive audio nervosa because "everything matters".

 

Claims that files that are bit perfect copies, played back through the same reproduction chain, from lossless sources that have been buffered in RAM in exactly the same form sounding different don't strike me as credible. Where are the differences to be found? USB cables sounding different? I am not sure yet. So far have heard hubs that I thought changed sound for the worse, but without hubs all the USB cables seem the same (though I have not used any expensive high end USB cables yet). Now blind testing protocols seem like they should be able to tell us what is real, and what is snake oil. But ABX is not an enjoyable process to take part in I must say. I would prefer if we had a testing method that worked well, that most could agree upon. Then we could separate what really matters from what doesn't.

 

 

 

[Paul R]

I missed this when you originally posted it.

 

 

[CG]

The following is NOT meant to be snotty or otherwise negative...

 

What problem are you trying to solve here?

 

By that, I don't mean trying to find an alternate and useful testing protocol (which seems to have good merit, at least to me). Rather, just why is a testing protocol needed in the first place? I can understand why researchers may need that, or even some manufacturers. But people who wish to listen to music for pleasure? I don't fully understand. What am I missing?

 

[esldude]

Well in my opinion, at least some very expensive audio items are of dubious value. Like $1000/per meter cables. Exactly what can make it cost that much? Probably that someone is willing to pay as the construction cost isn't anywhere near that. So seems like lots of money is wasted on things that really don't matter. No worse than a Rolex watch which isn't the accurate watch for time keeping, but is a good status symbol I suppose.

 

If you read such forums it won't take long before you run across some seemingly outrageous claims about something effecting sound quality. Such testing might answer whether such has merit.

 

Now you can just plug it all up and listen happily of course. Like many hobbies some people want to get deeper into understanding what is happening. Likely at first to improve their experience. Too much can of course kill the pleasure of it. Some of us do get pleasure out of learning more about our hobbies. We sometimes can get a holier than thou attitude about it. Which isn't right, and I try to avoid.

 

The aim here is to separate fact from fiction in the end.

 

[Mark Powell]

We have another thread where the original question seems to have been lost as it is now a welter of statistics and a discussion of testing methodologies.

 

I have no interest in, or knowledge of, either.

 

Here is a simple blind test.

 

I come to your house with a truckload of equipment, of many varying prices, cables included. Maybe not loudspeakers, they are perhaps too big.

 

Between the loudspeakers there is a screen of some kind. Behind that I am changing the pieces, one at a time. you tell me which you like the best. Only you, you don't give a damn what any spurious 'panel' thinks, it is your money you are spending. No mumbo-jumbo needed either.

 

After all, you did say 'your preferences under your control'

 

Why all the 'methodology'?

 

[esldude]

Mark you have described a blind test.

 

There is actually no mumbo-jumbo. Just an attempt at somewhat clear and valid communication of a procedure.

 

[Mark Powell]

Yes, I did describe a blind test. I said so, on the 4th line down. It has to be blind, otherwise a 'Rolex type status symbol' might come into it.

 

Regards