Jump to content
IGNORED

AB Testing


 Share

Recommended Posts

  • 4 years later...
Hey guys what are your thoughts on AB listening tests. I have some opinions on it, but I will hold off until we get some other opinions this time.

I think a lot depends on the situation and the question you are trying to answer.

 

Specific questions are more usefully answered with AB tests (I assume you are talking about more casual testing rather than strict blind tests). Some components are difficult to test in AB situations - amplifiers being a good example where as it's easy to connect two sources to one amplifier, connecting one source and a pair of speakers to two amplifiers for quick switching is less easy.

 

Some questions are also better answered over the course of longer term testing - the "do I enjoy this sound more than the previous sound?" is only found out with testing over a period of a few days/weeks.

 

Also most people don't have the luxury of having a manufacturer shipping a $15k DAC to use for 6 weeks for testing - we have to rely on a dealer who will give us an hour or two.

 

Just a few of my thoughts... Like almost everything I think the "which is appropriate - AB testing or longer term testing?" question has no absolute answers and the correct answer is "Yes" (a combination of both).

 

Eloise

Eloise

---

...in my opinion / experience...

While I agree "Everything may matter" working out what actually affects the sound is a trickier thing.

And I agree "Trust your ears" but equally don't allow them to fool you - trust them with a bit of skepticism.

keep your mind open... But mind your brain doesn't fall out.

Link to comment
Share on other sites

As Eloise intimates, often it's the long term listening that is the true test especially when it's apples vs apples (in regards gear in a similar price bracket).

 

There has been times when I've tried a piece of kit and 30 seconds in I knew it was a no-go... Saying that, this experience tends to be at the, er, budget end of the spectrum.

 

Once you get into the high-end, this is where subtleties come into play which requires more than a simple A/B shoot-out over a couple of beers!

Naim 282/250/hi-cap/cd5xs/dac/stageline, mac book pro/fidelia/amarra hifi/halide bridge, rega p3/24, focal utopia scala

Link to comment
Share on other sites

Perhaps I am biased by my training in math and physics, but I have a fondness for double-blind AB over long-term tests. My fear with long-term testing is that it could involve "listener burn-in" and be prone to influence from what one knows about or expects from a particular system. At the same time, double-blind AB testing involves switches, so it does not allow testing cables (should one want to test them).

Link to comment
Share on other sites

Here's my take:

 

AB testing: for hearing differences (usually immediately obvious to me)

 

Longer-term listening: for deciding which I prefer (not always obvious to me at all, sometimes it is, sometimes it isn't)

 

Mani.

Main: Okto dac8PRO -> 6x Neurochrome 286 mono amps -> Tune Audio Anima speakers + 2x Rotel RB-1590 stereo amps -> 4x subs
Office: MOTU UltraLite-mk5 -> 6x Neurochrome 286 mono amps -> Impulse H2 speakers
Vinyl: Thöress Phono Enhancer -> RME ADI-2 Pro

Link to comment
Share on other sites

Here's my take:

 

AB testing: for hearing differences (usually immediately obvious to me)

 

Longer-term listening: for deciding which I prefer (not always obvious to me at all, sometimes it is, sometimes it isn't)

Much more succinctly put than I managed...

Eloise

---

...in my opinion / experience...

While I agree "Everything may matter" working out what actually affects the sound is a trickier thing.

And I agree "Trust your ears" but equally don't allow them to fool you - trust them with a bit of skepticism.

keep your mind open... But mind your brain doesn't fall out.

Link to comment
Share on other sites

Just another point...

 

I'm not at all convinced by the 'got to be double-blind, else biases may play a part' argument. I think we all develop along our audiophile paths. Sure, in the early days expectation bias may well play a part. But for my own part, the longer I've been in the pursuit of good sound the less I give a damn about how something 'should' sound from its looks and/or price. Most of my uber expensive stuff either sits in my basement or has been sold off.

 

Mani.

Main: Okto dac8PRO -> 6x Neurochrome 286 mono amps -> Tune Audio Anima speakers + 2x Rotel RB-1590 stereo amps -> 4x subs
Office: MOTU UltraLite-mk5 -> 6x Neurochrome 286 mono amps -> Impulse H2 speakers
Vinyl: Thöress Phono Enhancer -> RME ADI-2 Pro

Link to comment
Share on other sites

Just another point...

 

I'm not at all convinced by the 'got to be double-blind, else biases may play a part' argument. I think we all develop along our audiophile paths. Sure, in the early days expectation bias may well play a part. But for my own part, the longer I've been in the pursuit of good sound the less I give a damn about how something 'should' sound from its looks and/or price. Most of my uber expensive stuff either sits in my basement or has been sold off.

It depends on what the aim of your testing is Mani (IMO)

 

If you are trying to decide which device suit you the most, then you are right being blind has no bearing. If you feel happier because of the look / cost (one way or the other) combined with the sound you hear then that is the best product for you (so long as you can afford it).

 

On the other hand if you are trying to be as objective as possible to come up with a consensus which is the better product in terms of sound on an absolute level, then you have to approach the situation "blind".

 

Eloise

Eloise

---

...in my opinion / experience...

While I agree "Everything may matter" working out what actually affects the sound is a trickier thing.

And I agree "Trust your ears" but equally don't allow them to fool you - trust them with a bit of skepticism.

keep your mind open... But mind your brain doesn't fall out.

Link to comment
Share on other sites

engine springtime irate whirlwind frightened rouse submissive beforehand basin patch commentate naught goblet kwashiorkor fresco gundog unsafe wake you'll exposure equally .

 

 

I think user vstoavvgswmvs brings up some really valid points..

Link to comment
Share on other sites

On the other hand if you are trying to be as objective as possible to come up with a consensus which is the better product in terms of sound on an absolute level, then you have to approach the situation "blind".

 

I understand what you're saying Eloise, but I'm not sure I agree. My system has totally transformed from that which I described over two years ago here http://www.computeraudiophile.com/f8-general-forum/show-us-your-equipment-kit-3220/index2.html. There is no question in my mind that my current setup is a quantum leap better than my previous setup. No blind listening involved at all.

 

Again, I think it has a lot to do with 'maturity' in one's development. You get to a point where the sound transcends all other factors. This isn't faked or forced... it just is. Interestingly, most of the 'audiophile crowd' I seem to 'hang out' with have had very similar experiences to me - mere 'audio jewellery' giving way to truly musical and interesting setups. These are not necessarily mutually exclusive of course, but too often are... IMO too!

 

Mani.

Main: Okto dac8PRO -> 6x Neurochrome 286 mono amps -> Tune Audio Anima speakers + 2x Rotel RB-1590 stereo amps -> 4x subs
Office: MOTU UltraLite-mk5 -> 6x Neurochrome 286 mono amps -> Impulse H2 speakers
Vinyl: Thöress Phono Enhancer -> RME ADI-2 Pro

Link to comment
Share on other sites

I think I fall somewhere in the middle on this. If I really don't like the sound of something, it is obvious within a minute or two. Rapid A/B switching is useful to weed out gear I do not even care to consider.

 

Long term listening is necessary after that point to train myself to hear what a piece of gear has to offer. This is the point that seems to horrify the crowd that labels themselves as "objectivists," but is totally rational. Human hearing is designed to rapidly detect threats in the environment, not rapidly discern subtle differences in sound from different equipment. Not at least, until the person involved has listened to the source for a significant amount of time.

 

Which means there is also, perhaps, some additional value in A/B testing after long term listening.

 

Paul

Anyone who considers protocol unimportant has never dealt with a cat DAC.

Robert A. Heinlein

Link to comment
Share on other sites

It depends on what the aim of your testing is Mani (IMO)

 

If you are trying to decide which device suit you the most, then you are right being blind has no bearing. If you feel happier because of the look / cost (one way or the other) combined with the sound you hear then that is the best product for you (so long as you can afford it).

 

On the other hand if you are trying to be as objective as possible to come up with a consensus which is the better product in terms of sound on an absolute level, then you have to approach the situation "blind".

 

Eloise

 

As mani mentioned, certainly for the purpose of immediately telling differences, blind AB testing is most objective.

 

On the other hand, determining what best approaches the ideal - equipment that transmits what it's being fed without adding or detracting - for me at least involves listening to many different recordings with the same piece of equipment and trying to hear any "sameness" of sound or presentation among them. The piece of equipment that allows these different recordings to sound most different from each other will be the one that best approaches the ideal of being faithful to the music and imparting the least sound of its own.

 

That sort of process requires some fairly long term listening for each piece of equipment just to get through the various different recordings. You're not likely to have the time to do this unless you own the equipment or have it on long-term loan. I'm not sure how you would set this up on an A/B basis with two different pieces of equipment. With the time and number of recordings involved, I think it might be a matter of instinctively feeling more content and comfortable with one piece than it would be any sort of exacting, quantitative comparison.

One never knows, do one? - Fats Waller

The fairest thing we can experience is the mysterious. It is the fundamental emotion which stands at the cradle of true art and true science. - Einstein

Computer, Audirvana -> optical to EtherREGEN -> microRendu -> ISO Regen -> Pro-Ject Pre Box S2 DAC -> Spectral DMC-12 & DMA-150 -> Vandersteen 3A Signature.

Link to comment
Share on other sites

Peter Aczel over at the Audio Critic writes pretty strongly around this and is a firm believer in the theory of it (not that I've hardly ever done it) and he is understandably hard on the "glossier" hifi journals which promote pieces of equipment using nice metaphors rather than direct comparisons with other items that do the same function. I used to buy those mags for their eye-candy but after reading Peter's material I haven't bought one. That's not to say there is a fabulous ABX testing source somewhere out there, and I still pick up those magazines on the shelf and thumb through them to see what's coming out next but the reviews are valueless to me. I'd rather look at sites like this to see amassed opinions plus the odd comparison and that's what has driven my last few purchases, along with a store test-out.

 

To a non-sound tech person like me, doing a real ABX test, if it were practical for a piece of equipment, would be a starting point, either with just human ears and/or some measuring equipment, despite all the intricacies of the latter. It's so easy for the something like the volume level to be up by a couple of decibels on a different piece of kit and some less-experienced audiophiles (me probably included) think it's an order of magnitude better than the predecessor.

 

I guess the other question I'd raise (and I'm sure someone else has) is that if the difference between two items under comparison is really that hard to pick up over time (when sound levels are equal :-) ), then surely the choice doesn't matter as one isn't really 'losing out' ? As people keep saying, it's what sounds good to you. Sorry I am certain this second point is covered elsewhere ad nauseam so I don't want to "flog a dead horse" if it is.

Synology DS211J NAS-->CAPS2-->Bifrost -->Audioengine A5+ actives

same-->Macbook Pro 2010-->Audirvana-->Audiinst HUD MX-1-->Grado 325i headphones

same-->wireless bridge-->Sonos-->MF VDAC1-->Yaqin MC100B-->Celestion A3

Link to comment
Share on other sites

In the end, it boils down to do I want the objectively best choice or the one that works best for me?

 

If I want the objective best choice, I'd need to do blind AB, anything else, as discussed extensively on this forum, will be biased (see McGurk etc.). I know some people would disagree, but on this one I'm personally convinced, having a scientific background.

 

But then comes reality. I tried the above with analog interconnects. I got about 10 from 3 different dealers (mostly at the same time), and then tried to find somebody willing enough to crouch behind my rack and change the cables without me knowing. My wife just called me crazy, and unlike the good old days in high school when I did this exercise last, I couldn't find any friend motivated enough for this exercise either (they didn't call me crazy, but maybe just not in my face).

 

So I ended up comparing them without blinding. Cannot remember how many times I switched (imagine the combination possibilities with 10 cables). In the end, the only thing I really achieved was a very solid polishing of the cinch plugs on DAC and amp. I ended up buying the King Cobra because 1) I somehow had the impression it sounded best, 2) I liked the looks and 3) it was from my main dealer who I really like.

 

On the DAC side, luckily the availability here in Switzerland was limited, so I only took home 3 DACs, and ended up buying in the end the one that comes from the same manufacturer as the amp, so I have consistent looks and a common remote. Co-incidentally I also really liked the sound. No blind testing involved here either, but a couple of A/B sessions.

 

So by now I've mentally given up on full blown A/B testing. I just try it in my chain for a while and see if I like the effect and whether the perceived improvement is worth it to me. This is what I plan in the near future for a USB/SPIDF and potentially a second power amp for bi-amping.

 

Does that assure I'll get the best objective value for money? Clearly no. But honestly, given the number of new products, especially e.g. in the DAC area, where there's a new one each week, can anybody even try to listen to all of them and draw reasonable conclusions? I think the answer is no. But at least I feel I have the right system for me, which is in the end what it is all about.

 

P.S. Unfortunately, reading this site doesn't really help, as you are continuously reminded that there are a gazillion alternatives out there.

Link to comment
Share on other sites

No one noticed this was my entry for necro-post of the year?

 

Sorry, I was listening to my right speaker only :)

 

Peter

“We are the Audiodrones. Lower your skepticism and surrender your wallets. We will add your cash and savings to our own. Your mindset will adapt to service us. Resistance is futile.” - (Quote from Star Trek: The Audiophile Generation)

Link to comment
Share on other sites

Peter Aczel over at the Audio Critic writes pretty strongly around this and is a firm believer in the theory of it (not that I've hardly ever done it) and he is understandably hard on the "glossier" hifi journals which promote pieces of equipment using nice metaphors rather than direct comparisons with other items that do the same function. I used to buy those mags for their eye-candy but after reading Peter's material I haven't bought one. That's not to say there is a fabulous ABX testing source somewhere out there, and I still pick up those magazines on the shelf and thumb through them to see what's coming out next but the reviews are valueless to me. I'd rather look at sites like this to see amassed opinions plus the odd comparison and that's what has driven my last few purchases, along with a store test-out.

 

To a non-sound tech person like me, doing a real ABX test, if it were practical for a piece of equipment, would be a starting point, either with just human ears and/or some measuring equipment, despite all the intricacies of the latter. It's so easy for the something like the volume level to be up by a couple of decibels on a different piece of kit and some less-experienced audiophiles (me probably included) think it's an order of magnitude better than the predecessor.

 

I guess the other question I'd raise (and I'm sure someone else has) is that if the difference between two items under comparison is really that hard to pick up over time (when sound levels are equal :-) ), then surely the choice doesn't matter as one isn't really 'losing out' ? As people keep saying, it's what sounds good to you. Sorry I am certain this second point is covered elsewhere ad nauseam so I don't want to "flog a dead horse" if it is.

 

I've been an audiophile for over 40 years and it is a fun hobby for me. I like listening to different pieces of equipment, finding new music to listen to and discussing my experiences with fellow audiophiles and music lovers. Audiophiles have never agreed on what sort of equipment is best - eg valves/tubes vs transistor amplifiers, horns vs bass reflex speakers, vinyl records vs reel to reel tape vs digital. But we enjoy having good natured discussions about our preferences.

 

I don't regard people like Peter Aczel or the Hydrogen Audio forum as fellow audiophiles. They do not have good natured discussions about the audio preferences with other audiophiles. I just looked up Peter Aczel's site and he describes the valve sound as a 'big lie'. Well I like listening to valves/tubes and I'm not a lier at all. When someone calls my a lier I would rather not have anything to do with them. I don't regard these guys as fellow audiophiles, they are 'anti audiophiles' for me and I don't want to have anything to do with them.

 

I have no problem with people discussing A/B tests or ABX tests in a good natured manner, as are most (but not all) discussions on this site. There are a few exceptions on Computer Audiophile where the 'objectivists' go over the top and start being rude with people who have honest disagreements with them. So A/B tests or DBX tests are fine with me as long as they aren't supposed to be the 'be all and end all' of audio judgements. For me these days when someone brings up the subject of DBX tests, it tends to be a 'dog whistle' comment about how much they don't like audiophiles such as myself having fun listening to single ended tube amps in an entirely unscientific manner (it's a hobby for heaven's sake!).

System (i): (Stack Audio Link/MoOde > 2Qute+MCRU psu; Gyrodec/SME V/Ortofon 2M Black/EAT E-Glo Petit/Magnum Dynalab FT101A) > Glow Amp One > Klipsch RP-600M

System (ii): iUSB3.0 Nano/Allo USB Signature/MoOde > Bel Canto uLink+AQVOX psu > Chord Hugo > (Tandy LX5; JBL LSR305 ; Audeze LCD-3)

Link to comment
Share on other sites

Good Necro-post wgscott. Apparently wasn't much interest in this topic nearly 4 years ago. And now it looks like it is a topic of interest.

 

First off, AB or ABX testing is about the least fun one can have as an audiophile. Hard to think of any type listening more tiring or less enjoyable. That doesn't mean it is of no value or without a place.

 

Secondly, AB and especially AB blind testing is about the most inconvenient thing you could do with your audio system. It is easier with say the ABX plug-in of Foobar or other software for some purposes. One can even record signals from other equipment and do it through Foobar. But other than basic files, swapping cables, or amps or source or anything, doing it blind, doing it with enough people or enough trials as an amateur audiophile is darned difficult and inconvenient. So I think it better left to genuine research by scientists or as group efforts of audio clubs. An alternative would be serious efforts by publications of one sort or another, but unfortunately that hasn't been a regular event anywhere. Probably would need to be an audio consumer reports type publication or website that had subscriber funding and took none from makers of this equipment. I doubt you will get much ad space if you show an expensive piece of gear is no better than cheaper gear. Nor would such makers have an incentive to loan gear and even take that risk regardless of the eventual outcome.

 

Now the typical alternative is swapping stuff and listening sighted. For quick comparisons that is okay.

 

Next is people who say long term listening is the alternative for finer discrimination. I agree to a point, but not too much. I think what happens as often as not is time for you to 'convince yourself'. And I don't really mean convince. I mean once you think you have a favorite you get more comfortable with the idea over time. This happens if there is a real difference of course. Something may really be better. I think it also happens when there is little to no difference as well. Which is why at some level of marginal audibility it isn't the gold standard. Please note, I am not saying it has no place or that it isn't viable for some purposes, just that I think there are even better ways.

 

Now in terms of making equipment choices everyone wants two things. The most discerning way to make the best choice, and one that makes them feel they have the best choice.

 

When you know nothing else, properly done ABX blind testing can determine if something is audibly different from something else. I also think it has (when done properly) nearly the best level of discrimination. There is simply too much evidence that near the margins of human perception, non-blind testing is swayed by too many factors and also has a strong tendency to indicate difference where none exists. This is true for all humans and for all of our senses.

 

Unfortunately, blind testing is probably the worst at making a person feel they have made the best choice. So of the two desires: discernment, and confidence in the person(s) it is maybe tops at the first and worst in the latter.

 

Long term listening is tops in inspiring confidence, but perhaps only middling in discernment. It is better than short sighted listening. Like play this amp for three songs and then play that one. The discernment is less (though not zero) and the confidence is less though better than in blind testing.

 

The other alternative is measurements. Done right I think it is the most discriminating as I believe in general measurements have exceeded our hearing ability. It suffers however in interpretation. Some easily measurable differences are of no consequence, and others are. Determining whether it is or not is the problem.

 

You can make some decisions at some level without listening. For instance differences in sound level that would be below not only the threshold of hearing, but below the brownian motion of the air can be dismissed as not audible. Some physical structure of the ear itself puts limits on what might be heard, and can guide measurements. Some is known about processing done on sound by our ear/brain that put some limits on it. In time with more knowledge of the brain and real-time scanning of it in action we might be able to determine what measurements at what level represent full transparency without any listening tests by humans whatsoever.

 

But for the present translating measurements into what is audible or not must also involve some listening. I think research done by psycho-acoustics already tells us plenty. And that double-blind testing is the most discriminating at finding those limits. Not all possibilities are fully fleshed out and that leads people to doubt such conclusions. Which means when long term listening leads them with confidence into believing a difference they don't believe either measurements or blind test results to the contrary.

 

So everyone wants high discernment and high confidence that they have made the right choices in comparing.

 

Long term listening: middling discernment with very high confidence.

 

Blind tests: high discernment with nearly no confidence in listeners.

 

Measurements: highest discernment (with problems of interpretation) and more confidence than blind testing, but still low confidence in listeners.

 

Finally, it seems everyone thinks they are special. That unlike these other easily biased humans they are very careful, very discerning and listen with unusual care that allows them to believe in their sighted listening despite protests they might not represent the highest discernment.

 

Think about it this way. How fast can a man run 100 meters? Well Usain Bolt manages it well under 10 seconds as do a few other people. That however wouldn't generally be the right answer. For one thing there is no absolutely correct answer for everyone. I don't know what the average would be, but I can assure you it is more than 10 seconds. When young you might get close to 10 seconds and when older further away. There isn't even a correct answer for anyone at all times. Mr. Bolt is maybe one in a million or less. In a distribution curve he is way up at one end away from everyone else. There could be someone even faster. We can confidently say there isn't anyone ready to do 5 seconds.

 

Hearing ability is like that. Someone somewhere has the most sensitive and discriminating hearing in the world. We don't know who it is. Some number among us have hearing considerably above average and we all lose some of that with age. It also varies over time from day to day. There is enough known that we can say some things are not heard by anyone. And we can set pretty good limits for the great majority of the people. Some things lots of people claim to hear are like saying they can easily do the 5 second 100 meter run. It isn't credible or likely. Others claims are like more than half the people thinking they surely can run just the 10 second 100 meters. Just not at all likely to be true. Yet, so often, so many audiophiles all want to believe they are the Usain Bolt of the audiophile world.

 

Now any of us can grab a stopwatch or get a friend to work one, mark off 100 meters and quickly determine just about how close to being Usain Bolt we are. Hearing and hearing ability is a bit trickier. I do think we as a group could put together some reasonably simple tests to do the equivalent of running the 100 meters and timing yourself. Even then people are just too desirous of saying the tests aren't testing the real thing with real music (even when it uses music) and would not believe the results. If the results indicated contradiction with their personal basic hearing ability and long term listening impressions, they are going to disbelieve the test.

 

Hearing isn't a competition (maybe an idea the Audiophile Olympics to see who hears the tiniest differences), and tests are not fun. People don't like failing them or even facing the risk of it in their hobbies. Nor is there any need. Until they make outlandish claims that others don't wish to accept. And there is the problem. Maybe someone will figure a way out of it one day.

And always keep in mind: Cognitive biases, like seeing optical illusions are a sign of a normally functioning brain. We all have them, it’s nothing to be ashamed about, but it is something that affects our objective evaluation of reality. 

Link to comment
Share on other sites

For me these days when someone brings up the subject of DBX tests, it tends to be a 'dog whistle' comment about how much they don't like audiophiles such as myself having fun listening to single ended tube amps in an entirely unscientific manner (it's a hobby for heaven's sake!).

 

Hi Richard,

 

I don't think this last remark is a fair one... The people who prefer an objective view with regard to audio in all its aspects, at times discuss and defend their objective point of view quite fiercely, but so far I have yet to see any evidence of "personal dislike of audiophiles such as yourself", at least here on CA.

 

On another note, this topic has been started by Chris C. himself, and I am absolutely sure he does not do any "dog whistling" here :)

 

As for tube amps... I know two Dutch brothers who have been in audio and DIY about as long as you are... Both come from a technical background. During the last 10 to 15 years the have been working together building their tube-amps and loudspeakers, and making fun of their efforts "Well, those tubies might not pass the Hi-Fi-norm, but they sure make the Fat Ladies sing!"

 

Regards,

Peter

“We are the Audiodrones. Lower your skepticism and surrender your wallets. We will add your cash and savings to our own. Your mindset will adapt to service us. Resistance is futile.” - (Quote from Star Trek: The Audiophile Generation)

Link to comment
Share on other sites

Well said. The only point I contend you might want to reconsider is discrimination over time. You learn to recognize what you are hearing.

 

Take a person raised in a big city, full of 24X7 traffic, people, electronic, and other sounds, and who has never been out of the city. Plop them down in at midsummer in the middle of a large forest in Northern California (with a guide and suitable survival gear) and for a week, they won't be able to discern most birds from bugs. But they will learn what they are hearing, and as they gain experience, they will discern more and more information from the sounds they re hearing.

 

Audio listening is exactly like that.

 

-Paul

 

 

 

Good Necro-post wgscott. Apparently wasn't much interest in this topic nearly 4 years ago. And now it looks like it is a topic of interest.

 

First off, AB or ABX testing is about the least fun one can have as an audiophile. Hard to think of any type listening more tiring or less enjoyable. That doesn't mean it is of no value or without a place.

 

Secondly, AB and especially AB blind testing is about the most inconvenient thing you could do with your audio system. It is easier with say the ABX plug-in of Foobar or other software for some purposes. One can even record signals from other equipment and do it through Foobar. But other than basic files, swapping cables, or amps or source or anything, doing it blind, doing it with enough people or enough trials as an amateur audiophile is darned difficult and inconvenient. So I think it better left to genuine research by scientists or as group efforts of audio clubs. An alternative would be serious efforts by publications of one sort or another, but unfortunately that hasn't been a regular event anywhere. Probably would need to be an audio consumer reports type publication or website that had subscriber funding and took none from makers of this equipment. I doubt you will get much ad space if you show an expensive piece of gear is no better than cheaper gear. Nor would such makers have an incentive to loan gear and even take that risk regardless of the eventual outcome.

 

Now the typical alternative is swapping stuff and listening sighted. For quick comparisons that is okay.

 

Next is people who say long term listening is the alternative for finer discrimination. I agree to a point, but not too much. I think what happens as often as not is time for you to 'convince yourself'. And I don't really mean convince. I mean once you think you have a favorite you get more comfortable with the idea over time. This happens if there is a real difference of course. Something may really be better. I think it also happens when there is little to no difference as well. Which is why at some level of marginal audibility it isn't the gold standard. Please note, I am not saying it has no place or that it isn't viable for some purposes, just that I think there are even better ways.

 

Now in terms of making equipment choices everyone wants two things. The most discerning way to make the best choice, and one that makes them feel they have the best choice.

 

When you know nothing else, properly done ABX blind testing can determine if something is audibly different from something else. I also think it has (when done properly) nearly the best level of discrimination. There is simply too much evidence that near the margins of human perception, non-blind testing is swayed by too many factors and also has a strong tendency to indicate difference where none exists. This is true for all humans and for all of our senses.

 

Unfortunately, blind testing is probably the worst at making a person feel they have made the best choice. So of the two desires: discernment, and confidence in the person(s) it is maybe tops at the first and worst in the latter.

 

Long term listening is tops in inspiring confidence, but perhaps only middling in discernment. It is better than short sighted listening. Like play this amp for three songs and then play that one. The discernment is less (though not zero) and the confidence is less though better than in blind testing.

 

The other alternative is measurements. Done right I think it is the most discriminating as I believe in general measurements have exceeded our hearing ability. It suffers however in interpretation. Some easily measurable differences are of no consequence, and others are. Determining whether it is or not is the problem.

 

You can make some decisions at some level without listening. For instance differences in sound level that would be below not only the threshold of hearing, but below the brownian motion of the air can be dismissed as not audible. Some physical structure of the ear itself puts limits on what might be heard, and can guide measurements. Some is known about processing done on sound by our ear/brain that put some limits on it. In time with more knowledge of the brain and real-time scanning of it in action we might be able to determine what measurements at what level represent full transparency without any listening tests by humans whatsoever.

 

But for the present translating measurements into what is audible or not must also involve some listening. I think research done by psycho-acoustics already tells us plenty. And that double-blind testing is the most discriminating at finding those limits. Not all possibilities are fully fleshed out and that leads people to doubt such conclusions. Which means when long term listening leads them with confidence into believing a difference they don't believe either measurements or blind test results to the contrary.

 

So everyone wants high discernment and high confidence that they have made the right choices in comparing.

 

Long term listening: middling discernment with very high confidence.

 

Blind tests: high discernment with nearly no confidence in listeners.

 

Measurements: highest discernment (with problems of interpretation) and more confidence than blind testing, but still low confidence in listeners.

 

Finally, it seems everyone thinks they are special. That unlike these other easily biased humans they are very careful, very discerning and listen with unusual care that allows them to believe in their sighted listening despite protests they might not represent the highest discernment.

 

Think about it this way. How fast can a man run 100 meters? Well Usain Bolt manages it well under 10 seconds as do a few other people. That however wouldn't generally be the right answer. For one thing there is no absolutely correct answer for everyone. I don't know what the average would be, but I can assure you it is more than 10 seconds. When young you might get close to 10 seconds and when older further away. There isn't even a correct answer for anyone at all times. Mr. Bolt is maybe one in a million or less. In a distribution curve he is way up at one end away from everyone else. There could be someone even faster. We can confidently say there isn't anyone ready to do 5 seconds.

 

Hearing ability is like that. Someone somewhere has the most sensitive and discriminating hearing in the world. We don't know who it is. Some number among us have hearing considerably above average and we all lose some of that with age. It also varies over time from day to day. There is enough known that we can say some things are not heard by anyone. And we can set pretty good limits for the great majority of the people. Some things lots of people claim to hear are like saying they can easily do the 5 second 100 meter run. It isn't credible or likely. Others claims are like more than half the people thinking they surely can run just the 10 second 100 meters. Just not at all likely to be true. Yet, so often, so many audiophiles all want to believe they are the Usain Bolt of the audiophile world.

 

Now any of us can grab a stopwatch or get a friend to work one, mark off 100 meters and quickly determine just about how close to being Usain Bolt we are. Hearing and hearing ability is a bit trickier. I do think we as a group could put together some reasonably simple tests to do the equivalent of running the 100 meters and timing yourself. Even then people are just too desirous of saying the tests aren't testing the real thing with real music (even when it uses music) and would not believe the results. If the results indicated contradiction with their personal basic hearing ability and long term listening impressions, they are going to disbelieve the test.

 

Hearing isn't a competition (maybe an idea the Audiophile Olympics to see who hears the tiniest differences), and tests are not fun. People don't like failing them or even facing the risk of it in their hobbies. Nor is there any need. Until they make outlandish claims that others don't wish to accept. And there is the problem. Maybe someone will figure a way out of it one day.

Anyone who considers protocol unimportant has never dealt with a cat DAC.

Robert A. Heinlein

Link to comment
Share on other sites

Well said. The only point I contend you might want to reconsider is discrimination over time. You learn to recognize what you are hearing.

 

Take a person raised in a big city, full of 24X7 traffic, people, electronic, and other sounds, and who has never been out of the city. Plop them down in at midsummer in the middle of a large forest in Northern California (with a guide and suitable survival gear) and for a week, they won't be able to discern most birds from bugs. But they will learn what they are hearing, and as they gain experience, they will discern more and more information from the sounds they re hearing.

 

Audio listening is exactly like that.

 

-Paul

 

One point about what you are saying Paul. Sure, the city person might hear bugs and birds the same until they gain experience. Lets say it takes them two weeks just pretending. Once they have learned the difference we can move them to another rural setting and they don't yet again need two weeks to discern bugs and birds. They will do it the first night.

 

Dennis

And always keep in mind: Cognitive biases, like seeing optical illusions are a sign of a normally functioning brain. We all have them, it’s nothing to be ashamed about, but it is something that affects our objective evaluation of reality. 

Link to comment
Share on other sites

One point about what you are saying Paul. Sure, the city person might hear bugs and birds the same until they gain experience. Lets say it takes them two weeks just pretending. Once they have learned the difference we can move them to another rural setting and they don't yet again need two weeks to discern bugs and birds. They will do it the first night.

 

Dennis

Oh yes, to a degree. But move them say, to the Northeast, someplace like Vermont say, and it will take them some little while to learn the new bug and bird sounds. Maybe a few hours, or maybe a day, or maybe a couple days. :)

 

-Paul

Anyone who considers protocol unimportant has never dealt with a cat DAC.

Robert A. Heinlein

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share



×
×
  • Create New...