AB Testing

[esldude]

Oh yes, to a degree. But move them say, to the Northeast, someplace like Vermont say, and it will take them some little while to learn the new bug and bird sounds. Maybe a few hours, or maybe a day, or maybe a couple days.

 

-Paul

 

Agreed Paul.

 

Now down south where I live, the bugs are the ones that might tote you away. The birds just make noise.

[bluesman]

Sorry I got to this party late. As many point out, A/B testing is hard to do well - "all other things being equal" is a tough requirement to meet. Subjective first impressions can be misleading, but at least for me they're usually right on the big issues. And, as also stated by others, the subtleties noticed in longer listening sessions and over time are usually just that - subtle and not deal breakers. But for most of us, sensory memory is not as accurate as we'd like, which is why "rediscovering" sources and equipment we abandoned for "better" ones is so revealing to me. I also notice that I seem to like things in proportion to the strength of my desire to like them.

 

I've used A/B testing in a different way that has been effective, though. I've recorded the same passages on the same instrument multiple times with changes like new strings on the guitar, different mouthpiece on the trumpet etc - and my "golden-eared" friends have never even considered the possibility of such change. Beyond what I disguised as shifts between preamps and amps, guesses over the years have ranged from changes in bias (the voltage, not the thought process) to crossover frequencies to equalization to interconnects to.......

 

When we want to hear a difference, we usually manage to do so. Truly controlled A/B testing would probably reduce this.

[thesurfingalien]

Hi Chris,

 

I assume we are talking about A/B testing in the form of (D)BT...

 

On some occasions I have given my "2 cents" with regards to this topic here on CA, but there is an additional comment I would like to make...

 

For ease of discussion I will stick to my all-time favorite: audio-cables

 

 

First of all, from experience I know that a fully controlled (D)BT is not easy to set up, and I do recognize some of the objections people have against them. However...

 

Cable manufacturers have been feeding us with rather woolly talk about "why" their top-of-the-line cables outperforms all others without presenting us with rock-solid proof this is indeed the case. Given the (sometimes?) exorbitant price levels, one (at least I) would expect this to be available.

 

What I am wondering here is why cable-manufacturers do not select (based on the ability to discern "introduced" differences) and hire a small group of people and train them to be comfortable in DBT listening sessions. My point here is that once people see doing a DBT as a job where there can be no pass or fail, all stress should be eliminated. The same thing goes for listening fatigue... By testing people first you can determine what, on average, the span of required attention is and adjust testing-time accordingly.

 

If the results of such listening-sessions were to be presented, it would provide adequate insight about performance. To be honest, even if I could afford spending US$ 29.000,00 on two 2.5 meters long cables without blinking my eyes, I would like to have at lease some of that rock-solid proof instead of just relying on my ears... But that, obviously, is just a personal view.

 

Unfortunately, I have yet to see any kind of information like this, and not only for cables...

 

Regards,

Peter

[esldude]

Why do cable companies not do such testing? I think the answer is the obvious one.

 

They do not stand to gain anything from it. At this point people believe in what they are selling and they are making good money. One company might do such a test and show superiority of their cable, but then there can only be one or some limited number of winners. Without doing this they don't have to risk their profitable business, and hope to be one of the winners.

 

Now it would be different if a company knew or thought they knew how superior their designs were and couldn't drum up any business. Then it would behoove them to prove a difference. That isn't the case currently.

 

There at least in the realm of interconnects seems to be no measurements that make you think they could be audibly different. Only in basic LCR parameters do they measure different and those can be controlled to be a non-issue in terms of having audible effects with the vast majority of consumer equipment.

 

There doesn't appear to be any DBT's showing IC's to sound different.

 

Only sighted listening results in people finding them to sound different.

 

Measurements say no difference, DBT's say no difference, and sighted listeners say big difference.

 

Either sighted (long term shall we say) listening is much more discerning than measurements and blind tests or sighted listening is reporting a difference that isn't actually there.

 

Interconnects are interesting in this regard because they seem to have the easiest job in audio when kept to a couple meters or less. Seem to be technically closest to perfection in all of audio. Yet just like all the other more difficult parts you have the schism of blind testing and measurement agreeing and sighted listening in opposition. Either sighted listening is right or not.

 

So there you have the age old division in audio at the consumer level. Sighted listeners may never agree with the other side(s).

 

Sighted listeners have repeatedly been shown to indicate differences when none exist near audibility margins. That makes them suspect at making the finest conclusions in my book. Perhaps blind tests will fail to get statistically significant results just a bit above the actual margins. But whether you do or don't perceive a difference the cut-off point seems much more repeatable. At best sighted listening is like using a measuring instrument with a +/- 20% margin of error. When you get close to the point the instrument (sighted listener in this case) can no longer measure the difference you have to include that +/- 20%. So things can be determined sighted, but the dividing lines are rather broad at best. Plus you are using an instrument that has a tendency to show something when there is nothing to show. Such an 'instrument' isn't useless, but is quite far from ideal.

[bdiament]

Hi Chris,

 

Hey guys what are your thoughts on AB listening tests. I have some opinions on it, but I will hold off until we get some other opinions this time.

 

Not sure if I'm misinterpreting the question as a lot of folks seem to equivocate "AB listening tests" with ABX or blind testing, etc. My take is to interpret the term as it is: comparing two items, be they hardware, software, recordings, etc.

 

I find AB tests have their positive and their negative aspects but overall, I find them more than a little bit useful. I like to do comparisons where "A" and "B" are the only variables, so if I'm comparing say, two D-A converters, I want to use a system with which I'm very familiar, in a familiar room, using recordings I know intimately.

 

I also use AB tests all the time in my mastering work. The overwhelming majority of recordings I receive for mastering are made using typical studio techniques (closely placed, multiple microphones, overdubs, mixdowns, control room monitoring that is, to put it mildly, not honest). Most of these need some help in the mastering, either to try and "undo" (to the extent possible) what has been done to them - such as being EQ'd to make things sound "right" on poor monitoring - or to provide some remediation for bad instrument-to-instrument balances in a mix, etc.

 

So, for example, when I apply EQ, I'll invariably AB the sound with and without the EQ. This, like other instances of AB'ing in my experience, is where the negatives *can* come into play. I say "can" because they won't *necessarily* enter the picture, IF the listener is experienced and keeps in mind an internal reference for where they want to "go". In the absence of this (or if there is a lack of listening experience), one risks what I call "getting lost in A-B Land".

 

I should add that while I do find AB'ing useful for some things, I find long term listening to ultimately be more informative. And for things like cables, AB'ing can be fruitless unless one significantly lengthens the A and the B. (In my experience, cables need time and will change sonics from when first installed to when they've played music for a while. Even fully burned-in cables will, to my ears, not sound the same right after they've been moved, so when I compare cables, it is more long term than a quick AB.)

 

Getting lost in A-B Land can manifest itself a number of ways, prime among these, falling into the trap of deeming one (A or B) to be the reference. Of course, if one deems A or B "perfect" that isn't a problem but in my experience, perfection is a goal more often than it is the actual reality. If a non-perfect (i.e., real world) A or B becomes the reference, the comparison can quickly lose meaning.

 

One example that comes to mind occurred a lot back in the days of Dolby - whether Dolby B for cassettes or Dolby A in the studio. Many folks will hear the accentuated treble of a Dolby encoded recording that has not been properly decoded. Fewer will be aware of other aspects of the sonics, such as changes in dynamics, temporal response, etc. For this example, let's just stay with the difference in treble. I knew many folks who preferred leaving the Dolby decoding off for Dolby encoded recordings because when they turned the decoding on and off, the correct version (decoded) sounded "dull" to them in comparison to the non-decoded, treble "enhanced" version. They got lost in A-B Land.

 

I've seen this in the mastering room too, at many sessions where the client attended. They'd hear me switch in the EQ and in those instances where the EQ was providing a boost, would almost always prefer it to the bypass of the EQ. The reverse would often occur when the EQ involved a cut. Many folks prefer "more" and after getting lost in the AB, don't seem to differentiate between "more" and "too much"; there is only "more" and "not more". (I think a similar thing occurs when folks audition certain DACs, which I call "detail enhancers". A bit of spurious harmonic distortion and some will hear more "detail", even if it is not contained in the recording.)

 

So much for the downside. For me, AB testing is a part of daily life. Same when I'm tuning my guitar: I listen to one string against another - that's AB'ing isn't it? I adjust tuning based on whether my ears tell me one is flatter or sharper than the other. Even when comparing that initial string to the sound of the tuning fork (a real reference in this case), I'm AB'ing the sound (the pitch) of the string vs. that of the tuning fork.

 

When I picked my guitar, I AB'd a number of the same model, first A vs. B until I preferred one, then the one I preferred against C, until I found the one I wanted to take home with me.

 

I also enjoy AB'ing different single-malt scotches on occasion - though experience says these tests must be short. Talk about getting lost in A-B Land. ;-}

 

Best regards,

Barry

Soundkeeper Recordings

Barry Diament Audio

[thesurfingalien]

Hi Chris,

 

Not sure if I'm misinterpreting the question as a lot of folks seem to equivocate "AB listening tests" with ABX or blind testing, etc. My take is to interpret the term as it is: comparing two items, be they hardware, software, recordings, etc.

 

Best regards,

Barry

 

Hi Barry,

 

I was not sure about what Chris intended to ask either, so I made my own interpretation of it and responded accordingly. On the other hand, it seems to be a bit of a strange question if Chris meant A/B testing the way you refer to it (in general) as I assume that, for example, most potential purchases are evaluated this way...

 

Regards,

Peter

[bdiament]

Hi Peter,

 

Hi Barry,

 

I was not sure about what Chris intended to ask either, so I made my own interpretation of it and responded accordingly. On the other hand, it seems to be a bit of a strange question if Chris meant A/B testing the way you refer to it (in general) as I assume that, for example, most potential purchases are evaluated this way...

 

Regards,

Peter

 

I don't know. I've made most of my purchases without AB'ing. Oh I compared cables after the larger components were chosen but things like speakers and digital converters (and microphones for my work) were not AB'd at all. I simply sat down and spent some time listening to them.

 

In a way, I suppose these were AB'd too but not to other gear -- these were compared to an internal reference.

 

I'm sure Chris will let us know how he meant the question.

 

Best regards,

Barry

Soundkeeper Recordings

Barry Diament Audio

[spdif-usb]

Everything audio can be steered. If you listen for high frequencies, you might not hear low frequencies. If you listen for low frequencies, you might not hear high frequencies. If you listen for different things, you will remember different things. If you listen to a recording of a human speech that contains the letter 's', even if this 's' has been completely masked out of the recording, you might still hear an 's'. This can happen to any normal person, if the speech makes much more logical sense with than without the 's'. It making logical sense is what causes the human brain to fill in the missing 's'; the brain sort of expects there to be an 's', and this is why you hear an 's'. This was demonstrated in a psychoacoustics experiment by Poppy Crum, at AES Workshop 2009. The mp3 codec is another fine example of how it's possible to omit information from an audio signal without people really noticing, all because of the simple fact their brain fills in this information for them.

 

So, let's assume blind testing removes all forms of important bias. Let's just A/B compare two different stereo systems, using a music recording that contains two guitars. The test subject has never listened to this specific musical performance before, and therefore doesn't know how many guitars are there. After having listened to the same recording on both stereo systems, A first, B second, the test subject reports stereo system B is more revealing than system A because system A made the two guitars sound as if they're only one guitar instead of two and, afterwards, system B very clearly revealed the fact there's actually two guitars instead of only one. So far, obviously, nothing extraordinary has ocurred.

 

However, the test subject now knows that there's two guitars. So, next, if the test subject switches back to system A, the test subject's brain fills in the missing information, causing the test subject to still hear two guitars. System A didn't reveal that there's two guitars, system B did. This can be called objective evidence of there being an audible difference. However, the fact the test subject now knows what the music is supposed to sound like is what's causing the test subject to no longer hear this specific difference. A new form of bias has now been added as a direct result from knowledge of what the music previously sounded like. This is called cognitive bias.

 

There is absolutely no reason to assume expectation bias is necessarily always a stronger form of bias than cognitive bias. Had the test subject listened to system B first, A second (instead of A first, B second), the test result might, and probably would, have ended up being completely different. Which leads me to conclude that, if there's only one test subject involved, A/B testing is very often no more meaningful than sighted listening, except maybe if said test subject has trained hearing as well as knows how to conduct a listening test properly (and that's still only a maybe). The vast majority of people fail to even begin to understand why, so they naturally resort to making the wildest assumptions about everything. It's the I-understand-perfectly-how-all-this-works syndrome.

 

If differences exist, but can't be heard when blinded, the reason why they can't is not necessarily always because they're too small. During a blinded listening test, one is obviously aware of the fact there will be questions asked on whether differences could or could not be heard. This awareness naturally forces people to listen for differences rather than for pleasure. Like I said, if people listen for different things, they will remember different things (and, last time I checked, differences and pleasure were different things). Again, this shows how perfectly easy it really is for bias to creep in through the back door when people aren't trained to critically listen.

[mav52]

Good Necro-post wgscott. Apparently wasn't much interest in this topic nearly 4 years ago. And now it looks like it is a topic of interest.

 

First off, AB or ABX testing is about the least fun one can have as an audiophile. Hard to think of any type listening more tiring or less enjoyable. That doesn't mean it is of no value or without a place.

 

Secondly, AB and especially AB blind testing is about the most inconvenient thing you could do with your audio system. It is easier with say the ABX plug-in of Foobar or other software for some purposes. One can even record signals from other equipment and do it through Foobar. But other than basic files, swapping cables, or amps or source or anything, doing it blind, doing it with enough people or enough trials as an amateur audiophile is darned difficult and inconvenient. So I think it better left to genuine research by scientists or as group efforts of audio clubs. An alternative would be serious efforts by publications of one sort or another, but unfortunately that hasn't been a regular event anywhere. Probably would need to be an audio consumer reports type publication or website that had subscriber funding and took none from makers of this equipment. I doubt you will get much ad space if you show an expensive piece of gear is no better than cheaper gear. Nor would such makers have an incentive to loan gear and even take that risk regardless of the eventual outcome.

 

Now the typical alternative is swapping stuff and listening sighted. For quick comparisons that is okay.

 

Next is people who say long term listening is the alternative for finer discrimination. I agree to a point, but not too much. I think what happens as often as not is time for you to 'convince yourself'. And I don't really mean convince. I mean once you think you have a favorite you get more comfortable with the idea over time. This happens if there is a real difference of course. Something may really be better. I think it also happens when there is little to no difference as well. Which is why at some level of marginal audibility it isn't the gold standard. Please note, I am not saying it has no place or that it isn't viable for some purposes, just that I think there are even better ways.

 

Now in terms of making equipment choices everyone wants two things. The most discerning way to make the best choice, and one that makes them feel they have the best choice.

 

When you know nothing else, properly done ABX blind testing can determine if something is audibly different from something else. I also think it has (when done properly) nearly the best level of discrimination. There is simply too much evidence that near the margins of human perception, non-blind testing is swayed by too many factors and also has a strong tendency to indicate difference where none exists. This is true for all humans and for all of our senses.

 

Unfortunately, blind testing is probably the worst at making a person feel they have made the best choice. So of the two desires: discernment, and confidence in the person(s) it is maybe tops at the first and worst in the latter.

 

Long term listening is tops in inspiring confidence, but perhaps only middling in discernment. It is better than short sighted listening. Like play this amp for three songs and then play that one. The discernment is less (though not zero) and the confidence is less though better than in blind testing.

 

The other alternative is measurements. Done right I think it is the most discriminating as I believe in general measurements have exceeded our hearing ability. It suffers however in interpretation. Some easily measurable differences are of no consequence, and others are. Determining whether it is or not is the problem.

 

You can make some decisions at some level without listening. For instance differences in sound level that would be below not only the threshold of hearing, but below the brownian motion of the air can be dismissed as not audible. Some physical structure of the ear itself puts limits on what might be heard, and can guide measurements. Some is known about processing done on sound by our ear/brain that put some limits on it. In time with more knowledge of the brain and real-time scanning of it in action we might be able to determine what measurements at what level represent full transparency without any listening tests by humans whatsoever.

 

But for the present translating measurements into what is audible or not must also involve some listening. I think research done by psycho-acoustics already tells us plenty. And that double-blind testing is the most discriminating at finding those limits. Not all possibilities are fully fleshed out and that leads people to doubt such conclusions. Which means when long term listening leads them with confidence into believing a difference they don't believe either measurements or blind test results to the contrary.

 

So everyone wants high discernment and high confidence that they have made the right choices in comparing.

 

Long term listening: middling discernment with very high confidence.

 

Blind tests: high discernment with nearly no confidence in listeners.

 

Measurements: highest discernment (with problems of interpretation) and more confidence than blind testing, but still low confidence in listeners.

 

Finally, it seems everyone thinks they are special. That unlike these other easily biased humans they are very careful, very discerning and listen with unusual care that allows them to believe in their sighted listening despite protests they might not represent the highest discernment.

 

Think about it this way. How fast can a man run 100 meters? Well Usain Bolt manages it well under 10 seconds as do a few other people. That however wouldn't generally be the right answer. For one thing there is no absolutely correct answer for everyone. I don't know what the average would be, but I can assure you it is more than 10 seconds. When young you might get close to 10 seconds and when older further away. There isn't even a correct answer for anyone at all times. Mr. Bolt is maybe one in a million or less. In a distribution curve he is way up at one end away from everyone else. There could be someone even faster. We can confidently say there isn't anyone ready to do 5 seconds.

 

Hearing ability is like that. Someone somewhere has the most sensitive and discriminating hearing in the world. We don't know who it is. Some number among us have hearing considerably above average and we all lose some of that with age. It also varies over time from day to day. There is enough known that we can say some things are not heard by anyone. And we can set pretty good limits for the great majority of the people. Some things lots of people claim to hear are like saying they can easily do the 5 second 100 meter run. It isn't credible or likely. Others claims are like more than half the people thinking they surely can run just the 10 second 100 meters. Just not at all likely to be true. Yet, so often, so many audiophiles all want to believe they are the Usain Bolt of the audiophile world.

 

Now any of us can grab a stopwatch or get a friend to work one, mark off 100 meters and quickly determine just about how close to being Usain Bolt we are. Hearing and hearing ability is a bit trickier. I do think we as a group could put together some reasonably simple tests to do the equivalent of running the 100 meters and timing yourself. Even then people are just too desirous of saying the tests aren't testing the real thing with real music (even when it uses music) and would not believe the results. If the results indicated contradiction with their personal basic hearing ability and long term listening impressions, they are going to disbelieve the test.

 

Hearing isn't a competition (maybe an idea the Audiophile Olympics to see who hears the tiniest differences), and tests are not fun. People don't like failing them or even facing the risk of it in their hobbies. Nor is there any need. Until they make outlandish claims that others don't wish to accept. And there is the problem. Maybe someone will figure a way out of it one day.

 

Agree 100% interesting observation

[spdif-usb]

Oh and, I almost forgot... When we start to listen to music, we expect to hear music. Because we know alot of things about what music sounds like, choosing music as the audible content in A/B testing is what inevitably causes bias. I think it's safe to asssume that, if the experiment that was conducted by Poppy Crum works with a recorded voice, it'll also work with recorded music. So, A/B testing can be reliable to prove an audible difference exists, but not if the audible difference exists in music. Therefore, A/B testing can be used to compare two stereo systems, but not if music is being played back on them. Unfortunately however, most stereo systems have been designed to sound good only with music.

 

I am sick and tired of pseudoscientific interpretations of what it is A/B testing can and can't do, and I am sick and tired of these interpretations being abused to discredit audiophiles. Like any other artform, the importance of music is in its subtle details. Since A/B testing has a strong tendency to obscures these subtle details, I hate it with a passion when it comes to music listening and gear comparison. In the musical part of the universe, there is way too much mystification going on mostly because of how A/B land forces its shadows. It's like using a big sharp lawnmower to try and prove lawn A has prettier flowers than lawn B, by studying the riced and semi-decomposed remnants.

[esldude]

Everything audio can be steered. If you listen for high frequencies, you might not hear low frequencies. If you listen for low frequencies, you might not hear high frequencies. If you listen for different things, you will remember different things. If you listen to a recording of a human speech that contains the letter 's', even if this 's' has been completely masked out of the recording, you might still hear an 's'. This can happen to any normal person, if the speech makes much more logical sense with than without the 's'. It making logical sense is what causes the human brain to fill in the missing 's'; the brain sort of expects there to be an 's', and this is why you hear an 's'. This was demonstrated in a psychoacoustics experiment by Poppy Crum, at AES Workshop 2009. The mp3 codec is another fine example of how it's possible to omit information from an audio signal without people really noticing, all because of the simple fact their brain fills in this information for them.

 

So, let's assume blind testing removes all forms of important bias. Let's just A/B compare two different stereo systems, using a music recording that contains two guitars. The test subject has never listened to this specific musical performance before, and therefore doesn't know how many guitars are there. After having listened to the same recording on both stereo systems, A first, B second, the test subject reports stereo system B is more revealing than system A because system A made the two guitars sound as if they're only one guitar instead of two and, afterwards, system B very clearly revealed the fact there's actually two guitars instead of only one. So far, obviously, nothing extraordinary has ocurred.

 

However, the test subject now knows that there's two guitars. So, next, if the test subject switches back to system A, the test subject's brain fills in the missing information, causing the test subject to still hear two guitars. System A didn't reveal that there's two guitars, system B did. This can be called objective evidence of there being an audible difference. However, the fact the test subject now knows what the music is supposed to sound like is what's causing the test subject to no longer hear this specific difference. A new form of bias has now been added as a direct result from knowledge of what the music previously sounded like. This is called cognitive bias.

 

There is absolutely no reason to assume expectation bias is necessarily always a stronger form of bias than cognitive bias. Had the test subject listened to system B first, A second (instead of A first, B second), the test result might, and probably would, have ended up being completely different. Which leads me to conclude that, if there's only one test subject involved, A/B testing is very often no more meaningful than sighted listening, except maybe if said test subject has trained hearing as well as knows how to conduct a listening test properly (and that's still only a maybe). The vast majority of people fail to even begin to understand why, so they naturally resort to making the wildest assumptions about everything. It's the I-understand-perfectly-how-all-this-works syndrome.

 

If differences exist, but can't be heard when blinded, the reason why they can't is not necessarily always because they're too small. During a blinded listening test, one is obviously aware of the fact there will be questions asked on whether differences could or could not be heard. This awareness naturally forces people to listen for differences rather than for pleasure. Like I said, if people listen for different things, they will remember different things (and, last time I checked, differences and pleasure were different things). Again, this shows how perfectly easy it really is for bias to creep in through the back door when people aren't trained to critically listen.

 

The mp3 codec is another fine example of how it's possible to omit information from an audio signal without people really noticing, all because of the simple fact their brain fills in this information for them.

 

Actually this is not correct. The brain isn't filling in the missing info on an MP3. That info was masked and is info that would not have been perceived by the ear/brain anyway, so it being removed isn't noticed.

 

However, the test subject now knows that there's two guitars. So, next, if the test subject switches back to system A, the test subject's brain fills in the missing information, causing the test subject to still hear two guitars.

 

Well now this depends. Was system A defective and didn't reproduce with enough fidelity that one could hear two rather than one guitar? If so, and if defective enough, even after knowing there is two guitars a person might switch back and say they still couldn't pick up but one. Granted if it is close, if it is marginal one might hear two on system A after hearing it on system B. It will depend on how different they are. If A lacks enough fidelity, even with the knowledge there are two guitars you might not hear two. Not always would a person's brain fill in to that extent.

 

If differences exist, but can't be heard when blinded, the reason why they can't is not necessarily always because they're too small.

 

Not sure I even agree with this statement. But if I did, I don't see that it follows with sighted pleasure listening being more discerning. Listening for pleasure according to your thinking one listens for different things. Still for the experience to be different due to a difference in system performance what is heard must be different unless the differences in the listening experience lie elsewhere. And though noticing different things because of listening for pleasure and not just for difference one is open to many more potentially biasing factors than when blind. It doesn't seem likely to be more discerning or reliable even if what is discerned with pleasure listening is something unlike what is discerned with blind listening for differences.

 

There is absolutely no reason to assume expectation bias is necessarily always a stronger form of bias than cognitive bias.

 

We do know that cognitive bias is much more likely with sighted listening whether for pleasure or sighted listening for differences. And expectation bias is also more likely sighted. Your constructed complaints actually appear to reinforce the idea sighted listening is more easily biased.

[esldude]

Oh and, I almost forgot... When we start to listen to music, we expect to hear music. Because we know alot of things about what music sounds like, choosing music as the audible content in A/B testing is what inevitably causes bias. I think it's safe to asssume that, if the experiment that was conducted by Poppy Crum works with a recorded voice, it'll also work with recorded music. So, A/B testing can be reliable to prove an audible difference exists, but not if the audible difference exists in music. Therefore, A/B testing can be used to compare two stereo systems, but not if music is being played back on them. Unfortunately however, most stereo systems have been designed to sound good only with music.

 

I am sick and tired of pseudoscientific interpretations of what it is A/B testing can and can't do, and I am sick and tired of these interpretations being abused to discredit audiophiles. Like any other artform, the importance of music is in its subtle details. Since A/B testing has a strong tendency to obscures these subtle details, I hate it with a passion when it comes to music listening and gear comparison. In the musical part of the universe, there is way too much mystification going on mostly because of how A/B land forces its shadows. It's like using a big sharp lawnmower to try and prove lawn A has prettier flowers than lawn B, by studying the riced and semi-decomposed remnants.

 

An oft heard complaint is ABX testing often doesn't use music. Now I reading it is wrong if it uses music?

 

Your entire first paragraph is a painfully contorted bit of 'logic'. You end with the idea AB testing can be used to compare systems, but not if it music to be played back for testing. Lots left out in the middle, and you cannot get there I do not believe. For one thing there are tests using music that do show positive results. So obviously you can use music and get results indicating differences.

 

And complaining most audio systems have been designed only to sound good with music would really only apply to systems with less than good fidelity that have compromises in places that let music still sound enjoyable. Some systems have been put together and designed simply to be accurate. Accurate enough to be enjoyable with musical recordings. Also capable of accurately replaying speech and other sounds.

 

So your contention AB testing obscures fine details doesn't seem well founded or supported except by the fact such results contradict your sighted listening. We know sighted listening has some discrimination issues at the margins. You seem to be working up counter problems for blind listening without much to back up the idea other than the fact such an assumption resolves contradictions between sighted and blind listening in favor of the outcome you prefer. That isn't evidence, it is wishful thinking.

[spdif-usb]

Actually this is not correct. The brain isn't filling in the missing info on an MP3. That info was masked and is info that would not have been perceived by the ear/brain anyway, so it being removed isn't noticed.

The reason why it would not have been perceived by the ear/brain anyway is often the fact the brain fills in the missing information anyway.

Not always would a person's brain fill in to that extent.

My point exactly. Not always would it be possible to prove a person's brain does or does not fill in to that extent and, as a result, bias can creep in to an extent where blind testing becomes flawed.

Listening for pleasure according to your thinking one listens for different things.

Not just according to my thinking, but according to every sane neuroscientist on the planet. I find that listening for pleasure magnifies differences, whereas listening for differences obscures them because listening for differences causes me to not enjoy the music much, thereby effectively keeping me from discerning differences in my emotional response to the music (musical satisfaction is the reason why I listen to music, and when something's blocking it, I can't objectively compare it).

We do know that cognitive bias is much more likely with sighted listening whether for pleasure or sighted listening for differences.

The irony is in the fact you cannot also objectively prove it because... well, please tell me... how exactly does one objectively compare things like pleasure? :grin:

[spdif-usb]

An oft heard complaint is ABX testing often doesn't use music. Now I reading it is wrong if it uses music?

Yes, it's odd, isn't it? :grin:

For one thing there are tests using music that do show positive results. So obviously you can use music and get results indicating differences.

Yes, of course. However, there also are tests using music that do not show positive results. My point was about the logical reason why there are tests that don't. Somehow, you seem to entirely have missed that.

 

You immediately start off saying you understand perfectly how human bias affects not just human hearing, but human emotional response to music as well, even. This IMO clearly shows your arguments are based on advanced pseudoscience rather than auditory neuroscience.

 

And complaining most audio systems have been designed only to sound good with music would really only apply to systems with less than good fidelity that have compromises in places that let music still sound enjoyable.

Not a complaint, more like an objective observation. :grin:

Some systems have been put together and designed simply to be accurate.

Some are, most aren't. Those are my words.

So your contention AB testing obscures fine details doesn't seem well founded or supported except by the fact such results contradict your sighted listening. We know sighted listening has some discrimination issues at the margins. You seem to be working up counter problems for blind listening without much to back up the idea other than the fact such an assumption resolves contradictions between sighted and blind listening in favor of the outcome you prefer. That isn't evidence, it is wishful thinking.

In fact, I'm using the same kind of wishful thinking as you. You're saying blind testing is the more objective testing method when it comes to accuracy in pure terms of musical satisfaction; I call BS on that because the pot is calling the kettle black.

[esldude]

Nope, you still have it wrong about how MP3 encoding works. It works mainly on masking, not on the listener's brain filling in missing information. Here is a more detailed account if you wish:

 

MP3: The Definitive Guide: Sample Chapter 2

 

or here:

 

Perceptual Coding: How Mp3 Compression Works

 

Or any number of other sources.

 

They will explain that MP3 is based upon psycho-acoustical models taking advantage of masking. They determine what sounds in the original signal would not be heard anyway. So throwing that information away will not be noticed as it wouldn't be heard.

 

It is not predicated upon the idea that information that could be heard was left out and will be filled in by the brain. That simply is not the idea behind it nor how it works.

 

As for objectivity in pleasure, I did not say I was comparing pleasure. I writing about the amount of bias whether the listening is done for pleasure or otherwise. Two different things.

[chg]

Nope, you still have it wrong about how MP3 encoding works. It works mainly on masking, not on the listener's brain filling in missing information. Here is a more detailed account if you wish:

 

MP3: The Definitive Guide: Sample Chapter 2

 

or here:

 

Perceptual Coding: How Mp3 Compression Works

 

Or any number of other sources.

 

They will explain that MP3 is based upon psycho-acoustical models taking advantage of masking. They determine what sounds in the original signal would not be heard anyway. So throwing that information away will not be noticed as it wouldn't be heard.

 

It is not predicated upon the idea that information that could be heard was left out and will be filled in by the brain. That simply is not the idea behind it nor how it works.

 

As for objectivity in pleasure, I did not say I was comparing pleasure. I writing about the amount of bias whether the listening is done for pleasure or otherwise. Two different things.

 

I am now 100% convinced that you really do absolutely nothing else with your days other than spread your objectivist, save others from themselves agenda. Sad really. Any post with even a hint of subjective discussion you feel the need to jump in and repeat yourself over and over. You will never stop! Well, hopefully, eventually others will get tired of it and some day you'll get banned. You really add nothing positive to CA.

[Paul R]

Then why, if MP3's only throw away sounds that "would not be heard anyway" are they so easy to pick out as having poorer quality sound than say, an AIFF file?

 

Obviously, there is something wrong in your logic, and in the idea MP3s sound exactly the same as the files they originate from.

 

-Paul

 

 

Nope, you still have it wrong about how MP3 encoding works. It works mainly on masking, not on the listener's brain filling in missing information. Here is a more detailed account if you wish:

 

MP3: The Definitive Guide: Sample Chapter 2

 

or here:

 

Perceptual Coding: How Mp3 Compression Works

 

Or any number of other sources.

 

They will explain that MP3 is based upon psycho-acoustical models taking advantage of masking. They determine what sounds in the original signal would not be heard anyway. So throwing that information away will not be noticed as it wouldn't be heard.

 

It is not predicated upon the idea that information that could be heard was left out and will be filled in by the brain. That simply is not the idea behind it nor how it works.

 

As for objectivity in pleasure, I did not say I was comparing pleasure. I writing about the amount of bias whether the listening is done for pleasure or otherwise. Two different things.

[esldude]

I am now 100% convinced that you really do absolutely nothing else with your days other than spread your objectivist, save others from themselves agenda. Sad really. Any post with even a hint of subjective discussion you feel the need to jump in and repeat yourself over and over. You will never stop! Well, hopefully, eventually others will get tired of it and some day you'll get banned. You really add nothing positive to CA.

 

I suppose my contributions don't equal mis-representing how MP3 encoding works. Sorry for that.

 

I suppose the one who needs to get tired for me to get banned is Chris. Why don't you take it up with him? Really is a shame that one calls for a ban for presented ideas with some evidence beyond emotional feelings behind them. And so it goes.

[Paul R]

I suppose my contributions don't equal mis-representing how MP3 encoding works. Sorry for that.

 

I suppose the one who needs to get tired for me to get banned is Chris. Why don't you take it up with him? Really is a shame that one calls for a ban for presented ideas with some evidence beyond emotional feelings behind them. And so it goes.

 

Ah, don't be silly. But you do come off as awfully arrogant sometimes. If you insist on doing that, you are going to continue making yourself a target.

 

-Paul

[esldude]

Then why, if MP3's only throw away sounds that "would not be heard anyway" are they so easy to pick out as having poorer quality sound than say, an AIFF file?

 

Obviously, there is something wrong in your logic, and in the idea MP3s sound exactly the same as the files they originate from.

 

-Paul

 

My logic is not in error here. Yours is. You are presuming the MP3's were intended to sound identical and indistinguishable from the original file. In the early days at least, it was meant to get very close, and be a trade off in file size and fidelity. That is why they can be picked out. Even the people who developed the MP3 encoding didn't think at lower bit rates it was transparent. They wanted it efficient to allow it to be transparent enough.

 

They throw away sounds that would not be heard according to the model and adjust what might be heard based upon available bits to allocate. In any case, they don't work by taking advantage of what your brain fills in. If they did I suppose would could go the other way and ask, why doesn't the brain fully in the info? But there really is no point, the supposition about how MP3 works was incorrect. And it really is as simple as that.

[The Computer Audiophile]

I'm sure Chris will let us know how he meant the question.

 

Actually I have no idea what I was thinking because I started the thread about 65 days after starting the site nearly five years ago :~)

[esldude]

Ah, don't be silly. But you do come off as awfully arrogant sometimes. If you insist on doing that, you are going to continue making yourself a target.

 

-Paul

 

 

As arrogant as posting what is simply incorrect information on MP3's and continuing to insist on it being so?

 

Hey no one likes being told they are wrong, even when they are. This stuff about MP3's and how they are intended to work was wrong. Simply wrong. I am not apologizing for pointing that out. I guess that seems arrogant. Not sure what to do about that other than say nothing and let it go.

[sandyk]

They will explain that MP3 is based upon psycho-acoustical models taking advantage of masking. They determine what sounds in the original signal would not be heard anyway. So throwing that information away will not be noticed as it wouldn't be heard

Hi Dennis

As is so often the case, the "Experts" simply got it wrong !!!

I would be happy to UL for you a 320KBS MP3 and a copy of the original CD track from a "Yello" album. (Personal Use ONLY)

I have already done that a while back for a sceptic from another forum (also a member here) who found it hard to believe what was missing in the MP3 version. The differences aren't suble in the least compared with a track we both listened to a while back. Of course, if you are now using a Class D amplifier to drive those nice ESL speakers, then again, you may not hear the differences .(stir, stir and a big grin)

Kind Regards

Alex

[chg]

Hey no one likes being told they are wrong, even when they are.

 

How ironic...

[esldude]

Hi Dennis

As is so often the case, the "Experts" simply got it wrong !!!

I would be happy to UL for you a 320KBS MP3 and a copy of the original CD track from a "Yello" album. (Personal Use ONLY)

I have already done that a while back for a sceptic from another forum (also a member here) who found it hard to believe what was missing in the MP3 version. The differences aren't suble in the least compared with a track we both listened to a while back. Of course, if you are now using a Class D amplifier to drive those nice ESL speakers, then again, you may not hear the differences .(stir, stir and a big grin)

Kind Regards

Alex

 

Alex, and everyone, hold on. I said MP3's work by throwing away what we wouldn't hear according to their model. And this how they work. They didn't allocate enough bits to be fully transparent. But that was the basis and presumably with enough bits you won't miss what is thrown away. Surely a modern encoding for MP3 with 128 kbps is pretty amazing as it throw away like 90% of the bits and still sounds pretty recognizable. Now it isn't perfect, you can ABX and get positive results with those. Up around twice that it gets tougher than you might imagine if you haven't done it. But still not perfect though you are throwing away something like 80% of the bits. Pretty good proof their model has some validity though again even the creators wouldn't call it perfect.