Jump to content
IGNORED

Was Meyer and Moran debunked by Robert Stuart?


esldude

Recommended Posts

What is below is from a thread George Graves started a few weeks ago. It became a mess even though no one had access to the paper at that time. So I am starting this again by quoting George's first post from the other thread. Hoping it can stay a bit cleaner now that the AES paper is out.

 

 

I recently found out that the oft-cited and notorious Meyer and Moran paper on high-res audibility was debunked in a new paper delivered at this month's 137th AES (Audio Engineering Society) Convention in LA by no less an authority on digital audio than Robert J. Stuart of Meridian. I have been unable to find the paper posted online, but I do have this capsule description that was posted by Mark Waldrep, the head of AIX Records, who attended:

 

"I finished the afternoon by attending a few paper sessions. The first was titled, “The Audibility of Typical Digital Audio Filters in a High-Fidelity Playback System”. Although it may not be obvious from the title of the paper, this is the first AES publication that refutes the Meyer/Moran research that has been so often quoted as “proof” that CD specification PCM audio is enough for music reproduction (Meyer and Moran’s research has been widely discredited including by myself because of the lack of real high-resolution content used during the study).

Robert Stuart and his colleagues conclude: “first there exist audible signals that cannot be encoded transparently by a standard CD; and second, an audio chain used for such experiments must be capable of high-fidelity reproduction.” In other words, CDs aren’t good enough. This paper was given the top award by the AES organization. This is a very important finding."

 

If anyone here has access to this paper, or more information on it, please post it as this is critical to a number of strongly held beliefs. Chief among which, is that hi-resolution audio is a high-end marketing gimmick that adds no "real" value to recorded music, and also that there is no difference between DACs, and that one can encode/decode digital audio multiple times with no perceptible change in SQ.

And always keep in mind: Cognitive biases, like seeing optical illusions are a sign of a normally functioning brain. We all have them, it’s nothing to be ashamed about, but it is something that affects our objective evaluation of reality. 

Link to comment

I do not have the paper myself. But have seen some excerpts from it. There are a few curiosities that will get attention.

 

The signature headline results are trained listeners did 160 trials getting 56% correct which is right at the level for 95% confidence when comparing high sample rate 24 bit audio to low pass digitally filtered 16 bit audio.

 

The first curious thing is the digital low pass filter was said to be like that used in mastering and playback equipment. But there were two used. One was 22,050 hz brick wall and one was a 24 khz brickwall. What is curious is they used merely 500 hz wide transition bands between the cutoff and the flat response. Typically you would see 2050 hz and 4000 hz transition bands in consumer playback gear. Some are even a bit less steep than that with a roll off starting a little sooner than 20 khz and/or not reaching full stopband levels at the half sample rate point of 22.05 khz or 24 khz respectively.

 

Another curious thing is the low passed 16 bit content was dithered with rectangular dither or not dithered at all. They also used a signal consisting of wide dynamic range of approximately 16 bits. Better choices for dither would have been triangular or shaped.

 

Yet another curious result reported by those who have the paper is when they low passed the content without bringing it to 16 bits the results were chance. That would lead me to conclude the 16 bits is what was audible more so than the filter. But without the paper I don't know how they responded to that.

 

It would make sense to me to use filters steeper than normally found to see if the result is audible. This they did. My very next thought would be to use filters of different steepness to see how steep is too steep. Or in this case my next test would have used those wider filters like normally seen to see if this was detected. This step they skipped going right ahead with notion we need more bandwidth for less steep filters. One of the statements quoted from the paper says in effect, it is now accepted that high sample rates don't sound different than low rates due to signal above 20khz, but due to time domain performance of the filtering. Seems like a leap to that conclusion. Especially as it seems there was no effect with even the steep filtering at 24 bit.

 

Of course I have not seen the whole paper. Perhaps someone here has it.

And always keep in mind: Cognitive biases, like seeing optical illusions are a sign of a normally functioning brain. We all have them, it’s nothing to be ashamed about, but it is something that affects our objective evaluation of reality. 

Link to comment

Hi 'dude.

 

Have you seen this: Audibility of Typical Digital Filters in a Hi-Fi Playback System - Hydrogenaudio Forums

 

And maybe this: Conclusive "Proof" that higher resolution audio sounds different

 

The second thread is the background to an issue that is related to some of the problems experienced here. Over on 'whatsbestforum.com', Amirm, the proprietor, had a longstanding claim that he can clearly hear HF features that were previously thought to be inaudible, and moreover he shows some text files that purport to substantiate his claim, being generated in A/B testing using Foobar IIRC.

 

All this AFAIK fell apart on the issue of trust.

 

I'd be fascinated to see this paper, but personally I think the absolute audibility of high frequencies is about as significant in the real world as the demonstrated audibility of absolute phase.

Mike zerO Romeo Oscar November

http://wakibaki.com

Link to comment

Yes some of the excerpts from the paper came from the HA thread.

 

I took part in the "Conclusive Proof" thread. The issue with the AIX files were initially a .2 db level difference and different timing. The level was fixed, but the resampling software appeared to be rather poor.

 

Some of that latter discussion was about the jangling keys file which I could successfully detect in Foobar ABX. That too turned out to be an old poor resampling issue in my case. I resampled the original with Sox which does a much better job and I could no longer detect it. Amir said he still could. Which is hard to swallow because what is left with modern resampling is just about nothing. I do think Amir uses very efficient headphones, picks very short areas that decay near the noise floor and listens much louder than you could with music just playing. Some things become audible that way, but pretty much have no bearing on regular listening. I am not claiming Amir is lying which some do.

And always keep in mind: Cognitive biases, like seeing optical illusions are a sign of a normally functioning brain. We all have them, it’s nothing to be ashamed about, but it is something that affects our objective evaluation of reality. 

Link to comment

I notice that Bob Stuart's company has a history with Meyer and Moran. Their paper was prompted by what they felt were overly extravagant claims by Meridian, who are, of course, in the throes of launching a new distribution format.

 

Doubtless coincidental.

 

I looked at their site (Meridian). It says '...to reverse the trend, in which sound quality has been continually sacrificed for convenience...'

 

Sound Quality, that Lamb upon the Altar of Convenience. Who bears the guilt? All of us who listen to MP3s?

 

God said to Abraham, 'kill me a son'

Abe said 'God, you must be puttin' me on'

God said 'No' - Abe said 'What?'

God said 'You can do what you want Abe, but...

...the next time you see me comin' you'd better run...'

Abe said 'Where you want this killin' done?'

God said 'Out on Highway 61'

 

It's enough to bring tears to your eyes.

Mike zerO Romeo Oscar November

http://wakibaki.com

Link to comment
I notice that Bob Stuart's company has a history with Meyer and Moran. Their paper was prompted by what they felt were overly extravagant claims by Meridian, who are, of course, in the throes of launching a new distribution format.

 

Doubtless coincidental.

 

I looked at their site (Meridian). It says '...to reverse the trend, in which sound quality has been continually sacrificed for convenience...'

 

Sound Quality, that Lamb upon the Altar of Convenience. Who bears the guilt? All of us who listen to MP3s?

 

 

 

It's enough to bring tears to your eyes.

In that AVS Forum thread I previously linked, Amirm has explained, re-explained, and re-re-explained why the Meyer & Moran 'paper' is flawed to bejeezus. There are certain individuals who will try to discredit audiophiles on several audio related forums every single chance they get, i.e. by very extremely repetitively claiming their flawed theories as fact and by blatantly clinging to their own confirmation bias, to no end and while also showing very clearly that they have no real knowledge whatsoever of psychoacoustics. The paper by Bob Stuart also briefly touches upon why ABX testing is severely flawed, BTW. The main reason why it is flawed can, for convenience, basically be found in my forum signature.

If you had the memory of a goldfish, maybe it would work.
Link to comment

I kind if all delends what you really want to prove doesn't it? If one really doesn't habe a dog in that hunt, it comes down to "does this sound better to me in my system with my music?" It doesn't really matter all that much why, because the why can, like Dennis shows with his experience, always be found.

 

Me, I am more in the camp of it sounds better because my system's faults are less (or more) vulnerable to this or that factor. Noise, filters, processing, whatever.

 

The person involved usually really can hear a difference, and isn't all that interested in the technical reason why- they just want to be thrilled by the music all over again. I mean, you don't go to a Star Wars movie expecting technical scientific accuracy now, do you? You go for enjoyment. Same is mostly true for music. :)

 

Trouble starts when one group or another insists that the other side is wrong wrong wrong, utterly ruining the enjoyment for the other groups. Then out come the sharpened sticks and stones and general craziness. Academia again, but sometimes without any veneer of politeness.

 

Fact is, a number of people on the gasbag forums are so convinced they are on the inside of the truth, they act like missionaries in darkest Africa, in the 19th century. They are not really scientific, they are religious zealots. Some like that in every group, and in every group, it seriously weakens the group's message. Lay out the facts yes, tell people that those facts mean they are absolutely wrong and their observations and experiences are fantasy? Stupid thing to do. Unless they want to pick a fight.

 

So yeah, there is a lot of proof here that one cannot hear the diifference between high res and redbook files. There are mountains of contradictory evidence that suggest people can. That is about the limit of the facts though. Drawing a conclusion that one side or the other is suffering from delusion in the form of expectation bias or whatever is not supported by the evidence though. On either side.

 

Just my $0.02.

Anyone who considers protocol unimportant has never dealt with a cat DAC.

Robert A. Heinlein

Link to comment
I do not have the paper myself. But have seen some excerpts from it. There are a few curiosities that will get attention.

 

The signature headline results are trained listeners did 160 trials getting 56% correct which is right at the level for 95% confidence when comparing high sample rate 24 bit audio to low pass digitally filtered 16 bit audio.

 

The first curious thing is the digital low pass filter was said to be like that used in mastering and playback equipment. But there were two used. One was 22,050 hz brick wall and one was a 24 khz brickwall. What is curious is they used merely 500 hz wide transition bands between the cutoff and the flat response. Typically you would see 2050 hz and 4000 hz transition bands in consumer playback gear. Some are even a bit less steep than that with a roll off starting a little sooner than 20 khz and/or not reaching full stopband levels at the half sample rate point of 22.05 khz or 24 khz respectively.

 

Another curious thing is the low passed 16 bit content was dithered with rectangular dither or not dithered at all. They also used a signal consisting of wide dynamic range of approximately 16 bits. Better choices for dither would have been triangular or shaped.

 

Yet another curious result reported by those who have the paper is when they low passed the content without bringing it to 16 bits the results were chance. That would lead me to conclude the 16 bits is what was audible more so than the filter. But without the paper I don't know how they responded to that.

 

It would make sense to me to use filters steeper than normally found to see if the result is audible. This they did. My very next thought would be to use filters of different steepness to see how steep is too steep. Or in this case my next test would have used those wider filters like normally seen to see if this was detected. This step they skipped going right ahead with notion we need more bandwidth for less steep filters. One of the statements quoted from the paper says in effect, it is now accepted that high sample rates don't sound different than low rates due to signal above 20khz, but due to time domain performance of the filtering. Seems like a leap to that conclusion. Especially as it seems there was no effect with even the steep filtering at 24 bit.

 

Of course I have not seen the whole paper. Perhaps someone here has it.

 

If I'm understanding your description, kind of ludicrous to test with bandwidths that virtually no consumer DACs use. Would like to see reliable testing of real world scenarios, like Redbook, 192k, and DSD fed to a consumer sigma-delta DAC to determine whether folks can discriminate between in-DAC upsampling of 44.1k versus in-DAC upsampling of 192k, or between those and DSD that doesn't go through the upsampling or SDM steps.

 

It would also be nice to see the whole paper. From time to time I've considered an AES library subscription, but I worry about not having enough time to read the various papers I'm interested in and consequently wasting my money.

One never knows, do one? - Fats Waller

The fairest thing we can experience is the mysterious. It is the fundamental emotion which stands at the cradle of true art and true science. - Einstein

Computer, Audirvana -> optical Ethernet to Fitlet3 -> Fibbr Alpha Optical USB -> iFi NEO iDSD DAC -> Apollon Audio 1ET400A Mini (Purifi based) -> Vandersteen 3A Signature.

Link to comment
The paper by Bob Stuart also briefly touches upon why ABX testing is severely flawed, BTW. The main reason why it is flawed can, for convenience, basically be found in my forum signature.

 

 

Have you actually seen the Bob Stuart paper. If so, you are unique because although it was part of an AES convention session there does not seem to be a convention preprint available.

Link to comment

It would also be nice to see the whole paper. From time to time I've considered an AES library subscription, but I worry about not having enough time to read the various papers I'm interested in and consequently wasting my money.

 

You can buy individual papers and convention papers but I don't think this one is included.

Link to comment
I fail to see how calling people stupid is any less provocative than the behaviour you are complaining about.
Nah, they're not stupid. Just that you could put their brain inside the head of a shrimp and still have enough space left in there to add a 1kW Class A amp. :P
If you had the memory of a goldfish, maybe it would work.
Link to comment
If I'm understanding your description, kind of ludicrous to test with bandwidths that virtually no consumer DACs use. Would like to see reliable testing of real world scenarios, like Redbook, 192k, and DSD fed to a consumer sigma-delta DAC to determine whether folks can discriminate between in-DAC upsampling of 44.1k versus in-DAC upsampling of 192k, or between those and DSD that doesn't go through the upsampling or SDM steps.

 

It would also be nice to see the whole paper. From time to time I've considered an AES library subscription, but I worry about not having enough time to read the various papers I'm interested in and consequently wasting my money.

 

Well, there is the contentious topic of filter audibility. I can see testing with an extraordinarily steep filter to see can it be heard. That is what they did hear, and under very special conditions with highly advanced playback equipment it just barely is. That is one part of the summary being over looked. I will quote from the abstract:

 

Two main conclusions are offered: first, there exist audible signals that cannot be encoded transparently by a standard CD; and second, an audio chain used for such experiments must be capable of high-fidelity reproduction.

 

So much steeper than normal filtering, and over an exceptional system. It was in a very quiet room, with unusually wide dynamic range material over speakers with supposedly exemplary performance to 40 khz. And it just barely by a choice or two met the criterion for audibility. It did meet it however. 56% correct choices out of 160 total choices.

 

Now if you widen filters will that just make it go away as being audible? Seems like a good chance, but you can't jump to that conclusion either. If you have a less quiet room will it no longer be audible? That seems like a very good bet. If you have speakers with less extension will it no longer be detectable? That doesn't seem like a bad bet at all. If your speakers themselves are a filter at 20 khz would the other filtering matter? Not likely.

 

So it looks like good work, but really only a good start for most people's purposes. 48 khz material has a 4 khz transition band, this test used 500 hz. That is 8 times steeper a filter. I would have been much more impressed if they then used the wider filter regardless of the results. Of course instead, it ties in with their launch of a new music distribution format and encoding/decoding scheme which yes of course says you need more bandwidth so you can do the filtering in a way that is better than done at 44 and 48 khz rates.

 

Now I understand commercial ventures have to make a living. But this test says such filtering is barely audible under extraordinary conditions. The pitch for the new MQA highbandwidth format. Things like "It lets you feel every last bit of emotion in the music" The idea it will be a big step forward. If it is, it will mostly be due to better mastering, almost all of which would come across on CD. I imagine as it doesn't feed into Meridian's needs they won't be doing the test again with wider filters like used in actual consumer products.

And always keep in mind: Cognitive biases, like seeing optical illusions are a sign of a normally functioning brain. We all have them, it’s nothing to be ashamed about, but it is something that affects our objective evaluation of reality. 

Link to comment
In that AVS Forum thread I previously linked, Amirm has explained, re-explained, and re-re-explained why the Meyer & Moran 'paper' is flawed to bejeezus. There are certain individuals who will try to discredit audiophiles on several audio related forums every single chance they get, i.e. by very extremely repetitively claiming their flawed theories as fact and by blatantly clinging to their own confirmation bias, to no end and while also showing very clearly that they have no real knowledge whatsoever of psychoacoustics. The paper by Bob Stuart also briefly touches upon why ABX testing is severely flawed, BTW. The main reason why it is flawed can, for convenience, basically be found in my forum signature.

 

Well, like I am doing being skeptical of Meridian, one can be skeptical of M&M. I don't consider Amir an uninterested party either. M&M is most often criticized for using SACD's with no ultrasonic content. Well, what is ignored is some of it did have ultrasonic content. The results were no better for those. And that plenty of people were extolling how much nicer these SACD's sounded including those very releases without ultrasonic content.

 

What was more useful in their results to me was a confirmation that a 16 bit AD/DA conversion was transparent according to their results. I think many would have said even taking CD and subjecting it to another AD/DA loop would have been degrading. One might quibble and say their conditions of testing could have been better etc. But at a very minimum it shows that such an AD/DA conversion is nowhere near obvious. If it is detectable you have to be pretty persnickety to tease it out.

 

I agree with Amir about much casual blind testing being pretty flawed. Of course one of the big reasons is the memory thing. You really only have direct memory of aural events for perhaps 15 seconds or possibly less. Much of the stuff bandied about on HA for this and other reasons is somewhat to terribly flawed. But then so are things bandied about on more audiophile friendly forums. No level matching and listening to overly long periods of time yet feeling comfortable making detailed determinations of quality being a couple of examples.

 

Many like to complain we don't know everything about hearing so we can't draw conclusions. I don't agree we know plenty, but not everything. One thing not known is if a tiny, barely detectable flaw, like these filters, then leads to long term dissatisfaction of playback. Audiophiles tend to say yes. But they tend to say yes about plenty of things that just aren't so. And I would tend to say no. If you barely can detect something under special conditions, under most more casual conditions it will just be background noise of no consequence. But in fact the answer to that question is not known. Cracking it would not be easy either. Much easier to develop playback fidelity that is beyond reproach then you don't have to worry about that problem.

 

Despite one particular poster's contention I am out to get audiophiles or show them wrong that is not the case. I am interested in what is the truth. I think jumping to 384 khz sample rates just in case is rather wasteful at a minimum if correctly done 48 khz is enough. Or if we need more then fine.

 

I also think the answer to flawed ABX tests is to spread the information about how to do them. Not just write them off, and go on our merry subjectivist way.

And always keep in mind: Cognitive biases, like seeing optical illusions are a sign of a normally functioning brain. We all have them, it’s nothing to be ashamed about, but it is something that affects our objective evaluation of reality. 

Link to comment

What Amir is saying, AFAIK, is that he wants the audio files delivered in their recorded sample rate rather than being at the mercy of some engineer's idea of how to do 16/44 downsampling - his contention being that he can do this downsampling himself & most probably more correctly (or, at least have the ability to redo it again if he gets it wrong).

 

From your points above, I think you agree 100% with this stance?

 

His criticism of M & M's paper was in their disregard for the standards needed for reasonably valid DB tests i.e no apparent knowledge of MUSHRA or BS1116 standards or evidence of having applied these standards.

 

I think one of his main complaints about the M&M paper is that they haven't used trained listeners & his logic that if such is not the case then a null result is the most likely outcome.

 

All of this seems reasonable to me.

 

As for the Meridian paper, I haven't read it yet but from the quoted bits & the criticisms of it, it seems to be a more rigorous treatment of the area.

 

Your last point about ABX tests done properly is really quizzical as the main objection being raised against Amir's Foobar ABX positive results is one of trust - i.e. the test itself is too easy to be gamed & therefore is open to being distrusted.

Link to comment

Despite one particular poster's contention I am out to get audiophiles or show them wrong that is not the case. I am interested in what is the truth. I think jumping to 384 khz sample rates just in case is rather wasteful at a minimum if correctly done 48 khz is enough. Or if we need more then fine.

 

I also think the answer to flawed ABX tests is to spread the information about how to do them. Not just write them off, and go on our merry subjectivist way.

 

Naw- not me. I tend to agree with your results, just not always with your conclusions based on those results.

 

I think ABX tests have some basic flaws to them, and the only way to overcome those flaws is also to do longer term, relaxed listening. Sometime sit can take days for a person to realize on what they are hearing. Or not hearing. Or... whatever. Not going down that rabbit hole again. :)

Anyone who considers protocol unimportant has never dealt with a cat DAC.

Robert A. Heinlein

Link to comment
What Amir is saying, AFAIK, is that he wants the audio files delivered in their recorded sample rate rather than being at the mercy of some engineer's idea of how to do 16/44 downsampling - his contention being that he can do this downsampling himself & most probably more correctly (or, at least have the ability to redo it again if he gets it wrong).

 

From your points above, I think you agree 100% with this stance?

 

His criticism of M & M's paper was in their disregard for the standards needed for reasonably valid DB tests i.e no apparent knowledge of MUSHRA or BS1116 standards or evidence of having applied these standards.

 

I think one of his main complaints about the M&M paper is that they haven't used trained listeners & his logic that if such is not the case then a null result is the most likely outcome.

 

All of this seems reasonable to me.

 

As for the Meridian paper, I haven't read it yet but from the quoted bits & the criticisms of it, it seems to be a more rigorous treatment of the area.

 

Your last point about ABX tests done properly is really quizzical as the main objection being raised against Amir's Foobar ABX positive results is one of trust - i.e. the test itself is too easy to be gamed & therefore is open to being distrusted.

 

Yes I agree. Especially about letting us have master tapes in the native format or as close as is possible. We can do our own re-formatting if it is important. Even if I think 192khz is unneeded, if that were how the original was done that would be my preference. A copy exactly like the original.

And always keep in mind: Cognitive biases, like seeing optical illusions are a sign of a normally functioning brain. We all have them, it’s nothing to be ashamed about, but it is something that affects our objective evaluation of reality. 

Link to comment
Naw- not me. I tend to agree with your results, just not always with your conclusions based on those results.

 

I think ABX tests have some basic flaws to them, and the only way to overcome those flaws is also to do longer term, relaxed listening. Sometime sit can take days for a person to realize on what they are hearing. Or not hearing. Or... whatever. Not going down that rabbit hole again. :)

 

I did not have you in mind when I posted that Paul.

And always keep in mind: Cognitive biases, like seeing optical illusions are a sign of a normally functioning brain. We all have them, it’s nothing to be ashamed about, but it is something that affects our objective evaluation of reality. 

Link to comment
.....

 

I think ABX tests have some basic flaws to them, and the only way to overcome those flaws is also to do longer term, relaxed listening. Sometime sit can take days for a person to realize on what they are hearing. Or not hearing. Or... whatever. Not going down that rabbit hole again. :)

Yep, I agree completely & yet the contention often made when an ABX test returns a positive result is along the lines of "well if they have to try that hard to hear a difference, then what relevance does it have to normal listening".

 

Is this not some form of illogical thinking? Insisting on the far from normal listening mode imposed by double blind tests as some yardstick & when positive results returned, condemning these results by claiming that normal listening would not return such a noticeable difference?

Link to comment
It would make sense to me to use filters steeper than normally found to see if the result is audible. This they did. My very next thought would be to use filters of different steepness to see how steep is too steep.

 

I would say there's much more to digital filters than just steepness. Shape of the slope, etc.

Signalyst - Developer of HQPlayer

Pulse & Fidelity - Software Defined Amplifiers

Link to comment
I would say there's much more to digital filters than just steepness. Shape of the slope, etc.

 

Yes, of course, but I don't have the information beyond the size of the transition band used in this case.

And always keep in mind: Cognitive biases, like seeing optical illusions are a sign of a normally functioning brain. We all have them, it’s nothing to be ashamed about, but it is something that affects our objective evaluation of reality. 

Link to comment
Despite one particular poster's contention I am out to get audiophiles or show them wrong that is not the case. I am interested in what is the truth. I think jumping to 384 khz sample rates just in case is rather wasteful at a minimum if correctly done 48 khz is enough. Or if we need more then fine.

 

For some testing material, you can take the 352.8/24 test files from 2L and try different conversion algorithms and output formats to see what output resolution you think is enough for your case.

 

Coming up with universal "perfect for everyone" format is really hard task. At least you'd need to be sure that you have the best gear and ears to determine what is the best... Because I know I have neither, I rather play safe from technical perspective. So I like to come up with parameters where I know nothing at all is lost no matter how you look at it.

Signalyst - Developer of HQPlayer

Pulse & Fidelity - Software Defined Amplifiers

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now



×
×
  • Create New...