Misleading Measurements

[Jud]

Might be interesting for folks discussing this topic to read the following Wikipedia article:

 

https://en.m.wikipedia.org/wiki/New_Coke

[pkane2001]

41 minutes ago, Jud said:

Might be interesting for folks discussing this topic to read the following Wikipedia article:

 

https://en.m.wikipedia.org/wiki/New_Coke

 

Thanks, Jud, but I meant something stronger than a Coke 😄

 

44 minutes ago, pkane2001 said:

🤪 I need a drink...

 

[Audiophile Neuroscience]

2 hours ago, opus101 said:

 

Not completely sure its relevant here but Amir misappropriated ADI's measurements of their DAC chip to malign Schiit. If you need a link I'll look it up.

Thanks, as an example of "misleading measurements" and biased reporting rendering agenda-based measurements meaningless.😃

[Audiophile Neuroscience]

2 hours ago, pkane2001 said:

 

🤪 I need a drink...

 

So we're clear then about Amir's untrustworthy measurements, that's the main thing !

[Audiophile Neuroscience]

2 hours ago, Jud said:

Might be interesting for folks discussing this topic to read the following Wikipedia article:

 

https://en.m.wikipedia.org/wiki/New_Coke

 

Was it @Summit or @semente or someone else who brought this up earlier. i didn't read the full article but the previous poster referred to the propensity to initially choose a strong quality like the increased sweetness of Pepsi is often overturned by longer exposure to the product. I think the suggestion was that even well conducted short burst ABX tests may be misleading.

[opus101]

1 hour ago, Audiophile Neuroscience said:

Thanks, as an example of "misleading measurements" and biased reporting rendering agenda-based measurements meaningless.😃

 

This post is indeed a prime example of misleading measurements. You need to go to the AD5547 datasheet to see why though, the small print gives away that the two FFTs Amir places side by side are not comparable.

 

https://www.audio “science” review/forum/index.php?threads/review-battle-of-schiit-audio-dacs.5487/post-121858

 

Seems someone doesn't want a link pointing to ASR, you need to get rid of those quote marks, close up the spaces and add .com before it becomes valid.

[Audiophile Neuroscience]

28 minutes ago, opus101 said:

 

This post is indeed a prime example of misleading measurements. You need to go to the AD5547 datasheet to see why though, the small print gives away that the two FFTs Amir places side by side are not comparable.

 

https://www.audio “science” review/forum/index.php?threads/review-battle-of-schiit-audio-dacs.5487/post-121858

Thanks. I can see the FFT for THD of the AD DAC is the same as in the data sheet but not sure (due to lack of technical knowledge) what small print gives away that the two FFTs Amir places side by side are not comparable.

[opus101]

The key insight is that the AD5547 is a multiplying DAC and this FFT shows it operating in that mode - with its reference voltage being fed a sinewave. So the digital input is being held constant, unlike in the FFT on the right.

[Clockmeister]

Oh this is very interesting and exactly part of my remit on a daily basis specifically power supplies of all forms.

 

Part of of the CE validation is noise immunity of devices under test, be it dac's, streamers or power supplies. Where an AM modulation is introduced into the DUT (Immunity testing) to determine any adverse effects effects while being viewed ideally under a real time spectrum analyzer and respectable scope with bode plot ability. Other items required would be a line injector (both current and voltage) LISN (line impedance stabilization networks) both line AC voltages and DC characterization. Along with quality high voltage differential probes for ac measurements and at least a 1ma current probe. If you have the budget then an RF current probe from looking at the conducted emissions in real time for that deep dive into the power noise up to 1Ghz.

Normal CISPER windows are 9Khz-30Mhz, 30-300Mhz and up to 1Ghz

 

So many variables to look at and it does effect the sound without question recently I looked at a well know dac that is in the super dac claimed section of a well know internet site claimed to be one of the best measured devices of that crop of far east D to A's

 

Well not sure how it came to even have a CE mark (lol) as there was no ac line filtration what so ever, bad clock harmonics upto and above 2Ghz, plus very interesting 'audiophile rank capacitors' plus the ubiquitous 575 clock.

 

Sound was edgy, forward, solid and powerful yet two dimensional, aggressive and fatiguing, yet on the APX555 it measured very fine indeed, also on my R&S UPV it did the same, sound wise it pretty dire.

 

After some basic remedial work and RF attention it became much more enjoyable most of this work was power supply related. See post below of the before and after RF measurements.

 

dac measurements

 

Ac mains quality does vary enormously, we have a full power analyzer suite from both Tektronix,  Lecroy and Rigol  happy to share with you.

[pkane2001]

3 hours ago, Audiophile Neuroscience said:

Thanks. I can see the FFT for THD of the AD DAC is the same as in the data sheet but not sure (due to lack of technical knowledge) what small print gives away that the two FFTs Amir places side by side are not comparable.

 

So the fact that he didn't read a small print on a data sheet proves his incompetence or that he just wasn't paying attention? Or is it that his measurements of the device show that it has a very poor linearity, high jitter and starts to clip below 0dBFS? The latter is what I got out of it. We all seem to get what we want from measurements, I guess. I get information about the measured device, you -- information about the person.

[opus101]

7 minutes ago, pkane2001 said:

I get information about the measured device, you -- information about the person.

 

I missed the place where @Audiophile Neuroscience got information about Amir from this.

[pkane2001]

16 minutes ago, opus101 said:

I missed the place where @Audiophile Neuroscience got information about Amir from this.

 

 

5 hours ago, Audiophile Neuroscience said:

example of "misleading measurements" and biased reporting rendering agenda-based measurements

 

[Clockmeister]

I firmly believe Amir has an agenda, the premise of the asr site is good, however in reality its inhabited by biases (for both camps plus the usual over riding personalities stamping their correctness to the subject involved) maybe AP gave a preference on his purchase price? or a long term loan maybe? he purchased it outright with his own funds?. Not to bothered, however like with all equipment and the internet, information is the key, how you use  and apply it is down to the individual. So if said individual has an issue or score to settle then objectivity goes out of the window.

[opus101]

Ah that's not 'from this' - seems he brought that to the discussion prior.

[opus101]

8 minutes ago, Clockmeister said:

I firmly believe Amir has an agenda

 

I'm not sure if belief is relevant here, he's stated openly that he has a mission.

[Clockmeister]

Just now, opus101 said:

 

I'm not sure if belief is relevant here, he's stated openly that he has a mission.

 

Apologies Opus101 I did not realise he had publicly stated this, nothing wrong being on a mission provided you are open and transparent with your basic fundamental principles and not swayed by various outside influences

[pkane2001]

13 minutes ago, opus101 said:

Ah that's not 'from this' - seems he brought that to the discussion prior.

 

Well, actually it was in response to your statement that "Amir misappropriated ADI's measurements of their DAC chip to malign Schiit", so mostly it was 'from this'. And how would you interpret the word "malign" if not as an accusation of malicious intent?

[opus101]

1 minute ago, pkane2001 said:

 

Well, actually it was in response to your statement that "Amir misappropriated ADI's measurements of their DAC chip to malign Schiit", so mostly it is 'from this'. And how would you interpret the word "malign" if not as an accusation of malicious intent?

 

It wasn't intended to imply malicious intent - but he did bring Schiit's design competence into doubt through what appears to be a misunderstanding of the DS. In saying not 'from this' it was shorthand for 'not from the measurements I drew attention to' seeing as the context (established by yourself and the title of the thread) was firmly measurements.

[Allan F]

12 hours ago, Jud said:

Might be interesting for folks discussing this topic to read the following Wikipedia article:

 

https://en.m.wikipedia.org/wiki/New_Coke

 

FWIW, many years ago I took the Pepsi Challenge and it took me about three milliseconds to correctly identify Coke, which I prefer.

[Summit]

10 hours ago, Audiophile Neuroscience said:

 

Was it @Summit or @semente or someone else who brought this up earlier. i didn't read the full article but the previous poster referred to the propensity to initially choose a strong quality like the increased sweetness of Pepsi is often overturned by longer exposure to the product. I think the suggestion was that even well conducted short burst ABX tests may be misleading.

 

OT

Yes in a test there we are listening only for a few seconds to one product and they quickly change to another product for a few seconds, before changing again and again it’s common to choose the ones which have a sound that “stands out”. In this type of tests we are completely focused on noticing the difference between two products instead of the sound quality, which can’t be evaluated and decided after just a few seconds. This favouring of a type of sound that stands out is not very evident in the beginning of a test, but steadily increase by the amount of switching back and forward rapidly.

 

Related to this type of test method is the problem called forced choice bias. Forced choice means that we are forced to pick one of two even if we can't hear any difference between them, or worse don’t feel comfortable that the difference we hear also is desirable.

  

A well conducted short ABX test is good for distinguishing if there are any audible difference between two products or not. Or if we can hear a difference between sample A and B and many other things that can be easily distinguished.

 

It’s not very hard to identify Coke from Pepsi one time or two, but after 7 sips of Pepsi and 7 sips of Coke it’s much harder to distinguishing them. In food and beverage test we are not forced to do that 10 times, but in audio.

[pkane2001]

19 hours ago, fas42 said:

Another interesting bundle of thoughts I just came across, when someone asked for some noise making ideas ... https://www.eevblog.com/forum/projects/intentionally-injecting-noise-into-supply-rails/

 

As if he was reading this, Amir just posted measurements of an AudioQuest powerstrip, measuring its advertised line noise filtration capabilities. Apparently it does filter above 30kHz, but leaves all of the junk in the audio band. Who knows, it may actually be useful for some audio devices that don't have good filtration at the power supply.

[Jud]

17 hours ago, Jud said:

Might be interesting for folks discussing this topic to read the following Wikipedia article:

 

https://en.m.wikipedia.org/wiki/New_Coke

 

15 hours ago, Audiophile Neuroscience said:

 

Was it @Summit or @semente or someone else who brought this up earlier. i didn't read the full article but the previous poster referred to the propensity to initially choose a strong quality like the increased sweetness of Pepsi is often overturned by longer exposure to the product. I think the suggestion was that even well conducted short burst ABX tests may be misleading.

 

4 hours ago, Summit said:

 

OT

Yes in a test there we are listening only for a few seconds to one product and they quickly change to another product for a few seconds, before changing again and again it’s common to choose the ones which have a sound that “stands out”. In this type of tests we are completely focused on noticing the difference between two products instead of the sound quality, which can’t be evaluated and decided after just a few seconds. This favouring of a type of sound that stands out is not very evident in the beginning of a test, but steadily increase by the amount of switching back and forward rapidly.

 

Related to this type of test method is the problem called forced choice bias. Forced choice means that we are forced to pick one of two even if we can't hear any difference between them, or worse don’t feel comfortable that the difference we hear also is desirable.

  

A well conducted short ABX test is good for distinguishing if there are any audible difference between two products or not. Or if we can hear a difference between sample A and B and many other things that can be easily distinguished.

 

It’s not very hard to identify Coke from Pepsi one time or two, but after 7 sips of Pepsi and 7 sips of Coke it’s much harder to distinguishing them. In food and beverage test we are not forced to do that 10 times, but in audio.

 

I don't know whether there's academic work to back this up, but I have the general notion that in terms of sensory stimuli, discrimination and pattern matching are two different things.

 

Regarding discrimination, as @Audiophile Neuroscience and @Summit are saying, there seems to be a propensity to choose on the basis of a particular quality - sweetness when tasting soft drinks (wonder whether it's sweetness or something else that predominates in blind wine tastings?), loudness in audio (for example, blind violin listening tests, even when done by very accomplished violinists; or the hi res listening test done by former member Julf with the assistance of the principal of BIS recordings, where forum members by a large margin chose a file that was, unknown to them, 1dB louder than the rest as sounding best).

 

Sensory memories are quite short - perhaps 4 seconds in the case of audio; I don't know about taste, but I imagine it's not terribly different. It's not at all surprising we'd need a strong, very recognizable quality to impress us as different in that short amount of time. And I think many of us have experienced the sense of being at sea, of having nothing to go on, when such a strong stimulus is absent and/or a comparison takes longer than 4 seconds. So the fact that so many blind tests show an inability to discriminate isn't surprising, but fully expected.

 

Pattern matching is very different. There's the New Coke example - sweetness is preferred in a quick discrimination test, but when millions of customers sat back with a New Coke on a hot summer day and it just didn't taste "right" - didn't fit the familiar, comfortable pattern - they noticed, despite passage of a lot more than a few seconds since they'd tasted an original Coke. And of course in audio there's the familiar example many people have mentioned of a family member's voice on the phone - you recognize it instantly as matching the pattern of dad, mom, sibling, spouse, child, even if months have passed, and even on a connection that doesn't provide the best audio quality. People are so good at audio pattern matching that, for example, they are recruited to try to discern patterns in Jupiter's radio emissions. In fact we will even discern patterns that don't really exist - that's what optical and audible illusions are.

 

Which of these are we doing when we enjoy listening to music? I certainly don't think we're trying to do loudness comparisons. I'd say our enjoyment arguably lies in how much the tone of a violin or acoustic guitar, or the sound of a human voice, reminds us of the real thing, and excites the same emotions as the real thing did.

 

A few years ago I conducted a little test here in the forums. One acoustic guitar (classic Gibson, big round sound hole) was playing in one channel, a very different acoustic guitar (a very old Epiphone with small f-holes like a violin) in the other. The task was to determine which guitar was playing in which channel. Self-declared objectivists were given a selection that repeated 2 seconds on and 2 seconds off for 30 seconds. A rapid comparison, as an objectivist can tell you, is best for discrimination when doing the most common form of blind testing, i.e., listening to selection A, then quickly comparing it to selection B. So that's why they got that sort of test, even though there was no sequential comparison - both guitars were playing throughout the 2 seconds, one in each channel. Self-declared subjectivists were simply given a 30-second selection, because they don't tend to be concerned about quick comparisons, the length of echoic memory, and suchlike.

 

The guitars sounded different enough that most people got it right. But not surprisingly, a higher proportion of those listening to the 30 second selection got it right than those who had only 2 second repeated snippets to work with. (Only 1 person out of 45, who said he was using an iPhone and earbuds, could not distinguish between guitars in the 30 second selection.) Statistically the results should only have occurred 6% of the time by chance, so there is some level of assurance this was a real effect.

 

It seems to me that the act of trying to discriminate between sequential selections by blind testing may not be the best way to model what we actually are doing when we sit down to enjoy our music, to enjoy the familiar patterns of instruments or vocals.

 

What does this have to do with measurements? Well, how do we determine what levels of various measurements of distortions, for example, are audible and thus important? By discrimination tests. And I wonder how well such tests simulate our ability to detect the pattern of something that "sounds right." After all, the taste discrimination tests that led to New Coke did not seem to do a very good job developing something that "tasted right" to legions of people accustomed to the taste pattern of original Coke.

 

 

 

[fas42]

32 minutes ago, pkane2001 said:

 

As if he was reading this, Amir just posted measurements of an AudioQuest powerstrip, measuring its advertised line noise filtration capabilities. Apparently it does filter above 30kHz, but leaves all of the junk in the audio band. Who knows, it may actually be useful for some audio devices that don't have good filtration at the power supply.

 

IME, all audio devices don't have good filtration at the power supply - that is, it's trivially easy to introduce some electrical noise making device or process into the environment, and hear the impact on the SQ ... just read the reports of companies struggling to get good sound in a show situation, and how they had to deal with the "lousy hotel power!"

[fas42]

27 minutes ago, Jud said:

A few years ago I conducted a little test here in the forums. One acoustic guitar (classic Gibson, big round sound hole) was playing in one channel, a very different acoustic guitar (a very old Epiphone with small f-holes like a violin) in the other. The task was to determine which guitar was playing in which channel. Self-declared objectivists were given a selection that repeated 2 seconds on and 2 seconds off for 30 seconds. A rapid comparison, as an objectivist can tell you, is best for discrimination when doing the most common form of blind testing, i.e., listening to selection A, then quickly comparing it to selection B. So that's why they got that sort of test, even though there was no sequential comparison - both guitars were playing throughout the 2 seconds, one in each channel. Self-declared subjectivists were simply given a 30-second selection, because they don't tend to be concerned about quick comparisons, the length of echoic memory, and suchlike.

 

 

Sorry, I think this type of testing is complete nonsense - of zero value, in every possible way. Yesterday, I got a result - a negative one, in fact - which to me is of vastly greater value: I played a track of a live blues performance, quite recently recorded - I have played this track close to a 100 times; and I know exactly what the sense of it can be ... it was a fail on the new digital speakers of mine - nothing was missing, all the bits were there; but the vitality, the sense of the occasion was poorly rendered - it was, boring ...

 

Okay, I could explain why that was so - the system hadn't warmed up enough, there was too much interference from somewhere, there was some issue that I still haven't unearthed, etc - but that's not the point; what I instantly knew at that moment was that I had a handle, in terms of some piece of music, that would tell whether the rig was firing at an acceptable level.

[Audiophile Neuroscience]

 

11 hours ago, pkane2001 said:

 

So the fact that he didn't read a small print on a data sheet proves his incompetence or that he just wasn't paying attention? Or is it that his measurements of the device show that it has a very poor linearity, high jitter and starts to clip below 0dBFS? The latter is what I got out of it. We all seem to get what we want from measurements, I guess. I get information about the measured device, you -- information about the person.

 

"Incompetence" is your word not mine - any inference "or information about the person" in this regard is yours also. 

 

That said, whether incompetence, ignorance, agenda driven bias or some other reason makes a measurement untrustworthy, it is still untrustworthy