Why you can't trust measurements

[GoldenOne]

I figured this may be something worth starting a dedicated thread for.
There are all sorts of reasons as to why measurements may be unreliable, misleading, confusing, or just inconsistent. And I've made a video discussing some of these topics.

 

 

[GoldenOne]

6 hours ago, AudioDoctor said:

People should have to watch this video before being allowed to post here. Thanks @GoldenOne for taking the time to make this, I found it informative.

Yeah I think there's quite a few people that read the title but didn't watch the video :P

 

The video is not about whether or not measurements overall are/aren't important. But why measurements shouldn't always be taken at face value without additional context or scrutiny

 

 

[GoldenOne]

18 hours ago, yamamoto2002 said:

RMAA itself is excellent app but people use it with too carelessly and their measurement result is totally unreliable, there is a 15dB to 20dB SNR discrepancy on the same onboard DAC, almost always contaminated with ground loop problems, sometimes it seems measurement is simply failed with accidental click noise or other issues.

I don't think the issue is with RMAA itself.
RMAA as software is fine and very useful, but you can use any ADC you want with it, so the results are going to be heavily dependent on what ADC someone is using and how they have things set up.

If you use RMAA with an ADI-2 Pro FS R for example like Archimago (or myself prior to getting the APx555) then you can get very good results.
If you just use a basic focusrite interface then you're not gonna have such a good time.

Additionally, when it comes to motherboard audio there are also so many other factors. The PSU you're using, what other hardware such as GPU/CPU are in that particular PC etc, is it under load or idle.
How noisy a PC is can vary massively.

 

 

18 hours ago, yamamoto2002 said:

About intersample overs, I read somewhere, some oversampler of earlier DAC has no tolerance and waveform is flipped by integer arithmetic overflow!

I saw this actually but can't remember for the life of me what DAC it was. I think it might have been an earlier denafrips DAC? I had some interesting and similar behaviour when testing the Ares 2, I didn't do an intersample overs test but when using the slow filter, just doing an impulse response caused the waveform to wrap around as shown below.

I'm really not sure how this would happen but it seems it did :P
It didn't happen with Sharp OS though

This would most definitely be MUCH more problematic than how intersample overs are handled on modern DACs (which is that they simply clip/max out). But luckily I don't think this is something that happens with most DACs.
 

[GoldenOne]

18 hours ago, Speedskater said:

There is a whole lot of back story about that video.
I have not sorted it all out yet.
But it takes a lot of misbehavior to get banned from Audio Science Review forum.

This video isn't about ASR

I did however post in the thread they have going on there about this video. It was instantly removed of course but there's an archive.org page here if anyone would like to read it:

https://web.archive.org/web/20220421140602/https://www.audio “science” review/forum/index.php?threads/why-you-cant-trust-audio-measurements-by-goldensound.33222/page-4

(You must remove the spaces/quotes from 'audio “science” review'. Audiophilestyle automatically puts them in so the link doesn't work)

[GoldenOne]

1 minute ago, firedog said:

Link doesn't work archive says there's no such page-even when it lists all the archived pages from the site.

Try this: shorturl.at/lzCR2

[GoldenOne]

2 hours ago, JoshM said:

I didn’t see the SHF thread before it got removed, but I’m guessing it’s the guy who recently posted on ASR that six of nine of the budget components recommended by Amir that the poster bought failed or measured differently from what Amir found. The poster raised the possibility that some companies were sending cherry-picked units, and requested to send his units to Amir to verify (since, as usual, people didn’t believe the poster, even though he did his own measurements, blind tests, etc.). In response, Amir said (contrary to his “we measure anything for free” claim), that the poster would have to pay him $200 per piece of equipment to confirm the issues, then locked the thread.  

 

What that all illustrated for me (and aligns with my own experiences with these brands) is that certain companies lauded by ASR have major failure rates if you search relevant threads (on Head-Fi, any retailer with user reviews, and sometimes ASR itself) for words like “fail,” “issue,” “problem,” or “died.” But anyone overtly raising the possibility that this is a systematic issue on ASR is shut down. 
 

Of course, when it comes to disfavored brands, like Schiit, Amir will measure multiple units to find faults, conduct teardowns to inspect for minor solder splatter, etc. Favored companies are, of course, spared this critical eye. It’s science, you know. 

I've had a couple products measure differently from the measurements amir showed. Not necessarily by a huge degree but enough that if amir were to put the unit I'd received on his ranking list it'd be several spots lower.

 

Unfortunately with Amir's position/audience it would be silly to assume that manufacturers WERENT sending cherry picked units. His recommendation sells a lot of product so there's big incentive to make sure he gets a well performing unit.

 

Realistically a fairer way to do things would be for Amir to ask the manufacturer to reimburse him for the purchase of a retail unit from a distributor. This would ensure that he's testing something that will be in consumer hands. Not a cherry picked unit.

 

In regards to failure of devices though. Unfortunately reliability is measured best by user reports. It's not really something Amir can test so that's not really fair to put on him. 

A product can measure well but die 2 months later.

 

Also, on the note of having to pay to get stuff measured, if anyone would like anything measured I'll do a full set for free on my APx555. Just cover shipping.

[GoldenOne]

2 minutes ago, pkane2001 said:

 

This was brought up here before, but I'm not sure what the answer is. Are your measurements done with the same AP project settings as those that Amir is using? If not, there could be differences in settings (as per your video) that could account for some of the differences.

It depends. Amir has not shared his configuration publicly and he's hardly going to give it to me I imagine so I have no idea if it's the same or not.

 

But for the basic bench mode 1khz test where most of the settings can be pretty much assumed and the bandwidth is shown, I can match his config for that specific one. And in that situation I've had some devices which did not perform the same.

 

And to be clear that's not any 'fault' of amir. He's measuring the device given to him.

The issue is that some manufacturers may cherry pick and there is no way for amir (or viewers of his site) to know if that's happening or not

[GoldenOne]

Just now, JoshM said:

I think it’s unfair to manufacturers for some, not not others, to receive that kind of scrutiny. (He’s noted that Topping, for example, gives him products for free without asking it be returned, but has not done teardowns on all of those products, so it’s clearly not solely a permission issue.) If not all people who submit products agree to teardowns, he can either not do teardowns at all or purchase units himself.

On this note I'm quite curious to see internals of the new topping LA90.

 

Afaik topping hasn't done a discrete amp before so it'll be interesting to see what's inside this.

 

But whilst topping usually has at least partial internal views in their marketing material and full internal shots available elsewhere, there are no views of the insides available and they've seemingly asked Amir not to open it.....

 

[GoldenOne]

Just now, pkane2001 said:

 

Yeah, the 1kHz dashboard, while revealing, isn't painting the whole picture. I much prefer sweeps with varying levels and frequency showing noise, THD/IMD, linearity. With those tests, there are some parameters that can differ significantly without being visible on the AP chart, that's why I'm asking. 

 

 

There absolutely are. But unfortunately as mentioned Amir does not share his configuration publicly so there's no way to verify.

[GoldenOne]

1 minute ago, pkane2001 said:

 

Topping seems very concerned about competition copying their designs. Lately they started to scratch off all chip/component labels from every PCB before shipping. Looks kinda ugly. Wonder what they did in a discrete design LA90 to obscure it :)

 

That's probably pretty valid tbh. 

 

I'll be getting one in soonish and will open it up to have a look. Hopefully it's not too obscured inside.

 

[GoldenOne]

17 minutes ago, Iving said:

 

I think Golden and Amir have had problems in the past. I tried to help them stay cool. There's only so much that one man can do.

The video is not about amir.

Nor do disagreements between Amir and myself have anything to do with why certain market segments sales are heavily influenced by objective measurements

[GoldenOne]

On 5/24/2022 at 4:50 AM, JoshM said:

Also, relevant to @GoldenOne’s video, it thought Bob’s illustration of how amplifier distortion interacts with level was fascinating:

 

https://www.superbestaudiofriends.org/index.php?threads/tutorial-amplifier-distortion-vs-amplitude-a-visual-guide.12279/

 

It’s worth pondering what level is most realistic for one’s use of a product, and it might not be the level at which most measurements are performed. 

This is something that I personally feel is very important. For various reasons.

Firstly, I've noticed that some of the devices with stronger 'House Sounds' including DACs such as the Schiit Yggdrasil, show quite drastic changes in distortion level and profile in relation to amplitude.

 

In addition to this, some devices have slightly different distortion at their very uppermost output levels than they do in the rest of the output range.
The Gustard X26 pro for example, which is an excellent DAC, does seem to be somewhat 'optimising' for full scale output to look better on ASR-style tests.

 

You can see that distortion drops a few dB in the top ~6dB.
And in fact even when not limited by THD itself, you can visually see how the harmonics change.

At full scale 0dBfs output it looks like this:

All harmonics at or below about -140dB

But then take it to -3dBfs output and suddenly the 3rd order harmonic jumps up almost 15dB.

 

But ignoring DACs and just looking at amps, as has been mentioned, we don't listen to amps at 4v output. In fact on the vast majority of headphones that level would probably destroy your ears. So it's much more apt to test the device in the range that you're likely to use it.
Personally I offer measurements at 4v (to compare to ASR if people wish), 700mV (as this is a reasonably loud listening level on a few over ear headphones like an Arya) and 50mV (as this is appropriate for IEMs).

If you test something like an HE9 at 4V on 32 Ohm, you'll get about 0.3% THD+N (50dB SINAD). 
But if you test at a more realistic 700mV, you get a reduction to 0.02% THD+N (23dB increase to 73dB SINAD)

It seems silly to me to only test things at 4v as it's not realistic for amps in most cases.
But more to the point, on both DACs and Amps THD+N vs Level behaviour can reveal some pretty interesting stuff