Why you can't trust measurements

[pkane2001]

On 4/20/2022 at 11:07 AM, GoldenOne said:

I figured this may be something worth starting a dedicated thread for.
There are all sorts of reasons as to why measurements may be unreliable, misleading, confusing, or just inconsistent. And I've made a video discussing some of these topics.

 

 

 

So perhaps a better title would be "Why you can't trust audio measurements that are not well documented". Everything in the video speaks to the differences in setup and configuration of the test. As long as these are properly documented, there's no reason to not trust the measurements as they can then be reproduced by others. The only reason to question well-documented measurements is if someone else gets a significantly different result with the same configuration.

[pkane2001]

10 minutes ago, botrytis said:

 

Or if someone has such an expensive testing rig that no one else can afford....

 

$30k or so isn't out of the realm of the possible. Many audiophile components are way more expensive than that. If you really don't want to spend all that cash on an APx555, you could rent one for under $1k a day :)

 

 

 

[pkane2001]

3 minutes ago, botrytis said:

Well, Kippel rigs are 100K for the basic one and go up from there.

 

I don't have 30K IN ALL THE AUDIO EQUIPMENT I OWN. I could buy my next car for 30K - just me (mine is a 2007 and my wife's is a 2001). 

 

Forget that new car, or a new house, just get the Klippel analyzer ;)

 

[pkane2001]

1 minute ago, yamamoto2002 said:

In Japan, a Ferrari dealer sells Audio Precision products🙂

 

That's awesome! I'd recommend a package deal for both :)

 

[pkane2001]

12 minutes ago, botrytis said:

 

Yeah, I want to get kicked out of my house. My wife would say something, and it won't be pretty....

 

I hear you! But you have to have priorities in life. A house + wife vs Klipple… it’s a tough decision!

[pkane2001]

24 minutes ago, fas42 said:

52 minutes ago, botrytis said:

Human ears are poor quality, compared to dogs and other species. Humans are a vision and tactile centric species.

 

Yet a conductor in the storm of an orchestral crescendo drowning him in sound, from a flotilla of instruments feet away, all giving their all, can then point to a particular player in a section - and say, "You did that wrong!" ...

 

And no dog I know can do that! That proves it: human ears are better.

 

[pkane2001]

12 minutes ago, botrytis said:

 

You would be amazed. Some dogs yeowl or bark when hearing noise they don't like. Others don't care.

 

Some dogs are discriminating audiophiles, others are not.

[pkane2001]

21 minutes ago, Allan F said:

 

Not necessarily. You, incorrectly IMO, assume that there is a direct relationship between measurements and sound quality. Measurements may or may not explain the difference.

 

There is a direct relationship between measurements and transparency. Sound quality is a subjective metric and what I prefer may differ significantly from you or from anyone else. Using measurements designed to test for transparency to judge sound quality assumes that you prefer transparent playback. Not everyone does.

 

[pkane2001]

10 minutes ago, firedog said:

Not really. It's pretty easy.

 

Probably even easier if you insult the owner of the site and accuse him of improprieties, like @GoldenOne had done publicly. I assume he'll never be allowed back on ASR. But it would've been an interesting conversation about testing methodologies and blind/ABX testing. By the way, Amir does offer his APx project files to anyone who asks. Can't say if he'll share them with those who he has personal beef with, though.

 

[pkane2001]

40 minutes ago, JoshM said:

I don’t think this is quite fair. Amir and other ASR folks regularly accuse every other site of improprieties. If he can’t take it, he shouldn’t dish it out. Amir’s reviews were being reprinted in full by a Topping/SMSL distributor. Whether he knew it or not is up for debate. At the very least, he’s admitted that Topping sends him gear that he can keep for free, which is the type of thing that would be proof of bias for any other reviewer. Regardless of whether he needed to, @GoldenOne apologized and was still banned. 

 

There’s also the strange dynamic where Amir insists his subjective views on things like speakers are superior to others’ (and even superior to the measurements themselves), but everyone else’s measurements and subjective impressions are suspect, if not outright worthless. (His defenses of MQA were another odd tangent where measurements suddenly weren’t important.) So he clearly thinks subjective impressions matter…but only if they’re his. 
 

The crux of the issue, for me, is that measurements are neither everything nor nothing. They provide us with valuable information, but humans can make mistakes when conducting measurements, and there’s reasonable debate about which measurements truly matter, how to best present them, and how they relate to what we do (or don’t) hear. That’s what Golden was getting at, and I don’t quite understand Amir’s defensiveness at what seems like an undeniable, even banal, point. 

 

Josh,

 

If you're commenting on my post, then I don't think it's either fair or unfair, as I didn't make any value judgements. In fact, I thought the banning of GoldenOne on ASR was overdone and inappropriate at the time. But there is too much mud slinging still going on with accusations without evidence (from both sides). The intolerance and the inability to see things from the other side is, in my view, the crux of the bigger problem, not measurements vs subjective opinions.

 

 

[pkane2001]

3 hours ago, JoshM said:

 

That's fair. I read your response as, "It's not surprising given that he insulted Amir." My reaction was simply that Amir clearly doesn't apply the same standard to himself.

 

I think the issue is that much of what happens at ASR is "appeal to authority," rather than "science." This, for example, was just posted:

 

 

 

I don't know how anyone is supposed to react to that. As we've all seen during the pandemic, doctors disagree with one another. If I'd have listened to the doctor who once told me "We can't do anything about your pain besides give you opioids," I'd be in bad shape today. My mother had to diagnose herself with a rare disease that multiple doctors missed, likely saving her life. Others in that thread noted similar personal experiences with doctors, but Amir has no response to that.

 

There's a reason why appeals to authority are identified as a logical fallacy. Amir has justified MQA by saying "Well, are you Bob Stuart!?" over and over again. Yet simultaneously the expertise of someone like Mike Moffat is dismissed. However, if we're supposed to trust Stuart on MQA, why shouldn't we also trust Moffat on the Yggy? Appeal to authority is why Amir made such a big stink about @Archimago being anonymous. But either evidence speaks for itself, without reference to someone's name and CV, or we're not really dealing with science. 

 

There is, of course, some truth to the fact that how much you trust any evidence is at least partly based on the behavior and track record of the person presenting the evidence. We can't all replicate measurements at home because we can't all afford AP equipment (or have the time to conduct the measurements). But that's exactly one of @GoldenOne's points.

 

Ultimately, every ASR debate comes down to "Amir must be the unquestioned authority," which isn't really conducive to robust inquiry, scientific or otherwise. 

 

In my view, the more people doing measurements and checking each others' work, the better. But Amir's work is no more above criticism than anyone else's. (Indeed, based on track record, it's just as reasonable for someone to trust Amir less than someone like Arch or Golden!)

 

Josh,

 

Doctors may disagree with one another, but I still won't take advice to drink bleach to kill the virus from someone with no background in medicine. 

 

People  come to ASR to read Amir's opinion and then disagree with him and fight him on it. In reality, (and to most knowledgeable folks) Amir's opinion isn't interesting. His measurements are. And while there are those who criticise his measurements, their opinions are just as irrelevant as Amir's unless they can produce measurements that demonstrate that Amir borked his. Arguing about his "biases" or his "preferences" makes sense only if you give any weight to his opinion and subjective parts of his reviews. Personally, I don't, and have no reason to defend these. In fact, when ASR was just starting, the reviews only had measurements. Readers insisted that Amir give his opinion and his subjective reviews. So, now, here we are, discussing opinions which are not objective, not scientific, and unimportant.

 

I disagree that we are completely helpless and can't reproduce any of Amir's measurements at home without major investment. Investment in time and learning, yes, but not necessarily money. I've been doing this for years, and others have done, as well. It takes a little ingenuity and the desire to find things out for yourself. But then, there are many others that would rather be told what to think and what to like. While I can understand their motivation, such as lack of time, or lack of interest, I find it much more less acceptable to be told to drink bleach by someone -- I'd much rather have science and measurements presented to me to make up my own mind than to listen to an opinion of someone "in authority".

 

You can see people disagreeing with Amir in nearly every review thread. Few have facts, most have opinions. Of course, I'd also like to have more independent measurements. More real information, more facts and less opinions is a very desirable outcome to my mind. Which is why I said I was disappointed when GoldenOne was banned from ASR. But, I can also understand why Amir did it and it had nothing to do with measurements or objectivism.

 

 

 

[pkane2001]

1 hour ago, botrytis said:

Well, the way Amir dismissed and banned ELAC designer. Andrew Jones, speaks volume about his arrogance. Andrew Jones was nothing but a gentlemen. Jones told Amir that his measurements were done above the maximum SPL if the speakers. Also, he offered to look at the speaker and ship back to him on his coin. Amir was a ponce and banned him.

 

I was not aware of what happened. In any case, I'm less interested in Amir's personality, or his disagreements with others than I am in the measurements.

 

1 hour ago, opus101 said:

Whilst I agree with a large part of what you wrote @pkane2001 I disagree with this bit. The Andrew Jones example that @botrytis has just given shows that its not necessary for AJ to come up with his own measurements to show that Amir's are erroneous.

 

Sure, a mistake is a mistake and can often be spotted without duplicating the measurements. I apparently missed this whole thing with AJ. Can someone point me to the thread where this happened? This is not the first time this incident is mentioned.

 

[pkane2001]

37 minutes ago, opus101 said:

This post contains about as close as you'll get on AS to a link to ASR (down at the bottom in italics) : https://audiophilestyle.com/forums/topic/30381-mqa-is-vaporware/?do=findComment&comment=1137966

 

Thanks! Just checked it out. Possibly the thread was sanitized after the fact, but I found nothing inappropriate in what's there now of Amir's test. For one, he did all the Klipple measurements within the specs of the speaker, not exceeding any ratings. And as I said, that is the more interesting part to me, not his listening reports or his subjective ratings.

 

Amir's subjective listening to the speaker produced some noise/resonance from the cabinet. Maybe he exceeded the spec SPL, maybe not, this is not clear. AJ says he was able to reproduce the problem, but only near the maximum SPL range. From what I can tell, a few owners also reported the same problem. So, maybe it's not a huge issue, but it is good that it is documented. By the way, the test track that Amir claims caused this problem is a single female voice, singing in a quiet recording. To try to get 140 SPL out of the speaker on this track sounds like an insane thing to do. 

 

The measurements are still there for anyone to use or to dispute, but I don't see any disputes about these.

[pkane2001]

1 hour ago, botrytis said:

It was sanitized for sure. If you read the rules of that forum, one rule is like do not argue with the owner of the forum. No questioning is allowed.

 

I was on another forum, since 2003, and had about 32-35K posts. I was banned because one mod didn't like my posts. He banned me and also erased all of my posts. It was like I never existed. I was happy to walk away from that forum. It was another Audio forum.

 

Yeah, I think that's why people start their own forums. I gravitate to ASR not because it's perfect or because of Amir, but because of other technical folks that participate there. I love to experiment and tinker, and that's where I hear get to read others think and if there's something I could've done better. That's how I learn.

 

And yes, I've been banned from other audio forums before, also :) and this despite the fact that I own and moderate multiple forums with many thousands of members for another high-end, technical hobby. I've banned a total of a couple of people in the 10+ years on those other forums, primarily because they went haywire and couldn't stop with the personal insults and cursing out others, never because someone disagreed with me.

 

[pkane2001]

23 minutes ago, TheBatsEar said:

 

Ahoi, i just joined to tell you that is a false statement. Andrew Jones wasn't banned.

 

Stop it with posting facts! This is the Objective sub-forum, after all. We don't like facts here, only speculation ;)

[pkane2001]

45 minutes ago, Iving said:

https://forums.stevehoffman.tv/threads/just-got-banned-by-asr-for-pointing-out-the-inconvenience-truth.1144772/

 

The OP is not me btw!

 

Didn't realize he got banned! Too bad, he isn't a bad guy, just a bit annoying, posting the same things in nearly every thread, including the latest one he started, where he demanded that Amir prove him wrong by inviting him to his house for a blind test :)

 

I'm guessing that Amir didn't take to the idea, based on the above..

 

[pkane2001]

42 minutes ago, JoshM said:

I didn’t see the SHF thread before it got removed, but I’m guessing it’s the guy who recently posted on ASR that six of nine of the budget components recommended by Amir that the poster bought failed or measured differently from what Amir found. The poster raised the possibility that some companies were sending cherry-picked units, and requested to send his units to Amir to verify (since, as usual, people didn’t believe the poster, even though he did his own measurements, blind tests, etc.). In response, Amir said (contrary to his “we measure anything for free” claim), that the poster would have to pay him $200 per piece of equipment to confirm the issues, then locked the thread.  

 

What that all illustrated for me (and aligns with my own experiences with these brands) is that certain companies lauded by ASR have major failure rates if you search relevant threads (on Head-Fi, any retailer with user reviews, and sometimes ASR itself) for words like “fail,” “issue,” “problem,” or “died.” But anyone overtly raising the possibility that this is a systematic issue on ASR is shut down. 
 

Of course, when it comes to disfavored brands, like Schiit, Amir will measure multiple units to find faults, conduct teardowns to inspect for minor solder splatter, etc. Favored companies are, of course, spared this critical eye. It’s science, you know. 

 

Umm, no, not what happened. None of his components failed. Wayne did have one unit with a bad display from the start. He tried to prove that they all sounded different. His claim was that since they all measured well (by Amir) he wanted others to acknowledge that his hearing was detecting things that were beyond measurements. Meanwhile, he had problems with channel imbalances, equalizing levels, and at one point told me he wasn't interested in learning or understanding technology or measurements. This was after about a month of me trying to help him use DeltaWave properly (he failed). And no, I've not seen a single successful measurement from him, and he gave up trying after a while.

 

In the last thread he started, he insisted that Amir should organize a blind test for him to come over to Amir's house to prove to him that he can hear these differences. His one theory was that this may be due to "golden samples" sent to Amir by manufacturers. He had no measurements, and no real evidence to back up his claims, but he was very vocal and insistent about what he heard in other threads. Don't know what got him banned in the end.

 

[pkane2001]

22 minutes ago, Jud said:

 

Which for me raises the interesting question of to what extent veteran designers are able to engage in specmanship that doesn't have an audible effect.  I too have noticed the earlier slagging of Schiit and the later appearance of a couple of their units relatively high up on the Holy SINAD List.

 

Schiit invested in AP measurement equipment and started doing actual automated QA testing on their equipment before shipping, and also publishing these test reports. While some of their equipment still doesn't measure well, they are one of only a few manufacturers that publish a complete set of measurements. Good for them, regardless of where they wind up on the SINAD chart :)

[pkane2001]

4 minutes ago, JoshM said:

I can only go by what he said, where he referenced measuring his units and conducting blind tests. I see you were trying to help him some in another thread.
 

My read of the thread is that he assumed (correctly!) that no one would believe him about his measurements or blind tests. (Here we’re back to blind tests by others being dismissed. A “no true Scotsman” issue.) So he asked if Amir would assist with blind tests or measure his units.

 

Amir then requested $200 per unit for measurements. Then a mod locked the thread and (apparently) banned him. From what I read, nothing the original poster said was mean or offensive.

I have no reason to double the original poster’s claims. Nor do I know them to be true. But shutting him down for daring to raise the possibility that certain brands might have reliability issues that don’t get caught by ASR’s reviews seems to be what raised hackles. 
 

The overarching issue in my view is that not all products over there are subjected to the same measurements (as @GoldenOne noted) or the same critical analysis when it comes to construction and reliability. 
 

I’ll say that personally (starting before ASR even existed), I’ve owned five products by the two most-praised brands at ASR. Three failed. One literally sparked when plugged in. Threads and reviews elsewhere indicate such experiences aren’t uncommon. Meanwhile, solder spatter on other brands is labeled a reliability or safety risk. It’s just not ethical reviewing practices, which requires that all products are subjected to roughly the same analysis. I realize that lots of journalistic ethics are out the window in the internet age, but they still matter to me. 

 

It's important to know what measurements are and what they are good for. Certainly, poor measurements can point to poor design and/or poor engineering. Or, they may demonstrate intentional deviation from transparency.

 

Good measurements are no guarantee of reliability and hour-long test of a DAC or other component isn't going to reveal longer-term issues with the design. GoldenOne or Amir are not testing for this. Reliability testing is a much more complicated and involved process and is better done by test labs with dedicated people and equipment over a much longer period of time, with the ability to vary temperature, humidity, mechanical vibration, simulating various source and downstream component, and accounting for many other factors. Or you can just read the posts and complaints by existing owners :)

[pkane2001]

10 minutes ago, JoshM said:

I agree. I think it would be perfectly fine and ethical to say, “I’m not mentioning warranties, or reliability, or conducting teardowns. Just the same suite of measurements. Caveat emptor.” But I don’t think it’s ethical to conduct teardowns and speculate about reliability for some reviews, but not others. 

 

It would be unethical for him to conduct a teardown of a product that the owner doesn't want torn down :) Where permission is given, Amir usually does and posts a separate thread for these. 

[pkane2001]

17 minutes ago, GoldenOne said:

I've had a couple products measure differently from the measurements amir showed. Not necessarily by a huge degree but enough that if amir were to put the unit I'd received on his ranking list it'd be several spots lower.

 

This was brought up here before, but I'm not sure what the answer is. Are your measurements done with the same AP project settings as those that Amir is using? If not, there could be differences in settings (as per your video) that could account for some of the differences.

[pkane2001]

3 minutes ago, GoldenOne said:

It depends. Amir has not shared his configuration publicly and he's hardly going to give it to me I imagine so I have no idea if it's the same or not.

 

But for the basic bench mode 1khz test where most of the settings can be pretty much assumed and the bandwidth is shown, I can match his config for that specific one. And in that situation I've had some devices which did not perform the same.

 

And to be clear that's not any 'fault' of amir. He's measuring the device given to him.

The issue is that some manufacturers may cherry pick and there is no way for amir (or viewers of his site) to know if that's happening or not

 

Yeah, the 1kHz dashboard, while revealing, isn't painting the whole picture. I much prefer sweeps with varying levels and frequency showing noise, THD/IMD, linearity. With those tests, there are some parameters that can differ significantly without being visible on the AP chart, that's why I'm asking. 

 

 

[pkane2001]

3 minutes ago, GoldenOne said:

On this note I'm quite curious to see internals of the new topping LA90.

 

Afaik topping hasn't done a discrete amp before so it'll be interesting to see what's inside this.

 

But whilst topping usually has at least partial internal views in their marketing material and full internal shots available elsewhere, there are no views of the insides available and they've seemingly asked Amir not to open it.....

 

 

Topping seems very concerned about competition copying their designs. Lately they started to scratch off all chip/component labels from every PCB before shipping. Looks kinda ugly. Wonder what they did in a discrete design LA90 to obscure it :)

 

[pkane2001]

5 minutes ago, fas42 said:

It has been repeated an infinite number of times ... personal experience, 35 years ago, made this obvious. 

 

You're right, Frank! You've repeated this an infinite number of times 😉 

 

[pkane2001]

5 hours ago, opus101 said:

Nah its definitely only a finite number. It was only five years ago that that number he quoted was 30.

You’ll need to account for the number of messages per year, month, week, day, hour, second, millisecond, etc. As far as infinities go, what we're dealing with here might be one of those larger, uncountable infinities, as per Georg Cantor :)