Why you can't trust measurements

[Josh Mound]

On 4/23/2022 at 9:03 AM, pkane2001 said:

 

Probably even easier if you insult the owner of the site and accuse him of improprieties, like @GoldenOne had done publicly. I assume he'll never be allowed back on ASR. But it would've been an interesting conversation about testing methodologies and blind/ABX testing. By the way, Amir does offer his APx project files to anyone who asks. Can't say if he'll share them with those who he has personal beef with, though.

 

I don’t think this is quite fair. Amir and other ASR folks regularly accuse every other site of improprieties. If he can’t take it, he shouldn’t dish it out. Amir’s reviews were being reprinted in full by a Topping/SMSL distributor. Whether he knew it or not is up for debate. At the very least, he’s admitted that Topping sends him gear that he can keep for free, which is the type of thing that would be proof of bias for any other reviewer. Regardless of whether he needed to, @GoldenOne apologized and was still banned. 

 

There’s also the strange dynamic where Amir insists his subjective views on things like speakers are superior to others’ (and even superior to the measurements themselves), but everyone else’s measurements and subjective impressions are suspect, if not outright worthless. (His defenses of MQA were another odd tangent where measurements suddenly weren’t important.) So he clearly thinks subjective impressions matter…but only if they’re his. 
 

The crux of the issue, for me, is that measurements are neither everything nor nothing. They provide us with valuable information, but humans can make mistakes when conducting measurements, and there’s reasonable debate about which measurements truly matter, how to best present them, and how they relate to what we do (or don’t) hear. That’s what Golden was getting at, and I don’t quite understand Amir’s defensiveness at what seems like an undeniable, even banal, point. 

[Josh Mound]

1 hour ago, pkane2001 said:

 

Josh,

 

If you're commenting on my post, then I don't think it's either fair or unfair, as I didn't make any value judgements. In fact, I thought the banning of GoldenOne on ASR was overdone and inappropriate at the time. But there is too much mud slinging still going on with accusations without evidence (from both sides). The intolerance and the inability to see things from the other side is, in my view, the crux of the bigger problem, not measurements vs subjective opinions.

 

 

That's fair. I read your response as, "It's not surprising given that he insulted Amir." My reaction was simply that Amir clearly doesn't apply the same standard to himself.

 

I think the issue is that much of what happens at ASR is "appeal to authority," rather than "science." This, for example, was just posted:

 

 

 

I don't know how anyone is supposed to react to that. As we've all seen during the pandemic, doctors disagree with one another. If I'd have listened to the doctor who once told me "We can't do anything about your pain besides give you opioids," I'd be in bad shape today. My mother had to diagnose herself with a rare disease that multiple doctors missed, likely saving her life. Others in that thread noted similar personal experiences with doctors, but Amir has no response to that.

 

There's a reason why appeals to authority are identified as a logical fallacy. Amir has justified MQA by saying "Well, are you Bob Stuart!?" over and over again. Yet simultaneously the expertise of someone like Mike Moffat is dismissed. However, if we're supposed to trust Stuart on MQA, why shouldn't we also trust Moffat on the Yggy? Appeal to authority is why Amir made such a big stink about @Archimago being anonymous. But either evidence speaks for itself, without reference to someone's name and CV, or we're not really dealing with science. 

 

There is, of course, some truth to the fact that how much you trust any evidence is at least partly based on the behavior and track record of the person presenting the evidence. We can't all replicate measurements at home because we can't all afford AP equipment (or have the time to conduct the measurements). But that's exactly one of @GoldenOne's points.

 

Ultimately, every ASR debate comes down to "Amir must be the unquestioned authority," which isn't really conducive to robust inquiry, scientific or otherwise. 

 

In my view, the more people doing measurements and checking each others' work, the better. But Amir's work is no more above criticism than anyone else's. (Indeed, based on track record, it's just as reasonable for someone to trust Amir less than someone like Arch or Golden!)

[Josh Mound]

2 hours ago, pkane2001 said:

 

Josh,

 

Doctors may disagree with one another, but I still won't take advice to drink bleach to kill the virus from someone with no background in medicine. 

 

People  come to ASR to read Amir's opinion and then disagree with him and fight him on it. In reality, (and to most knowledgeable folks) Amir's opinion isn't interesting. His measurements are. And while there are those who criticise his measurements, their opinions are just as irrelevant as Amir's unless they can produce measurements that demonstrate that Amir borked his. Arguing about his "biases" or his "preferences" makes sense only if you give any weight to his opinion and subjective parts of his reviews. Personally, I don't, and have no reason to defend these. In fact, when ASR was just starting, the reviews only had measurements. Readers insisted that Amir give his opinion and his subjective reviews. So, now, here we are, discussing opinions which are not objective, not scientific, and unimportant.

 

I disagree that we are completely helpless and can't reproduce any of Amir's measurements at home without major investment. Investment in time and learning, yes, but not necessarily money. I've been doing this for years, and others have done, as well. It takes a little ingenuity and the desire to find things out for yourself. But then, there are many others that would rather be told what to think and what to like. While I can understand their motivation, such as lack of time, or lack of interest, I find it much more less acceptable to be told to drink bleach by someone -- I'd much rather have science and measurements presented to me to make up my own mind than to listen to an opinion of someone "in authority".

 

You can see people disagreeing with Amir in nearly every review thread. Few have facts, most have opinions. Of course, I'd also like to have more independent measurements. More real information, more facts and less opinions is a very desirable outcome to my mind. Which is why I said I was disappointed when GoldenOne was banned from ASR. But, I can also understand why Amir did it and it had nothing to do with measurements or objectivism.

 

People can do some measurements at home. But if the response is, “The APx555 is better” or “You can’t trust [Jude/Marv/Bob/Arch/Golden…]’s measurements, only mine,” then there’s not much left to say. 
 

There have, of course, been instances when people have produced measurement that conflicted with Amir’s, and it hasn’t mattered. The (in)famous Yggy “droop” comes to mind. Multiple people measured Yggys and didn’t find it. (I had one at the time and didn’t find it myself.) There have also been much less flattering measurements of various Topping products elsewhere. Was this borking, unit-to-unit variation, something else? I have no idea.
 

Andrew Jones was already mentioned, and that’s an example that points to the subjective decisions involved in conducting a measurement, such as, “What’s the right SPL to highlight?”

 

Like you, I totally discount his subjective views. But he does go to great lengths to argue that his listening impressions are better than others’. I find this silly, particularly given that according to his own account he didn’t score very well on Harman’s How to Listen. That said, it does at least highlight that, in reality, some people are (due to hearing ability, training, or both) better listeners than others. That’s a whole other can of worms, though, which raises lots of interesting questions.

 

While it’s only in the past few years that Amir has given explicit reviews, from the beginning he passed judgment on (and insulted) companies and products. For example, his 2017 Schiit Modi 2 review said, “Either they don't measure their DACs, don't care or don't know how to design a clean DAC. Yes, -85 db distortion products is not likely audible and hence the reason they get away with such poor performance. But the engineer in me wants to throw up on it.”
 

Besides the language involved, there are all kinds of claims in there related to what’s audible and why we should still care about something that supposedly isn’t audible. It’s that tone and condescension that rub many people the wrong way, not measurements per se. This is why it’s bad that Amir (and some of his critics) conflate his work with measurements, objectivism, or any other label. It muddies what we’re actually talking about, and creates a situation where someone like Golden can’t make his wide-ranging video without Amir taking it as a personal attack.

[Josh Mound]

19 hours ago, fas42 said:

 

What's going on ... that thread is no longer there?

I didn’t see the SHF thread before it got removed, but I’m guessing it’s the guy who recently posted on ASR that six of nine of the budget components recommended by Amir that the poster bought failed or measured differently from what Amir found. The poster raised the possibility that some companies were sending cherry-picked units, and requested to send his units to Amir to verify (since, as usual, people didn’t believe the poster, even though he did his own measurements, blind tests, etc.). In response, Amir said (contrary to his “we measure anything for free” claim), that the poster would have to pay him $200 per piece of equipment to confirm the issues, then locked the thread.  

 

What that all illustrated for me (and aligns with my own experiences with these brands) is that certain companies lauded by ASR have major failure rates if you search relevant threads (on Head-Fi, any retailer with user reviews, and sometimes ASR itself) for words like “fail,” “issue,” “problem,” or “died.” But anyone overtly raising the possibility that this is a systematic issue on ASR is shut down. 
 

Of course, when it comes to disfavored brands, like Schiit, Amir will measure multiple units to find faults, conduct teardowns to inspect for minor solder splatter, etc. Favored companies are, of course, spared this critical eye. It’s science, you know. 

[Josh Mound]

1 hour ago, firedog said:

Recently he's found several Schiit models that measure well, and has praised Schiit.

It would be impossible to get around them measuring well. But, as I said, he’s done teardowns on some of those Schiit products and nitpicked solder spatter, claiming without evidence that it could cause failures. Meanwhile, products from other companies not only aren’t subjected to teardowns, but well-documented issues with failures and other issues are dismissed. 

[Josh Mound]

43 minutes ago, pkane2001 said:

 

Umm, no, not what happened. None of his components failed. Wayne did have one unit with a bad display from the start. He tried to prove that they all sounded different. His claim was that since they all measured well (by Amir) he wanted others to acknowledge that his hearing was detecting things that were beyond measurements. Meanwhile, he had problems with channel imbalances, equalizing levels, and at one point told me he wasn't interested in learning or understanding technology or measurements. This was after about a month of me trying to help him use DeltaWave properly (he failed). And no, I've not seen a single successful measurement from him, and he gave up trying after a while.

 

In the last thread he started, he insisted that Amir should organize a blind test for him to come over to Amir's house to prove to him that he can hear these differences. His one theory was that this may be due to "golden samples" sent to Amir by manufacturers. He had no measurements, and no real evidence to back up his claims, but he was very vocal and insistent about what he heard in other threads. Don't know what got him banned in the end.

 

I can only go by what he said, where he referenced measuring his units and conducting blind tests. I see you were trying to help him some in another thread.
 

My read of the thread is that he assumed (correctly!) that no one would believe him about his measurements or blind tests. (Here we’re back to blind tests by others being dismissed. A “no true Scotsman” issue.) So he asked if Amir would assist with blind tests or measure his units.

 

Amir then requested $200 per unit for measurements. Then a mod locked the thread and (apparently) banned him. From what I read, nothing the original poster said was mean or offensive.

I have no reason to doubt the original poster’s claims. Nor do I know them to be true. But shutting him down for daring to raise the possibility that certain brands might have reliability issues that don’t get caught by ASR’s reviews seems to be what raised hackles. 
 

The overarching issue in my view is that not all products over there are subjected to the same measurements (as @GoldenOne noted) or the same critical analysis when it comes to construction and reliability. 
 

I’ll say that personally (starting before ASR even existed), I’ve owned five products by the two most-praised brands at ASR. Three failed. One literally sparked when plugged in. Threads and reviews elsewhere indicate such experiences aren’t uncommon. Meanwhile, solder spatter on other brands is labeled a reliability or safety risk. It’s just not ethical reviewing practices, which requires that all products are subjected to roughly the same analysis. I realize that lots of journalistic ethics are out the window in the internet age, but they still matter to me. 

[Josh Mound]

52 minutes ago, Jud said:

 

Which for me raises the interesting question of to what extent veteran designers are able to engage in specmanship that doesn't have an audible effect.  I too have noticed the earlier slagging of Schiit and the later appearance of a couple of their units relatively high up on the Holy SINAD List.

I think this comes down to “incentives matter.” ASR helped cultivate a market for budget SINAD-chasing DACs and headamps. That wasn’t Schiit’s initial goal, but they realized they could easily make both that would retail for ~$100 each and have better warranties than similarly-priced competitors. It was a smart business decision, and I tend to tell people who are measurements-focused to just buy the budget Schiit stack and be done with their audio journey. 

[Josh Mound]

5 minutes ago, pkane2001 said:

 

It's important to know what measurements are and what they are good for. Certainly, poor measurements can point to poor design and/or poor engineering. Or, they may demonstrate intentional deviation from transparency.

 

Good measurements are no guarantee of reliability and hour-long test of a DAC or other component isn't going to reveal longer-term issues with the design. GoldenOne or Amir are not testing for this. Reliability testing is a much more complicated and involved process and is better done by test labs with dedicated people and equipment over a much longer period of time, with the ability to vary temperature, humidity, mechanical vibration, simulating various source and downstream component, and accounting for many other factors. Or you can just read the posts and complaints by existing owners :)

I agree. I think it would be perfectly fine and ethical to say, “I’m not mentioning warranties, or reliability, or conducting teardowns. Just the same suite of measurements. Caveat emptor.” But I don’t think it’s ethical to conduct teardowns and speculate about reliability for some reviews, but not others. 

[Josh Mound]

5 minutes ago, GoldenOne said:

I've had a couple products measure differently from the measurements amir showed. Not necessarily by a huge degree but enough that if amir were to put the unit I'd received on his ranking list it'd be several spots lower.

 

Unfortunately with Amir's position it would be silly to assume that manufacturers WERENT sending cherry picked units.

 

Realistically a fairer way to do things would be for Amir to ask the manufacturer to reimburse him for the purchase of a retail unit from a distributor. This would ensure that he's testing something that will be in consumer hands. Not a cherry picked unit.

 

In regards to failure of devices though. Unfortunately reliability is measured best by user reports. It's not really something Amir can test so that's not really fair to put on him. 

A product can measure well but die 2 months later.

 

Also, on the note of having to pay to get stuff measured, if anyone would like anything measured I'll do a full set for free on my APx555. Just cover shipping.

Totally agree. We really need some comprehensive surveys of owners about reliability by brand, product, or ideally both.

[Josh Mound]

20 minutes ago, pkane2001 said:

 

It would be unethical for him to conduct a teardown of a product that the owner doesn't want torn down :) Where permission is given, Amir usually does and posts a separate thread for these. 

I think it’s unfair to manufacturers for some, but not others, to receive that kind of scrutiny. (He’s noted that Topping, for example, gives him products for free without asking it be returned, but has not done teardowns on all of those products, so it’s clearly not solely a permission issue.) If not all people who submit products agree to teardowns, he can either not do teardowns at all or purchase units himself.

 

I’ll add that — even as a proud anti-corporate pinko who’s a big “consumer movement” Naderite — I still recognize that people’s livelihoods, no matter how big the company, are impacted by reviews. So bending over backwards to be fair to all companies only seems right to me. The zeal to try to help consumers, which I share, doesn’t obviate that.

[Josh Mound]

1 minute ago, pkane2001 said:

 

Yeah, the 1kHz dashboard, while revealing, isn't painting the whole picture. I much prefer sweeps with varying levels and frequency showing noise, THD/IMD, linearity. With those tests, there are some parameters that can differ significantly without being visible on the AP chart, that's why I'm asking. 

 

 

I suspect I’ll be hitting you up for some advice on best practices using your Multitone with the Cosmos ADC. I have some vintage DACs I want to write up.

[Josh Mound]

On 5/5/2022 at 6:38 PM, JoshM said:

I agree. I think it would be perfectly fine and ethical to say, “I’m not mentioning warranties, or reliability, or conducting teardowns. Just the same suite of measurements. Caveat emptor.” But I don’t think it’s ethical to conduct teardowns and speculate about reliability for some reviews, but not others. 

Apropos of the previous discussion about reliability, I happened upon this exchange while perusing a review over at ASR. This poster’s experience is very similar to (though worse than) mine.

 

 

Here’s Amir’s response:

 

[Josh Mound]

On 5/20/2022 at 6:54 AM, firedog said:

I have no data to back this up, but: I think Amir's measurements of devices under $1000 and under $500 is pretty influential. My impression is that there's a serious segment of people who only want to spend a few hundred on an amp or DAC and since it's often impossible to audition units anyway, they decide to buy units that measure exceptionally well, as their first criterion.

 

It also seems like some manufacturers are responding to this, by making units that have great measurements and by publishing those measurements. Schiit and some of the better known Chinese brands come to mind. This wasn't a "thing" a few years ago.

 

I also bought a Schiit Modi 3E for this reason. Measured well(not by Amir) , had the features I wanted, and was cheap. I'm very happy with it. I wasn't going to go to the trouble of auditioning multiple items for a $125 outlay. I'm sure many others have a similar POV. It btw, didn't embarass itself when I connected it to my expensive system.

 

The recent extremely positive reviews of the Topping DAC (Absolute Sound, subjective only) and and Pre (Stereophile, subjective plus measurements) in which there was agreement that the two under $1000 pieces were as good SQ wise (and for the pre, measurement wise) as pretty much anything on the market in their segment is telling: Even the mainstream publications are feeling the pressure from the market to review such relatively inexpensive devices that measure as state of the art. Once upon a time, neither of those publications would review such products, as they don't have dealer networks and are only available online.

I think this is all correct. What I find puzzling (even concerning) is that many of these folks seem to be constantly “upgrading” their DACs to more expensive models, despite already owning “perfect” measuring DACs. In that sense, I don’t quite get the brand of objectivism at play. Indeed, it looks a lot like the old subjectivist FOMO/FOTM pattern, and certain ASR-favored brands seem to be fueling this by constantly introducing new models.  

 

The early Topping DACs reviewed by ASR were generally $100 to $200 and measured “perfectly,” but perhaps lacked reliability and customer service. Now, however, many of those models are out of production, and there are new Topping DACs in the $500 to $900 range. 
 

But what’s the point, from a measurements-first perspective? Something like the Modi 3 is $125, has “perfect” measurements, and has good reliability and customer support. I can’t see someone who’s a hard-core measurements-only objectivist doing anything other than buying a Modi 3 and being done with DACs forever. 

[Josh Mound]

Also, relevant to @GoldenOne’s video, it thought Bob’s illustration of how amplifier distortion interacts with level was fascinating:

 

https://www.superbestaudiofriends.org/index.php?threads/tutorial-amplifier-distortion-vs-amplitude-a-visual-guide.12279/

 

It’s worth pondering what level is most realistic for one’s use of a product, and it might not be the level at which most measurements are performed. 

[Josh Mound]

On 5/24/2022 at 3:35 AM, firedog said:

Can't say it really bothers me. Don't disagree with Amir. If I order a Chinese DAC from AliExpress, I'm taking into account that I am unlikely to get warranty service for it in a reasonable or cost effective manner. I'd assume that people buying these devices are taking that into account. But if I buy a Topping DAC that's supposed to be "state of the art" and "reference quality" for $700, then I'm figuring that if  it's defective, I can get another and I still won't have spent as much as I would have on a much more expensive 
DAC from a "conventional" supplier. Same for if I don't agree after purchase that it is truly state of the art. I took a relatively small monetary risk that's supposed to give me a non-proportional sonic payoff. If I don't really get that, I haven't lost a significant amount relative to buying something much higher priced with warranty and service. 

 

If you aren't willing to take those risks, you shouldn't be buying such products.

 

And when Stereophile and TAS recommend those same products, I don't think they are standing behind them any more than Amir is. 

In terms of ASR’s practices, this gets back to some products receiving teardowns, but not others. Or some manufacturers getting scrutinized for reliability, but not others. Consistency matters in reviews, both in fairness to companies and to provide the consumer with actionable info.
 

In terms of the general assumption that people who are buying devices based on rave “objectivist” reviews know what they’re getting into, I’m not so sure. If a casual audiophile keeps reading that something is “state of the art,” I’m guessing they don’t think it’s likely to be a lemon. By definition, reliability and service are part of a quality product. I’m not calling on reviewers to conduct extensive reliability tests (though a Consumer Reports or Wirecutter for the audio would be awesome), but things like warranty and customer service should get at least some consideration in a review. 
 

IMHO, our disposable culture is bad. I’d rather spend more for a product that will last. That said, it was somewhat defensible to say “caveat emptor” when the product was $100. But now that some of the products in question are $500-$800, I don’t think that’s tenable. That’s a lot of money for most people. More importantly, that’s why I brought up the Schiit Modi 3 in my first post. To my mind, any “objectivist” recommending a $500-$800 DAC with a functionally nonexistent warranty over a Modi 3 (or Modius if balanced is preferred) when the Schiit products are cheaper, have better warranties and service, and also measure “perfectly” isn’t really providing their readers with the best information. 

[Josh Mound]

15 hours ago, GoldenOne said:

This is something that I personally feel is very important. For various reasons.

Firstly, I've noticed that some of the devices with stronger 'House Sounds' including DACs such as the Schiit Yggdrasil, show quite drastic changes in distortion level and profile in relation to amplitude.

 

In addition to this, some devices have slightly different distortion at their very uppermost output levels than they do in the rest of the output range.
The Gustard X26 pro for example, which is an excellent DAC, does seem to be somewhat 'optimising' for full scale output to look better on ASR-style tests.

 

You can see that distortion drops a few dB in the top ~6dB.
And in fact even when not limited by THD itself, you can visually see how the harmonics change.

At full scale 0dBfs output it looks like this:

All harmonics at or below about -140dB

But then take it to -3dBfs output and suddenly the 3rd order harmonic jumps up almost 15dB.

 

But ignoring DACs and just looking at amps, as has been mentioned, we don't listen to amps at 4v output. In fact on the vast majority of headphones that level would probably destroy your ears. So it's much more apt to test the device in the range that you're likely to use it.
Personally I offer measurements at 4v (to compare to ASR if people wish), 700mV (as this is a reasonably loud listening level on a few over ear headphones like an Arya) and 50mV (as this is appropriate for IEMs).

If you test something like an HE9 at 4V on 32 Ohm, you'll get about 0.3% THD+N (50dB SINAD). 
But if you test at a more realistic 700mV, you get a reduction to 0.02% THD+N (23dB increase to 73dB SINAD)

It seems silly to me to only test things at 4v as it's not realistic for amps in most cases.
But more to the point, on both DACs and Amps THD+N vs Level behaviour can reveal some pretty interesting stuff

 

I don't find reports on headamps at what will be ear-damaging levels for 99% of headphones to be very helpful, either. 

 

As far as DACs, I wonder if such issues at least partly account for differing subjective opinions. If one's listening to mostly clipped, "loudness wars"-era files, their impression of a DAC might be very different from someone listening to more dynamic, softer masterings, particularly if odd-order harmonics creep into audible territory at some levels but not others. IOTW, the original "Californication" CD isn't likely to sound the same as the original "Peter Gabriel IV" CD on DACs with distortion profiles that vary significantly by level. 

[Josh Mound]

Following up on the earlier discussion of reliability and review ethics:

 

I’m having a hard time seeing how a 30% failure rate for the ~140 Topping PA5 units bought by ASR member hasn’t caused Amir to update his review of the PA5, in which he raved about its superb engineering. Buyers have reported that the company has asked them to disassemble and fix the amp themselves and generally given them the runaround. Amir’s response has been to defend the company and assure ASR readers that it will fix things. Meanwhile, people are reporting that their replacement amps are failing, too. 
 

“It is abundantly clear that ton of great engineering has gone into the design of Topping PA5.” 🤭

 

 

 

 

 

 

[Josh Mound]

On 7/31/2022 at 7:09 PM, fas42 said:

Enthusiastic audiophiles are seen by most to be in the same category as model train buffs - a peculiar bunch of people who spend lots of money on something that is of little relevance to anything else in life, apart from the pleasure of fiddling with things. No-one spends any time, energy, or money analysing the bigger picture in model trains - and the same applies in audio.

Neil Young would like a word!

 

https://lemelson.mit.edu/resources/neil-young