Why you can't trust measurements

[firedog]

19 hours ago, Speedskater said:

But it takes a lot of misbehavior to get banned from Audio Science Review forum.

Not really. It's pretty easy.

[firedog]

1 hour ago, GoldenOne said:

This video isn't about ASR

I did however post in the thread they have going on there about this video. It was instantly removed of course but there's an archive.org page here if anyone would like to read it:

https://web.archive.org/web/20220421140602/https://www.audio “science” review/forum/index.php?threads/why-you-cant-trust-audio-measurements-by-goldensound.33222/page-4

(You must remove the spaces/quotes from 'audio “science” review'. Audiophilestyle automatically puts them in so the link doesn't work)

Link doesn't work archive says there's no such page-even when it lists all the archived pages from the site.

[firedog]

14 minutes ago, JoshM said:

Of course, when it comes to disfavored brands, like Schiit

Recently he's found several Schiit models that measure well, and has praised Schiit.

[firedog]

On 5/18/2022 at 10:55 PM, Iving said:

 

I know this full well.

 

You may recall I supported your position throughout that difficult "gauntlet" episode. See my record on ASR.

 

I meant to suggest - lightheartedly only - that you would be aware of the play of budget equipment measuring "well" on ASR - and that Schiit were part of that equation.

 

I favour a Mani phono over a Cambridge one. Nothing to do with measurements.

I have no data to back this up, but: I think Amir's measurements of devices under $1000 and under $500 is pretty influential. My impression is that there's a serious segment of people who only want to spend a few hundred on an amp or DAC and since it's often impossible to audition units anyway, they decide to buy units that measure exceptionally well, as their first criterion.

 

It also seems like some manufacturers are responding to this, by making units that have great measurements and by publishing those measurements. Schiit and some of the better known Chinese brands come to mind. This wasn't a "thing" a few years ago.

 

I also bought a Schiit Modi 3E for this reason. Measured well(not by Amir) , had the features I wanted, and was cheap. I'm very happy with it. I wasn't going to go to the trouble of auditioning multiple items for a $125 outlay. I'm sure many others have a similar POV. It btw, didn't embarass itself when I connected it to my expensive system.

 

The recent extremely positive reviews of the Topping DAC (Absolute Sound, subjective only) and and Pre (Stereophile, subjective plus measurements) in which there was agreement that the two under $1000 pieces were as good SQ wise (and for the pre, measurement wise) as pretty much anything on the market in their segment is telling: Even the mainstream publications are feeling the pressure from the market to review such relatively inexpensive devices that measure as state of the art. Once upon a time, neither of those publications would review such products, as they don't have dealer networks and are only available online.

[firedog]

1 hour ago, PeterG said:

 

I found the Topping DAC review deeply disappointing, even disturbing, from a publication I have often relied upon.  They did indeed assert the SQ was "reference level".  But then the only comparison offered was to a $600 DAC, and then concluded that you might go either way.  So these are two world-beaters at less than $1000?  It also bothered me that they repeated the no euphonics chant numerous times, almost as if to hedge their assertion--I am not aware of any euphoric reference level DACs.  This was very different than a few years ago when a number of publications asserted the Yggy was better than anything up to $X in price or something like that--it was much less credible.

 

Similarly, the Absolute Sound rave review of the B&W 805 D4 seemed to evade the near universal question around the 800 series--too much treble?

 

I'm not big on measurements in general, but they sure would have helped ground both of these discussions

 I think that yes, Steven Stone was basically saying the sound was reference level. The impressive thing to me was that he did blind testing of the two DACs, and he also used his own master recordings of acoustic music, which he obviously knows very well. He said they sounded exactly as they should. I think that's a pretty impressive recommendation. I haven't heard the Topping DAC, but if I was in the market to spend that kind of money and couldn't audition, that to me would be a serious recommendation.

 

The pre and DAC have measurements elsewhere (ASR, Stereophile, and a couple of other places).

[firedog]

7 hours ago, PeterG said:

Stone provided virtually no substantiation for his assertion that the Topping is reference level.  As you note, there were no measurements. (I'm OK with that)  Worse, there were no meaningful comparisons--a $600 Gustard?  Please, be serious.  Who's even heard of Gustard?  Also--"sounded exactly as they should"--what does that even mean?  Compared to what?

You are arguing with yourself. Of course he didn't - it's a subjective review, just like 99% of audio reviews. Gustard is a pretty well known brand, with some highly regarded PRO and consumer devices. That YOU haven't heard of them doesn't mean much. It's common to compare similar items in a hi-fi review. That's what Stone did. 

 

Meaningful comparisons? What do think is meaningful? He compared it to another under $1000 DAC under blind conditions - rarely done in the audiophile reviewing world.

 

He played back his own master recordings. "Sounded exactly as they should" - the meaning is totally clear, except if you are making a special effort not to understand the sentence. If you make your own recording, you know exactly how it "should" sound. And he said the DAC did exactly that level of reproduction. Maybe he should have phrased it differently so it would be phrased the way you'd like it to be.

 

In the end, it's a subjective review. If you think it's worthless, don't read it. IMO it was more substantial and more seriously done than virtually all other subjective reviews of audio equipment we commonly see. 

[firedog]

3 hours ago, JoshM said:

I think this is all correct. What I find puzzling (even concerning) is that many of these folks seem to be constantly “upgrading” their DACs to more expensive models, despite already owning “perfect” measuring DACs. In that sense, I don’t quite get the brand of objectivism at play. Indeed, it looks a lot like the old subjectivist FOMO/FOTM pattern, and certain ASR-favored brands seem to be fueling this by constantly introducing new models.  

 

The early Topping DACs reviewed by ASR were generally $100 to $200 and measured “perfectly,” but perhaps lacked reliability and customer service. Now, however, many of those models are out of production, and there are new Topping DACs in the $500 to $900 range. 
 

But what’s the point, from a measurements-first perspective? Something like the Modi 3 is $125, has “perfect” measurements, and has good reliability and customer support. I can’t see someone who’s a hard-core measurements-only objectivist doing anything other than buying a Modi 3 and being done with DACs forever. 

At ASR they have difficulty when you ask them why the measurements matter when they are beyond what is considered "audibility". Does my Chinese DAC with Sinad of 113 or 114 sound as good as their "state of the art" DACs with SINAD of 121? And does it sound better than a DAC with SINAD of 109?

 

Some of them will say all DACs with measurements past a certain threshold sound the same. But others will buy the best measuring DAC they can afford, even if it is well above that threshold. It's either: a version of FOMO; a kind of obsession with the numbers; or (and some of them acknowledge this) buying something that they consider to have the best engineering, as shown by the best measurements. 

On 5/21/2022 at 5:49 PM, Allan F said:

Re "sonic differences that cannot be measured", I believe that he means that listening to its sound may result in specific changes to a circuit's design that are not suggested directly by the DAC's measurements themselves.

 

I admit the possibility that this exists. What would be interesting to hear from manufacturers that measure, then listen, and then "adjust by ear" - what they think is happening when they reconfigure a device, the measurements stay the same, yet they think it sounds different?  And I'd be interested to know if they blind test their two versions of the component, or is it possibly self induced confirmation bias?

 

Bruno Putzeys has said (at least as far as amps) that very extensive measurements ( I believe he referred to about 20 different ones he conducts, some of which put the amp in extreme situations) of the right type WILL predict amp performance and sound. Sort of not just what you measure, but how you measure it. He said when he gets those right, he knows the amp will sound good and the way he intended. He also said he can color any amp to sound as he wishes. As an  example, he said he could give the MacIntosh sound to Class D - if he wanted to.

 

My takeaway from this is that perhaps Amir and some others aren't doing all the right measurements, not that the right ones don't exist yet. If I was Bruno, and I'd figured this out, I certainly wouldn't make it public knowledge, it has great proprietary value. 

[firedog]

1 minute ago, Windows X said:

 

He really posted that? Wasn't the point in his website is about recommending budget products with good measurements to his readers? If products have a lot of reports about defects, he should put his priority on readers' interests first. This made me wonder if it's either his pride / ego of being right or he has some companies invested on his website and need to keep that up.

Can't say it really bothers me. Don't disagree with Amir. If I order a Chinese DAC from AliExpress, I'm taking into account that I am unlikely to get warranty service for it in a reasonable or cost effective manner. I'd assume that people buying these devices are taking that into account. But if I buy a Topping DAC that's supposed to be "state of the art" and "reference quality" for $700, then I'm figuring that if  it's defective, I can get another and I still won't have spent as much as I would have on a much more expensive 
DAC from a "conventional" supplier. Same for if I don't agree after purchase that it is truly state of the art. I took a relatively small monetary risk that's supposed to give me a non-proportional sonic payoff. If I don't really get that, I haven't lost a significant amount relative to buying something much higher priced with warranty and service. 

 

If you aren't willing to take those risks, you shouldn't be buying such products.

 

And when Stereophile and TAS recommend those same products, I don't think they are standing behind them any more than Amir is.