Why you can't trust measurements

[Josh Mound]

On 5/24/2022 at 3:35 AM, firedog said:

Can't say it really bothers me. Don't disagree with Amir. If I order a Chinese DAC from AliExpress, I'm taking into account that I am unlikely to get warranty service for it in a reasonable or cost effective manner. I'd assume that people buying these devices are taking that into account. But if I buy a Topping DAC that's supposed to be "state of the art" and "reference quality" for $700, then I'm figuring that if  it's defective, I can get another and I still won't have spent as much as I would have on a much more expensive 
DAC from a "conventional" supplier. Same for if I don't agree after purchase that it is truly state of the art. I took a relatively small monetary risk that's supposed to give me a non-proportional sonic payoff. If I don't really get that, I haven't lost a significant amount relative to buying something much higher priced with warranty and service. 

 

If you aren't willing to take those risks, you shouldn't be buying such products.

 

And when Stereophile and TAS recommend those same products, I don't think they are standing behind them any more than Amir is. 

In terms of ASR’s practices, this gets back to some products receiving teardowns, but not others. Or some manufacturers getting scrutinized for reliability, but not others. Consistency matters in reviews, both in fairness to companies and to provide the consumer with actionable info.
 

In terms of the general assumption that people who are buying devices based on rave “objectivist” reviews know what they’re getting into, I’m not so sure. If a casual audiophile keeps reading that something is “state of the art,” I’m guessing they don’t think it’s likely to be a lemon. By definition, reliability and service are part of a quality product. I’m not calling on reviewers to conduct extensive reliability tests (though a Consumer Reports or Wirecutter for the audio would be awesome), but things like warranty and customer service should get at least some consideration in a review. 
 

IMHO, our disposable culture is bad. I’d rather spend more for a product that will last. That said, it was somewhat defensible to say “caveat emptor” when the product was $100. But now that some of the products in question are $500-$800, I don’t think that’s tenable. That’s a lot of money for most people. More importantly, that’s why I brought up the Schiit Modi 3 in my first post. To my mind, any “objectivist” recommending a $500-$800 DAC with a functionally nonexistent warranty over a Modi 3 (or Modius if balanced is preferred) when the Schiit products are cheaper, have better warranties and service, and also measure “perfectly” isn’t really providing their readers with the best information. 

[Josh Mound]

15 hours ago, GoldenOne said:

This is something that I personally feel is very important. For various reasons.

Firstly, I've noticed that some of the devices with stronger 'House Sounds' including DACs such as the Schiit Yggdrasil, show quite drastic changes in distortion level and profile in relation to amplitude.

 

In addition to this, some devices have slightly different distortion at their very uppermost output levels than they do in the rest of the output range.
The Gustard X26 pro for example, which is an excellent DAC, does seem to be somewhat 'optimising' for full scale output to look better on ASR-style tests.

 

You can see that distortion drops a few dB in the top ~6dB.
And in fact even when not limited by THD itself, you can visually see how the harmonics change.

At full scale 0dBfs output it looks like this:

All harmonics at or below about -140dB

But then take it to -3dBfs output and suddenly the 3rd order harmonic jumps up almost 15dB.

 

But ignoring DACs and just looking at amps, as has been mentioned, we don't listen to amps at 4v output. In fact on the vast majority of headphones that level would probably destroy your ears. So it's much more apt to test the device in the range that you're likely to use it.
Personally I offer measurements at 4v (to compare to ASR if people wish), 700mV (as this is a reasonably loud listening level on a few over ear headphones like an Arya) and 50mV (as this is appropriate for IEMs).

If you test something like an HE9 at 4V on 32 Ohm, you'll get about 0.3% THD+N (50dB SINAD). 
But if you test at a more realistic 700mV, you get a reduction to 0.02% THD+N (23dB increase to 73dB SINAD)

It seems silly to me to only test things at 4v as it's not realistic for amps in most cases.
But more to the point, on both DACs and Amps THD+N vs Level behaviour can reveal some pretty interesting stuff

 

I don't find reports on headamps at what will be ear-damaging levels for 99% of headphones to be very helpful, either. 

 

As far as DACs, I wonder if such issues at least partly account for differing subjective opinions. If one's listening to mostly clipped, "loudness wars"-era files, their impression of a DAC might be very different from someone listening to more dynamic, softer masterings, particularly if odd-order harmonics creep into audible territory at some levels but not others. IOTW, the original "Californication" CD isn't likely to sound the same as the original "Peter Gabriel IV" CD on DACs with distortion profiles that vary significantly by level. 

[Rexp]

'To get an obvious question out of the way: Yes, I did listen to the Ayazi without the Master Time. By its lonesome, the Ayazi reproduced music with less resolution and timbral accuracy and created a spatially smaller, less lifelike sound. Music sounded duller and less compelling.' Alex Halberstadt

 

Overall, Ideon's Ayazi did well on the test bench. I didn't find any difference in its performance when fed USB data directly or via the 3R Master Time Black Star.—John Atkinson

 

https://www.stereophile.com/content/ideon-audio-ayazi-mk2-da-processor-3r-master-time-black-star-usb-clock

[Confused]

9 hours ago, Rexp said:

'To get an obvious question out of the way: Yes, I did listen to the Ayazi without the Master Time. By its lonesome, the Ayazi reproduced music with less resolution and timbral accuracy and created a spatially smaller, less lifelike sound. Music sounded duller and less compelling.' Alex Halberstadt

 

Overall, Ideon's Ayazi did well on the test bench. I didn't find any difference in its performance when fed USB data directly or via the 3R Master Time Black Star.—John Atkinson

 

https://www.stereophile.com/content/ideon-audio-ayazi-mk2-da-processor-3r-master-time-black-star-usb-clock

 

There are at least two possibilities here:

 

1. Mr Halberstadt was imagining the improvement from the Master Time, hence the measurements correctly show no improvement in performance.

 

2. The improvement in sound quality from the Master Time is real, but the correct measurements to show why this might be the case were not performed.

 

If I was the editor of a magazine that planned to publish this, I would set up a blind test with Mr Halberstadt (and any other golden eared hacks that I had available) to see if he can reliably pick up the resolution and timbral accuracy and the spatially smaller, less lifelike sound, without the Master Time.

 

If he could not, I'd get him to rewrite the article. If he could, I'd get Mr Atkinson to take another look at how to perform his measurements.

 

As things stand, the article does not tell us much, apart from demonstrating that the audio press can be a bit useless at times.

[Rexp]

35 minutes ago, Confused said:

 

There are at least two possibilities here:

 

1. Mr Halberstadt was imagining the improvement from the Master Time, hence the measurements correctly show no improvement in performance.

 

2. The improvement in sound quality from the Master Time is real, but the correct measurements to show why this might be the case were not performed.

 

If I was the editor of a magazine that planned to publish this, I would set up a blind test with Mr Halberstadt (and any other golden eared hacks that I had available) to see if he can reliably pick up the resolution and timbral accuracy and the spatially smaller, less lifelike sound, without the Master Time.

 

If he could not, I'd get him to rewrite the article. If he could, I'd get Mr Atkinson to take another look at how to perform his measurements.

 

As things stand, the article does not tell us much, apart from demonstrating that the audio press can be a bit useless at times.

The fact is the reviewer heard a significant difference which was not measurable. What fact(s) lead you to believe either the reviewer was wrong or the measurements faulty?

[Confused]

2 hours ago, Rexp said:

The fact is the reviewer heard a significant difference which was not measurable. What fact(s) lead you to believe either the reviewer was wrong or the measurements faulty?

I have no reason to believe the measurements are “faulty”. It is of course possible that different or additional measurements could be performed.

 

What I see as a “fact” is that the ultimate conclusion from the measurements and the listening tests are contradictory.

 

Do you think that it is 100% impossible that the reviewer was not influenced by expectation bias?

 

It is of course possible that what the reviewer heard was real but not picked up by the specific measurements that were performed.

[Superdad]

50 minutes ago, Confused said:

It is of course possible that what the reviewer heard was real but not picked up by the specific measurements that were performed.

This. 

[Confused]

26 minutes ago, Superdad said:

This. 

Do you think that it is 100% impossible that the reviewer was not influenced by expectation bias?

[Iving]

but wait

 

there's more

 

yes - perceptual anomaly one possibility - means a subjective difference but no actual difference in pre-ear sound environments - whether measured or not

 

yes - sensitivity another possibility - a subjective difference accountable to a difference in pre-ear sound environments which was not measured

 

also -

 

even if subjective delta matched a measured objective delta - the two are not necessarily causally related.

 

they could be unrelated co-phenomena.

 

there could be a third hidden factor aetiological to both.

 

only an intervention study where sound environment manipulated to observe a corresponding subjective delta starts to establish causal relationship.

 

convincing data near impossible with single reviewer - 

and multi participant studies all sorts selection and expense problems.

 

thus people can say what they like and cannot be conclusively contradicted.

 

all the uncertainty makes a lot of room for individual hobby fun, forum conflicts - and industry confidence trickery.

 

antidote to all above is education of all types - and learning to think for ourselves

[Iving]

7 minutes ago, Iving said:

only an intervention study where sound environment manipulated to observe a corrrsponding subjective delta starts to establish causal relationship.

 

This ^ the starting point for development of theories about how sound accounts for subjective musical enjoyment.

 

and even if this high ambition approached experimentally, we have only measured pre-ear. What the brain does with sound deltas we can barely fathom.

 

there is no advanced explanation of human consciousness.

 

we have a long way to go and a lot of scope for disagreement before "truth" eventually revealed.

[Confused]

22 minutes ago, Iving said:

convincing data near impossible with single reviewer - 

and multi participant studies all sorts selection and expense problems.

Some very fine wise words in your earlier post Mr Iving. You kind of stated what I am thinking, although you have phrased it far better than I could.

 

Although re the quoted words above, I do not think we need expensive multi participant studies in this particular case. What we have is a reviewer who has clearly stated that he can hear very specific differences with the $4K Master Time in the chain. This would take only two people and a couple of hours to establish if he could do this "blind". If he could, this would be convincing data from one reviewer, and thus provide good reason to investigate further in the measurements.

[Iving]

3 minutes ago, Confused said:

if he could do this "blind". If he could, this would be convincing data from one reviewer

 

it would

 

blind tests v unlikely to get a false positive

 

but all sorts explanations possible for negatives

[Superdad]

3 hours ago, Confused said:

Do you think that it is 100% impossible that the reviewer was not influenced by expectation bias?

I was not in the room where it happened so have no idea what was actually heard or imagined.

Then again, I don't see why IDEON charges $4,000 for what is essentially a copy of our original 2015 USB REGEN ($175) with exact same SMSC hub chip, some enhanced power networks, and some unknown clocking. (And that whole layout with the daughter board is terrible for USB signal integrity.) 

[Confused]

58 minutes ago, Superdad said:

I was not in the room where it happened so have no idea what was actually heard or imagined.

Then again, I don't see why IDEON charges $4,000 for what is essentially a copy of our original 2015 USB REGEN ($175) with exact same SMSC hub chip, some enhanced power networks, and some unknown clocking. (And that whole layout with the daughter board is terrible for USB signal integrity.) 

 

Personally, I get a far greater expectation bias invoked SQ hike from a $4000 item than I do from a $175 product. 🙂

 

Joking apart, I know the Master Time has excellent clocking properties because it has "advanced femtoclock anti-jitter technology" written on it in big letters on the circuit board.

[fas42]

6 hours ago, Iving said:

all the uncertainty makes a lot of room for individual hobby fun, forum conflicts - and industry confidence trickery.

 

antidote to all above is education of all types - and learning to think for ourselves

 

There's lots of uncertainty ... but very little trickery, in the normal sense. The whole thing falls in the same arena as the massive, personal cosmetics industry - do you want to send all the executives of these companies to prison, for defrauding the public?

[fas42]

It's interesting how so many people can't stand back from what results after making some changes to a system; there is almost a gritted determination to find some improvement - there must be, because I have put so much energy, or money into doing this; it better be better !!!

 

I have had far too many disappointments over the years to see any use for this attitude - a rig is what it is: it may be incredibly expensive, with layers of bling to bedazzle, but if it sounds like a mess, well, it is a mess ...

[Iving]

6 hours ago, fas42 said:

The whole thing falls in the same arena as the massive, personal cosmetics industry

 

no it doesn't

 

6 hours ago, fas42 said:

very little trickery, in the normal sense

 

confidence trickery - believe what i say because of the way i say it - don't examine merits 

 

it's clear what i mean

 

6 hours ago, fas42 said:

do you want to send all the executives of these companies to prison

 

don't be silly

 

all goading and off topic

[fas42]

1 hour ago, Iving said:

 

confidence trickery - believe what i say because of the way i say it - don't examine merits 

 

it's clear what i mean

 

It is? From https://en.wikipedia.org/wiki/Confidence_trick,

 

Quote

A confidence trick is an attempt to defraud a person or group after first gaining their trust. Confidence tricks exploit victims using their credulity, naïveté, compassion, vanity, confidence, irresponsibility, and greed. Researchers have defined confidence tricks as "a distinctive species of fraudulent conduct [...] intending to further voluntary exchanges that are not mutually beneficial", as they "benefit con operators ('con men') at the expense of their victims (the 'marks')

 

So, say, every person who was purchased an expensive audio cable, which costs many, many times more then any person could conceive that it would cost to manufacture, has been defrauded ... correct?

[Iving]

@fas42 since you are quoting me haven't you got anything interesting or worthwhile to say about "why you can't trust measurements" - particularly mapping your remarks to my incisive breakdown of the whole subject in 3 posts. all you have to offer is an off beam unjustified parallel with lipstick, a childish imprisonment gambit, and a pick over what's meant by confidence trickery. the only contribution you ever make you made "decades ago" - and now you are just hijacking and spoiling another thread.

[fas42]

Yes, silly of me to be annoyed by the seemingly obligatory put down of people who produce tweaks - who are supposedly always doing for the money, as compared to having a belief in the value of their product.

[kumakuma]

 

Deleted

[kumakuma]

55 minutes ago, fas42 said:

Yes, silly of me to be annoyed by the seemingly obligatory put down of people who produce tweaks - who are supposedly always doing for the money, as compared to having a belief in the value of their product.

 

“Stupidity is a more dangerous enemy of the good than malice. One may protest against evil; it can be exposed and, if need be, prevented by use of force. Evil always carries within itself the germ of its own subversion in that it leaves behind in human beings at least a sense of unease. Against stupidity we are defenseless. Neither protests nor the use of force accomplish anything here; reasons fall on deaf ears; facts that contradict one’s prejudgment simply need not be believed – in such moments the stupid person even becomes critical – and when facts are irrefutable they are just pushed aside as inconsequential, as incidental. In all this the stupid person, in contrast to the malicious one, is utterly self satisfied and, being easily irritated, becomes dangerous by going on the attack. For that reason, greater caution is called for when dealing with a stupid person than with a malicious one. Never again will we try to persuade the stupid person with reasons, for it is senseless and dangerous.”

 

Dietrich Bonhoeffer

 

[fas42]

https://www.webmd.com/mental-health/what-is-a-mob-mentality ...

 

Quote

Mob mentality, herd mentality, pack mentality, groupthink, or crowd psychology — the concept has many names. These all boil down to the same idea: Individuals are influenced by a larger group. Regardless of whether that group includes people in your class, your neighborhood, or an entire nation, you may experience mob mentality.

In the 1950s, researchers conducted a famous conformity experiment that showed how readily people conform or change their behavior to match social norms. It involved:

• A single participant was put in a room with seven undercover accomplices.
• Asch presented the group with four lines, and the goal was to determine which two lines were the same length.
• Even though the answer was obvious, the undercover accomplices purposefully gave an incorrect answer.
• The real participant answered last.
• The intent of the experiment was to see if the real participant would give a false answer — conforming with the accomplices — even if the correct answer was clear.

‌The results were surprising. About one-third of the real participants answered incorrectly. They conformed to the wrong answer given by the rest of the group

 

Why did they do it? When the participants heard the incorrect answer, some started to believe the incorrect answer was actually correct. The participants conformed mainly because they wanted to fit in with the rest of the group and thought the rest of the group was more informed than they were.

 

 

.

[botrytis]

Maybe it is more along the lines of this paper. It applies to many thinks taken to religious extremes....

 

 

nihms868078.pdf

[kumakuma]

12 hours ago, fas42 said:

https://www.webmd.com/mental-health/what-is-a-mob-mentality ...

 

.

 

I'm having a hard time seeing how this relates to the subject being discussed in this thread. Can you elaborate?