Misleading Measurements

[Jud]

15 hours ago, jabbr said:

Kinda like I could care less what ASR measures a Pass Labs component at ...

Thought of you the last time I read an ASR Pass Labs review.   

[Jud]

On 7/2/2020 at 8:50 AM, pkane2001 said:

 

Correct. But there are many more opportunities to mess up a DAC than a USB cable.

 

Though there are plenty of opportunities to introduce noise into a system involving a number of boxes and cables that include both of these. I would love to see more work done on system topology, components and cabling and how all of these affect system noise. I've seen a few articles about topics like grounding, but I haven't had good luck finding detailed and precise information on how to configure a system and then how to measure the results in easy to digest form for the audio layperson (i.e., non-electrician).

[Jud]

4 hours ago, pkane2001 said:

 

A USB cable has a very simple design and the USB spec defines the parameters of what it has to be able to accomplish, and to what tolerances. Anything else the cable is doing is superfluous or possibly detrimental to its purpose of carrying data.

 

If you want to know how anything in the chain is affecting noise in the audio system -- measure the output of the system. The output of a DAC is what is fed into an amp. If the noise is not detectable there, then it's either non-existent, or it's properly filtered out by the receiver circuits.

 

Unlike the claims by Frank and others, I usually test a large portion of my system all wired together. Sometimes including two PCs (one is a Mac), attached RAID storage, a Wi-Fi network, a USB-to-Ethernet converter, a couple of USB and ethernet cables (one is 100ft long), a DAC, and sometimes even an amp with a simulated load. All in one shot. If this doesn't introduce any measurable additional noise into my system compared to, say, a Lush^2 cable feeding the same DAC directly from a battery-powered PC, then I'm not worried that anything is introducing unwanted noise. 

I was thinking along the lines of practical tutorials like "OK, you've got a ground hum. Here's how to find what's responsible and eliminate it." 

 

Even better, since I think hum and noise that isn't consciously audible may subconsciously affect enjoyment (cf. the Iowa gambling task experiment), what is the equipment and what are the measurements to take to determine noise levels when there's no audible hum or noise?

[Jud]

I can think of two reasons I might like to see measurements that are below generally accepted thresholds of human hearing.

 

1. Lab bench vs. in situ, something John mentioned - it's possible a component in a home in less than the (hopefully) ideal conditions of a lab test might perform somewhat differently. This relates to Paul's point about "better engineered." But I haven't seen measurements like resistance to noise carried into a DAC from external connections, for example, so it's difficult to tell what's really better engineered in this context.

 

2. I think it is possible that noise below generally accepted hearing thresholds might have subconscious effects. Is anyone aware of papers measuring hearing thresholds that use something like an fMRI rather than conscious verbal responses? Experimenters do take account of the difference between tests that use instrumental measurements and those that require a conscious verbal response, but I don't recall seeing academic papers on this topic in the area of human hearing.

[Jud]

56 minutes ago, pkane2001 said:

No, not rich, just many years of putting these together.

We’ll give you a “Wow!” anyway. 😉

[Jud]

1 minute ago, Audiophile Neuroscience said:

How is WOW measured i wonder ?

Just about the same way as flutter, but slower.

[Jud]

Might be interesting for folks discussing this topic to read the following Wikipedia article:

 

https://en.m.wikipedia.org/wiki/New_Coke

[Jud]

17 hours ago, Jud said:

Might be interesting for folks discussing this topic to read the following Wikipedia article:

 

https://en.m.wikipedia.org/wiki/New_Coke

 

15 hours ago, Audiophile Neuroscience said:

 

Was it @Summit or @semente or someone else who brought this up earlier. i didn't read the full article but the previous poster referred to the propensity to initially choose a strong quality like the increased sweetness of Pepsi is often overturned by longer exposure to the product. I think the suggestion was that even well conducted short burst ABX tests may be misleading.

 

4 hours ago, Summit said:

 

OT

Yes in a test there we are listening only for a few seconds to one product and they quickly change to another product for a few seconds, before changing again and again it’s common to choose the ones which have a sound that “stands out”. In this type of tests we are completely focused on noticing the difference between two products instead of the sound quality, which can’t be evaluated and decided after just a few seconds. This favouring of a type of sound that stands out is not very evident in the beginning of a test, but steadily increase by the amount of switching back and forward rapidly.

 

Related to this type of test method is the problem called forced choice bias. Forced choice means that we are forced to pick one of two even if we can't hear any difference between them, or worse don’t feel comfortable that the difference we hear also is desirable.

  

A well conducted short ABX test is good for distinguishing if there are any audible difference between two products or not. Or if we can hear a difference between sample A and B and many other things that can be easily distinguished.

 

It’s not very hard to identify Coke from Pepsi one time or two, but after 7 sips of Pepsi and 7 sips of Coke it’s much harder to distinguishing them. In food and beverage test we are not forced to do that 10 times, but in audio.

 

I don't know whether there's academic work to back this up, but I have the general notion that in terms of sensory stimuli, discrimination and pattern matching are two different things.

 

Regarding discrimination, as @Audiophile Neuroscience and @Summit are saying, there seems to be a propensity to choose on the basis of a particular quality - sweetness when tasting soft drinks (wonder whether it's sweetness or something else that predominates in blind wine tastings?), loudness in audio (for example, blind violin listening tests, even when done by very accomplished violinists; or the hi res listening test done by former member Julf with the assistance of the principal of BIS recordings, where forum members by a large margin chose a file that was, unknown to them, 1dB louder than the rest as sounding best).

 

Sensory memories are quite short - perhaps 4 seconds in the case of audio; I don't know about taste, but I imagine it's not terribly different. It's not at all surprising we'd need a strong, very recognizable quality to impress us as different in that short amount of time. And I think many of us have experienced the sense of being at sea, of having nothing to go on, when such a strong stimulus is absent and/or a comparison takes longer than 4 seconds. So the fact that so many blind tests show an inability to discriminate isn't surprising, but fully expected.

 

Pattern matching is very different. There's the New Coke example - sweetness is preferred in a quick discrimination test, but when millions of customers sat back with a New Coke on a hot summer day and it just didn't taste "right" - didn't fit the familiar, comfortable pattern - they noticed, despite passage of a lot more than a few seconds since they'd tasted an original Coke. And of course in audio there's the familiar example many people have mentioned of a family member's voice on the phone - you recognize it instantly as matching the pattern of dad, mom, sibling, spouse, child, even if months have passed, and even on a connection that doesn't provide the best audio quality. People are so good at audio pattern matching that, for example, they are recruited to try to discern patterns in Jupiter's radio emissions. In fact we will even discern patterns that don't really exist - that's what optical and audible illusions are.

 

Which of these are we doing when we enjoy listening to music? I certainly don't think we're trying to do loudness comparisons. I'd say our enjoyment arguably lies in how much the tone of a violin or acoustic guitar, or the sound of a human voice, reminds us of the real thing, and excites the same emotions as the real thing did.

 

A few years ago I conducted a little test here in the forums. One acoustic guitar (classic Gibson, big round sound hole) was playing in one channel, a very different acoustic guitar (a very old Epiphone with small f-holes like a violin) in the other. The task was to determine which guitar was playing in which channel. Self-declared objectivists were given a selection that repeated 2 seconds on and 2 seconds off for 30 seconds. A rapid comparison, as an objectivist can tell you, is best for discrimination when doing the most common form of blind testing, i.e., listening to selection A, then quickly comparing it to selection B. So that's why they got that sort of test, even though there was no sequential comparison - both guitars were playing throughout the 2 seconds, one in each channel. Self-declared subjectivists were simply given a 30-second selection, because they don't tend to be concerned about quick comparisons, the length of echoic memory, and suchlike.

 

The guitars sounded different enough that most people got it right. But not surprisingly, a higher proportion of those listening to the 30 second selection got it right than those who had only 2 second repeated snippets to work with. (Only 1 person out of 45, who said he was using an iPhone and earbuds, could not distinguish between guitars in the 30 second selection.) Statistically the results should only have occurred 6% of the time by chance, so there is some level of assurance this was a real effect.

 

It seems to me that the act of trying to discriminate between sequential selections by blind testing may not be the best way to model what we actually are doing when we sit down to enjoy our music, to enjoy the familiar patterns of instruments or vocals.

 

What does this have to do with measurements? Well, how do we determine what levels of various measurements of distortions, for example, are audible and thus important? By discrimination tests. And I wonder how well such tests simulate our ability to detect the pattern of something that "sounds right." After all, the taste discrimination tests that led to New Coke did not seem to do a very good job developing something that "tasted right" to legions of people accustomed to the taste pattern of original Coke.

 

 

 

[Jud]

2 hours ago, manisandher said:

 

Hey Jud, when I conducted the ABX test with Mans, each sample lasted for about 15 seconds. There was then a 5-10 second delay before the next sample started. The whole 10x ABX took around 15 minutes to undertake.

 

 

In the ABX, I was primarily picking up an ever-so-slight edginess in the piano in one and not in the other sample (both bit-identical to each other).

 

Hi Mani,

 

When you've got a recognizable "tell," I'd argue this is pattern recognition. I'd also speculate your ability to find the "tell" was helped along by years of adjusting XXHE parameters and hearing the results, which ISTM is training to recognize the telltale pattern(s). (This is a reason I'm very interested in how training effects relate to perceptual tests.)

 

2 hours ago, manisandher said:

 

By the end of the ABX, I was totally exhausted (having conducted a couple of non-ABX tests beforehand too). The experience was absolutely nothing like listening to music for pleasure. I suspect this is the main reason why ABX tests often 'fail'.

 

I'd speculate the closer the experience is to a test of discrimination, the more exhausting. But again, though it wasn't a strong signal (and thus took effort, leaving you a bit worn out), I'd opine you were looking for your "tell," and therefore doing pattern recognition.

 

2 hours ago, manisandher said:

 

I'm reading, and very much enjoying, 'The Master and His Emissary' by Iain McGilchrist at the moment. Maybe an ABX test is more of a left-brain activity, whereas listening to music for enjoyment is more of a right-brain activity? Not easy to reconcile the two.

 

Mani.

 

I think a lot of the "right-brain/left-brain" stuff has been debunked at this point.  

[Jud]

4 minutes ago, pkane2001 said:

 

Why? When Amir started measuring Schiit equipment, their performance was mediocre at best (that's the measurable performance). After getting criticized and after seeing poor results from a third party, Schiit implemented a test regime to QA all their equipment using an AP analyzer. They also started publishing detailed measurement reports for each product. Not only did these published measurements improve their image, but the performance of their equipment rose by an order of magnitude, from mediocre to top of the line. Instead of spending time and effort to defend the poor quality of their older products, their products now get a recommendation from Amir because they performs well. And his measurements confirm what's published by Schiit, so the consumer gets a confirmation that the manufacturer isn't exaggerating. A little third-party verification can work to the benefit of the consumer. Not a bad thing, IMO.

 

Interesting. This does somewhat suggest at least the possibility that parts and circuits can be tailored to produce a desired set of measurements, perhaps without substantial alterations in the sound the designers want (if they were able to recognize such a sound - both of the principals have decades of design experience).

[Jud]

5 minutes ago, pkane2001 said:

Yet another explanation, is of course, that with distortions below a threshold of audibility, the devices all sound the same.

 

This possibly implicates some of the things I was talking about earlier regarding training effects and pattern recognition. I doubt someone new to XXHE would have heard the "tell" @manisandher did. But if you can hear an effect with training, that means it's not inaudible, you're just not consciously noticing it. And even if you don't consciously notice it, it's possible that effect/distortion might be bothering you greatly.

 

(In the Iowa Gambling Task experiment I keep mentioning, subjects were given "good" and "bad" decks of cards. Subjects with "bad" decks showed elevated galvanic skin response - that is, they literally broke out in a sweat - on average several rounds of play before they consciously realized they had a bad deck and asked for a new one. So might we be made uncomfortable enough subconsciously by audio distortion we don't consciously notice to break out in a sweat? Interesting to think about. And if it were true, what would it mean for levels of audibility that have been established by tests that require conscious verbal responses?)

[Jud]

1 hour ago, The Computer Audiophile said:

Second flush all the way :~)

 

This is the best tea in the world: http://www.makaibari.com/en/buy-tea/silver-tips-imperial.aspx

 

Here's where to order it, though they don't appear to have any in stock at the moment. Lots of other excellent teas, though: https://www.silvertipstea.com

[Jud]

4 hours ago, The Computer Audiophile said:

No way!

 

Organic Nepali Golden Black

 

https://youngmountaintea.com/products/nepali-golden-black

 

:~)

 

We'll each have to try the other.

 

[Jud]

7 hours ago, ray-dude said:

 

Holy crap, $300/pound!!  This is the UPOCC of darjeelings!!  I'm very tempted....

Last OT on this: I think a little less expensive sold thru the site I recommended, though still not cheap. I’ve had it. Loved the tea, but though I enjoy a nice cup very much, it’s not gonna change my life.

[Jud]

8 hours ago, Audiophile Neuroscience said:

 

 

 

 

 

I also think that like many others that ABX tests can be exhausting and anxiety provoking and is not how we generally make higher order perceptual connections whether sensory discriminations or recognizable patterns.

 

It appears there are some things that can't be forced and as counter-intuitive as it may seem you cannot try to do it. The harder you try the more you fail. Falling asleep is one and so is penile erections ( he quickly adds.... It is a recurrent theme in patients I have treated). The sentence I must fall asleep now and focusing on it usually begets failure. It is a letting go process. Complex perception I think is in that category

But I think it isn’t quite so simple as “ABX tests are hard.” (Not saying that this is your position.) *Why* are they hard (or not, as in your other example 😉)?

 

For vision, the brain is extremely specialized, down to some groups of neurons recognizing horizontal features, others vertical. I’m speculating that recognizing when a familiar piano sounds just a very little bit “off” may be handled differently in the brain than trying to recall whether a musical passage you heard 20 seconds ago sounds different than one you’re listening to now. (Note “trying to recall” in the second half of the sentence but not the first.)

 

There’s been plenty of academic work done on that “trying to recall” bit. Here from a researcher’s page are tests you can try at home:  http://deutsch.ucsd.edu/psychology/pages.php?i=209

 

(Then ask yourself whether sequential A/B comparison of musical passages more closely resembles the easier or the more difficult test.)

[Jud]

2 hours ago, pkane2001 said:

 

Sorry about the infinite loop, David. Unfortunately, recycled claims, just like those you dredge out in the quote above, are what keeps it going. 

 

Alex reacted badly to ASR measurements of his product(s), but provided nothing to refute them other than the "there are many happy customers evidence". In fact, his engineer eventually fixed the issue reported with the power supply that was causing the mains frequency to leak into the powered device. This was with a product designed to provide AC/DC conversion and a supposedly perfect DAC isolation, according to marketing and repeated claims. So was the measurement misleading or were the claims by the manufacturer? I know my answer. Alex knows it too, since his product now includes a fix for it, despite all the protests.

 

Chris has been doing and saying many things about ASR and Amir specifically, but whenever I press for details, he just tells me to connect the dots myself. Not very convincing evidence there, either, I'm afraid.

 

A manufacturer certainly doesn't want bad news to be published about their product. Neither does an audiophile who invested hard-earned cash into the product. Nor the owner of a web forum which is sponsored by these same manufacturers and audiophiles. That's understandable. But that's why measurements are important. Everyone has a bias and everyone has a hidden agenda. Measurements do not.

 

I'll agree to drop it, but only if you do it, as well.

 

 

 

Am I reading this correctly to say the original condition was the highest harmonic at -105dB with the LPS-1 and MeanWell, versus -115dB with the other supply (at a different mains frequency?)? And the fix (in the LPS-1.2?) brought the original level of -105dB down to what?

[Jud]

49 minutes ago, pkane2001 said:

I've yet to see measurements of LPS-1.2, maybe Alex can share those or someone should send LPS-1.2 to Amir for measurement.

 

So the fix you mentioned occurred in the Meanwell or LPS-1?

[Jud]

14 minutes ago, Bill Brown said:

 

Agree!

 

In Texas our tea is iced, though I differ from lots and drink it unsweetened

OK, I said I wouldn’t go OT any more, but I have to note my agreement. Live in a climate like TX, OK, NM etc., and sugary drinks don’t quench thirst IMO.

[Jud]

50 minutes ago, pkane2001 said:

 

You're asking for feedback from some amorphous group that's picking and choosing what measurements they like, and arguing that it's misleading. Unless this is a rhetorical question, you may want to identify at least some members of that group, and ask them. Or, at least, point to some examples where they are being misleading. @Archimago was one that you identified in the OP. Can you perhaps point to where he's sharing something that's misleading or hypocritical (words you used throughout this thread)? Not that I want to argue on his behalf, but my perception is that what he's been sharing is not misleading in the least.

 

I'll jump in with a what rather than a who: I have seen praise for low jitter figures in Benchmark DACs by people who have elsewhere said the importance of jitter is greatly exaggerated. (Benchmark themselves have stated that asynchronous USB input and DSD are unnecessary, then built both into their DACs; and have praised at least the former in their marketing. This is entirely aside from the DAC's performance - I don't own one but certainly the measured performance appears to be excellent.)

[Jud]

1 hour ago, Audiophile Neuroscience said:

SO, could it be possible that this 'double down' acoustic intermodulation of tones be in some way partly responsible for the inherent difficulties of playback systems to get larger scale works right? It simply doesn't matter how good the recording or playback is there will be artifact created by added intonation which colors the sound.

 

45 minutes ago, bluesman said:

That’s exactly my belief.

 

@esldude and I discussed this quite some time back.

 

I wondered if reproduction of ultrasonics helped reproduce some of the natural harmonics of musical instruments. He maintained any intermodulation of that type from home speakers over and above the audible frequencies captured at the mics would be distortion. I'm not exactly sure if Paul is arguing the opposite.

 

I'm quite sure at least some speaker designers are well aware of the possibility such intermodulation may occur and take steps to minimize it insofar as possible. I'd guess it would show up pretty clearly with some speaker tests.

[Jud]

19 minutes ago, Jud said:

I'd guess it would show up pretty clearly with some speaker tests.

 

In fact free software like REW used with a calibrated mic should show it with test tones at various frequencies, so you can check whether this might be a concern with your speakers.

[Jud]

10 minutes ago, Audiophile Neuroscience said:

Those acoustic intermodulations are recreated anew in the air on playback, which in effect doubles down on the intermodulations already captured and faithfully reproduced.

 

I'm saying why theorize about this? Run test tones through your speakers (Audacity or other free software should work), and use REW or other free software with a calibrated mic to see if you're getting intermodulation products at what could be audible levels.

[Jud]

8 minutes ago, fas42 said:

 

Again, modulation doesn't occur "in the air" - it may in the driver, but that's a completely separate issue.

 

 

Another proposition that's simply tested. Measure with mic at speaker(s), measure with mic at listening position.

[Jud]

10 hours ago, Audiophile Neuroscience said:

The latter must create a new set of sum and difference tones in your living room which add to the original captured sum and difference tones.

 

The question is, must it indeed and if so to what degree? It seems like something that might make sense, but:

 

- To what degree are speaker designers aware of this and design speakers to minimize the effect?

 

- I've seen science defined as "A beautiful theory slain by an ugly fact." 🙂 So the theory sounds at least somewhat sensible, but let's do some measurements and see if we discover any contrary ugly facts.

[Jud]

3 hours ago, bluesman said:

 

I suspect there are measurable parameters (e.g. phase differences?) that would enable separation of IM products into those on the recording and those created by the same instrumental interactions on playback.  This could be a new opportunity to make a truly meaningful measurement

 

I'm probably being naive, but this sounds overcomplicated. Why not create an Audacity file with 4k and 5k tones, and do a mic measurement at speakers and listening position to see if you get 1k products and at what level?

 

You can make this as multi-tonal as you like to simulate the complexity of music.