Misleading Measurements

[pkane2001]

14 minutes ago, fas42 said:

 

IME, all audio devices don't have good filtration at the power supply - that is, it's trivially easy to introduce some electrical noise making device or process into the environment, and hear the impact on the SQ ... just read the reports of companies struggling to get good sound in a show situation, and how they had to deal with the "lousy hotel power!"

Well, not all, as the JDS Labs Atom in the review showed no difference with filtered and unfiltered power, and Amir’s power line is pretty dirty.

[pkane2001]

Just now, Audiophile Neuroscience said:

 

 

"Incompetence" is your word not mine - any inference "or information about the person" in this regard is yours also. 

 

That said, whether incompetence, ignorance, agenda driven bias or some other reason makes a measurement untrustworthy, it is still untrustworthy

 

 

I’m starting to detect a tiny hint of bias in your comments about Amir

[Audiophile Neuroscience]

50 minutes ago, Jud said:

 

 

 

I don't know whether there's academic work to back this up, but I have the general notion that in terms of sensory stimuli, discrimination and pattern matching are two different things.

 

Regarding discrimination, as @Audiophile Neuroscience and @Summit are saying, there seems to be a propensity to choose on the basis of a particular quality - sweetness when tasting soft drinks (wonder whether it's sweetness or something else that predominates in blind wine tastings?), loudness in audio (for example, blind violin listening tests, even when done by very accomplished violinists; or the hi res listening test done by former member Julf with the assistance of the principal of BIS recordings, where forum members by a large margin chose a file that was, unknown to them, 1dB louder than the rest as sounding best).

 

Sensory memories are quite short - perhaps 4 seconds in the case of audio; I don't know about taste, but I imagine it's not terribly different. It's not at all surprising we'd need a strong, very recognizable quality to impress us as different in that short amount of time. And I think many of us have experienced the sense of being at sea, of having nothing to go on, when such a strong stimulus is absent and/or a comparison takes longer than 4 seconds. So the fact that so many blind tests show an inability to discriminate isn't surprising, but fully expected.

 

Pattern matching is very different. There's the New Coke example - sweetness is preferred in a quick discrimination test, but when millions of customers sat back with a New Coke on a hot summer day and it just didn't taste "right" - didn't fit the familiar, comfortable pattern - they noticed, despite passage of a lot more than a few seconds since they'd tasted an original Coke. And of course in audio there's the familiar example many people have mentioned of a family member's voice on the phone - you recognize it instantly as matching the pattern of dad, mom, sibling, spouse, child, even if months have passed, and even on a connection that doesn't provide the best audio quality. People are so good at audio pattern matching that, for example, they are recruited to try to discern patterns in Jupiter's radio emissions. In fact we will even discern patterns that don't really exist - that's what optical and audible illusions are.

 

Which of these are we doing when we enjoy listening to music? I certainly don't think we're trying to do loudness comparisons. I'd say our enjoyment arguably lies in how much the tone of a violin or acoustic guitar, or the sound of a human voice, reminds us of the real thing, and excites the same emotions as the real thing did.

 

A few years ago I conducted a little test here in the forums. One acoustic guitar (classic Gibson, big round sound hole) was playing in one channel, a very different acoustic guitar (a very old Epiphone with small f-holes like a violin) in the other. The task was to determine which guitar was playing in which channel. Self-declared objectivists were given a selection that repeated 2 seconds on and 2 seconds off for 30 seconds. A rapid comparison, as an objectivist can tell you, is best for discrimination when doing the most common form of blind testing, i.e., listening to selection A, then quickly comparing it to selection B. So that's why they got that sort of test, even though there was no sequential comparison - both guitars were playing throughout the 2 seconds, one in each channel. Self-declared subjectivists were simply given a 30-second selection, because they don't tend to be concerned about quick comparisons, the length of echoic memory, and suchlike.

 

The guitars sounded different enough that most people got it right. But not surprisingly, a higher proportion of those listening to the 30 second selection got it right than those who had only 2 second repeated snippets to work with. (Only 1 person out of 45, who said he was using an iPhone and earbuds, could not distinguish between guitars in the 30 second selection.) Statistically the results should only have occurred 6% of the time by chance, so there is some level of assurance this was a real effect.

 

It seems to me that the act of trying to discriminate between sequential selections by blind testing may not be the best way to model what we actually are doing when we sit down to enjoy our music, to enjoy the familiar patterns of instruments or vocals.

 

What does this have to do with measurements? Well, how do we determine what levels of various measurements of distortions, for example, are audible and thus important? By discrimination tests. And I wonder how well such tests simulate our ability to detect the pattern of something that "sounds right." After all, the taste discrimination tests that led to New Coke did not seem to do a very good job developing something that "tasted right" to legions of people accustomed to the taste pattern of original Coke.

 

 

 

 

Hi Jud, I agree that pattern matching is extremely relevant in sensory perception - perceptual grouping, recognition ( and therefore discrimination), memory function and organization and even creativity.

 

I wholly agree about needing a longer time,and would add uninterrupted and relaxed time period, to allow perceptual connections to occur, especially when the component parts to the pattern are not always immediately available as might be the case in say the sound of someone's voice or the appearance of their face etc

 

I mentioned creativity above, there is for me some fascinating stuff on this and speculation on my part about how this might relate to listening tests, longer time durations, and patterns/ perceptual connections. Steve Jobs was apparently quoted as saying "creativity is just connecting things". Our capacity for pattern-recognition, Bor argues, is the very source of human creativity. Oddly enough, or perhaps not odd at all, John Cleese lectures on creativity and shares some interesting insights also. perception might be regarded in itself as a creative process, some people are better than others.It is a right brain function.

 

There are many other aspects regarding perception that could be discussed and that was the premise of a website I formed back when CA (not AS) was (for me) a hostile place to discuss such things. It may be a worthwhile thread if there are enough interested people

[Audiophile Neuroscience]

36 minutes ago, pkane2001 said:

I’m starting to detect a tiny hint of bias in your comments about Amir

Only the biased measurements 🤣 I have a bias against bias.😇🤷‍♂️

[opus101]

4 hours ago, pkane2001 said:

Well, not all, as the JDS Labs Atom in the review showed no difference with filtered and unfiltered power, and Amir’s power line is pretty dirty.

 

Let me guess - no common-mode measurements were made in that review?

[pkane2001]

6 hours ago, opus101 said:

 

Let me guess - no common-mode measurements were made in that review?

Why would that be important in measuring the effect of a power line filter?

[opus101]

Because a significant amount of power line noise is common-mode.

[pkane2001]

Just now, opus101 said:

Because a significant amount of power line noise is common-mode.

But what does that have to do with the filter? It should filter any noise, regardless of the source. How would you measure it differently to account for common mode noise?

[opus101]

Normally power line filters have elements to filter common-mode noise (a CM choke for example) as well as normal mode noise (LC filtering perhaps). Wouldn't comprehensive measurements wish to characterize both?

[pkane2001]

32 minutes ago, opus101 said:

Normally power line filters have elements to filter common-mode noise (a CM choke for example) as well as normal mode noise (LC filtering perhaps). Wouldn't comprehensive measurements wish to characterize both?

Maybe. But if the power line carries a lot of common mode noise as you say, Amir’s test would show the combination of what the filter did to both, normal and common mode noise. To me that’s what matters in the end: does the filter eliminate noise, and if it does, does this have an effect on the output of my audio system.
 

As an engineer, I might like to know more about the filters and the types of noise they reject, but as a consumer, I wouldn’t know what part of my power line noise is common mode and what is normal; I’d just want to know that all noise is not going to make it to my overly sensitive, audiophile components 

 

 

[Clockmeister]

As I have pointed out out in previous post line filtration ideally common and differential mode dual line filters would be desirable to keep the upper frequency rubbish out of the sensitive areas of the equipment.

Plus its a requirement for CE

[pkane2001]

20 minutes ago, Clockmeister said:

As I have pointed out out in previous post line filtration ideally common and differential mode dual line filters would be desirable to keep the upper frequency rubbish out of the sensitive areas of the equipment.

Plus its a requirement for CE

 

Don't know enough about CE certification, but my impression is that it has to do with noise produced/emitted by a component rather than noise filtration of already dirty power. Is this not correct?

[Clockmeister]

23 minutes ago, pkane2001 said:

 

Don't know enough about CE certification, but my impression is that it has to do with noise produced/emitted by a component rather than noise filtration of already dirty power. Is this not correct?

 Paul

 

In essence it is bring many products on to a level playing field for conducted, radiated immunity RF, common mode noise, line harmonics that could cause interference to other nearby products whether audio or not by distribution through said product. These are CE directives you have your own standard in the US called CL.

 

However concerning audio I have many files and hours of data relating to insertion loss filter modelling and trails specifically for very sensitive equipment from instrumentation to audio. The better the filter design at the incoming ac point, the less crap floating around inside the box that can be affected by said mains (ac) bourne noise.

 

 

[opus101]

2 hours ago, pkane2001 said:

Maybe. But if the power line carries a lot of common mode noise as you say, Amir’s test would show the combination of what the filter did to both, normal and common mode noise. To me that’s what matters in the end: does the filter eliminate noise, and if it does, does this have an effect on the output of my audio system.

 

I'd also presume readers would want to know what the filter did to both. But separate, distinct measurement setups are required for each which is why I guessed Amir hadn't made the CM measurements. I've not read the review - am I wrong in my guess?

[manisandher]

15 hours ago, Jud said:

Sensory memories are quite short - perhaps 4 seconds in the case of audio...

 

Hey Jud, when I conducted the ABX test with Mans, each sample lasted for about 15 seconds. There was then a 5-10 second delay before the next sample started. The whole 10x ABX took around 15 minutes to undertake.

 

15 hours ago, Jud said:

It's not at all surprising we'd need a strong, very recognizable quality to impress us as different in that short amount of time.

 

In the ABX, I was primarily picking up an ever-so-slight edginess in the piano in one and not in the other sample (both bit-identical to each other).

 

15 hours ago, Jud said:

Which of these are we doing when we enjoy listening to music?

 

By the end of the ABX, I was totally exhausted (having conducted a couple of non-ABX tests beforehand too). The experience was absolutely nothing like listening to music for pleasure. I suspect this is the main reason why ABX tests often 'fail'.

 

I'm reading, and very much enjoying, 'The Master and His Emissary' by Iain McGilchrist at the moment. Maybe an ABX test is more of a left-brain activity, whereas listening to music for enjoyment is more of a right-brain activity? Not easy to reconcile the two.

 

Mani.

[pkane2001]

49 minutes ago, opus101 said:

 

I'd also presume readers would want to know what the filter did to both. But separate, distinct measurement setups are required for each which is why I guessed Amir hadn't made the CM measurements. I've not read the review - am I wrong in my guess?

 

He did not do common-mode noise testing, at least none was mentioned in the review.

 

[plissken]

On 7/6/2020 at 8:31 PM, Audiophile Neuroscience said:

I don't trust Amir's measurements. They would need to be reproduced.

 

His measurements are reproduced with regularity. Also Schiit sending him a Modius for testing speaks for itself.

[Clockmeister]

30 minutes ago, plissken said:

 

His measurements are reproduced with regularity. Also Schiit sending him a Modius for testing speaks for itself.

 

This unfortunately tells me a great deal about that company, any quality audio equipment producing company worth its salt would have a AP555x or similar in the design section for basic audio THD/SINAD/BW and FFT base line markers. Why do you need to send it to non recognised third party with a public agenda for affirmation?

 Which are way beyond all forms of electrical and mechanical testing that any of the online or printed media periodicals can obtain

 

You would have consistency, accuracy and total repeatability, I have had the privilege 😂 of seeing PM's lab up close, its a slight step up form Amir's front room. How about a feature "who regulates the unofficial audio measurement mouth pieces". That would be worth a read.

 

You see people will follow which champion they feel comfortable with and 'suits' they current purpose or agenda relating to which particular equipment badge they are currently sporting or audio banner they are following.

Audio sound quality is far more than just an AP555x /R&S UPV etc these are tools which will help you in your quest to produce desirable saleable equipment bu they are not the holy grail of sound quality con-try to popular belief.

 

Here's a suggestion audio precision could donate (long term loan) a fully spec'd 555x to AS for its own testing put purposes with the pre requisite that its installed in a suitable location nd testing environment. 

 

 

[Jud]

2 hours ago, manisandher said:

 

Hey Jud, when I conducted the ABX test with Mans, each sample lasted for about 15 seconds. There was then a 5-10 second delay before the next sample started. The whole 10x ABX took around 15 minutes to undertake.

 

 

In the ABX, I was primarily picking up an ever-so-slight edginess in the piano in one and not in the other sample (both bit-identical to each other).

 

Hi Mani,

 

When you've got a recognizable "tell," I'd argue this is pattern recognition. I'd also speculate your ability to find the "tell" was helped along by years of adjusting XXHE parameters and hearing the results, which ISTM is training to recognize the telltale pattern(s). (This is a reason I'm very interested in how training effects relate to perceptual tests.)

 

2 hours ago, manisandher said:

 

By the end of the ABX, I was totally exhausted (having conducted a couple of non-ABX tests beforehand too). The experience was absolutely nothing like listening to music for pleasure. I suspect this is the main reason why ABX tests often 'fail'.

 

I'd speculate the closer the experience is to a test of discrimination, the more exhausting. But again, though it wasn't a strong signal (and thus took effort, leaving you a bit worn out), I'd opine you were looking for your "tell," and therefore doing pattern recognition.

 

2 hours ago, manisandher said:

 

I'm reading, and very much enjoying, 'The Master and His Emissary' by Iain McGilchrist at the moment. Maybe an ABX test is more of a left-brain activity, whereas listening to music for enjoyment is more of a right-brain activity? Not easy to reconcile the two.

 

Mani.

 

I think a lot of the "right-brain/left-brain" stuff has been debunked at this point.  

[Summit]

3 hours ago, manisandher said:

 

Hey Jud, when I conducted the ABX test with Mans, each sample lasted for about 15 seconds. There was then a 5-10 second delay before the next sample started. The whole 10x ABX took around 15 minutes to undertake.

 

 

In the ABX, I was primarily picking up an ever-so-slight edginess in the piano in one and not in the other sample (both bit-identical to each other).

 

 

By the end of the ABX, I was totally exhausted (having conducted a couple of non-ABX tests beforehand too). The experience was absolutely nothing like listening to music for pleasure. I suspect this is the main reason why ABX tests often 'fail'.

 

I'm reading, and very much enjoying, 'The Master and His Emissary' by Iain McGilchrist at the moment. Maybe an ABX test is more of a left-brain activity, whereas listening to music for enjoyment is more of a right-brain activity? Not easy to reconcile the two.

 

Mani.

 

What sounds different or to recognize spacial patterns (in a familiar system) is the easy part and can be tested by discrimination tests like you did. What “sounds right” is also relatively easy if we are to choose between a play back system and real musicians in the flesh playing live.

 

To determine which of two product, in a whole audio system, which sounds most correct is much harder because the audio system and recordings can never sound exactly like real musicians on stage. We are therefor in reality to determine, not which product that sound like real musicians on stage, but which that sounds closest to how it should sound, or more exact which gear that fool us best.

 

Few things are more difficult to establish as if one gear in an audio system actually “sounds right” or not. To determine if one specific part of a chain sounds right is next to impossible, because how do we know that the rest of the audio chain “sounds right” and the diversion from “sounds right” is to be blamed on the gear we are examining and not any other gears?

 

The main problem (as I see it) is that we have no real reference system in which we can compare gear. What we have is many thousands of audio system in as many homes that in practice are used as reference systems. Many systems also seems to be in a constant state of change.

 

Measurements remove many of the problems associated to lack of universal reference system, different in preference, the quality of the recordings etc. The only problem is that the correlation between measurements and what audiophiles contemplate “sounds right” is weak for high end gear.  

[plissken]

4 hours ago, Clockmeister said:

Why do you need to send it to non recognised third party with a public agenda for affirmation?

 

Your logic seems a bit broken. Are you saying miscellaneous can't measure and can't put out a regimen that you could use to reproduce or debunk said measurements. 

 

Could you see yourself suggesting shortcomings in the the current testing and suggest others or do you just want to throw tomatoes?

[pkane2001]

4 hours ago, Clockmeister said:

 

This unfortunately tells me a great deal about that company, any quality audio equipment producing company worth its salt would have a AP555x or similar in the design section for basic audio THD/SINAD/BW and FFT base line markers. Why do you need to send it to non recognised third party with a public agenda for affirmation?

 

Why? When Amir started measuring Schiit equipment, their performance was mediocre at best (that's the measurable performance). After getting criticized and after seeing poor results from a third party, Schiit implemented a test regime to QA all their equipment using an AP analyzer. They also started publishing detailed measurement reports for each product. Not only did these published measurements improve their image, but the performance of their equipment rose by an order of magnitude, from mediocre to top of the line. Instead of spending time and effort to defend the poor quality of their older products, their products now get a recommendation from Amir because they perform well. And the consumer gets a confirmation that the manufacturer isn't exaggerating by seeing independent measurements. A little third-party verification can work to the benefit of the consumer. Not a bad thing, IMO.

[Jud]

4 minutes ago, pkane2001 said:

 

Why? When Amir started measuring Schiit equipment, their performance was mediocre at best (that's the measurable performance). After getting criticized and after seeing poor results from a third party, Schiit implemented a test regime to QA all their equipment using an AP analyzer. They also started publishing detailed measurement reports for each product. Not only did these published measurements improve their image, but the performance of their equipment rose by an order of magnitude, from mediocre to top of the line. Instead of spending time and effort to defend the poor quality of their older products, their products now get a recommendation from Amir because they performs well. And his measurements confirm what's published by Schiit, so the consumer gets a confirmation that the manufacturer isn't exaggerating. A little third-party verification can work to the benefit of the consumer. Not a bad thing, IMO.

 

Interesting. This does somewhat suggest at least the possibility that parts and circuits can be tailored to produce a desired set of measurements, perhaps without substantial alterations in the sound the designers want (if they were able to recognize such a sound - both of the principals have decades of design experience).

[pkane2001]

8 minutes ago, Jud said:

 

Interesting. This does somewhat suggest at least the possibility that parts and circuits can be tailored to produce a desired set of measurements, perhaps without substantial alterations in the sound the designers want (if they were able to recognize such a sound - both of the principals have decades of design experience).

 

Another possible explanation is that the fans of a product or a manufacturer will often overlook (overhear?) significant problems that might be obvious to others. This might explain the variety of different opinions on how a specific piece of equipment "sounds". But that's just a theory, meant to show that there are other possible explanations. Other than the BIAS thread studies I've shared, I have no other data to back it up 

 

Yet another explanation, is of course, that with distortions below a threshold of audibility, the devices all sound the same.

[Jud]

5 minutes ago, pkane2001 said:

Yet another explanation, is of course, that with distortions below a threshold of audibility, the devices all sound the same.

 

This possibly implicates some of the things I was talking about earlier regarding training effects and pattern recognition. I doubt someone new to XXHE would have heard the "tell" @manisandher did. But if you can hear an effect with training, that means it's not inaudible, you're just not consciously noticing it. And even if you don't consciously notice it, it's possible that effect/distortion might be bothering you greatly.

 

(In the Iowa Gambling Task experiment I keep mentioning, subjects were given "good" and "bad" decks of cards. Subjects with "bad" decks showed elevated galvanic skin response - that is, they literally broke out in a sweat - on average several rounds of play before they consciously realized they had a bad deck and asked for a new one. So might we be made uncomfortable enough subconsciously by audio distortion we don't consciously notice to break out in a sweat? Interesting to think about. And if it were true, what would it mean for levels of audibility that have been established by tests that require conscious verbal responses?)