Misleading Measurements

[vmartell22]

On 9/22/2020 at 12:01 PM, Blake said:

Speaking of the topic of this thread, I'm not sure if this has been posted about and discussed here on AS, but I found it to be an interesting read as I have always had some suspicions about ASR:

 

https://www.superbestaudiofriends.org/index.php?threads/audio-science-review-review.9827/

 

 

Well, I do understand how ASR can rub people the wrong way. I remember when the fine peeps at ASR were losing their shit over the Yggy glitch....  Kept looking at the graphs and was thinking "well, this is obviously beyond any human's hearing thresholds - what is the big deal?" - yet well, it was a big deal over at ASR... ah well  - nobody's perfect, but all in all, we need ASR - they help keep manufacturers honest...

 

v

[vmartell22]

After 41 pages, suddenly had a realization re: the title of the thread and first post. The problem with it is that well, properly taken measurements are not misleading by nature (let's not consider errors in measurement for the moment - sure mistakes can happen leading to the wrong conclusion)

 

That said, need to clarify - why are measurements NEVER misleading? Well, because this thread and in general, in audiophile discussions this is looked at  incorrectly. Remember, these are engineered devices - how can a designer/manufacturer/engineer know that her device is working properly, according to spec? Measurements! - well, test plans/specs with functional and other kind of testing. All relying on measurements.

 

All that measurements prove is that the device is operating correctly according to spec.

 

That's it. Think of a medical device, for example, a pulse oximeter. Guess what ? That's all you need to know. You want that device to operate correctly within spec. 

 

The problem is that with audio, the output instead of raw cold numbers is well... sound. Sound that goes thru our VERY FALLIBLE human hearing to our brain which can influence the perception in many ways - ah that's all the difference. And the root of all our disagreement. Because ( I have said this. MANY times, I accept it, apologies for the repetition) to assign qualities like "musicality" to an audio device is as silly as assigning a quality like "empathy" to a medical device... sure all patients want that. And is kind of related. BUT of course, not applicable to a device!

 

Measurements are not misleading - they only show whether the device is operating correctly or not.

 

Of course - the argument can be continued from there.  For example, the great late Peter Aczel said, "any properly designed amplifier is completely transparent". Measurements help us determine the "properly designed" part. But this is kind of another discussion isn't it?

 

v

 

[vmartell22]

17 hours ago, vmartell22 said:

 

 

Measurements are not misleading - they only show whether the device is operating correctly or not.

 

 

 

 

10 minutes ago, manueljenkin said:

I have nothing against measurements, they are a good way to ensure "components" of a system are not broken. If you have 1000 systems being made, measurements can be used to check if the capacitor or any other component in one of the system has failed by comparing with the other measurements that's all! 

 

 

We agree! great! Great minds think alike... 

 

v

[vmartell22]

6 hours ago, pkane2001 said:

Maybe you can reach out to him and have him post his results.

 

And his methodology!   

[vmartell22]

14 minutes ago, manueljenkin said:

Well his profession involves rigorous statistics, and he often persuades me to do ABX so I doubt he has any problem with the methodology. Shoot me all your questions precisely (what you expect, what numbers etc). and I'll ask him when he's free.

 

Has the experiment been peer reviewed as well as the results?

 

v

[vmartell22]

13 hours ago, manueljenkin said:

I don't see any certifiable peer reviews on the "conclusions" asr and the echo chamber makes either. I don't see any IEEE certifications or similar at ASR. It's just one person's opinion, and his personal interpretations of what he sees in a chart (extreme possibility of bias). Just because the site name has the word "science" in it doesn't mean it automatically gets credibility as a reliable source of information. Fyi there's a subreddit called r/sciencememes.

 

If that doesn't have a peer review to be accepted by you and the objective club, no reason this has to be. So here's a task, get the asr place and analysis articles certified and let's talk 😊 about us refuting it. Not much time to refute a non reliable source of information with massive holes. It is non reliable/inconclusive, end of story, things can swing either way. For myself, I personally experience the changes described with usb cables, software swaps, etc despite sine squiggle measurements of an analyzer approving zero change. No one has to "prove" it to me. I accept certain things that are real don't have a clear explanation yet.

 

The thing is whatever you find wrong with ASR's approach, at least it is supported by formally accepted principles - that's a good start even if the individual members are not leading lights of the IEEE or ACM or AES or..

 

https://en.wikipedia.org/wiki/Straw_man

 

The thing is that all we have here is argument by unknown/unverified authority...  and I am not saying that one should not compare, experiment, etc - all I am saying is that without reviewed and accepted rigor, you statician friend's results are no different from just another anecdote/opinion... no proof of anything...

 

Really have no desire to continue the argument - You do you!  Be Happy! - I will get out of our hair... not sure why I even replied in the first place - I will confess, prbly the reason is that "argument by authority" always makes me reply...  ah well 

 

we will never convince each other... no point I guess... such is life!

 

peace

 

v

[vmartell22]

49 minutes ago, manueljenkin said:

No this is not straw-man. I put forward extremely legit points. Supported by formally accepted principles? Like what? That you should be fussing about 110db SNR vs 120db SNR because to them "lower number is always better"? Yeah let's ignore audibility thresholds and relative weights. That everything about sq is determined by a limited set of measurements? You have completely ignored the entire points I have made and the real arguments. Here, let me quote them for you one more time.

 

I did not say his is a "proof" of something. I just said it's about as valid as, if not more than the conclusions written at ASR. I am saying the current measurements you have donot give you the authority to bully someone who perceives a change (opinion bullying to destroy their credibility). As of now, things are inconclusive, and in such a situation, the benefit of doubt is on the subjectivist's side, and I would rather be interested in someone exploring the causation properly now. Trying to cocoon things will do no one any benefit

 

Well

 

1.- You kindly mentioned "audibility thresholds" above - it looks like you are finally seeing things correctly  - I love bringing people to the right approach - that is a big component and one of the reasons  subjective claims are basically unprovable... but as long as you know...

 

2.- In previous post I indicated that measurements are never misleading,  they just prove that the devices are operating correctly. The difference of course is not trying to adjudicate other magical properties to the devices. Next thing we know people are gonna say a certain dac or amplifier has "a refined sound with hints of vanilla and notes of peppermint and clorox"  :D

 

3.- The "benefit of the doubt" does not apply to science and engineering endeavours at all... yes, is on the subjective idea, because, it cannot be on the other side by definition. Nothing gets the benefit of the doubt. Prove every claim.

 

v

[vmartell22]

21 hours ago, fas42 said:

 

And where on the spec sheet for some consumer audio gear would I find that?

 

It would be on the design document - from there, after a risk analysis,  a test plan is created. Part of the test plan is functional testing.  Whatever you are aiming for, like 100% noise rejection, there will be a test case for that. The test case will specify thresholds, tolerances, etc. Data points will be taken using... measurement devices, not by ear. And pass/fail will be determined.

 

We are going off in many tangents.  IMHO the OP was pretty much about what do you do when a device measures well and you THINK you don't like the sound or when it measures terribly and you THINK you love the sound.

 

Might the measurements be misleading, asks the OP?

 

Of course not. What should you do? Well, what the audio enthusiasts have been doing all this time. Buy based on other factors. Some will rationalize and try to make it about audio, others will happily admit they just wanted the object of desire.

 

That's it. Close the thread. I will be going over to a music discussion forum.  EVERYTHING is subjective on that matter, so at least having the endless discussions, flame wars, arguments via logical fallacies make more sense to me...

 

 

v

[vmartell22]

20 minutes ago, fas42 said:

 

Yes, that's how it should work ... so why do all real world audio components then fail ... ? It's trivially easy for me to inject some real world electrical noise into the environment of the replay setup, and hear the SQ degrade ...

 

Meaning, that the test cases, if used, are not strict enough - engineering of audio circuitry doesn't cut the mustard; and therefore the consumer then has to complete the work that should have been done in the factory.

 

 

 

Injecting noise... well that IS definitely a test case. Remember the part about thresholds and tolerances?  some devices will pass or not based on least or more stringent conditions - it all depends on the pass/fail criteria defined by the designer and course will reflect the design philosophy. Human hearing? not part of formal testing.

 

OF COURSE, things escape the testers, or maybe devices are not tested at all. Or the test plan is badly designed. Or the device  is shoddily built. Or there is a manufacturing QC problem.  But that does not mean we should discard formal measurements and go by ear. 

 

v

 

 

[vmartell22]

7 hours ago, MarkusBarkus said:

...I would just add that there would also be a few assumptions underlying the test/QA plan, that might constitute "normal usage/conditions." 
 

Perhaps the test case: "Turn on arc welder on adjacent power feed..." would not constitute a normal test case. And that exception wouldn't likely appear in consumer documentation: Warning, do not run device while neighbor is arc welding." Although the the caveats that do appear sometimes read as if written for consumers new to the planet.

 

Not that we're voting, but may I cast one for measure and listen?

 

Of course - problem is that listening is in the "ear of the earholder"- I might declare a device as ok but some other person might say is crap. Then you add biases - I am pretty sure there could be listeners that will declare anything over $10K as super fantastic no matter what...

 

So yeah let's is ok to listen - is just not reliable.. by definition...

 

v