Jump to content
IGNORED

Misleading Measurements


Recommended Posts

On 11/14/2020 at 7:59 AM, pkane2001 said:

 

I can, but I don't know if you can. And that's the point -- your experience is nothing like mine. What you hear is a mystery to most people, even though you keep talking about it all the time. There's nothing objective to what you're reporting or to your metric. It's purely subjective. 

It's not really mystery to me though. I may not be able to hear as good as his system, but I do have experiences closer to his. And so do many others here. 😊

Link to comment
17 hours ago, vmartell22 said:

After 41 pages, suddenly had a realization re: the title of the thread and first post. The problem with it is that well, properly taken measurements are not misleading by nature (let's not consider errors in measurement for the moment - sure mistakes can happen leading to the wrong conclusion)

 

That said, need to clarify - why are measurements NEVER misleading? Well, because this thread and in general, in audiophile discussions this is looked at  incorrectly. Remember, these are engineered devices - how can a designer/manufacturer/engineer know that her device is working properly, according to spec? Measurements! - well, test plans/specs with functional and other kind of testing. All relying on measurements.

 

All that measurements prove is that the device is operating correctly according to spec.

 

That's it. Think of a medical device, for example, a pulse oximeter. Guess what ? That's all you need to know. You want that device to operate correctly within spec. 

 

The problem is that with audio, the output instead of raw cold numbers is well... sound. Sound that goes thru our VERY FALLIBLE human hearing to our brain which can influence the perception in many ways - ah that's all the difference. And the root of all our disagreement. Because ( I have said this. MANY times, I accept it, apologies for the repetition) to assign qualities like "musicality" to an audio device is as silly as assigning a quality like "empathy" to a medical device... sure all patients want that. And is kind of related. BUT of course, not applicable to a device!

 

Measurements are not misleading - they only show whether the device is operating correctly or not.

 

Of course - the argument can be continued from there.  For example, the great late Peter Aczel said, "any properly designed amplifier is completely transparent". Measurements help us determine the "properly designed" part. But this is kind of another discussion isn't it?

 

v

 

The resolution requirements of an oximeter and an audio chain is not the same. The former gives a reading that is used to make more of less a binary decision (whether the person is alright or needs treatment) and even if it were more than a binary decision it's mainly based on "cut offs".

 

Contrast that to audio where we donot have any proper correlation factors to each parameter being measured (or not measured), the comparison gets broken and not extendable. Atleast ADC is somewhat closer to the oximeter in that they both capture and store a representation of what is already present, while a audio reproduction system needs to create the whole thing again, and adds complexity to the analysis.

 

As for Peter aczel, it's just one person's opinon and that too very open ended. What are the necessary tests and parameter cut offs, with respective weights to have a "Conclusive" analysis of sound quality of a system is still an area of mystery. I have nothing against measurements, they are a good way to ensure "components" of a system are not broken. If you have 1000 systems being made, measurements can be used to check if the capacitor or any other component in one of the system has failed by comparing with the other measurements that's all! It's only relative to other implementations of the same design, it's not a valid conclusive metric to compare final performance of different designs as of now.

Link to comment
11 hours ago, pkane2001 said:

 

So you too hear amazing sound coming out of the built-in laptop speakers? ;)

 

I can hear quite nice sound coming out of my phone speakers the moment I swapped my music player software to usb audio player Pro. I also hear much better sound out of a superlux hd562 + a decent amp combo than hd600 with a sub-par amp (mind you, this sub par amp is better than most mobiles, it's a dap).

 

I feel I am more discerning of aberrations in digital playback chain, than from a transducer so much so that my split up in pricing for components would be 40% for dac, 40% for amp and 20% for the transducer.

Link to comment
10 minutes ago, pkane2001 said:


Would it be asking for too much to use at least a shred of something objective in your arguments in an objective forum, Frank?

Well the designs being successful and well received for decades is a fact, and there exists a big population of engineers and others who have heard the "asr approved" gear and these and preferred the voodoo, as you say! All of these are factual. Maybe there's more to it after all, a gremlin you just don't seem to understand, or refuse to comprehend.

Link to comment
On 11/14/2020 at 8:54 AM, manueljenkin said:

Let me share my thoughts on this topic. Most of this is a copypasta (with minor modifications) from my messages in another forum, so you might be having answers even before your questions 😜.


1. Most of the people from objectivist cult speak for themselves, but frame it in a way as if it were universal truth for everyone. Not everyone needs to have the same needs or interests as they do. Armchair opinions, thinking every audible parameter is covered with a limited subset of measurements (most of them static or steady state) form a major part of such conversations. Oh you prefer a different amp topology or design? They claim that you MUST be hearing and preferring distortion,  completely dismissing every other possibility. As of today, we can't CONCLUDE things, especially relating to audibility limits, with the limited set of measurements being done generally.


2. Science often can't prove that something can never occur. It's just a framework to predict the behavior and outcome of something using past experiences, experiments and reasoning. They may anytime be over written. For this, I get a counter argument of something like say planck's constant and speed of light is unlikely to be overwritten anytime soon. True, but it is important to remember that there were so much abstractions earlier before we got to the level of quantum physics understanding we have today. We are not sticking to the John Dalton's model anymore which probably was the best approximation a couple of centuries ago. Comparing audio system analysis to Planck's constant or speed of light is strawman-hat argument since they are direct physical phenomenon which have now been well understood to quite a good level, while audio is cognition related. There is still plenty of research happening on cognition, especially audio. Music isn't really a mystery, cognition is. It's quite hard to probe and correlate what performs what functions there and we only get a black box view, and ears being super tied to the brain doesn't help it much. Spatial properties and object detection is still something not well understood beyond a basic abstraction. The door is wide open in this area for potential changes.


3. Throwing the burden of proof on the one who hears changes. Well telling there shouldn't be a difference is a claim too! Burden of proof must be on either ways of claim. And again, you can't prove there will be no perceivable difference without getting cognition solved. Sure you got "some measurements" but as said above, they are not Conclusive and certainly doesn't grant you permission to demean someone else's choices and experiences.


4. Assumption of weights to different performance parameters (measured or un-measured). You certainly can't make claims on what measurements correlate to perceivable changes, the weights for different performance metric on audibility and what type of parameters need to be measured additionally unless you understand cognition properly. A big part of understanding cognition is to check what types of input it is responsive and not responsive to. Cognition is also relating to precision of detectability of different parameters (including non-desirable influences from parameters), and the location estimation is a use case for which the brain developed to this precision.

As expected @pkane2001 has conveniently ignored all the valid points against asr I've mentioned here.

Link to comment
8 minutes ago, pkane2001 said:


What points and why would I be defending ASR? It’s not my site, I post there just like I post here. Why would you expect me to defend anyone against your attacks? Very strange discussion, indeed.

It's not an attack it's a valid argument against the current state of measurements you so happen to promote actively.

 

And in fact you're the one who's attacking opinions differing from you by using manipulative words to either distract from the thread or try to belittle the credibility of the person in context.

Link to comment
6 hours ago, pkane2001 said:


Provide objective evidence for any of your arguments, and not what someone somewhere claimed that they heard. There are probably thousands of different measurements on ASR, and I’m certainly not going to defend every one of them. 

I have a friend who does DBT on pretty much every gear he buys. He doesn't like to be around forums, but you can reach out to him if you're interested.

 

He runs a rme adi2 (which is touted by asr measurements to be completely impervious to upstream changes wrt usb, system scheduling) and found enough improvements on it using sonore switch. He has done enough DBT on this aspect.

 

He finds asr measurements to be almost irrelevant to the results he has got via DBT on many devices.

 

Just one of many examples.

 

Link to comment
5 hours ago, vmartell22 said:

 

Has the experiment been peer reviewed as well as the results?

 

v

I don't see any certifiable peer reviews on the "conclusions" asr and the echo chamber makes either. I don't see any IEEE certifications or similar at ASR. It's just one person's opinion, and his personal interpretations of what he sees in a chart (extreme possibility of bias). Just because the site name has the word "science" in it doesn't mean it automatically gets credibility as a reliable source of information. Fyi there's a subreddit called r/sciencememes.

 

If that doesn't have a peer review to be accepted by you and the objective club, no reason this has to be. So here's a task, get the asr place and analysis articles certified and let's talk 😊 about us refuting it. Not much time to refute a non reliable source of information with massive holes. It is non reliable/inconclusive, end of story, things can swing either way. For myself, I personally experience the changes described with usb cables, software swaps, etc despite sine squiggle measurements of an analyzer approving zero change. No one has to "prove" it to me. I accept certain things that are real don't have a clear explanation yet.

Link to comment
2 hours ago, pkane2001 said:

 

Let me summarize the 43 pages of this thread for you, so you don't have to keep repeating it:

 

ASR sucks, we don't know everything, measurements are wrong because we hear things, and noise hidden in two, exactly the same digital files is different because one was recorded with an LPS and the other with a switching supply. Got it. Carry on. 

You're misquoting me. Measurements need not be "wrong". Just that the notion that all parameters of sound quality can be conclusively determined by the limited measurements done at asr is wrong. I haven't tried the digital file differences yet, so can't say much but "I can believe it" since I've had similar experience on another operation.

Link to comment
22 minutes ago, The Computer Audiophile said:

Come on Paul, only 42.5 of the pages are about that :~)

 

I saw a great quote yesterday. I'm sure many have seen if before, but it was new to me. I'm also sure many in the Objective-Fi form will like and know it well. 

 

You can't convince a believer of anything; for their belief is not based on evidence, it's based on a deep seated need to believe.


― Carl Sagan

Well we have an outlier here I guess 😅. I used to "believe" usb cables should make no difference to an ASR certified dac 😛 and once in a while used to troll the believers, now I'm on the other side of the club 😁. Maybe we could apply uncertainty/probability based estimates to this quote!!

Link to comment
3 hours ago, vmartell22 said:

 

The thing is whatever you find wrong with ASR's approach, at least it is supported by formally accepted principles - that's a good start even if the individual members are not leading lights of the IEEE or ACM or AES or..

 

https://en.wikipedia.org/wiki/Straw_man

 

The thing is that all we have here is argument by unknown/unverified authority...  and I am not saying that one should not compare, experiment, etc - all I am saying is that without reviewed and accepted rigor, you statician friend's results are no different from just another anecdote/opinion... no proof of anything...

 

Really have no desire to continue the argument - You do you!  Be Happy! - I will get out of our hair... not sure why I even replied in the first place - I will confess, prbly the reason is that "argument by authority" always makes me reply...  ah well 

 

we will never convince each other... no point I guess... such is life!

 

peace

 

v

No this is not straw-man. I put forward extremely legit points. Supported by formally accepted principles? Like what? That you should be fussing about 110db SNR vs 120db SNR because to them "lower number is always better"? Yeah let's ignore audibility thresholds and relative weights. That everything about sq is determined by a limited set of measurements? You have completely ignored the entire points I have made and the real arguments. Here, let me quote them for you one more time.

 

I did not say his is a "proof" of something. I just said it's about as valid as, if not more than the conclusions written at ASR. I am saying the current measurements you have donot give you the authority to bully someone who perceives a change (opinion bullying to destroy their credibility). As of now, things are inconclusive, and in such a situation, the benefit of doubt is on the subjectivist's side, and I would rather be interested in someone exploring the causation properly now. Trying to cocoon things will do no one any benefit

Link to comment
On 11/14/2020 at 8:54 AM, manueljenkin said:

Let me share my thoughts on this topic. Most of this is a copypasta (with minor modifications) from my messages in another forum, so you might be having answers even before your questions 😜.


1. Most of the people from objectivist cult speak for themselves, but frame it in a way as if it were universal truth for everyone. Not everyone needs to have the same needs or interests as they do. Armchair opinions, thinking every audible parameter is covered with a limited subset of measurements (most of them static or steady state) form a major part of such conversations. Oh you prefer a different amp topology or design? They claim that you MUST be hearing and preferring distortion,  completely dismissing every other possibility. As of today, we can't CONCLUDE things, especially relating to audibility limits, with the limited set of measurements being done generally.


2. Science often can't prove that something can never occur. It's just a framework to predict the behavior and outcome of something using past experiences, experiments and reasoning. They may anytime be over written. For this, I get a counter argument of something like say planck's constant and speed of light is unlikely to be overwritten anytime soon. True, but it is important to remember that there were so much abstractions earlier before we got to the level of quantum physics understanding we have today. We are not sticking to the John Dalton's model anymore which probably was the best approximation a couple of centuries ago. Comparing audio system analysis to Planck's constant or speed of light is strawman-hat argument since they are direct physical phenomenon which have now been well understood to quite a good level, while audio is cognition related. There is still plenty of research happening on cognition, especially audio. Music isn't really a mystery, cognition is. It's quite hard to probe and correlate what performs what functions there and we only get a black box view, and ears being super tied to the brain doesn't help it much. Spatial properties and object detection is still something not well understood beyond a basic abstraction. The door is wide open in this area for potential changes.


3. Throwing the burden of proof on the one who hears changes. Well telling there shouldn't be a difference is a claim too! Burden of proof must be on either ways of claim. And again, you can't prove there will be no perceivable difference without getting cognition solved. Sure you got "some measurements" but as said above, they are not Conclusive and certainly doesn't grant you permission to demean someone else's choices and experiences.


4. Assumption of weights to different performance parameters (measured or un-measured). You certainly can't make claims on what measurements correlate to perceivable changes, the weights for different performance metric on audibility and what type of parameters need to be measured additionally unless you understand cognition properly. A big part of understanding cognition is to check what types of input it is responsive and not responsive to. Cognition is also relating to precision of detectability of different parameters (including non-desirable influences from parameters), and the location estimation is a use case for which the brain developed to this precision.

Here it is. You've made me quote this for the second time (and I had to since you are completely ignoring the valid arguments).

Link to comment
On 11/15/2020 at 8:53 AM, manueljenkin said:

The resolution requirements of an oximeter and an audio chain is not the same. The former gives a reading that is used to make more of less a binary decision (whether the person is alright or needs treatment) and even if it were more than a binary decision it's mainly based on "cut offs".

 

Contrast that to audio where we donot have any proper correlation factors to each parameter being measured (or not measured), the comparison gets broken and not extendable. Atleast ADC is somewhat closer to the oximeter in that they both capture and store a representation of what is already present, while a audio reproduction system needs to create the whole thing again, and adds complexity to the analysis.

 

As for Peter aczel, it's just one person's opinon and that too very open ended. What are the necessary tests and parameter cut offs, with respective weights to have a "Conclusive" analysis of sound quality of a system is still an area of mystery. I have nothing against measurements, they are a good way to ensure "components" of a system are not broken. If you have 1000 systems being made, measurements can be used to check if the capacitor or any other component in one of the system has failed by comparing with the other measurements that's all! It's only relative to other implementations of the same design, it's not a valid conclusive metric to compare final performance of different designs as of now.

Those "formally accepted" techniques are not used to assess absolute sonic performance, as done at asr. It is just to ensure whether the circuit behaves in line with the rest of the devices in the chain. And to an extent, the designer can relate to whether a design choice relates to a "measurable parameter", doesn't necessarily say anything about sq in terms of audible weights and thresholds. There are other areas like the oximeter example where the correlation of measured numbers and the analysis/task to be done is fairly well established, but it is not the case for audio (reason: read the previous post quote).

Link to comment
1 hour ago, vmartell22 said:

 

Well

 

1.- You kindly mentioned "audibility thresholds" above - it looks like you are finally seeing things correctly  - I love bringing people to the right approach - that is a big component and one of the reasons  subjective claims are basically unprovable... but as long as you know...

 

2.- In previous post I indicated that measurements are never misleading,  they just prove that the devices are operating correctly. The difference of course is not trying to adjudicate other magical properties to the devices. Next thing we know people are gonna say a certain dac or amplifier has "a refined sound with hints of vanilla and notes of peppermint and clorox"  :D

 

3.- The "benefit of the doubt" does not apply to science and engineering endeavours at all... yes, is on the subjective idea, because, it cannot be on the other side by definition. Nothing gets the benefit of the doubt. Prove every claim.

 

v

1. Audibility thresholds and weights is still and active area of research. So nothing can be analysed conclusively at this point of time. So if the claimers don't have proof, the same way the naysayers don't have proof either (until you solve cognition).

 

2. One can form any correlation they want based on experience, since the measurements donot seem to show the whole picture. Proving it, is unlikely, agreed. Doesn't mean it exists, doesn't mean it doesn't exist. It is uncertain, it's up to the audience to try and come to a conclusion. No one has the right to opinion bully them to "not try/explore".

 

3. I don't really know what you're trying to say here. You're saying we shouldn't doubt anything, and should have stayed with the 14th century thought that earth is flat? See point 1 again, has the answer to "prove every claim"? The claim that they cannot make any changes to sound quality is not proven either.

Link to comment
1 hour ago, John Dyson said:

IMO, avoiding scientific methods on purpose is masochistic - or perhaps wish for a forever hobby of tweaking. I AM convinced that there can be individuals who have trained their hearing to be accurate, but such training and testing also needs to be based upon scientific method.  Just using experience for a basis of such training is probably not sufficient.  It seems that there must to be a scientific basis somewhere , even if the technique appers to  have elements of being subjective.

 

I am not against scientific methods, if the structure is well correlated and conclusive. As of now it is not, so why run behind something inconclusive to derive conclusions? And you're certainly on an abusive tone here trying to say anyone who is disagreeing with the limited set of measurements, because they have experienced a different correlation, are wasting their money. Not really!

 

Also I have counted the number of times you have used the word "engineer" (atleast 7 times) as if it's your last resort to showcase your opinion with higher weight. I am also an engineer, so are many people in this forum who do believe in the changes not measured. You can call yourself as a good or bad engineer and I don't mind that (good and bad is qualitative, and can be personal) but I don't think that qualifies you to make absolutist statements on the performance indexes for measurements of sound reproduction systems.

 

1 hour ago, John Dyson said:

You should never get an argument from a true engineer or scientist about 'sounds better' -- that cannot be measured except by someone who is honestly listening.  However, there are objective measures&methods and most often there are good ways of testing for differing levels of technical quality.   Without the techniques needed for real engineering solutions, then we get  much too many long random walks of design (which I have recently been involved in) or misleading claims.

Well that's more of a task for a researcher who is into psychoacoustics than an engineer or a scientist. And that's the very same argument that invalidates anything that is being said wrt correlation to measurements without the approval of someone into psychoacoustic research.

 

2 hours ago, pkane2001 said:

 

I suggest you look up the definition of objectivity. Something very much lacking in all your arguments. Objectivity doesn't mean the "objectivist cult" that you've referred to. It's an underlying principle of all the sciences and engineering. Its primary purpose is to remove biases, opinions and fervent beliefs when trying to explain the natural world. All you've expressed here so far are opinions and beliefs, none of which qualify as objective.

 

I do know it's definition. So here's a sentence from the same page - "Next to unintentional and systematic error, there is always the possibility of deliberate misrepresentation of scientific results, whether for gain, fame, or ideological motives." Guess what? Thats exactly what is happening at ASR.

Link to comment

Again, our friend John, is trying to push something as end all be all, when it is just him speaking for himself. I'm sorry John, if the test results felt conclusive enough for us we would have actually embraced it. I make a significant portion of my electronics purchases - say a display, or a camera, or a lens with a lot of weight to measurements. And when I say measurement, I refer to a rigorous analysis as done at anandtech, or notebookcheck, and for cameras I would also put some weight on video/camera samples from early adopters and users at Vimeo/flickr. Unfortunately, the measurement at ASR is not at all rigorous.

 

I do have a few other places which I do trust a little bit for measurements. Goldenears used to be a great resource for headphone measurements, measuring true impulse responses and CSD. Today I find, this page to be a valuable source of information : https://reference-audio-analyzer.pro/en/report/hp/shure-srh-1840.php . It is not conclusive, but it sure is not misleading. All you see is graphs, not one person's opinion or interpretation at the end. They also measure amplifiers, but by a very simple metric yet measure certain things that ASR doesn't measure. They test it across various loads, and actually index it into equivalent class A, Class B, Class D performance (though I don't like this tag, since implementation >> buzzword, it gets the job done). Again it's just numbers and is good enough to know if the amp is suitable for my use case or not (and not a gauge for absolute performance relating to audibility). High oi, ok it's fine if my headphone is a flat impedance profile or very high impedance, provided the amp has other parameters straight. Now this is not a conclusive test either but there is no personal agenda like at ASR where they just correlate SNR to be an end all be all index when in reality that claim is unfounded. The amps that they tend to show in bad light at ASR are still well above the known audibility levels in those indexes. And personal opinions like "bad engineering" or whatever, when it really is not, it's just not overoptimized for the numbers ASR cares about. They are optimized for few other things that have been experienced to be of better sq perceived by humans, but not yet conclusively correlated, that's all.

 

And a lot of have experiences where what did not show a change in ASR still showed meaningful change in sound quality.

 

Link to comment
On 11/18/2020 at 6:41 PM, pkane2001 said:

 

I can use my tongue on the battery terminals to get a subjective observation of the remaining charge, but a voltmeter is going to produce a much more useful and accurate "observation". As long as the measurement device is properly calibrated and the error margins are known, there's no reason to keep relying on the difficult to perform and hard to validate subjective results.

What a wonderful straw-man hat argument as usual from @pkane2001 . A battery is used to power up a device. It's the devices requirements and correlation indexes that matter. (Unless you're a bot running on battery power 😅). And we have fairly established results correlating to the static/dynamic and voltage/current requirements of most loads, or atleast to the accuracy of what we want the load to perform at.

 

An audio reproduction system is used to cater to a human hearing system (ears+brain). So the correlation indexes that matter should come from humans. And it's not well defined yet! So you can't conclude using tests from ASR as conclusive means. End of story. 

Link to comment
5 hours ago, pkane2001 said:

 

Seems you totally missed the context. Oh well. But funny how everything winds up being about ASR.

 

Look, produce something interesting and objective for us to discuss, something other than your opinion or I'm simply going to stop responding to you.

I'm pretty much on line. You're the one missing the point. I gave a link to another place which I find to be a reasonable objective information source.

 

Also, thought for the day. Bacteria objectively existed before we could Identify them.

Link to comment
1 hour ago, yamamoto2002 said:

 

Fluorescent light glow starter and refrigerator are also problematic

 

It seems it is better to avoid office chair with gas lift as well on critical audio listening 🙂

 

https://support.displaylink.com/knowledgebase/articles/738618-display-intermittently-blanking-flickering-or-los

Turn off the lights, sit on the floor and meditate when listening 😊. Two tasks done in one shot.

 

Note: I've actually tried this, no clue of sound quality changes, but it sure is more enjoyable than otherwise.

 

Side note: display link is like the worst case scenario - adapting a standard made for a specific, although general purpose'y task, to do something that's generally not expected of it. Not that I'm saying hdmi, or displayport would be completely rid of the problems that displaylink would show, but they are less likely to be affected and/or show much lesser deviations/aberrations (or a different type of, subjectively less jarring) in similar scenarios.

Link to comment

Hearing is mostly an involuntary process. Not sure what is there to specifically say "to use ears properly". Maybe if this was regarding safe limits, it could have been phrased, "be careful". Or are you suggesting to have further rigorous training on mind (eg, yoga, etc) since a significant part of the hearing process happens in the brain?

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now



×
×
  • Create New...