Misleading Measurements

[pkane2001]

Just now, sandyk said:

It's not a personal attack. It's an observation. Otherwise , why have you gone to the trouble of designing  the S/W linked to in your signature ?

 

But of course it is! You just told me I don't like or enjoy music and that I don't belong on an Audiophile forum.

 

We are worlds removed, Alex, and not just geographically. Don't try to read anything into what I do or why I do it, you'll be wrong. 

 

[sandyk]

23 minutes ago, pkane2001 said:

 

But of course it is! You just told me I don't like or enjoy music and that I don't belong on an Audiophile forum.

 

We are worlds removed, Alex, and not just geographically. Don't try to read anything into what I do or why I do it, you'll be wrong. 

 

 I didn't say that you don't like or enjoy music, but you appear to get a great deal more enjoyment out of measurements of equipment than listening to music with them. In my case, I am fanatical about extracting as much as possible out of my equipment in order to keep enjoying music longer due to my advanced age (approaching 82) and hearing damage.

To do this, I end up listening to a lot more music than  many do, and perhaps get to enjoy it more too.

 

 Incidentally, even Archimago will tell you that people with hearing damage are sometimes able to hear things that others are unable to, just as J.D. found not so long back when picking up on actual distortion much more when he had a Eustachian tube blockage.

EDITED .

[pkane2001]

1 minute ago, sandyk said:

 I didn't say that don't like or enjoy music, but you appear to get a great deal more enjoyment out of measurements of equipment than listening to music with them. In my case, I am fanatical about extracting as much as possible out of my equipment in order to keep enjoying music longer due to my advanced age (approaching 82) and hearing damage.

 

Do you know anything about what I do or why I do it, Alex? Stop talking about me and my motivations and focus on the topic.

[fas42]

Well, I got around to listening the 3rd instance of the files Alex provided, and things had changed! At first there seemed to be a difference between the two original uploads, but this faded, and at the end all three were coming across as sounding close enough to identical.

 

But, things had changed on my laptop! Yesterday I installed a piece of software which added to the kernel of Windows, which allowed a capture of streaming audio; a restart was necessary to bring it on line. And, some instability, hopefully transient, was now introduced - trying to play a file using Media Monkey crashed Windows 10 - first time ever on this laptop. Plus, every so often the System process, the Windows kernel, goes into some strange space, spinning its wheels for no obvious reason, chewing up about 30% CPU - it was doing this this morning, and nothing one does seems to be able to throw it out of the loop; eventually it goes away, or I do a reboot.

 

Which could all add up to changing what I was hearing ... to be blunt, I don't want this sort of 'complexity' in the mix - too many complications already; I want this one to be at at the bottom of the pile ... sorry about that, Alex 🙂.

[fas42]

6 hours ago, manueljenkin said:

Well we have an outlier here I guess 😅. I used to "believe" usb cables should make no difference to an ASR certified dac 😛 and once in a while used to troll the believers, now I'm on the other side of the club 😁. Maybe we could apply uncertainty/probability based estimates to this quote!!

 

Nothing helps like having a rig right on the edge of either sounding brilliant, or nothing special - I have never had a system so clearly in one camp or the other as the one 3 decades ago, and it makes one highly sensitive to what's going on. And so you learn that the slightest tipping of one factor, in a certain direction, can be absolutely critical - it always, always comes comes down the thinking, I'm Removing Badness, in how one approaches optimising a setup.

[manueljenkin]

3 hours ago, vmartell22 said:

 

The thing is whatever you find wrong with ASR's approach, at least it is supported by formally accepted principles - that's a good start even if the individual members are not leading lights of the IEEE or ACM or AES or..

 

https://en.wikipedia.org/wiki/Straw_man

 

The thing is that all we have here is argument by unknown/unverified authority...  and I am not saying that one should not compare, experiment, etc - all I am saying is that without reviewed and accepted rigor, you statician friend's results are no different from just another anecdote/opinion... no proof of anything...

 

Really have no desire to continue the argument - You do you!  Be Happy! - I will get out of our hair... not sure why I even replied in the first place - I will confess, prbly the reason is that "argument by authority" always makes me reply...  ah well 

 

we will never convince each other... no point I guess... such is life!

 

peace

 

v

No this is not straw-man. I put forward extremely legit points. Supported by formally accepted principles? Like what? That you should be fussing about 110db SNR vs 120db SNR because to them "lower number is always better"? Yeah let's ignore audibility thresholds and relative weights. That everything about sq is determined by a limited set of measurements? You have completely ignored the entire points I have made and the real arguments. Here, let me quote them for you one more time.

 

I did not say his is a "proof" of something. I just said it's about as valid as, if not more than the conclusions written at ASR. I am saying the current measurements you have donot give you the authority to bully someone who perceives a change (opinion bullying to destroy their credibility). As of now, things are inconclusive, and in such a situation, the benefit of doubt is on the subjectivist's side, and I would rather be interested in someone exploring the causation properly now. Trying to cocoon things will do no one any benefit

[manueljenkin]

On 11/14/2020 at 8:54 AM, manueljenkin said:

Let me share my thoughts on this topic. Most of this is a copypasta (with minor modifications) from my messages in another forum, so you might be having answers even before your questions 😜.

1. Most of the people from objectivist cult speak for themselves, but frame it in a way as if it were universal truth for everyone. Not everyone needs to have the same needs or interests as they do. Armchair opinions, thinking every audible parameter is covered with a limited subset of measurements (most of them static or steady state) form a major part of such conversations. Oh you prefer a different amp topology or design? They claim that you MUST be hearing and preferring distortion,  completely dismissing every other possibility. As of today, we can't CONCLUDE things, especially relating to audibility limits, with the limited set of measurements being done generally.

2. Science often can't prove that something can never occur. It's just a framework to predict the behavior and outcome of something using past experiences, experiments and reasoning. They may anytime be over written. For this, I get a counter argument of something like say planck's constant and speed of light is unlikely to be overwritten anytime soon. True, but it is important to remember that there were so much abstractions earlier before we got to the level of quantum physics understanding we have today. We are not sticking to the John Dalton's model anymore which probably was the best approximation a couple of centuries ago. Comparing audio system analysis to Planck's constant or speed of light is strawman-hat argument since they are direct physical phenomenon which have now been well understood to quite a good level, while audio is cognition related. There is still plenty of research happening on cognition, especially audio. Music isn't really a mystery, cognition is. It's quite hard to probe and correlate what performs what functions there and we only get a black box view, and ears being super tied to the brain doesn't help it much. Spatial properties and object detection is still something not well understood beyond a basic abstraction. The door is wide open in this area for potential changes.

3. Throwing the burden of proof on the one who hears changes. Well telling there shouldn't be a difference is a claim too! Burden of proof must be on either ways of claim. And again, you can't prove there will be no perceivable difference without getting cognition solved. Sure you got "some measurements" but as said above, they are not Conclusive and certainly doesn't grant you permission to demean someone else's choices and experiences.

4. Assumption of weights to different performance parameters (measured or un-measured). You certainly can't make claims on what measurements correlate to perceivable changes, the weights for different performance metric on audibility and what type of parameters need to be measured additionally unless you understand cognition properly. A big part of understanding cognition is to check what types of input it is responsive and not responsive to. Cognition is also relating to precision of detectability of different parameters (including non-desirable influences from parameters), and the location estimation is a use case for which the brain developed to this precision.

Here it is. You've made me quote this for the second time (and I had to since you are completely ignoring the valid arguments).

[manueljenkin]

On 11/15/2020 at 8:53 AM, manueljenkin said:

The resolution requirements of an oximeter and an audio chain is not the same. The former gives a reading that is used to make more of less a binary decision (whether the person is alright or needs treatment) and even if it were more than a binary decision it's mainly based on "cut offs".

 

Contrast that to audio where we donot have any proper correlation factors to each parameter being measured (or not measured), the comparison gets broken and not extendable. Atleast ADC is somewhat closer to the oximeter in that they both capture and store a representation of what is already present, while a audio reproduction system needs to create the whole thing again, and adds complexity to the analysis.

 

As for Peter aczel, it's just one person's opinon and that too very open ended. What are the necessary tests and parameter cut offs, with respective weights to have a "Conclusive" analysis of sound quality of a system is still an area of mystery. I have nothing against measurements, they are a good way to ensure "components" of a system are not broken. If you have 1000 systems being made, measurements can be used to check if the capacitor or any other component in one of the system has failed by comparing with the other measurements that's all! It's only relative to other implementations of the same design, it's not a valid conclusive metric to compare final performance of different designs as of now.

Those "formally accepted" techniques are not used to assess absolute sonic performance, as done at asr. It is just to ensure whether the circuit behaves in line with the rest of the devices in the chain. And to an extent, the designer can relate to whether a design choice relates to a "measurable parameter", doesn't necessarily say anything about sq in terms of audible weights and thresholds. There are other areas like the oximeter example where the correlation of measured numbers and the analysis/task to be done is fairly well established, but it is not the case for audio (reason: read the previous post quote).

[sandyk]

45 minutes ago, fas42 said:

Which could all add up to changing what I was hearing ... to be blunt, I don't want this sort of 'complexity' in the mix - too many complications already; I want this one to be at at the bottom of the pile ... sorry about that, Alex 🙂.

This does not explain away the differences you reported hearing previously between ALL 3 versions of the TOTO-Africa file.

 

Perhaps capturing streaming on your laptop is not going to be as good a listening experience as local music playing, although if your Internet is now fast enough, it will give you a chance to listen to more variety and help guide your music purchasing habits.   

 

[The Computer Audiophile]

@sandyk You’ve been reported for posting subjective stuff in the objective forum. Please stop. 

[fas42]

8 minutes ago, sandyk said:

This does not explain away the differences you reported hearing previously between ALL 3 versions of the TOTO-Africa file.

 

Perhaps capturing streaming on your laptop is not going to be as good a listening experience as local music playing, although if your Internet is now fast enough, it will give you a chance to listen to more variety and help guide your music purchasing habits.   

 

 

All 3 versions of your Tyros files were downloaded, and played in Audacity, from local files imported into the program. As were the Toto files. But, this morning my laptop was in "different shape" - was this enough to throw what I was hearing? ... I've had this happen, many many times ...

[sandyk]

12 minutes ago, The Computer Audiophile said:

@sandyk You’ve been reported for posting subjective stuff in the objective forum. Please stop. 

Sure thing Chris

 Like most members here I would love to see some real Objective reports about Misleading Measurements in this thread that you started.

The only way to do this appears to be to present actual measurements that were later shown to be misleading by other measurements or DBTs.

[fas42]

The audio game just swims in confounders - being so-called objective about it is often just pretending that these confounders don't exist in the situation, or that they have been "taken care of" - oh, if life could always be so, umm, rational ... 😉

[vmartell22]

49 minutes ago, manueljenkin said:

No this is not straw-man. I put forward extremely legit points. Supported by formally accepted principles? Like what? That you should be fussing about 110db SNR vs 120db SNR because to them "lower number is always better"? Yeah let's ignore audibility thresholds and relative weights. That everything about sq is determined by a limited set of measurements? You have completely ignored the entire points I have made and the real arguments. Here, let me quote them for you one more time.

 

I did not say his is a "proof" of something. I just said it's about as valid as, if not more than the conclusions written at ASR. I am saying the current measurements you have donot give you the authority to bully someone who perceives a change (opinion bullying to destroy their credibility). As of now, things are inconclusive, and in such a situation, the benefit of doubt is on the subjectivist's side, and I would rather be interested in someone exploring the causation properly now. Trying to cocoon things will do no one any benefit

 

Well

 

1.- You kindly mentioned "audibility thresholds" above - it looks like you are finally seeing things correctly  - I love bringing people to the right approach - that is a big component and one of the reasons  subjective claims are basically unprovable... but as long as you know...

 

2.- In previous post I indicated that measurements are never misleading,  they just prove that the devices are operating correctly. The difference of course is not trying to adjudicate other magical properties to the devices. Next thing we know people are gonna say a certain dac or amplifier has "a refined sound with hints of vanilla and notes of peppermint and clorox"  :D

 

3.- The "benefit of the doubt" does not apply to science and engineering endeavours at all... yes, is on the subjective idea, because, it cannot be on the other side by definition. Nothing gets the benefit of the doubt. Prove every claim.

 

v

[manueljenkin]

1 hour ago, vmartell22 said:

 

Well

 

1.- You kindly mentioned "audibility thresholds" above - it looks like you are finally seeing things correctly  - I love bringing people to the right approach - that is a big component and one of the reasons  subjective claims are basically unprovable... but as long as you know...

 

2.- In previous post I indicated that measurements are never misleading,  they just prove that the devices are operating correctly. The difference of course is not trying to adjudicate other magical properties to the devices. Next thing we know people are gonna say a certain dac or amplifier has "a refined sound with hints of vanilla and notes of peppermint and clorox"  :D

 

3.- The "benefit of the doubt" does not apply to science and engineering endeavours at all... yes, is on the subjective idea, because, it cannot be on the other side by definition. Nothing gets the benefit of the doubt. Prove every claim.

 

v

1. Audibility thresholds and weights is still and active area of research. So nothing can be analysed conclusively at this point of time. So if the claimers don't have proof, the same way the naysayers don't have proof either (until you solve cognition).

 

2. One can form any correlation they want based on experience, since the measurements donot seem to show the whole picture. Proving it, is unlikely, agreed. Doesn't mean it exists, doesn't mean it doesn't exist. It is uncertain, it's up to the audience to try and come to a conclusion. No one has the right to opinion bully them to "not try/explore".

 

3. I don't really know what you're trying to say here. You're saying we shouldn't doubt anything, and should have stayed with the 14th century thought that earth is flat? See point 1 again, has the answer to "prove every claim"? The claim that they cannot make any changes to sound quality is not proven either.

[pkane2001]

3 minutes ago, manueljenkin said:

1. Audibility thresholds and weights is still and active area of research. So nothing can be analysed conclusively at this point of time. So if the claimers don't have proof, the same way the naysayers don't have proof either (until you solve cognition).

 

2. One can form any correlation they want based on experience, since the measurements donot seem to show the whole picture. Proving it, is unlikely, agreed. Doesn't mean it exists, doesn't mean it doesn't exist. It is uncertain, it's up to the audience to try and come to a conclusion. No one has the right to opinion bully them to "not try/explore".

 

3. I don't really know what you're trying to say here. You're saying we shouldn't doubt anything, and should have stayed with the 14th century thought that earth is flat? See point 1 again, has the answer to "prove every claim"? The claim that they cannot make any changes to sound quality is not proven either.

 

I suggest you look up the definition of objectivity. Something very much lacking in all your arguments. Objectivity doesn't mean the "objectivist cult" that you've referred to. It's an underlying principle of all the sciences and engineering. Its primary purpose is to remove biases, opinions and fervent beliefs when trying to explain the natural world. All you've expressed here so far are opinions and beliefs, none of which qualify as objective.

[fas42]

Unfortunately, changes in SQ that matter a great deal to the subjective experience don't lend themselves to being measured easily - I group the factors under a general umbrella of "integrity" - my first really good system, decades ago, and the current active speakers, have had nothing done to them to change their characteristics, in any conventional sense - all the tweaking applied to both should have had close to zero impact to all the things normally measured. But what has been done to both is to track down, yes, the 'confounders' which play such a great part in the perception of the playback - their integrity as functioning systems, and robustness against external interference mechanisms were worked on a great deal - and the listening shows the benefits.

 

There is no magic to this - no more than there is magic to, say, SpaceX getting their rockets to perform reliability - not have too many RUDs - Rapid Unpredicted Disassemblies ... us normal folk call these, spectacular explosions, 😉. There's a mighty big difference between having a RUD, and not having a RUD - which could be down to someone rushing the last bit of checking, or being distracted while assembling something ... now, how do you measure that ... ?

[John Dyson]

52 minutes ago, manueljenkin said:

1. Audibility thresholds and weights is still and active area of research. So nothing can be analysed conclusively at this point of time. So if the claimers don't have proof, the same way the naysayers don't have proof either (until you solve cognition).

 

2. One can form any correlation they want based on experience, since the measurements donot seem to show the whole picture. Proving it, is unlikely, agreed. Doesn't mean it exists, doesn't mean it doesn't exist. It is uncertain, it's up to the audience to try and come to a conclusion. No one has the right to opinion bully them to "not try/explore".

 

3. I don't really know what you're trying to say here. You're saying we shouldn't doubt anything, and should have stayed with the 14th century thought that earth is flat? See point 1 again, has the answer to "prove every claim"? The claim that they cannot make any changes to sound quality is not proven either.

Any good engineer that I know (and there are some on this list) will suggest double-blind or other such variant of tests that mitigate the effects of personal & external bias in measurements.   Some more specific cases, we have laws of physics involved, and such problems are often not very interesting to do measurements.  (If the laws of physics&math aren't obeyed, then why do measurements or tests anyway?)   When it superficially *appears* that the laws of physics are violated and tests appear to be properly done, then there are engineering/technology problems that biased the results.   Technology can be a problem, but math about bits isn't.

 

You should never get an argument from a true engineer or scientist about 'sounds better' -- that cannot be measured except by someone who is honestly listening.  However, there are objective measures&methods and most often there are good ways of testing for differing levels of technical quality.   Without the techniques needed for real engineering solutions, then we get  much too many long random walks of design (which I have recently been involved in) or misleading claims.   Sometimes 'real enginering' in specific situations  is difficult or impractical, but very little discussed here is resistant to good engineering and good science (except for sounds-good).    Stuff is already difficult enough to do  without adding the additional variable of prejudice or feeling.

 

Of anyone who posts here, and those of us with a  generally have a good engineering mind, I have been mixed up the worst with a design that is very difficult to measure -- and it is hell.  Now, I even more strongly regret that EVERYONE doesn't try to adopt scientific methods than before my project-from-hell.   Strongly held non-objective beliefs causes people to hurt themselves with faulty decision and research methods.   Emotions & feelings need not apply -- even though they will always be involved SOMEHOW.

 

The test about 'bits not being bits' (for example) is of no interest to a scientist, is because EVERYTHING will fall down and fail if 'bits aren't bits'.  

 

There is space for subjective evaluation (e.g. sounds better), and objective techniques (design improvement, testing, verification, etc.)   Since I am an engineer (almost to the core) and I know that my own listening is not a very reliable measurement method, I try to use scientific techniques when possible.   It would also be a good thing if more people would realize that their own hearing isn't very reliable for measurement purposes.*  IMO, avoiding scientific methods on purpose is masochistic - or perhaps wish for a forever hobby of tweaking.

 

* I AM convinced that there can be individuals who have trained their hearing to be accurate, but such training and testing also needs to be based upon scientific method.  Just using experience for a basis of such training is probably not sufficient.  It seems that there must to be a scientific basis somewhere , even if the technique appers to  have elements of being subjective.

 

John

 

[opus101]

59 minutes ago, John Dyson said:

There is space for subjective evaluation (e.g. sounds better), and objective techniques (design improvement, testing, verification, etc.)   Since I am an engineer (almost to the core) and I know that my own listening is not a very reliable measurement method, I try to use scientific techniques when possible.   It would also be a good thing if more people would realize that their own hearing isn't very reliable for measurement purposes.*  IMO, avoiding scientific methods on purpose is masochistic - or perhaps wish for a forever hobby of tweaking.

 

As another engineer here I disagree that measurement is the foundation of science. This would appear to be a dogma held by 'objectivists'.

 

I completely agree that 'my own listening is not a very reliable measurement method'. When I listen, I am making an observation, not a measurement. An observation is fundamentally a qualitative thing, not a quantitative thing.

[pkane2001]

13 minutes ago, opus101 said:

As another engineer here I disagree that measurement is the foundation of science. This would appear to be a dogma held by 'objectivists'.

 

I wouldn't say it's the foundation, but it is certainly one of them. Can you give an example of a hard science were measurements are not used as a key method for determining experimental results? 

[manueljenkin]

1 hour ago, John Dyson said:

IMO, avoiding scientific methods on purpose is masochistic - or perhaps wish for a forever hobby of tweaking. I AM convinced that there can be individuals who have trained their hearing to be accurate, but such training and testing also needs to be based upon scientific method.  Just using experience for a basis of such training is probably not sufficient.  It seems that there must to be a scientific basis somewhere , even if the technique appers to  have elements of being subjective.

 

I am not against scientific methods, if the structure is well correlated and conclusive. As of now it is not, so why run behind something inconclusive to derive conclusions? And you're certainly on an abusive tone here trying to say anyone who is disagreeing with the limited set of measurements, because they have experienced a different correlation, are wasting their money. Not really!

 

Also I have counted the number of times you have used the word "engineer" (atleast 7 times) as if it's your last resort to showcase your opinion with higher weight. I am also an engineer, so are many people in this forum who do believe in the changes not measured. You can call yourself as a good or bad engineer and I don't mind that (good and bad is qualitative, and can be personal) but I don't think that qualifies you to make absolutist statements on the performance indexes for measurements of sound reproduction systems.

 

1 hour ago, John Dyson said:

You should never get an argument from a true engineer or scientist about 'sounds better' -- that cannot be measured except by someone who is honestly listening.  However, there are objective measures&methods and most often there are good ways of testing for differing levels of technical quality.   Without the techniques needed for real engineering solutions, then we get  much too many long random walks of design (which I have recently been involved in) or misleading claims.

Well that's more of a task for a researcher who is into psychoacoustics than an engineer or a scientist. And that's the very same argument that invalidates anything that is being said wrt correlation to measurements without the approval of someone into psychoacoustic research.

 

2 hours ago, pkane2001 said:

 

I suggest you look up the definition of objectivity. Something very much lacking in all your arguments. Objectivity doesn't mean the "objectivist cult" that you've referred to. It's an underlying principle of all the sciences and engineering. Its primary purpose is to remove biases, opinions and fervent beliefs when trying to explain the natural world. All you've expressed here so far are opinions and beliefs, none of which qualify as objective.

 

I do know it's definition. So here's a sentence from the same page - "Next to unintentional and systematic error, there is always the possibility of deliberate misrepresentation of scientific results, whether for gain, fame, or ideological motives." Guess what? Thats exactly what is happening at ASR.

[pkane2001]

8 minutes ago, manueljenkin said:

I do know it's definition. So here's a sentence from the same page - "Next to unintentional and systematic error, there is always the possibility of deliberate misrepresentation of scientific results, whether for gain, fame, or ideological motives." Guess what? Thats exactly what is happening at ASR.

 

Yet another opinion. ASR is a big place with thousands of people posting. You claim there's deliberate misrepresentation happening there. I'm just going to keep asking you: provide objective evidence to back up your claims. Opinions are a dime-a-dozen. 

[asdf1000]

13 hours ago, The Computer Audiophile said:

Hi Manuel, I have one foot solidly in each camp (objective and subjective) :~)

 

Me too.

 

I want a component does what it says on the can ! i.e.  performs as the manufacturer themselves claim in their specs !! this is not asking much.... but needs 3rd party measurements unless you can do it yourself.

 

In this case I find ASR very useful because there have been numerous cases where a product did not perform as well as the manufacturer claimed... in the early days this could have been the fault of the measurement setup/method by ASR. But I think these days he's sorted out those issues, because his measurements seem to correlate well with some other 3rd party measurements and reputable manufacturers.

 

If it does do what the manufacturer claims, then the rest is listening with my ears (subjective tastes).

 

I don't think there's anything wrong with this approach to be honest.

 

 

[opus101]

1 hour ago, pkane2001 said:

 

I wouldn't say it's the foundation, but it is certainly one of them. Can you give an example of a hard science were measurements are not used as a key method for determining experimental results? 

 

I'm not saying measurements don't play an important role in science, of course they do. But I am saying they're not foundational, observation is foundational.

 

'Hard science' is used here because you believe other (non-hard) kinds of science aren't science? I'd say some kinds of science are characterized as 'hard' because of their heavy reliance on measurement so the intersection of 'hard science' with 'measurements being not a key method...' is indeed the empty set.

[pkane2001]

6 hours ago, opus101 said:

 

I'm not saying measurements don't play an important role in science, of course they do. But I am saying they're not foundational, observation is foundational.

 

'Hard science' is used here because you believe other (non-hard) kinds of science aren't science? I'd say some kinds of science are characterized as 'hard' because of their heavy reliance on measurement so the intersection of 'hard science' with 'measurements being not a key method...' is indeed the empty set.

 

Hard sciences was used to limit the scope and complexity of the question. Soft sciences use measurements as well, although frequently these do not employ measurement instruments and instead use counting statistics.