Misleading Measurements

[fas42]

46 minutes ago, pkane2001 said:

 

I can, but I don't know if you can. And that's the point -- your experience is nothing like mine. What you hear is a mystery to most people, even though you keep talking about it all the time. There's nothing objective to what you're reporting or to your metric. It's purely subjective. 

 

And my point is that's how simple it is ... normal audio playback makes it trivially easy to decide if the sound of, say, a piano coming form a room is the real thing, or merely playback. Turns out that when enough of the distortion clues are absent, that it becomes essentially impossible to pick, at least at a casual level of listening - and that the transition in this subjective experiencing is mostly not gradual; it operates like a switch in the brain - it's either on, or it's off. Then the exercise of refining the capability of a rig is to push it beyond this critical level of quality.

 

If a system can be developed to work reliably at this level, then objective assessments can be made - how many people are unable to judge reliably whether they are being fooled or not.

[pkane2001]

18 minutes ago, fas42 said:

 

And my point is that's how simple it is ... normal audio playback makes it trivially easy to decide if the sound of, say, a piano coming form a room is the real thing, or merely playback. Turns out that when enough of the distortion clues are absent, that it becomes essentially impossible to pick, at least at a casual level of listening - and that the transition in this subjective experiencing is mostly not gradual; it operates like a switch in the brain - it's either on, or it's off. Then the exercise of refining the capability of a rig is to push it beyond this critical level of quality.

 

If a system can be developed to work reliably at this level, then objective assessments can be made - how many people are unable to judge reliably whether they are being fooled or not.

 

Simple? Let's try to be objective here. Publish some DBT tests that demonstrate that your metric is useful to anyone but you.

 

[fas42]

3 minutes ago, pkane2001 said:

 

Simple? Let's try to be objective here. Publish some DBT tests that demonstrate that your metric is useful to anyone but you.

 

 

Let me get this straight ... you want to have DBT tests done that determine whether people prefer an audio system that "sounds like the real thing", versus one which always sounds like a stereo hifi?

[vmartell22]

After 41 pages, suddenly had a realization re: the title of the thread and first post. The problem with it is that well, properly taken measurements are not misleading by nature (let's not consider errors in measurement for the moment - sure mistakes can happen leading to the wrong conclusion)

 

That said, need to clarify - why are measurements NEVER misleading? Well, because this thread and in general, in audiophile discussions this is looked at  incorrectly. Remember, these are engineered devices - how can a designer/manufacturer/engineer know that her device is working properly, according to spec? Measurements! - well, test plans/specs with functional and other kind of testing. All relying on measurements.

 

All that measurements prove is that the device is operating correctly according to spec.

 

That's it. Think of a medical device, for example, a pulse oximeter. Guess what ? That's all you need to know. You want that device to operate correctly within spec. 

 

The problem is that with audio, the output instead of raw cold numbers is well... sound. Sound that goes thru our VERY FALLIBLE human hearing to our brain which can influence the perception in many ways - ah that's all the difference. And the root of all our disagreement. Because ( I have said this. MANY times, I accept it, apologies for the repetition) to assign qualities like "musicality" to an audio device is as silly as assigning a quality like "empathy" to a medical device... sure all patients want that. And is kind of related. BUT of course, not applicable to a device!

 

Measurements are not misleading - they only show whether the device is operating correctly or not.

 

Of course - the argument can be continued from there.  For example, the great late Peter Aczel said, "any properly designed amplifier is completely transparent". Measurements help us determine the "properly designed" part. But this is kind of another discussion isn't it?

 

v

 

[pkane2001]

8 hours ago, fas42 said:

 

Let me get this straight ... you want to have DBT tests done that determine whether people prefer an audio system that "sounds like the real thing", versus one which always sounds like a stereo hifi?

No, to determine if your metric helps with actually tuning a system so that others find it better. You know, something that actually helps improve the system for others, something that has a predictive value.

[fas42]

11 hours ago, pkane2001 said:

No, to determine if your metric helps with actually tuning a system so that others find it better. You know, something that actually helps improve the system for others, something that has a predictive value.

 

Audiophiles come in different shapes: there are those who aim to get the very best, conventional measurements happening, and that then whatever comes out of the speakers is, The Truth; also, those for whom the rig is a metric, which they use to separate "good" from "bad" recordings; and then there is another category who want to gain maximum enjoyment from whatever recording they put on, "measuring" is not part of the deal. What I do will be useful to the last group - for the others, my methods are not really relevant.

 

So far, people around me appreciate what I do. If the playback sounds bad, then that's a signal to me that the chain is faulty - you fix the fault, and the result is that the playback no longer sounds bad. To me, this automatically means "it's better" - others may not find such an approach useful, because, their interest is to see audio playback in a different light, as above.

[pkane2001]

20 minutes ago, fas42 said:

 

Audiophiles come in different shapes: there are those who aim to get the very best, conventional measurements happening, and that then whatever comes out of the speakers is, The Truth; also, those for whom the rig is a metric, which they use to separate "good" from "bad" recordings; and then there is another category who want to gain maximum enjoyment from whatever recording they put on, "measuring" is not part of the deal. What I do will be useful to the last group - for the others, my methods are not really relevant.

 

So far, people around me appreciate what I do. If the playback sounds bad, then that's a signal to me that the chain is faulty - you fix the fault, and the result is that the playback no longer sounds bad. To me, this automatically means "it's better" - others may not find such a approach useful, because, their interest is to see audio playback in a different light, as above.

Objectively, your metric is meaningless, Frank. Everyone here, every audiophile ever, tries to get what sounds like the most realistic sound reproduction. This is not a revelation you are trying to make it out to be.
 

Many here will tell your their system is transparent, realistic, wonderful sounding. In fact, I think I’ve told you this about my system a long time ago: I’ve been extremely happy with it for over 20 years, with very minor tweaks.
 

Using your metric doesn’t help anyone but you, since there’s no way to judge who’s achieved the ultimate nirvana and who is still far away and needs to do more work. What’s more, your metric doesn’t tell anyone what’s wrong or what needs to be adjusted. It’s a subjective judgement of quality. Perfectly fine if you use it for yourself, not at all useful to help others. 

[fas42]

2 minutes ago, pkane2001 said:

Objectively, your metric is meaningless, Frank. Everyone here, every audiophile ever, tries to get what sounds like the most realistic sound reproduction. This is not a revelation you are trying to make it out to be.

 

Getting the "most realistic" reproduction is what people say is part of the deal. However, as soon as it doesn't sound realistic, the excuses come flooding out: bad recording, acoustics of the listening area are wrong, speakers need to be shifted around, pointed in another direction, DSP correction is needed, need to get a better mastering of the album, etc, etc. Every one of those reasons is 'wrong', in my book ... that's the 'revelation' ...

 

2 minutes ago, pkane2001 said:

 

Many here will tell your their system is transparent, realistic, wonderful sounding. In fact, I think I’ve told you this about my system a long time ago: I’ve been extremely happy with it for over 20 years, with very minor tweaks.

 

 

Being happy with what you have is what matters - I won't hold a grudge against you for doing that, for 20 years +, 😉 .

 

2 minutes ago, pkane2001 said:

 

Using your metric doesn’t help anyone but you, since there’s no way to judge who’s achieved the ultimate nirvana and who is still far away and needs to do more work. What’s more, your metric doesn’t tell anyone what’s wrong or what needs to be adjusted. It’s a subjective judgement of quality. Perfectly fine if you use it for yourself, not at all useful to help others. 

 

The ultimate is that every recording that you have, works a treat. 'Difficult' recordings separate the men from the boys - I try these out on an unknown setup; and if they make a mess of it, then definitely it's "needs to do more work" time.

 

Experience, trial and error, all play a part in 'debugging' a rig - there could be many dozens of reasons why a system sounds "off"; best practice is to look at each possibility in turn, work your way through them; every setup will be unique. Which makes it impossible to specify what needs to be done as a simple set of steps.

[manueljenkin]

On 11/14/2020 at 7:59 AM, pkane2001 said:

 

I can, but I don't know if you can. And that's the point -- your experience is nothing like mine. What you hear is a mystery to most people, even though you keep talking about it all the time. There's nothing objective to what you're reporting or to your metric. It's purely subjective. 

It's not really mystery to me though. I may not be able to hear as good as his system, but I do have experiences closer to his. And so do many others here. 😊

[manueljenkin]

17 hours ago, vmartell22 said:

After 41 pages, suddenly had a realization re: the title of the thread and first post. The problem with it is that well, properly taken measurements are not misleading by nature (let's not consider errors in measurement for the moment - sure mistakes can happen leading to the wrong conclusion)

 

That said, need to clarify - why are measurements NEVER misleading? Well, because this thread and in general, in audiophile discussions this is looked at  incorrectly. Remember, these are engineered devices - how can a designer/manufacturer/engineer know that her device is working properly, according to spec? Measurements! - well, test plans/specs with functional and other kind of testing. All relying on measurements.

 

All that measurements prove is that the device is operating correctly according to spec.

 

That's it. Think of a medical device, for example, a pulse oximeter. Guess what ? That's all you need to know. You want that device to operate correctly within spec. 

 

The problem is that with audio, the output instead of raw cold numbers is well... sound. Sound that goes thru our VERY FALLIBLE human hearing to our brain which can influence the perception in many ways - ah that's all the difference. And the root of all our disagreement. Because ( I have said this. MANY times, I accept it, apologies for the repetition) to assign qualities like "musicality" to an audio device is as silly as assigning a quality like "empathy" to a medical device... sure all patients want that. And is kind of related. BUT of course, not applicable to a device!

 

Measurements are not misleading - they only show whether the device is operating correctly or not.

 

Of course - the argument can be continued from there.  For example, the great late Peter Aczel said, "any properly designed amplifier is completely transparent". Measurements help us determine the "properly designed" part. But this is kind of another discussion isn't it?

 

v

 

The resolution requirements of an oximeter and an audio chain is not the same. The former gives a reading that is used to make more of less a binary decision (whether the person is alright or needs treatment) and even if it were more than a binary decision it's mainly based on "cut offs".

 

Contrast that to audio where we donot have any proper correlation factors to each parameter being measured (or not measured), the comparison gets broken and not extendable. Atleast ADC is somewhat closer to the oximeter in that they both capture and store a representation of what is already present, while a audio reproduction system needs to create the whole thing again, and adds complexity to the analysis.

 

As for Peter aczel, it's just one person's opinon and that too very open ended. What are the necessary tests and parameter cut offs, with respective weights to have a "Conclusive" analysis of sound quality of a system is still an area of mystery. I have nothing against measurements, they are a good way to ensure "components" of a system are not broken. If you have 1000 systems being made, measurements can be used to check if the capacitor or any other component in one of the system has failed by comparing with the other measurements that's all! It's only relative to other implementations of the same design, it's not a valid conclusive metric to compare final performance of different designs as of now.

[vmartell22]

17 hours ago, vmartell22 said:

 

 

Measurements are not misleading - they only show whether the device is operating correctly or not.

 

 

 

 

10 minutes ago, manueljenkin said:

I have nothing against measurements, they are a good way to ensure "components" of a system are not broken. If you have 1000 systems being made, measurements can be used to check if the capacitor or any other component in one of the system has failed by comparing with the other measurements that's all! 

 

 

We agree! great! Great minds think alike... 

 

v

[pkane2001]

10 hours ago, manueljenkin said:

It's not really mystery to me though. I may not be able to hear as good as his system, but I do have experiences closer to his. And so do many others here. 😊

 

So you too hear amazing sound coming out of the built-in laptop speakers? ;)

 

[John Dyson]

6 hours ago, pkane2001 said:

 

So you too hear amazing sound coming out of the built-in laptop speakers? ;)

 

I believe that some people mistake the language 'sound quality' for 'enjoying the music'.   They are NOT the same thing.

I do lots of test playbacks, listening for various kinds of subjective problems (sorry about that, but no objective measurements available yet), and even though the processing might mitigate some technical problems, I might not enjoy the music experience.

 

Sometimes certain music sounds ugly to me, even though the sound is clean.

And -- vice versa.   Sometimes imperfect quality recordings can be enjoyable.

 

Emotions might confuse the thinking about enjoyment vs sound quality???

 

John

 

[fas42]

8 hours ago, pkane2001 said:

 

So you too hear amazing sound coming out of the built-in laptop speakers? ;)

 

 

If laptop speakers are good enough to make it quite clear that two versions of a track that by all respects should be identical, because the music data is identical, when played back via a very ordinary software tool, like Audacity - as I mentioned in another thread - then they are good enough to show up differences for more normal reasons 🙂.

 

Sound playback can listened to, to find something out - or, to enjoy what the musical message is about; to immerse oneself in the creativity of the musicians ... if something gets in the way of allowing the latter to happen, then I, for one, revert to the former mode of listening ... it's all about focus, you see ... 😜.

[sandyk]

1 hour ago, fas42 said:

If laptop speakers are good enough to make it quite clear that two versions of a track that by all respects should be identical, because the music data is identical, when played back via a very ordinary software tool, like Audacity - as I mentioned in another thread - then they are good enough to show up differences for more normal reasons 🙂.

See Frank's reply here :

https://audiophilestyle.com/forums/topic/60916-best-flac-converter-software/page/5/#comments

Quote

Well, Alex has done it again - those two files, mp4s, were loaded into Audacity, and output just using that software's playback function. One was clearly superior to the other, in terms of tonal qualities - it was unmistakable ... but when I tried comparing them, by inverting one and adding the other, I got a perfect null. Then used the command line utility "FC /B" to do an MS-DOS file comparison, byte by byte - and they matched ...

 

What's going on ... I'm certainly scratching my head ... 🙃

 I challenge Paul to show with his S/W what Frank and many others are able to hear.

 Even Paul should be able to hear the differences that Frank heard if Textbooks haven't brainwashed him into being unable to hear them !!! Want to prove me wrong Paul ? 

The Audible differences between the 2 files were NOT subtle, and neither were the Video differences if using a decent external monitor. 😜

 P.S. 

 The reply from Frank is 100% ON TOPIC

[pkane2001]

21 minutes ago, sandyk said:

See Frank's reply here :

https://audiophilestyle.com/forums/topic/60916-best-flac-converter-software/page/5/#comments

 

 I challenge Paul to show with his S/W what Frank and many others are able to hear.

 Even Paul should be able to hear the differences that Frank heard if Textbooks haven't brainwashed him into being unable to hear them !!! Want to prove me wrong Paul ? 

The Audible differences between the 2 files were NOT subtle, and neither were the Video differences if using a decent external monitor. 😜

 

Psychology is not one of my strengths, Alex, although I can certainly come up with a reasonable conjecture.

[sandyk]

13 minutes ago, pkane2001 said:

Psychiatry is not one of my strengths, Alex, although I can certainly come up with a reasonable conjecture.

You don't need Psychiatry , you only need those apparently superfluous things either side of your head.😜

 

BTW,  I am happy to provide the links via PM to the files that Frank listened to on request, but I doubt that you will take me up on the offer, even though it would take very little time to try them.

Not that I expect that your brain will permit you to hear what many others in the forum have no problems hearing though.

[pkane2001]

3 minutes ago, sandyk said:

You don't  Psychiatry , you only need those apparently superfluous things either side of your head.😜

 

BTW,  I am happy to provide the links via PM to the files that Frank listened to on request, but I doubt that you will take me up on the offer, even though it would take very little time to try them.

Not that I expect that your brain will permit you to hear what many others in the forum have no problems hearing though.

No thanks, Alex, I’ve seen and heard your evidence before. I have no interest in going down this path again. But do ask Frank to do a blind test, if you really want to know the answer. 

[sandyk]

Just now, pkane2001 said:

No thanks, Alex, I’ve seen and heard your evidence before. I have no interest in going down this path again. But do ask Frank to do a blind test, if you really want to know the answer. 

Your answer is as expected.

 The files you listened to previously had only minor differences. I have been able since then to create much more obvious differences. BTW, you can always use your eyes with these, as well as others in MY Profile such as the Aurora Music Video, and Miley Cyrus-Malibu  where the differences in Brightness and detail should be quite obvious unless you are using a crappy Laptop.

 

[pkane2001]

3 minutes ago, sandyk said:

Your answer is as expected.

 The files you listened to previously had only minor differences. I have been able since then to create much more obvious differences. BTW, you can always use your eyes with these, as well as others in MY Profile such as the Aurora Music Video, and Miley Cyrus-Malibu  where the differences in Brightness and detail should be quite obvious unless you are using a crappy Laptop.

 

Again, no thank you. The question of magic has been answered previously in this thread, and I have no need to keep checking if you’ve discovered it, yet again.

[fas42]

12 minutes ago, pkane2001 said:

No thanks, Alex, I’ve seen and heard your evidence before. I have no interest in going down this path again. But do ask Frank to do a blind test, if you really want to know the answer. 

 

Quite unnecessary - I had zero expectations of there being anything there; in fact, I queried Alex about the quality before I listened to a bar - Audacity was showing very severe limiting, clipping of the track, a "sea of red"; he said to proceed as is.

 

Just import both versions in, and solo first one and the other, while playing continuously - back and forth, back and forth; as obvious as someone pulling the leads off the tweeter, so to speak.

[sandyk]

10 minutes ago, fas42 said:

 

Quite unnecessary - I had zero expectations of there being anything there; in fact, I queried Alex about the quality before I listened to a bar - Audacity was showing very severe limiting, clipping of the track, a "sea of red"; he said to proceed as is.

 

Just import both versions in, and solo first one and the other, while playing continuously - back and forth, back and forth; as obvious as someone pulling the leads off the tweeter, so to speak.

https://www.dropbox.com/s/6ak9tyqrpglq1w9/Tyros 3 - The Power Of Love (Instrumental)HF.mp4?dl=0

https://www.dropbox.com/s/a8koifix231idw3/Tyros 3 - The Power Of Love (Instrumental)LF.mp4?dl=0
 If you are using an external monitor, look at the Keyboard in both from the start.

[pkane2001]

5 minutes ago, fas42 said:

 

Quite unnecessary - I had zero expectations of there being anything there; in fact, I queried Alex about the quality before I listened to a bar - Audacity was showing very severe limiting, clipping of the track, a "sea of red"; he said to proceed as is.

 

Just import both versions in, and solo first one and the other, while playing continuously - back and forth, back and forth; as obvious as someone pulling the leads off the tweeter, so to speak.

Frank, your imagination is very well documented here. Do a proper controlled DBT showing that you can get a p-value of less than 0.05 from 15 tries. It should be trivial to do based on how big the differences you claim are. But leave me out of it — this is not something I‘m the least bit interested in. Just like I don’t need to keep checking if Santa exists every few months.

[sandyk]

15 minutes ago, pkane2001 said:

But leave me out of it — this is not something I‘m the least bit interested in.

It should be, because it's a perfect example of MISLEADING MEASUREMENTS.

 

You appear to have a closed mind where only Measurements matter, just like most ASR members. 

 I will now exit this pointless thread before somebody complains.

 

[fas42]

15 minutes ago, pkane2001 said:

Frank, your imagination is very well documented here. Do a proper controlled DBT showing that you can get a p-value of less than 0.05 from 15 tries. It should be trivial to do based on how big the differences you claim are. But leave me out of it — this is not something I‘m the least bit interested in. Just like I don’t need to keep checking if Santa exists every few months.

 

And the trouble with doing such a test is that immediately people with the desire not to be curious about what's going on will either:

 

a) Completely ignore it, pretend that it never happened - if I just turn my back on it, it will eventually go away ...

 

b) Be certain that the testing was stuffed up; that there had to be a 'tell' - and frantically dive into every corner, of everything, to ferret it out

 

c) Turn around, and insist that 20 people get the same result - if that fails, insist that 200 people get the same result - if that fails, insist that 2,000 people get the same result - if that fails, ...

 

😝