Misleading Measurements

[John Dyson]

1 hour ago, pkane2001 said:

 

That's why there are more complete measurements being published. Nobody (except for some manufacturers) publish just the THD % -- and in those cases, I fully agree with you -- that's mostly meaningless, even when stated as 1kHz @ 0dBFS.

 

Here's an example of the types of evaluation I find useful, having performed these myself. I look for these when published by others, as they do provide a lot more detail about a device than just a single number.

 

Distortion vs. level with levels of individual harmonics, noise, etc:

 

Distortion vs Frequency:

 

And yes, Chris, these are mostly below threshold of audibility, although the noise floor is higher than I'd like to see.

 

Those are the good, minimum specs that are needed for reasonable judgements.  Hobby claims need that kind of characterization also (perhaps to a bit less detail just because of practicality.)

 

I am also skeptical of the effects of the input transducers.   I haven't seen the full specs recently -- but for those still into vinyl -- the impedance thing, esp for MM is important.   Since MM/MC preamps are common in the hobby realm, claims about distortion really need to include the impedance of the transducer.

 

But yep -- this is getting into the realm of relatively meaningful.

 

It WOULD be nice if a transparent explanation would be given for non-techie audiophiles also -- what does that 0.001% mean if it increase to 0.01% at 30kHz?    You know what I mean...   IMD is also important, where THD vs IMD have different importance at different frequency.   To do the specs in full detail can be onerous.  But, it isn't about the 'specs' per se...  Like, how bad is the IMD.  Is IMD even an issue in the design (it can be an issue, but hopefully a good design won't make it worse than what it should be.)

I can keep on rambling on about this without communicating more of what I intended  -- but as we all know (both people who have a preference for objective or subjective), raw numbers are meaningless unless the *effect* of those 'numbers' (however diminishingly small or big) on the sound is the important thing.

 

I'll sign off on this subject because my point is made -- and I only feel uncomfortable when being too much or too little focused on the objective specs, and also uncomfortable if the subjective effects aren't verified/tested.   When I suggest 'testing', I mean using experiments with controls.  These true subjective tests, with some reasonable scientific/statistical/blind method are very inconvenient, but can be amazingly beneficial to both the consumer and the engineer doing a design.  For the design, it is more about verification, but for the consumer, it can be about validation or choice.  (probably other reasons.)

 

I guess -- I try to say, don't discount ANY information source, and use whatever tools you can make available.  Doing things 'right' can be incredibly inconvenient, and happily, much of the time, we are lucky -- and HOPEFULLY whatever mistakes we make in design/testing and as a customer end up being inconsequential.

(To be polite, unless someone directly replies and effectively asks for response, I'll just lurk again :-)).

 

John

 

[Jud]

I can think of two reasons I might like to see measurements that are below generally accepted thresholds of human hearing.

 

1. Lab bench vs. in situ, something John mentioned - it's possible a component in a home in less than the (hopefully) ideal conditions of a lab test might perform somewhat differently. This relates to Paul's point about "better engineered." But I haven't seen measurements like resistance to noise carried into a DAC from external connections, for example, so it's difficult to tell what's really better engineered in this context.

 

2. I think it is possible that noise below generally accepted hearing thresholds might have subconscious effects. Is anyone aware of papers measuring hearing thresholds that use something like an fMRI rather than conscious verbal responses? Experimenters do take account of the difference between tests that use instrumental measurements and those that require a conscious verbal response, but I don't recall seeing academic papers on this topic in the area of human hearing.

[fas42]

Typical measurements are misleading, because they don't point out the weaknesses of the product - those put out by the manufacturer are a joke, because they merely form part of the marketing ... my 35 year old power amp had 10 times as much information printed in the included manual as compared to current units held in high regard - the word "pathetic" comes to mind.

 

No-one measures what really matters when the 'transparency' of a system reaches a certain level - things like, the ability to reject external interference - the consumer is completely blind to how it will really behave, until he has it in situ. Quite absurd, so no wonder the industry is such a mess,when looked at by an outsider ...

 

 

[Audiophile Neuroscience]

10 hours ago, The Computer Audiophile said:

Measurements are misleading just like statistics.

 

10 hours ago, semente said:

 

This particular case is not so much that measurements are misleading but whether a couple of measurements are enough to characterise audible performance; I defend that they're not.

 

 

9 hours ago, Summit said:

Measurements and statistics are not misleading per se, therefore better don't throw out the baby with the bathwater.

 

 

 

 

Measurements tell you no more or less than the test tool and its human operator are capable of. Errors can occur at both these levels. If entirely accurate the measure can only tell you what the particular tool is designed to look for within its limitations of sensitivity, specificity etc.

 

If you don't know how good the tool is that is used to make the measurement you don't know how good the measurement is.IOW you don't know how the tool measures up.

 

An absence of an accurate and meaningful measurement can sometimes just mean that a different tool (or operator) is required.

 

The meaning of the measurement is open to interpretation and thus subjective conclusion and open to errors whether intentional or not.

 

For all these reasons IMHO one must be skeptical of measurements, acknowledge they can indeed be misleading or meaningless, and not slavishly and blindly accept measurements over observation.

 

 

 

 

[Audiophile Neuroscience]

7 hours ago, The Computer Audiophile said:

Why do you want to see measurements that make no difference to any human ears?

 

Simply because I do not trust the 'measurement guy' to tell me what is and is not audible, or otherwise in some way influence audibility. It is logically apparent that "measurements that make no difference" are meaningless and may well be misleading. It is a double edged sword tho because i want to see all the data and not have someone tell me what is relevant or not because they say so.

[Audiophile Neuroscience]

9 hours ago, pkane2001 said:

If I ever need a stent, I'll spend my time to educate myself on what matters and interpret "facts" and measurements published by a manufacturer rather than just trust someone else to tell me

 

I would suggest it is difficult to interpret measurements published from a source with vested interests. For similar reasons I have issues trusting measurements or studies published by audio manufacturers and that includes Harman, irrespective of the lead name on the paper.

[Audiophile Neuroscience]

5 hours ago, Jud said:

Is anyone aware of papers measuring hearing thresholds that use something like an fMRI rather than conscious verbal responses? Experimenters do take account of the difference between tests that use instrumental measurements and those that require a conscious verbal response, but I don't recall seeing academic papers on this topic in the area of human hearing.

A couple of more recent papers but probably off topic for discussion attached here

 

2020 Assessment of temporal resolution of bone-conducted ultrasonic hearing using neuromagnetic measurements

2017 High-Resolution Audio with Inaudible High-Frequency Components Induces a Relaxed Attentional State without Conscious Awareness

 

 

2020 Assessment of temporal resolution of bone-conducted ultrasonic hearing using neuromagnetic measurements.pdf2017 High-Resolution_Audio_with_Inaudible_High-Frequenc.pdf

[PeterSt]

9 hours ago, The Computer Audiophile said:

This is interesting. I don't believe it's true that a better measuring product is better engineered, designed, better thought-out, or assembled.

 

Of course it is.

 

But see, I put this out of context and the context is your opinion about it. That's why you (think you) can put it like that.

But of course a better measuring product is better engineered etc. There are no coincidences here.

[PeterSt]

11 hours ago, The Computer Audiophile said:

Another perplexing thing to me is the reason why objective leaning people want to see measurements that are below the threshold of human hearing. If one is truly objective, then the measurements below human hearing don't matter. 

 

The point of my post from yesterday was that there is virtually no threshold of audibility. You (and millions more) seem to think this threshold exists. If it would exist, how can I and so many others point at the same virtue of random changes even without preparation, picking random tracks ?

(and I obviously refer to matters which can't be measured)

 

I appreciate it is too hard to grasp. But hopefully it comes across that at least one person (me) thinks that the whole subject of this thread is moot.

Every smallest change is audible to me and my system and none of them can be measured with known equipment (or it can't realistically be bought because it costs millions (literally)).

 

The crux is that every measurement I myself show, well regarded by you guys to be under the threshold of audibility is crucial because it is vastly audible. And then there is this whole plethora of "measurement" even under that level of today's possibilities of measurement, that still are audible.

 

- Improve your listening skills;

- Improve your system where needed (but it is my estimate this will be the least issue for most);

- get the proper tools to vary the results (this may impede your imagination). Now iterate to the first step.

 

The latter is totally crucial because if you don't have anything to vary, nothing will matter either. Think about this !

[Audiophile Neuroscience]

1 hour ago, PeterSt said:

But of course a better measuring product is better engineered etc. There are no coincidences here.

 

A better measuring product measures better with a particular tool and FWIW. That may or may not mean it is "better" in whatever way depending on what the goal is and how validly the measure reflects that goal ...if at all.

 

53 minutes ago, PeterSt said:

 

The point of my post from yesterday was that there is virtually no threshold of audibility. You (and millions more) seem to think this threshold exists.

 

I think there are obviously auditory perceptual thresholds and they are at least potentially measurable with the right tools and the right method. I think the "millions" you refer to Peter are those individuals who mistake results of test tone audiometry for perception of complex and dynamic musical passages. They mistake acuity at one level to complex perception at other levels and/or alternative ways of perceiving. 

 

 

Quote

- Improve your listening skills;

- Improve your system where needed (but it is my estimate this will be the least issue for most);

- get the proper tools to vary the results (this may impede your imagination). Now iterate to the first step.

 

+1

[semeniub]

I would like to propose a different position on the relevance of measurements, from a sound and vibration engineering, rather than EE perspective.

 

The last items in the audio chain before your ears are either the loudspeaker or headphone drivers. The measurement that I am then most interested in is the displacement and phase of the driver surface, as this is what is radiating sound. I can measure this very precisely using a laser vibrometer, at single points, or map over the entire surface of the driver.

 

The proposal then is if I have a reasonably accurate system and keep it constant, I can change out components such as DACs which differentiate themselves via “superior” EE measurement parameters, and see if there is any measurable difference in terms of driver displacements. The premise then is if you have not changed the driver displacement, you have not introduced a significant audible change with the introduction of the superior EE measuring component in the system.

 

I realize that this would be a new way of looking at the problem, and not everyone has access to laser vibrometer measurement facilities, but at least this would be a type of measurement parameter that would be guaranteed to have an effect on audibility. Maybe someone with the right connections to a lab could have a system set up there where 5 different DACs, having low to high SINAD numbers, could be tested as a feasibility study?

[PeterSt]

1 hour ago, Audiophile Neuroscience said:

A better measuring product measures better with a particular tool and FWIW. That may or may not mean it is "better" in whatever way depending on what the goal is and how validly the measure reflects that goal ...if at all.

 

2 hours ago, PeterSt said:

But of course a better measuring product is better engineered etc.

 

Despite what you make of it, am I right or not ?

Of course I am. 😉

[PeterSt]

7 minutes ago, semeniub said:

 

I realize that this would be a new way of looking at the problem

 

It wouldn't be really different from the way I record and represent changes (I still see nobody doing that), although that way would be more indirect than "viewing" the speaker driver's surface.

 

9 minutes ago, semeniub said:

Maybe someone with the right connections to a lab could have

 

You started out so nicely, and I was ready to invite you over, but you end up saying that you do not own (or can possess) said equipment ?

 

About everything which I change in the XXHighEnd software alone, is totally easily "feelable" at the woofers. It goes from nice sines (when intended to) to super square mess (sines still intended). So ... come over with that stuff ...

please. 🤪 

[semeniub]

11 minutes ago, PeterSt said:

 

It wouldn't be really different from the way I record and represent changes (I still see nobody doing that), although that way would be more indirect than "viewing" the speaker driver's surface.

 

 

You started out so nicely, and I was ready to invite you over, but you end up saying that you do not own (or can possess) said equipment ?

 

About everything which I change in the XXHighEnd software alone, is totally easily "feelable" at the woofers. It goes from nice sines (when intended to) to super square mess (sines still intended). So ... come over with that stuff ...

please. 🤪 

You are in the Netherlands, correct?

 

I think then that you should be able to find a few good acoustics labs nearby...

[Audiophile Neuroscience]

22 minutes ago, semeniub said:

I would like to propose a different position on the relevance of measurements, from a sound and vibration engineering, rather than EE perspective.

 

The last items in the audio chain before your ears are either the loudspeaker or headphone drivers. The measurement that I am then most interested in is the displacement and phase of the driver surface, as this is what is radiating sound. I can measure this very precisely using a laser vibrometer, at single points, or map over the entire surface of the driver.

 

The proposal then is if I have a reasonably accurate system and keep it constant, I can change out components such as DACs which differentiate themselves via “superior” EE measurement parameters, and see if there is any measurable difference in terms of driver displacements. The premise then is if you have not changed the driver displacement, you have not introduced a significant audible change with the introduction of the superior EE measuring component in the system.

 

I realize that this would be a new way of looking at the problem, and not everyone has access to laser vibrometer measurement facilities, but at least this would be a type of measurement parameter that would be guaranteed to have an effect on audibility. Maybe someone with the right connections to a lab could have a system set up there where 5 different DACs, having low to high SINAD numbers, could be tested as a feasibility study?

Nice. At least we are moving downstream, from the output of the DAC (comparing signals with signals) to the output of the sound wave transducer.

One day we should be able to use a vibrometer at the tympanic membrane level, even further downstream. One day it may be even possible to tune into the cochlear transduced signal. Oh wait, that's what the brain is for ...😁🤣

 

Just kidding, sounds like a very cool idea. The caveat would be the same as other measurements - how valid is the result (is it sensitive, is it reliable etc)

 

 

[semeniub]

1 minute ago, Audiophile Neuroscience said:

Nice. At least we are moving downstream, from the output of the DAC (comparing signals with signals) to the output of the sound wave transducer.

One day we should be able to use a vibrometer at the tympanic membrane level, even further downstream. One day it may be even possible to tune into the cochlear transduced signal. Oh wait, that's what the brain is for ...😁🤣

 

Just kidding, sounds like a very cool idea. The caveat would be the same as other measurements - how valid is the result (is it sensitive, is it reliable etc)

 

 

This concept just has to be tested out in a small feasibility study first. Some colleagues have used laser vibrometers to measure nano scale vibrations in materials, does that sound sensitive enough? I'll check with them sometime to see if they have any suggestions on how this could be done.

[Audiophile Neuroscience]

1 minute ago, semeniub said:

This concept just has to be tested out in a small feasibility study first. Some colleagues have used laser vibrometers to measure nano scale vibrations in materials, does that sound sensitive enough? I'll check with them sometime to see if they have any suggestions on how this could be done.

Intriguing. I suppose you'd attempt some kind of Fourier transform of component frequencies? and somehow try and correlate changes in vibrations or frequencies with perceived listening experiences. At the end of the day it would come down to how well you could analyze the vibrations to meaningful spectral or timing or spatial characteristics that match/predict perception. Could we measure changes for example that correlate with soundstage. Fascinating stuff

[PeterSt]

2 minutes ago, Audiophile Neuroscience said:

and somehow try and correlate changes in vibrations or frequencies with perceived listening experiences.

 

I am pretty sure that would not work. Showing differences, however, will be easy.

So it can't be qualified.

[Audiophile Neuroscience]

3 minutes ago, PeterSt said:

 

I am pretty sure that would not work. Showing differences, however, will be easy.

So it can't be qualified.

If you can demonstrate clear differences you can establish independent and dependent variables for a research hypothesis.Its a start.

[PeterSt]

3 minutes ago, Audiophile Neuroscience said:

I suppose you'd attempt some kind of Fourier transform of component frequencies?

 

Not even that, unless you fall back to test signals. So do notice that I was not talking about test signals with the sines and the squarish mess etc. - it is to be about real music, e.g. containing a nice long organ pipe. So it is the lot which changes a particular "instrument" / sound.

If such a thing would allow measurement from a single test signal (or 10 or 50 (IMD)) it would be easy. But it does not work like that at all.

 

So you have, for example, a choir in a church loudly singing and let's say that the baritones do not run into your woofers. What I say is that the quality of your woofer's output (implied by the organ) is influenced in the electronic's path by the other frequencies. Change a few XXHighEnd settings et voilá.  Or change to a different USB cable. Or change the configuration of it (oops). Change the computer processor's frequency.

Everything matters. In this example it could be the impedance response of the power amp which is influenced by higher frequencies (and the higher frequencies may change by whatever the change you applied), now allowing for better (or worse) bass control. ... Like I said earlier, test your imagination on what all could happen in order to explain things (which is thus what I always try).

[Audiophile Neuroscience]

4 minutes ago, PeterSt said:

 

Not even that, unless you fall back to test signals. So do notice that I was not talking about test signals with the sines and the squarish mess etc. - it is to be about real music, e.g. containing a nice long organ pipe. So it is the lot which changes a particular "instrument" / sound.

If such a thing would allow measurement from a single test signal (or 10 or 50 (IMD)) it would be easy. But it does not work like that at all.

 

So you have, for example, a choir in a church loudly singing and let's say that the baritones do not run into your woofers. What I say is that the quality of your woofer's output (implied by the organ) is influenced in the electronic's path by the other frequencies. Change a few XXHighEnd settings et voilá.  Or change to a different USB cable. Or change the configuration of it (oops). Change the computer processor's frequency.

Everything matters. In this example it could be the impedance response of the power amp which is influenced by higher frequencies (and the higher frequencies may change by whatever the change you applied), now allowing for better (or worse) bass control. ... Like I said earlier, test your imagination on what all could happen in order to explain things (which is thus what I always try).

I never said it would be easy 😁

[PeterSt]

13 minutes ago, Audiophile Neuroscience said:

If you can demonstrate clear differences you can establish independent and dependent variables for a research hypothesis.Its a start.

 

David, also see my last post;

 

Whatever I do with a single frequency (I know, that could be too easy anyway), I won't be able to influence THD on the bass response by whatever setting or cable I implied in my previous post. All what can happen is that I overdrive matters and put too much Wattage into the woofer so it starts to distort. But we obviously leave that out.

 

What could theoretically be done is that you'd run two "recordings" (of that vibro meter) of two situations which clearly feel different (hey, they should sound different, right ? - well, obviously they do) - and that you compare them and output the difference.

This is what I do with normal recording (and that works) but this is not qualifiable.  Only if you know which sounded best (is subjective) you could see the anomalies in the diff. But hey ...

And mind you please, when the bass got better, the highs could have gone worse ...

[PeterSt]

Btw, we turned in some kind of theory mode again; The thread is about the oddness of manufacturers showing measurements beyond audible threshold;

I claim this threshold is not at the position at all where a million people think it is and that thus something is wrong with the stipulation (maybe a stipulation is always wrong - haha).

So get the right "tools" and feel your woofers. If you start to feel a difference then all you need to learn is how to hear it. And I further claim that if you don't hear it, it is not beyond the audible threshold, no, it is only that you don't know what to listen for (in the midst of well chosen music which will be problem #1).

 

You don't need vibro meters to feel differences, unless we now change the subjectivity of hearing to subjectivity of feeling.

[semeniub]

I should have also been more clear in my earlier posts that you also can measure the vibrations of the tweeters as well, or even the piston motion of a single drive unit IEM headphone, so that you can measure much more than what you can feel in the woofers.

[semente]

17 hours ago, The Computer Audiophile said:

Publishing measurements that are meaningless to the consumer, with absolutely zero interpretation, can only be a disservice.

 

I agree.

But if an audiophile is unwilling to take the trouble of learning how to interpret measurements that is his own fault and loss.

Measurements and listening are complementary tools for performance assessment and both are indispensible for system building.

Measurements are the difference between a trial-and-error method which may occasionally lead to the fortunate accidental upgrade and taking control of your choices. They also help you to steer away from snake oil and hype, and to focus on what is essential.

 

17 hours ago, The Computer Audiophile said:

Take for example, all the specs produced via measurements and published for components. There is no interpretation done, just specs published. These specs are really measurements. Posting this stuff can't help a consumer.

 

Specs don't help the consumer not because they lack an interpretation – any evolved audiophile should be able to make his own elations – but because they're presented in an over-simplified manner.

For example a speaker manufacturer specifying frequency response of a speaker as 30Hz - 30kHz vs 34Hz–23kHz, ±3dB (on axis); 36Hz–20kHz, ±1.5dB, (on axis); 36Hz–10kHz (30° off axis); LF cutoff, –10dB: 29Hz

But more importantly a ±3dB range can accomodate very different curves from a nearly-flat curve with a simple tilt to a very bumpy zig-zag across the spectrum.

The same is true for other averaged specs like THD.

 

Things get even more complicated when you look at things like bass loading and dispersion (room interference) or the interaction between speakers and amplification.

It's the job of the evolved audiophile to learn the basic of how things work. If one is serious about it. Otherwise we're just throwing money into a bottomless chest... Box-swappers.

 

17 hours ago, The Computer Audiophile said:

It's really just a bunch of guys having fun with themselves, looking at the lowest measurements etc... Joe Sixpack can't be helped by this, even with zero interpretation. 

 

I disagree.

I don't care much about Amir's comments or conclusions (I draw my own and sometimes disagree with his interpretations) but I am convinced that once one are able to roughly correlate measurements with listening then one can safely dismiss speakers from a worth-listening-to list.

 

If Joe Sixpack want's it easy then he can use What Hi-Fi?'s star ratings. That's what I did when I was my late teens when I embraced this hobby.

He can follow the SINAD and Preference Ratings.

Or he can just spend his life navigating through endless amounts of gear in hope of hitting that accidental upgrade... No different that playing the lottery.