When do measurements correlate with subjective impressions

[4est]

In a recent post @plissken stated:

"Where certain measured characteristics track with your preferences, well I think knowing is better than not." in reference to measurements as they may pertain to someone's subjective preference."

Or perhaps by another poster @tmtomh:

"The only way to compile a large proportion of DACs that are significantly different (aside from power supplies, XLR vs RCA outputs, etc) is to look at analogue stages and in particular tubes (since ASR also has shown that op amp rolling doesn't produce significant differences), or to look at R2R DACs. ASR has reviewed some of both, especially the latter, and the measurements are there plain as day for those who are interested."

 

My question to you all is at what point would one consider something "true to the source" or indistinguishable in situ? Is there a an accepted number for THD, IMD or ?  that is considered close enough, or are we back into the 70 and 80s where we are chasing numbers down to zero? Realize this is a question about objective measurements seeking an objective outcome to subjective experiences.

[pkane2001]

2 hours ago, 4est said:

Realize this is a question about objective measurements seeking an objective outcome to subjective experiences.

 

That's doable. In fact, you can even do some of this for yourself, if curious. Quite a bit of content regarding this is available on ASR, but not much here, on AS. I can share some of the more relevant links, or just go there and search.

 

[4est]

52 minutes ago, pkane2001 said:

 

That's doable. In fact, you can even do some of this for yourself, if curious. Quite a bit of content regarding this is available on ASR, but not much here, on AS. I can share some of the more relevant links, or just go there and search.

 

I was hoping to do this here on AS tbh, but feel free to link their pages. I personally don't know how to measure exactly what I seem to hear between components. When I hear various DACs referred to as transparent it gives me pause. IME I hear trends and differences in them, both perhaps related to the analog aspects. I am looking to be brought up to date in order to understand, whilst kick starting the ob-fi side of things.

[pkane2001]

9 minutes ago, 4est said:

I was hoping to do this here on AS tbh, but feel free to link their pages. I personally don't know how to measure exactly what I seem to hear between components. When I hear various DACs referred to as transparent it gives me pause. IME I hear trends and differences in them, both perhaps related to the analog aspects. I am looking to be brought up to date in order to understand, whilst kick starting the ob-fi side of things.

 

Here's a good start:  https://www.audiosciencereview.com/forum/index.php?threads/audibility-thresholds-of-amp-and-dac-measurements.5734/post-127757

 

Don't go too deep into the thread, but the initial few pages should help you get going.

 

If you want to figure out levels of SINAD, THD, IMD, etc. that are audible to you, then try my DISTORT app:

 

https://www.audiosciencereview.com/forum/index.php?threads/beta-test-distort-audibility-of-distortions.10163/post-277173

 

Or you can go for Archimago's THD audibility blind test. In fact, do that first

 

 

[Speedskater]

When there are significant measurable differences between units, preferences may not track with the accuracy of the units.

Euphonic colorations (inaccuracies) are often preferred. 

[4est]

Are you suggesting that people don't want the truth? 

 

[Kal Rubinson]

1 hour ago, 4est said:

Are you suggesting that people don't want the truth? 

Why would it be different in audio? 

[fas42]

What the truth is, that the majority of people on both sides of the "battle" don't want to hear, is that not enough is measured about the performance of the system overall - and that the insertion of a different component normally alters the 'balance' of any higher resolution rig, because the engineering is always too inadequate to ensure that this doesn't happen. The fantasy is that all you have to do, using one of Paul;s beloved analogies, is to just stick a bigger engine under the hood - and your performance vehicle will always be better in every possible way ... any automotive engineer who believes this will be out the door so fast that his feet won't ... . Because, the car is a system , and every subsystem highly likely will have to be touched to ensure that the full potential of the greater power can be fully realised.

 

Until people learn to measure "properly", the debacle will continue - the answer is that the system has to be measured, not the component.

[pkane2001]

17 minutes ago, fas42 said:

What the truth is, that the majority of people on both sides of the "battle" don't want to hear, is that not enough is measured about the performance of the system overall - and that the insertion of a different component normally alters the 'balance' of any higher resolution rig, because the engineering is always too inadequate to ensure that this doesn't happen. The fantasy is that all you have to do, using one of Paul;s beloved analogies, is to just stick a bigger engine under the hood - and your performance vehicle will always be better in every possible way ... any automotive engineer who believes this will be out the door so fast that his feet won't ... . Because, the car is a system , and every subsystem highly likely will have to be touched to ensure that the full potential of the greater power can be fully realised.

 

Until people learn to measure "properly", the debacle will continue - the answer is that the system has to be measured, not the component.

Can always count on you, Frank, to set the record straight! And to use my favorite car analogy in the process

[Jud]

In the Audio Precision seminar at the first RMAF I attended, something like what Paul is talking about in post #4 was presented. Levels of various types of distortion in the signal were increased, and people were asked to raise their hands as the distortion became audible to them.

 

The one that I heard substantially before anyone else was slew rate limiting, and to me it was fingernails-on-chalkboard irritating, while from the reactions of others it seemed fairly innocuous to them. I own Spectral amplification, with a very fast slew rate. This led me to wonder whether I liked the Spectral stuff because I dislike slew rate limiting, or whether owning the Spectral stuff had made me more sensitive to something I was unaccustomed to hearing in my own system.

 

Just wanted to bring this up to raise the general idea that individual sensitivity to various forms of distortion may vary, and so @4est's original question may not have the same answers for everyone.

[sdolezalek]

37 minutes ago, Jud said:

This led me to wonder whether I liked the Spectral stuff because I dislike slew rate limiting, or whether owning the Spectral stuff had made me more sensitive to something I was unaccustomed to hearing in my own system.

 

Jud: I think this is a hugely important question because we know that our brains are capable of allowing us to filter out annoyances and highlight other aspects of sound we deem important for survival. I have frequently wondered why so little is studied about how far down a certain path our ears can take us.  Do those "adjustments" end after a few seconds, minutes, hours or months/years.  

 

For me, the eye opener was the ability to play around with Jussi's filters in HQ Player (or before that with the settings in Audirvana).  Finding ones that "sounded great" seemed easy at first, but I soon tired of filters i liked at first listen but tired of over days or weeks.  Others, that I had skipped over on first listen began to slowly reveal their strengths over time.  I certainly also learned that I was sensitive to different accuracies/inaccuracies than others were.  So there was no "best" filter, but there were certainly filters that to me made my system sound more like what I heard in live music or at least triggered a stronger emotional involvement in making me believe I was there in the real event.  

 

So how do we introduce clarity into a field where there may not be any right or wrong answers, but only answers that represent the set of compromises that best suit our own personal situation? I'm personally a big fan of Magneplanar speakers, as are a number of others here.  Interestingly, I have found that when those other fans also like certain DACS, amplifiers, etc. I am more likely to agree with their choices than I do with those who prefer very different sounding speakers.  It almost becomes like a choice between classical and jazz, neither one is a better choice, but if I like jazz and another jazz aficionado recommends something to me, I'm more likely to give it a listen.  It is why Spotify works.

 

Maybe we need a Spotify equivalent to help us choose which equipment we ought to like?

 

[Miska]

18 hours ago, 4est said:

My question to you all is at what point would one consider something "true to the source" or indistinguishable in situ? Is there a an accepted number for THD, IMD or ?  that is considered close enough, or are we back into the 70 and 80s where we are chasing numbers down to zero? Realize this is a question about objective measurements seeking an objective outcome to subjective experiences.

 

Lot of the objective reviews are just like amplifier measurements in 70s. Companies made amplifiers that had very low THD and IMD figures, yet they sounded bad. And engineers were puzzled as why because everything was measuring perfectly. Then one Finnish guy, Tapio Köykkä wrote about distorted discontinuous sounds, that were working fine on older style tube amps (for example his Circlotron-type tube bridge amp), although those had notably worse THD and IMD figures. Then other people found out about the same too. This was picked up by team of three on Technical Research Centre of Finland; Eero Leinonen, Matti Otala and John Curl, who devised a measurement method for this new type of distortion called TIM (Transient Intermodulation Distortion).

 

Then Otala & Lohstroh also designed a solid state amplifier that didn't exhibit this problem. Based on this amplifier design, a new Norwegian amplifier company actually appeared on the market, called Electrocompaniet. Later on, Otala moved on for example to work with Harman/Kardon to design the Citation XX power amplifier.

 

Matti Otala is truly someone in terms of electronics in Finland, and I would say also globally.

 

Nowadays most amplifiers behave better in this respect, but I can also see that the important measurement is not being used much these days, so who knows. But I've started using it on DACs, since these days it is possible to perform on DACs too to a reasonable extent.

 

There are also many aspects of a DAC that traditional THD+N and IMD measurements don't cover. Jitter measurements were devised later, but there are many more aspects as well. So it is not possible to conclude about "transparency" based on couple of measured figures, it is much more complex matter, especially for something as squeezed as RedBook. In addition, RedBook content itself has common problems, some that can be fixed by the DAC.

 

[4est]

1 hour ago, Miska said:

Nowadays most amplifiers behave better in this respect, but I can also see that the important measurement is not being used much these days, so who knows. But I've started using it on DACs, since these days it is possible to perform on DACs too to a reasonable extent.

 

There are also many aspects of a DAC that traditional THD+N and IMD measurements don't cover. Jitter measurements were devised later, but there are many more aspects as well. So it is not possible to conclude about "transparency" based on couple of measured figures, it is much more complex matter, especially for something as squeezed as RedBook. In addition, RedBook content itself has common problems, some that can be fixed by the DAC.

 

TY, I loved the history lesson btw.

When you say the "important measurement is not being used", are you referring to TIM specifically, or as in one of several measurements that aren't being used or provided? I intend to look further into pkane's links, but IMO your opinion and expertise would be invaluable. Not to place this onus upon you, but you seem to be the rare individual that capably performs measurements whilst using using your ears as a guide. Frankly, it is why I bought into your paradigm years ago, switching over from XXHighend to HQPlayer despite the learning curve and my initial lack of interest in DSD/SDM.

 

Somehow I need to square measurements with sonics. The notion that everything sounds the same within relatively similar SINAD/THD numbers just doesn't seem to be true IME.

[pkane2001]

2 hours ago, Miska said:

 

Lot of the objective reviews are just like amplifier measurements in 70s. Companies made amplifiers that had very low THD and IMD figures, yet they sounded bad. And engineers were puzzled as why because everything was measuring perfectly. Then one Finnish guy, Tapio Köykkä wrote about distorted discontinuous sounds, that were working fine on older style tube amps (for example his Circlotron-type tube bridge amp), although those had notably worse THD and IMD figures. Then other people found out about the same too. This was picked up by team of three on Technical Research Centre of Finland; Eero Leinonen, Matti Otala and John Curl, who devised a measurement method for this new type of distortion called TIM (Transient Intermodulation Distortion).

 

Then Otala & Lohstroh also designed a solid state amplifier that didn't exhibit this problem. Based on this amplifier design, a new Norwegian amplifier company actually appeared on the market, called Electrocompaniet. Later on, Otala moved on for example to work with Harman/Kardon to design the Citation XX power amplifier.

 

Matti Otala is truly someone in terms of electronics in Finland, and I would say also globally.

 

Nowadays most amplifiers behave better in this respect, but I can also see that the important measurement is not being used much these days, so who knows. But I've started using it on DACs, since these days it is possible to perform on DACs too to a reasonable extent.

 

There are also many aspects of a DAC that traditional THD+N and IMD measurements don't cover. Jitter measurements were devised later, but there are many more aspects as well. So it is not possible to conclude about "transparency" based on couple of measured figures, it is much more complex matter, especially for something as squeezed as RedBook. In addition, RedBook content itself has common problems, some that can be fixed by the DAC.

 

 

There aren't that many things that an audio device can mess up. It's either timing (phase), amplitude, or frequency.

 

THD+N captures some of that, especially when plotted against frequency and level. Jitter plots and impulse response cover timing. Spectrum analysis can show frequency anomalies. Ok, so you may not want to do averages over time, so maybe capture peak values for all of these. What else is there?

 

SINAD as a single number, jitter as a number, IMD as a number, etc. are all just very big averages, and therefore can't accurately represent the true nature of a DUT. They are simplifications. A nice THD+N/frequency or THD+N/level chart can provide a ton more information, just like a jitter spectrum or a phase plot.

 

 

[jabbr]

22 hours ago, 4est said:

In a recent post @plissken stated:

"Where certain measured characteristics track with your preferences, well I think knowing is better than not." in reference to measurements as they may pertain to someone's subjective preference."

Or perhaps by another poster @tmtomh:

"The only way to compile a large proportion of DACs that are significantly different (aside from power supplies, XLR vs RCA outputs, etc) is to look at analogue stages and in particular tubes (since ASR also has shown that op amp rolling doesn't produce significant differences), or to look at R2R DACs. ASR has reviewed some of both, especially the latter, and the measurements are there plain as day for those who are interested."

 

My question to you all is at what point would one consider something "true to the source" or indistinguishable in situ? Is there a an accepted number for THD, IMD or ?  that is considered close enough, or are we back into the 70 and 80s where we are chasing numbers down to zero? Realize this is a question about objective measurements seeking an objective outcome to subjective experiences.

All I can say is that I’ve tried a few of the ASR recommended DACs, including the Topping and they leave me flat. The measurements don’t correlate with great sound for me.

 

On the other hand @Miska posted some measurements here of certain DACs two of which I love including the Pro-ject  S2D (which benefits from a good power supply) and the IFi iMicro — which has terrific input isolation. 
 

I think the differences I hear — assuming a good basic product, are mostly to do with the output stages. 
 

For me, the measurements need to have a predictive value in terms of what I hear. 

[pkane2001]

3 minutes ago, jabbr said:

All I can say is that I’ve tried a few of the ASR recommended DACs, including the Topping and they leave me flat. The measurements don’t correlate with great sound for me.

 

On the other hand @Miska posted some measurements here of certain DACs two of which I love including the Pro-ject  S2D (which benefits from a good power supply) and the IFi iMicro — which has terrific input isolation. 
 

I think the differences I hear — assuming a good basic product, are mostly to do with the output stages. 
 

For me, the measurements need to have a predictive value in terms of what I hear. 

 

Haven't tried Topping DACs.  I do have a range of others, from pro to R2R. In most cases, I can't really hear the differences between properly level-matched, modern DACs with good measurements.

 

Some older ones I have from the 90's do generate more noise and more obvious distortions. A more recent (but still old) Emotiva DAC is one where I can hear the differences easily. It doesn't measure well. It also, has the curious quality of being sensitive to the USB cable I use with it. 

[Bill Brown]

After lots of years of listening (32! Oh my) and owning tons of gear I have come to believe a few things and still have some doubts about others.

 

Tube amps (which I have owned): they can sound subjectively glorious, but is it because of a euphonic distortion spectrum?  I think that a pattern of decreasing levels with harmonic order is important and that this has been demonstrated.  I also think that 2nd harmonic distortion (more prominent in tube amps) is not easily heard, and when it is, it tends to be euphonic (see Nelson Pass' experiments in this with his SE SS designs and second-harmonic distortion generator).  I also think that higher-order distortions, even at low levels, can be perceived and are unpleasant.  Finally, with regard to tube amps, many have a higher output impedance that interacts with the varying impedance of loudspeakers and produce frequency-response changes that are likely perceptible, and again, may cause subjectively pleasing FR changes.

 

I think that modern SS amps have decreased in distortion (especially higher-order) so much that it is imperceptible.  I own a Benchmark amp.  John Curl and Mattie Otala did significant work in this area and with TIM distortion.

 

But...are there "second-order" effects in amplifiers (or audio gear in general) that can't be measured that are important?  I struggle with this and am simply not sure.

 

I believe the contributions of Toole and Olive have been very important in loudspeaker design, directivity in particular.  I own JBL professional monitors.  But....are there effects with using drivers out of the frequency band in which they are radiating pistonically (Ka > 1)?  Is this affected by cone materials, the way in which they "ring."  Maybe there is.  Charles Hansen was convinced of this and it lead to his design for Avalon back in the day.  It also is a feature of Laurence Dickie's designs (the BMW Nautilus, and now for Vivid Audio) where all operate pistonically.

 

Earl Geddes, the definition of objectivism, has produced wonderful data re. distortion perception (mainly the lack thereof) in loudspeakers, but also developed his own metric for the perception of distortion in SS amplifiers.

 

I have found JA's measurements in Stereophile invaluable in all of the above.  It is clear to me that many of the perceptions of the subjective reviewers must be thought about carefully, though there are some which I have come to know well over the years, whose perceptions correlate with mine, and whom I trust....

 

I have also blind-tested digital filters as I thought I heard a significant difference and believe my thoughts were borne out....

[Jud]

1 hour ago, pkane2001 said:

It also, has the curious quality of being sensitive to the USB cable I use with it.

 

Measured differences? In what measures? (Curious as to what if any measurable differences a cable might make as part of a system.) Are there any differences between cables that meet spec?

[CG]

I've given an unreasonable amount of thought to this very topic over the years.  I don't have an answer yet - sorry!

 

But, here's a Gedanken experiment to consider...

 

Imagine that you just received the very latest Audio Precision test system, with all the options.  It truly is the state of the art.  You practice using it by making back to back measurements of the internal test generator.  With all the signal averaging on and everything set right, you can see that your AP unit has a residual distortion level of better than -120 dB.  Pretty great!

 

So, you measure your existing power amplifier and find that the 2nd harmonic is only 70 dB below a 1 watt 1000 Hz tone.  That doesn't seem like what you want, so you go buy a new amplifier that claims much lower harmonic distortion.  

 

Before connecting it up, you test the new amplifier with your new test gear.  Yup - the 2nd harmonic on this amplifier is -110 dB at the same power.  40 dB better.

 

Now you connect it into your system in place of the old amp.

 

The first thing you play sounds not so good.  There's a kind of crackling sound every now and then.

 

In engineering terms, WTF?  The amp sure tested pretty well.

 

So, which is wrong?  

 

The AP test system?

 

Or, your ears?

 

It turns out that neither are wrong.

 

Back to the test bench, you take the amplifier cover off and eventually find a wonky connection.  After tightening a screw, you retest the amp.  No difference.

 

Back to the system, you listen again and it's fine.

 

Again, WTF?

 

The AP system was doing exactly what you told it to do.  It applied a single tone and read the amplitude of the amplifier output spectrum over and over.  Loads of sweeps.  Because of the averaging function, you could see distortion way into the noise.  That's what averaging does - it assumes that the fundamental tone and the distortion tones are constant in amplitude, which probably is an ok assumption (maybe) and then assumes that the rest are just random chaotic events that don't repeat either in amplitude or frequency over all those sweeps.  So, much of the noise voltage just gets averaged away.

 

The crud you heard was the result of a bad connection that was random in nature and chaotic.  Some sweeps there was nothing.  Others, something.  But, the spectral content varied all over the place.  Overall, this may have increased the averaged noise floor by a dB or so.  Who can tell?

 

Now, if you had looked at the distortion with averaging turned off, the distortion likely would've been buried in the noise.  But, you'd probably have seen the crackling.  If there was a "peak" or "max hold" function activated, you would have captured the noise spikes.

 

So, it's a matter of what you're looking for.

 

How often have you seen a test result with averaging turned off?

 

Just one example of how incomplete the data set might be.

 

 

 

 

[CG]

9 minutes ago, Jud said:

 

Measured differences? In what measures? (Curious as to what if any measurable differences a cable might make as part of a system.) Are there any differences between cables that meet spec?

What spec?  🙂

[Jud]

Just now, CG said:

What spec?  🙂

 

USB cables have standard specs they are able to meet, I believe.

[Miska]

2 hours ago, pkane2001 said:

There aren't that many things that an audio device can mess up. It's either timing (phase), amplitude, or frequency.

 

With TIM, the problem was that the measurements being used were operating on static sine waveforms like in THD and IMD. While problem was with transient signals. Thus TIM is measured with mixture of square and sine waves (15 kHz sine + 3.18 kHz square wave at 4 : 1 level ratio, -3 dB at 100 kHz).

 

In these cases it was due to the amplifier going to internal saturation, due to internal bandwidth issues, during transients only.

 

And this happened already with vinyl and open reel tape recordings, for example in crescendos. Problem was found because some engineers believed their ears.

 

[pkane2001]

4 minutes ago, Jud said:

 

Measured differences? In what measures? (Curious as to what if any measurable differences a cable might make as part of a system.) Are there any differences between cables that meet spec?

 

Cable I was testing was Lush^2. It made a small but measurable difference in noise level compared to a no-name USB cable. It actually slightly increased the level of noise at the output of the Emotiva DAC. I didn't find Lush^2 to make any difference with other DACs.

 

I assume the increase in noise had to do with shielding/grounding configuration of Lush^2 picking up EMI or introducing a ground loop. (I was using the stock Lush^2 configuration it was shipped with).

[pkane2001]

54 minutes ago, Bill Brown said:

After lots of years of listening (32! Oh my) and owning tons of gear I have come to believe a few things and still have some doubts about others.

 

Tube amps (which I have owned): they can sound subjectively glorious, but is it because of a euphonic distortion spectrum?  I think that a pattern of decreasing levels with harmonic order is important and that this has been demonstrated.  I also think that 2nd harmonic distortion (more prominent in tube amps) is not easily heard, and when it is, it tends to be euphonic (see Nelson Pass' experiments in this with his SE SS designs and second-harmonic distortion generator).  I also think that higher-order distortions, even at low levels, can be perceived and are unpleasant.  Finally, with regard to tube amps, many have a higher output impedance that interacts with the varying impedance of loudspeakers and produce frequency-response changes that are likely perceptible, and again, may cause subjectively pleasing FR changes.

 

I think that modern SS amps have decreased in distortion (especially higher-order) so much that it is imperceptible.  I own a Benchmark amp.  John Curl and Mattie Otala did significant work in this area and with TIM distortion.

 

But...are there "second-order" effects in amplifiers (or audio gear in general) that can't be measured that are important?  I struggle with this and am simply not sure.

 

I believe the contributions of Toole and Olive have been very important in loudspeaker design, directivity in particular.  I own JBL professional monitors.  But....are there effects with using drivers out of the frequency band in which they are radiating pistonically (Ka > 1)?  Is this affected by cone materials, the way in which they "ring."  Maybe there is.  Charles Hansen was convinced of this and it lead to his design for Avalon back in the day.  It also is a feature of Laurence Dickie's designs (the BMW Nautilus, and now for Vivid Audio) where all operate pistonically.

 

Earl Geddes, the definition of objectivism, has produced wonderful data re. distortion perception (mainly the lack thereof) in loudspeakers, but also developed his own metric for the perception of distortion in SS amplifiers.

 

I have found JA's measurements in Stereophile invaluable in all of the above.  It is clear to me that many of the perceptions of the subjective reviewers must be thought about carefully, though there are some which I have come to know well over the years, whose perceptions correlate with mine, and whom I trust....

 

I have also blind-tested digital filters as I thought I heard a significant difference and believe my thoughts were borne out....

 

I'll point you to my own software, DISTORT, if you want to explore audibility of certain kinds of distortions. For example, you can try out only even harmonics or only odd. Or create a specific custom harmonic mix. Or, add a compressor-type effect. Or change noise floor, etc. I'm adding support for different filter types and phase controls to it, but that's still work in progress

 

[pkane2001]

9 minutes ago, CG said:

I've given an unreasonable amount of thought to this very topic over the years.  I don't have an answer yet - sorry!

 

But, here's a Gedanken experiment to consider...

 

Imagine that you just received the very latest Audio Precision test system, with all the options.  It truly is the state of the art.  You practice using it by making back to back measurements of the internal test generator.  With all the signal averaging on and everything set right, you can see that your AP unit has a residual distortion level of better than -120 dB.  Pretty great!

 

So, you measure your existing power amplifier and find that the 2nd harmonic is only 70 dB below a 1 watt 1000 Hz tone.  That doesn't seem like what you want, so you go buy a new amplifier that claims much lower harmonic distortion.  

 

Before connecting it up, you test the new amplifier with your new test gear.  Yup - the 2nd harmonic on this amplifier is -110 dB at the same power.  40 dB better.

 

Now you connect it into your system in place of the old amp.

 

The first thing you play sounds not so good.  There's a kind of crackling sound every now and then.

 

In engineering terms, WTF?  The amp sure tested pretty well.

 

So, which is wrong?  

 

The AP test system?

 

Or, your ears?

 

It turns out that neither are wrong.

 

Back to the test bench, you take the amplifier cover off and eventually find a wonky connection.  After tightening a screw, you retest the amp.  No difference.

 

Back to the system, you listen again and it's fine.

 

Again, WTF?

 

The AP system was doing exactly what you told it to do.  It applied a single tone and read the amplitude of the amplifier output spectrum over and over.  Loads of sweeps.  Because of the averaging function, you could see distortion way into the noise.  That's what averaging does - it assumes that the fundamental tone and the distortion tones are constant in amplitude, which probably is an ok assumption (maybe) and then assumes that the rest are just random chaotic events that don't repeat either in amplitude or frequency over all those sweeps.  So, much of the noise voltage just gets averaged away.

 

The crud you heard was the result of a bad connection that was random in nature and chaotic.  Some sweeps there was nothing.  Others, something.  But, the spectral content varied all over the place.  Overall, this may have increased the averaged noise floor by a dB or so.  Who can tell?

 

Now, if you had looked at the distortion with averaging turned off, the distortion likely would've been buried in the noise.  But, you'd probably have seen the crackling.  If there was a "peak" or "max hold" function activated, you would have captured the noise spikes.

 

So, it's a matter of what you're looking for.

 

How often have you seen a test result with averaging turned off?

 

Just one example of how incomplete the data set might be.

 

Turn off averaging, and use peak-hold. Problem solved?