Jump to content
IGNORED

Misleading Measurements


Recommended Posts

1 hour ago, Jud said:

 

Though there are plenty of opportunities to introduce noise into a system involving a number of boxes and cables that include both of these. I would love to see more work done on system topology, components and cabling and how all of these affect system noise. I've seen a few articles about topics like grounding, but I haven't had good luck finding detailed and precise information on how to configure a system and then how to measure the results in easy to digest form for the audio layperson (i.e., non-electrician).

 

A USB cable has a very simple design and the USB spec defines the parameters of what it has to be able to accomplish, and to what tolerances. Anything else the cable is doing is superfluous or possibly detrimental to its purpose of carrying data.

 

If you want to know how anything in the chain is affecting noise in the audio system -- measure the output of the system. The output of a DAC is what is fed into an amp. If the noise is not detectable there, then it's either non-existent, or it's properly filtered out by the receiver circuits.

 

Unlike the claims by Frank and others, I usually test a large portion of my system all wired together. Sometimes including two PCs (one is a Mac), attached RAID storage, a Wi-Fi network, a USB-to-Ethernet converter, a couple of USB and ethernet cables (one is 100ft long), a DAC, and sometimes even an amp with a simulated load. All in one shot. If this doesn't introduce any measurable additional noise into my system compared to, say, a Lush^2 cable feeding the same DAC directly from a battery-powered PC, then I'm not worried that anything is introducing unwanted noise. 

Link to comment
29 minutes ago, Jud said:


I was thinking along the lines of practical tutorials like "OK, you've got a ground hum. Here's how to find what's responsible and eliminate it." 

 

Even better, since I think hum and noise that isn't consciously audible may subconsciously affect enjoyment (cf. the Iowa gambling task experiment), what is the equipment and what are the measurements to take to determine noise levels when there's no audible hum or noise?

 

I won't be any good at writing a testing and measurement guide, but these do exist written by others. What I do is simply use a quality ADC to measure a test signal at the output. Software like REW and others (including DeltaWave) let me measure distortions, compare waveforms, check phase, and measure harmonic distortion. With DeltaWave, I can use any audio track as the test signal. Occasionally I use a digital storage oscilloscope to check for what goes on well above audible frequencies. That's about it. You can do quite a bit with just REW.

 

Link to comment
4 hours ago, semente said:

 

There's an interesting discussion going on at ASR where Amir has against his and others' expectations found that a speaker which was favoured in a Harman study and produces beautiful Spinoramas sounds bad. Many are calling bias, so entrenched are they in their beliefs regarding audibility that they refuse to accept the obvious. They've been Tooled to accept that nothing other than tonal balance matters...

 

 

Good observation is good science. Ignoring observation is dogma.


That’s only a controversy if one expects one’s person subjective opinion to be used to judge how something will sound to anyone else.
 

Amir has previously, and even in that same thread, admitted to making many obvious mistakes when judging something subjectively, sighted. The only controversy in my mind is why anyone objectively minded would take that one sighted test by Amir as nothing more than his personal opinion that has little relevance to anyone else.
 

His measurements are testable and repeatable, his one sighted speaker test, not so much. At least not without investing a few million $$$ into a research study, similar to what Harman did. There’s a good paper on the extent they went to construct as an unbiased testing facility as possible to test with as many people as possible. Do you think that if Toole believed that one person can be accurate 100% of the time that he’d have constructed this facility and ran multiple experiments with 40+ people involved?

 

This is just a simple case of argument by authority when people start believing that Amir has some super-human powers. He doesn’t.

Link to comment
8 minutes ago, semente said:

 

I disagree. It's not a matter of how it sounds (taste/subjective) but of identifying shortcomings (observation/objective).

 

 

Like I said, I am ever more inclined to believe that corners were cut in Harman's research. Methodology, small samples, untrained listeners...

 

 

I am not convinced that Amir is a particularly experienced listener but I find it perfectly possible for the Infinity to produce audible distortion(s).

 

You seem to be missing the point. One person's sighted opinion doesn't invalidate detailed, repeated, published and peer-reviewed scientific studies.  Even if that one person is Amir, or even if it was Toole himself. The fact that some people on ASR are calling Amir to task on this is the right thing to do. There's no controversy here, and Amir knows he can't defend his personal preference. Here's what he said:

 

Quote

The measurement data is provided for all. And developed and advanced by others. Listening results are mine because none of you can listen to it. I have offered to fix that by loaning out the speaker.

 

If you get anything out of my BIAS thread, you'll realize how many things can go wrong when biases are not properly controlled. Even if you're a trained listener with years of experience. There were no controls used in Amir's listening session, so why would anyone expect the result to be objective and rise to the level of useful evidence?

Link to comment
13 minutes ago, semente said:

 

Amir expected the speaker, which performed well in a Harman study, to perform but it didn't. Yes the test was sighted and Amir saw the measurements prior to listening but they're not better nor worse than other speakers' which he liked in the past.

 

Is it so difficult to conceive that the could be a problem?

Are you that biased against listening assessments that are not of the ABX type?

You put a lot of trust in the detailed, repeated, published and peer-reviewed scientific studies; I see scope for flaws and corner cutting and even commercially driven bias and personal taste.

 

Meanwhile a pier-reviewed paper about Covid 19 (hydroxichloroquine) published in none other than the Lancet and the New England Journal of Medicine magazines has just been found void and retracted...

Trust science, yes, but not blindly.

 

I'm not going to read into Amir's mind why he preferred one speaker over another. There are just too many possibilities, you seem to think that expectation bias is a simple process. It's not. Which is why, by simple Occam's razor, his one listening test is not enough to invalidate anything that was done in properly conducted, bias-controlled studies: there are just too many obvious and simple explanations for why he heard what he heard, totally unrelated to the actual differences.


Let me ask you, if the president of the USA claimed that hydroxchlorquine is the best treatment for Covid-19, would that mean that all the scientific studies that show the opposite are now suspect? Why would you trust Trump? Or why would you trust Amir?

 

The only reason I trust Amir on his measurements is because they've been replicated by many others, some by me. I know he's not making these up or fudging the results. He might make mistakes, but these are easy to validate and fix. His listening test cannot be verified or validated, and cannot be fixed. It is not objective, by definition.

Link to comment
10 minutes ago, The Computer Audiophile said:

If measurements for items that fall below human hearing are important, wouldn't it be just as important to note the color of the component being measured?

 

It's important to me to know that my car can reach 180mph even though it has a 150mph limiter, and 99% of the time I keep it under 100mph. So sue me. It's irrational.

 

I'm an engineer and like well-designed and built machinery with tolerances to handle extreme conditions. Same with audio. I like the idea of a perfectly transparent system. One that I don't need to worry about adding or subtracting anything to the sound, regardless of whether I can hear it or not. But, given a specific budget, I'll pick the best performing device (read: best measuring) among all those that fall below my threshold of audibility. 

 

Link to comment
14 minutes ago, John Dyson said:

It WOULD be nice if the measurements actually measured in-circuit/in-situ behavior, and comparisons were made with the priorities openly explained.   At least, when someone says that (for example) the 0.25dB down at 20kHz and 1dB down at 30kHz has the effect of significantly changing the sounds of the cymbals crashing -- the judgement can then be prioritized by the person reading the spec.   At least, I know how I would prioritize that interpretation of that raw frequency response spec, esp at 30kHz.  (BTW, the change in cymbals crashing might be caused more by dynamic input impedance effects, say, on MM preamps.)

 

That's why there are more complete measurements being published. Nobody (except for some manufacturers) publish just the THD % -- and in those cases, I fully agree with you -- that's mostly meaningless, even when stated as 1kHz @ 0dBFS.

 

Here's an example of the types of evaluation I find useful, having performed these myself. I look for these when published by others, as they do provide a lot more detail about a device than just a single number.

 

Distortion vs. level with levels of individual harmonics, noise, etc:

image.thumb.png.4bb2b212ef275eb11031cdd7bf6f21f5.png

 

Distortion vs Frequency:

image.thumb.png.831006dda903c9156c8fd917685afb3a.png

 

And yes, Chris, these are mostly below threshold of audibility, although the noise floor is higher than I'd like to see.

 

Link to comment
48 minutes ago, The Computer Audiophile said:

Serious question. 
 

DAC ABC has a noise floor of -150 dB

DAC XYZ has a noise floor of -140 dB

 

Will the average consumer be mislead into thinking something about these two DACs that the measurements don’t say?

 

I respect whatever opinion you have. 

 

The measurement is significant to those who understand what noise floor represents, and I'm very much against anyone else deciding for me that a measurement shouldn't be published because someone, somewhere, might be mislead because they didn't bother to understand what they are reading.

 

Link to comment
Just now, The Computer Audiophile said:

I think your someone somewhere argument falls pretty flat. It’s more like the 1% of people who use these components actually understand that the measurement is meaningless to all but those who get their kicks off engineering. 

 

Chris, why would you expect that more than that one percent will actually read these measurements? Who are you protecting? Can you point me to this average consumer you're talking about who has thousands of dollars to spend on audio equipment, will seek out published measurements, but still cannot be bothered to learn the meaning of noise floor?

Link to comment
Just now, The Computer Audiophile said:

I think you know how lithe real world works, but are purposely ignoring it. 
 

Joe Sixpack does a Google search for a product he’s interested in. He find the Stereophile review with measurements. There you have it. Took about 0.0002 seconds if Google is correct. 
 

I’m not protecting anyone. I’m just exposing hypocrisy when I see it.

 

You likely also know that there are numerous ways to interpret these measurements. Yet, you think all my friends who are busy with other things in their lives, saving other people’s lives in the ER, and just don’t have much free time, should educate themselves on all these measurements that really are meaningless for the practical use of the product. 
 

I get it, you like products that push the envelope further even though it has no relevance to your use of the product.

 

Your favorite reviewer at ASR posts measurements and in every review states if the measurement is likely to cause audible issues. Even if the measurement isn't very good, he says if it's low enough in level to be inaudible. Can your average Joe Sixpack be bothered to actually read a review instead of glancing at a single number? I'm really not getting this desire to hide information to protect the innocent consumer.

Link to comment
2 minutes ago, The Computer Audiophile said:

I’m not protecting anyone. I’m just asking you to accept reality you guys go nuts for that which can’t be heard. You go nuts for low DAC measurements and you go nuts for products that you consider snake oil even though they measure very well (look at some audiophile switch measurements). However, your going nuts is completely opposite depending the product.


Enough with the generalizations, Chris.  Please stop making these statements about “you guys”. You guys are not here to answer. Show me where I ever go nuts about noise floor or THD or jitter or whatever else you think “we” do. If you want to have a discussion with strawmen, then just say so and I’ll stop responding.

Link to comment
2 minutes ago, The Computer Audiophile said:

You’re the only one speaking for the class of people I’m addressing with this thread. 

 

There's a reason for that, and we both know it.

 

3 minutes ago, The Computer Audiophile said:

Have you ever looked at measurements for audiophile switches before purchasing a switch? I’m going to guess no because you don’t think switches can have an effect on sound quality. Yet, I’ll have to pry measurements of other things that can’t effect sound quality from your cold dead hands

 

Why yes, I've looked at audiophile switch measurements, at least those I could find. And found them to be well below audibility, in fact, below any measurement capability. This is as I'd have expected based on my understanding of networks. I've done a lot of testing and measurements in my own system, using different routers, switches, wi-fi networks, usb converters, optical networks, etc. I found no measurable differences (not just audible differences) that could be attributed to network equipment. 

 

As explained before, I prefer well-engineered products that improve transparency, even when I can't hear it. Audiophile switches do nothing to improve transparency of the system, cost a lot more, some generate a lot of heat and lower network speed. So, explain to me why I should be looking at one again? None of this sounds like good engineering.

 

And yes, measurements helped confirm all of this, so I'm still in favor of having them published for everyone to see and not hidden, as you seem to want, because someone might get the wrong impression. At the very least, the measurements confirm that the switch isn't causing harm to audio and that's already useful information.

Link to comment
44 minutes ago, Summit said:

 

Paul I’m curious, which source, DAC, amp and speaker do you have?  


This forum isn’t big enough for me to describe all my equipment :)

 

Three systems (one is 5.1) plus a ton of DACs audio interfaces, headamps, amps, headphones.
 

WiFi, Ethernet and USB are all in the digital audio path between the three floors. No PC in the main listening room.

Link to comment
23 minutes ago, Clockmeister said:

 

Here is a novel idea, measurements should be quoted at the point of entry and USE of the DUT (Dac/router/streamers/clock etc) while under  normal operating conditions for that DUT


I hope you meant at the exit and not the entry point. It doesn’t tell me anything about the DUT if the measurements are taken at the entry point.

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now



×
×
  • Create New...