Jump to content
IGNORED

When do measurements correlate with subjective impressions


4est

Recommended Posts

2 hours ago, 4est said:

Realize this is a question about objective measurements seeking an objective outcome to subjective experiences.

 

That's doable. In fact, you can even do some of this for yourself, if curious. Quite a bit of content regarding this is available on ASR, but not much here, on AS. I can share some of the more relevant links, or just go there and search.

 

Link to comment
17 minutes ago, fas42 said:

What the truth is, that the majority of people on both sides of the "battle" don't want to hear, is that not enough is measured about the performance of the system overall - and that the insertion of a different component normally alters the 'balance' of any higher resolution rig, because the engineering is always too inadequate to ensure that this doesn't happen. The fantasy is that all you have to do, using one of Paul;s beloved analogies, is to just stick a bigger engine under the hood - and your performance vehicle will always be better in every possible way ... any automotive engineer who believes this will be out the door so fast that his feet won't ... . Because, the car is a system , and every subsystem highly likely will have to be touched to ensure that the full potential of the greater power can be fully realised.

 

Until people learn to measure "properly", the debacle will continue - the answer is that the system has to be measured, not the component.


Can always count on you, Frank, to set the record straight! And to use my favorite car analogy in the process ;)

Link to comment
3 minutes ago, jabbr said:


All I can say is that I’ve tried a few of the ASR recommended DACs, including the Topping and they leave me flat. The measurements don’t correlate with great sound for me.

 

On the other hand @Miska posted some measurements here of certain DACs two of which I love including the Pro-ject  S2D (which benefits from a good power supply) and the IFi iMicro — which has terrific input isolation. 
 

I think the differences I hear — assuming a good basic product, are mostly to do with the output stages. 
 

For me, the measurements need to have a predictive value in terms of what I hear. 

 

Haven't tried Topping DACs.  I do have a range of others, from pro to R2R. In most cases, I can't really hear the differences between properly level-matched, modern DACs with good measurements.

 

Some older ones I have from the 90's do generate more noise and more obvious distortions. A more recent (but still old) Emotiva DAC is one where I can hear the differences easily. It doesn't measure well. It also, has the curious quality of being sensitive to the USB cable I use with it. 

Link to comment
4 minutes ago, Jud said:

 

Measured differences? In what measures? (Curious as to what if any measurable differences a cable might make as part of a system.) Are there any differences between cables that meet spec?

 

Cable I was testing was Lush^2. It made a small but measurable difference in noise level compared to a no-name USB cable. It actually slightly increased the level of noise at the output of the Emotiva DAC. I didn't find Lush^2 to make any difference with other DACs.

 

I assume the increase in noise had to do with shielding/grounding configuration of Lush^2 picking up EMI or introducing a ground loop. (I was using the stock Lush^2 configuration it was shipped with).

Link to comment
9 minutes ago, CG said:

I've given an unreasonable amount of thought to this very topic over the years.  I don't have an answer yet - sorry!

 

But, here's a Gedanken experiment to consider...

 

Imagine that you just received the very latest Audio Precision test system, with all the options.  It truly is the state of the art.  You practice using it by making back to back measurements of the internal test generator.  With all the signal averaging on and everything set right, you can see that your AP unit has a residual distortion level of better than -120 dB.  Pretty great!

 

So, you measure your existing power amplifier and find that the 2nd harmonic is only 70 dB below a 1 watt 1000 Hz tone.  That doesn't seem like what you want, so you go buy a new amplifier that claims much lower harmonic distortion.  

 

Before connecting it up, you test the new amplifier with your new test gear.  Yup - the 2nd harmonic on this amplifier is -110 dB at the same power.  40 dB better.

 

Now you connect it into your system in place of the old amp.

 

The first thing you play sounds not so good.  There's a kind of crackling sound every now and then.

 

In engineering terms, WTF?  The amp sure tested pretty well.

 

So, which is wrong?  

 

The AP test system?

 

Or, your ears?

 

It turns out that neither are wrong.

 

Back to the test bench, you take the amplifier cover off and eventually find a wonky connection.  After tightening a screw, you retest the amp.  No difference.

 

Back to the system, you listen again and it's fine.

 

Again, WTF?

 

The AP system was doing exactly what you told it to do.  It applied a single tone and read the amplitude of the amplifier output spectrum over and over.  Loads of sweeps.  Because of the averaging function, you could see distortion way into the noise.  That's what averaging does - it assumes that the fundamental tone and the distortion tones are constant in amplitude, which probably is an ok assumption (maybe) and then assumes that the rest are just random chaotic events that don't repeat either in amplitude or frequency over all those sweeps.  So, much of the noise voltage just gets averaged away.

 

The crud you heard was the result of a bad connection that was random in nature and chaotic.  Some sweeps there was nothing.  Others, something.  But, the spectral content varied all over the place.  Overall, this may have increased the averaged noise floor by a dB or so.  Who can tell?

 

Now, if you had looked at the distortion with averaging turned off, the distortion likely would've been buried in the noise.  But, you'd probably have seen the crackling.  If there was a "peak" or "max hold" function activated, you would have captured the noise spikes.

 

So, it's a matter of what you're looking for.

 

How often have you seen a test result with averaging turned off?

 

Just one example of how incomplete the data set might be.

 

Turn off averaging, and use peak-hold. Problem solved? 

Link to comment
7 minutes ago, Miska said:

 

With TIM, the problem was that the measurements being used were operating on static sine waveforms like in THD and IMD. While problem was with transient signals. Thus TIM is measured with mixture of square and sine waves (15 kHz sine + 3.18 kHz square wave at 4 : 1 level ratio, -3 dB at 100 kHz).

 

In these cases it was due to the amplifier going to internal saturation, due to internal bandwidth issues, during transients only.

 

And this happened already with vinyl and open reel tape recordings, for example in crescendos. Problem was found because some engineers believed their ears.

 

 

Do you have any evidence that TIM is a concern in any modern DAC implementations? I thought about adding TIM measurement to DeltaWave (and actually had a test version of this working) but found no useful information in those results.

Link to comment
1 minute ago, Miska said:

 

Time will show when I have collected enough data around it. But there are certainly differences between DACs in this respect and the components are in the audio band.

 

Highest figures I've measured have components around -65 dB, and it is not unusual to have them around -100 dB. So within range I know people can hear.

 

 

Would be curious to know which DACs show such a high level of TIM. Might be interesting to see how (or if) it affects other measurements and metrics.

Link to comment
3 minutes ago, CG said:

Maybe.  I think it would help in this one example, but that's just my opinion.  Doing this would not only obscure distortion measurements in many cases, but it would also freak a lot of people out.

 

Have you ever seen the results of that published anywhere?

 

And, just imagine the ensuing arguments!


I’ve certainly used that mode with oscilloscope, and it does look a lot more ugly than the average, but it does represent reality. Maybe it should be used in more cases and with some published results.

Link to comment
3 minutes ago, Jud said:

 

It's interesting to me, and I'll tell you why.

 

You ask someone new to astronomy to watch through a telescope for an asteroid. They ask "What does one look like through a telescope of this power? What should I watch for?" Your reply is "That's not very interesting to anyone. Try it yourself and you tell me." What are their chances of spotting an asteroid?

 

Replay that last conversation. Your response now is "Let me show you several examples so you get a sense of what to watch for." What are their chances of spotting an asteroid now?

 

So Paul, are you hearing the effects of jitter only at higher levels because it's only ever audible to humans at those levels, or because no one has trained you to notice what lower levels of jitter sound like?

 

What's the training effect? It seems to me this becomes important in determining whether we can be subconsciously affected by levels of distortion we haven't been trained to consciously notice.

 

Ha! You had to bring astronomy into it!  With the right telescope, an asteroid can be very easy to spot, even for a novice. Think of DISTORT as a very good telescope :)

 

Different types of jitter sound different, different amounts also sound different. But how can I tell if I can't hear low-level jitter because of lack of training or because it's not audible to me, or inaudible to everyone? How could I possibly know the answer to that? 

Link to comment
42 minutes ago, CG said:

Of course, I have no idea what the severity of that might be on the sound.  Lots and lots of variables.  So, no claims from me.  But, I know guys who made experiments and have found that very low frequency phase noise on the clocks seems to have an audible effect. By very low, I mean sub Hertz.  

 

That's where DISTORT can help. You can apply sub-Hertz modulation to a clock and see what sounds like. Apply it to a simple sine wave or to a full orchestra recording. You decide how much of each type of jitter to apply, from simple sine-wave, to random, to 1/f noise, to correlated and then listen, see if you can tell the effect.

Link to comment
14 minutes ago, jabbr said:


Hmm loaded wine, got the app to show up on screen (Ubuntu 18.04 LTS) but can’t get it to easily show results... wine is a weird environment (as expected) ...

 

It would be cool to load a file containing a phase noise plot and run that against the music file.

 

So it's just not displaying results or nothing is working? I'll spin up an Ubuntu VM and see if I can get it to work.

Link to comment
6 hours ago, jabbr said:


Hmm loaded wine, got the app to show up on screen (Ubuntu 18.04 LTS) but can’t get it to easily show results... wine is a weird environment (as expected) ...

 

It would be cool to load a file containing a phase noise plot and run that against the music file.

 

So I installed Ubuntu 18.04 and put Wine 5 on it, then installed Distort. Seems to work OK, or did you also get this far with yours?

 

image.thumb.png.df2e4c5cf393ff5cbcd0caeb3b18cde8.png

Link to comment
3 hours ago, barrows said:

next answer to the question of the thread:

 

When the component is the Mola Mola Tambaqui DAC.  See measurements here:

 

https://www.audiosciencereview.com/forum/index.php?threads/mola-mola-tambaqui-dac-and-streamer-review.10770/

 

That -110dBFS peak at 50kHz needs to be worked on! I'm not going to pay that much money and still have distortion at that level! ;)

Well, I probably wouldn't pay that much money for a DAC, period, but that really is one well-measuring DAC.

 

Link to comment
6 hours ago, jabbr said:


I got that far, then tried to load a music file.

 

I can load and process a file (click Save...) I can't play a file yet, because the audio library I'm using requires .NET 4.0, which isn't part of Wine 5 install. What happens when you try to generate a file? Just pick a WAV file you want to process, then pick the desired distortion, then click Save... button to generate the distorted file. I used Audacity to play it on Ubuntu.

 

I'll see if I can change the target .NET framework for the library, or install .NET 4.0 on Wine -- using it for the first time, so not remotely an expert 😄

 

Link to comment
  • 2 weeks later...
45 minutes ago, cat6man said:

 

I have to disagree with much of the above.

 

1.  i have spent 30+ years simulating complex digital telecommunications systems and, while often causing much consternation among my colleagues over this time (everyone like easy answers), i have found that MOST simulations oversimplify the problem and answer a question or problem, but via oversimplification or by not incorporating something (known or unknown), they do not solve THE question or problem.

do you have a realistic jitter spectrum?  what is the PDF (probability density function of the jitter)? what is the reconstruction filter?  it is technically obvious from the math (don't panic, not included here) that the combination and interaction of the jitter with the reconstruction filter are intimately related, so how is this handled?  some filters are not, in fact, even linear as some have lookup tables for filter coefficients that are data sequence dependent.

 

you may use your program to see if your particular waveform and jitter stimulus is/isn't audible to an individual (and there is value in that) but abstracting that to DACs in general is highly questionable.

 

2.  "Jitter is a fairly simple concept"?  sorry, but that just isn't correct.  jitter has a spectrum, it has a pdf, it may have various forms of correlation, the clock may pick up noise of various sorts which could be random, impulsive, RFI, etc.

 

i will try out your program, assuming it runs under WINE, as i'm curious to see what i can/cannot hear but i know enough about simulations to be suspicious of claiming relevance of a level below which jitter not not matter.

 

let's try a gedanken experiment.  assume i have a 50ps rms jitter clock and there is an impulsive noise (power line spike for example......A/C is pretty crappy) impacting that low level signal 1% of the time and causing the clock off instantaneously by 10x the RMS jitter (500ps).  i'm guessing the average RMS jitter measurement would not change at all but that the DAC would see a 10x timing offset 410 times a second (i.e. 1% of the time)...........maybe that could be audible?  i don't know but it certainly seems possible and something that would likely (?) not be measured with averaging turned on.

 

[technical analogy--feel free to skip]

an example i know intimately from simulation:  in a 4G LTE system, high speed data is controlled by a 'scheduler' that assigns time slots at the mobile and base station to different users.  in addition to assigning time slots to each mobile user, the scheduler must assign from a limited set of control channel slots in order to tell each mobile when its data time slot is coming up.  if there are not enough control channel time slots available to serve all the mobile users at a specific time, that mobile user is temporarily blocked and cannot send or receive data.

the metric commonly used in simulations was average control channel utilization, and it was thought that keeping that value below 70-80% on average was sufficient for good system operation.  however, simulation of many mobile data users with many different data usage profiles showed interesting results.  even with say 50% control channel utilization, the blocking could be as high as 5-20% as the distribution of the number of instantanous users could be highly skewed.  therefore, the solution turned out to be much more complex than expected as the tail of the distribution of #users needing control channel slots dominated overall performance and the average utilization was essentially irrelevant.

just a little non-audio example to show you where i'm coming from.

there is always a simple answer to complex problems, but it will often be wrong.

unfortunately, many people (and upper management) prefer a simple answer to a complex "well it depends" answer.

 

as always, YMMV

 

Sounds like you have some background to help contribute to DISTORT development (it is still work in progress, and will be for a while!)

 

1. MOST simulations oversimplify the problem

 

Yes, that's true. But are you saying simulations are never useful? A simulation is a way to study an otherwise hard to reproduce problem under controlled conditions.

 

DISTORT wasn't designed as a device simulation tool. It's designed as a distortion simulation tool. The purpose is to allow me and others to test the audibility (and visualization) of various types of distortions, in different combination, at different levels. I want an approximation that lets me study this complex interaction of jitter, non-linear transfer function, dither, feedback, phase errors, filters, etc., in the comfort of my own home, without investing hundreds of thousands of dollars into professional measurement equipment. If you have a specific area where you think DISTORT simulation can be made better, I'm all ears :)

 

2.  "Jitter is a fairly simple concept"?  sorry, but that just isn't correct.  jitter has a spectrum, it has a pdf, it may have various forms of correlation, the clock may pick up noise of various sorts which could be random, impulsive, RFI, etc.

 

Jitter as a concept is the error in the timing of a digital sample. That's simple enough, no? We can talk about complex sources of jitter and the effect in the frequency domain, along with many other measurements all we want, but the basic concept is simple. 

 

Again, I'm not simulating any specific device in DISTORT, but I am giving everyone a chance to play with different types of jitter, from random (white noise), to correlated, to 1/f noise, to arbitrary sine-waves in configurable amounts. If you think there are other signals with other PDFs that should be included, tell me which, and I'll add them. 

 

In fact, if you have a recording of the noise/signal modulating the clock, I can inject the exact jitter signature into any music file.

 

..........maybe that could be audible?  i don't know but it certainly seems possible and something that would likely (?) not be measured with averaging turned on.

 

DISTORT doesn't do any averaging. It injects arbitrary jitter signal directly into a WAV file, adjusting each sample individually. RMS jitter is not something that's used or computed by the software.

 

 

 

Link to comment
34 minutes ago, cat6man said:

maybe i could have been a little clearer.  the realistic modelling of jitter in a real system is not simple to me.  applying a different instantaneous time offset to each instance of the chosen reconstruction filter is mathematically simple but the interaction is not.  how does your program implement a digital reconstruction filter?  i do not have any measurements or data.  i'm a theory, not hardware, geek :)

 

DISTORT uses a polyphase interpolation filter. The final reconstruction filter is in the DAC playing the 'jittered' waveform, and so is not under DISTORT control.

 

34 minutes ago, cat6man said:

i understand your program doesn't measure or average jitter.  however, i assume you specify the jitter you want to add in some average or static sense?

 

Each jitter source is specified individually and can be combined for a total effect. Each is controlled by amplitude and/or frequency. Here's what the configuration looks like:

 

image.thumb.png.8efdcb44e141fb84879a087bcedb338e.png

 

34 minutes ago, cat6man said:

could you please post a link here (i'm lazy) to your program?  thanks.  i'd like to see what/if i can hear, plus then i can look at the program instead of asking dumb questions here about what it does.

 

Sure. It's in my signature and here: https://distortaudio.org/

 

It should work under WINE, although I'm not sure about sound playback through WINE device stack. Certainly you should be able to apply distortions to a WAV file, save it, and then play it back using your favorite player/device combination.

 

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now



×
×
  • Create New...