Why I never got that perfect null in difference testing

[esldude]

http://www.computeraudiophile.com/blogs/esldude/why-i-never-got-perfect-null-difference-testing-part-one-256/

 

I have put up a three part blog on this topic. Might be of interest to some of you.

 

I think there will be some interest in part three in particular.

 

Look forward to comments here or in the blog section.

 

At least many of you read the long thread where I attempted to do null testing of analog interconnects. These blogs come out of that testing. I was bothered by why even the same cable didn't null completely to itself. The nulls obtainable were quite good often -100 db or a bit more. But they weren't quite as perfect as possible. These blogs show some reasons why that was the case, and show the results have been improved upon.

[Julf]

At least many of you read the long thread where I attempted to do null testing of analog interconnects. These blogs come out of that testing. I was bothered by why even the same cable didn't null completely to itself. The nulls obtainable were quite good often -100 db or a bit more. But they weren't quite as perfect as possible. These blogs show some reasons why that was the case, and show the results have been improved upon.

 

The link you provided didn't work for me for some reason. I guess at least two of the parts are here and here.

[esldude]

Yes you got parts two and three. I fixed the link, so part one should link correctly now.

[Julf]

Yes you got parts two and three. I fixed the link, so part one should link correctly now.

 

Great! Thanks! Interesting findings! Just wonder if there would be a way to compensate for the gain creep?

[esldude]

Trying to straighten out the links.

 

Part One:

 

Why I never got that perfect null in difference testing part one - Blogs - Computer Audiophile

 

Part Two:

 

Why I never got that perfect null in difference testing part two - Blogs - Computer Audiophile

 

Part Three:

 

Why I never got that perfect null in difference testing part three - Blogs - Computer Audiophile

[RobbieC]

esldude,

 

I replied to Part One before I read the rest of it. I commented on your methodology but in all fairness it looks like you're kinda figuring it out after reading Part Two and Three.

 

As an exercise, consider sin(x). Now shift the sine wave slightly to the right, say sin(x - pi/16). Subtract the latter from the first, sin(x) - sin(x - pi/16). What do you have as a difference? This is why phase (timing) matters and why a faint signal will remain...no matter what, short of perfect timing (phase matching).

 

Second, while I believe analog interconnects have minor impact, your methodology doesn't go far at proving or disproving what is most at argument about analog interconnects: that they act as filters with the more expensive ones supposedly being the more neutral ones...supposedly. Using a few tones is certainly better than using a single but not sufficient to draw conclusions about the different levels of filtering between different brands or if there is even any significant difference at all.

 

That said, your work is a very good read and commendable if you don't mind me saying. Don't know if you're a scientist or engineer by trade but wouldn't be surprised if you are/were.

[RobbieC]

Btw, probability and statistics is one of my favorite branches of mathematics (and yes it can be consider singular).

 

Can anyone tell me what the probability is of getting two analog signals exactly aligned?

[esldude]

to RobbieC,

 

Yeah, your right. And some others have contacted me. I feel a bit dim-witted really. Picturing what was going on in my mind should have been obvious we are talking simply about supposition of out of phase sine waves. And vector addition can tell you how much out of phase they are. Funny how concepts that you can almost do in your head don't come to mind when you haven't used them in years.

 

I have also done this testing with multiple waves etc. At least in level, like happens with filtering of frequency response, there just doesn't seem to be anything going on. At least no where near the audible band of frequencies. Which is why I was looking to difference testing. When regular measurements show nothing, then difference testing shows any differences however they occur. As I have been finding out, that of course is limited by accuracy of measuring equipment in time and levels.

 

I hadn't thought of using the residual to measure the phase shift going on. Just stumbled upon it like I wrote it up. When a phase shift in the two compared files is present the residual differs by an even 6 db per octave. It is a FR issue as 6 db per octave frequency response would be plenty noticeable before subtracting the files.

 

My current view is measuring shows no frequency response issue that could matter with interconnects. It shows no conventional distortion that could matter. At least doing serial difference testing like this very minor gain shifts in the equipment (< 40 ppm) give residuals above the basic noise floors. The shifts in gain appear to be unrelated to the IC used. When that is compensated for lower residuals exist that are from differing phase in the two signals being compared. Those appear to be between a couple hundred picoseconds and a nanosecond while the exact phase shift varies over time even with locked clocks. That is not really surprising many of these clocks spec timing accuracy of 20 ppm or so. Just thinking of it before measuring and calculating I didn't realize the result would be as noticeable as it was above the noise floors.

 

Beyond those things which can be shown to be going on, there doesn't appear anything else left unless it is smaller in magnitude than these effects. So that is why I think the interconnects with most consumer equipment aren't altering the signal. If the signal isn't being altered by the interconnect it can't sound different due to the interconnect.

 

But other people catching my mistakes is a good reason to put stuff out in public.

[Julf]

Can anyone tell me what the probability is of getting two analog signals exactly aligned?

 

Nil, of course, because no 2 analog signals will ever be exactly the same - or if they are, there is no way to tell, because of quantum physics.

 

Barry will of course point out that either they are the same, or they aren't, and if they aren't exactly the same, it is not a valid null test.

[Julf]

But other people catching my mistakes is a good reason to put stuff out in public.

 

Exactly - that is why putting up stuff openly for peer review is so important. Remember Linus' Law - "given enough eyeballs, all bugs are shallow". So we appreciate the open way you have been conduction your experiments.

[mitchco]

Nelson Pass many years ago modeled speaker cables as filters: http://passdiy.com/pdf/spkrcabl.pdf and performed a number of tests/measures.

 

I tried lining up two analog signals (sine waves) here: FLAC vs WAV vs MP3 vs M4A Experiment - Blogs - Computer Audiophile

 

Similarly, I have had fun playing with null (difference) testing a variety of scenarios:

 

FLAC vs WAV Part 2 Final Results - Blogs - Computer Audiophile

 

JRiver vs JPLAY Test Results - Blogs - Computer Audiophile

 

16/44 vs 24/192 Experiment - Blogs - Computer Audiophile

 

Sonic Signatures: The Art and the Science - Blogs - Computer Audiophile

 

Cheers, Mitch

[RobbieC]

Nil, of course, because no 2 analog signals will ever be exactly the same - or if they are, there is no way to tell, because of quantum physics.

 

Barry will of course point out that either they are the same, or they aren't, and if they aren't exactly the same, it is not a valid null test.

 

Nice Julf! Not sure about your reasoning but there is in fact zero probability of aligning two continuous curves (that are exactly the same) exactly...just as there is zero probability that a continuously distributed random variable will equal any exact value within the interval in which it is defined. A mathematical reality of probability. Typically written, P(X=exact value) = 0, where X is a continuously distributed random variable.

 

Mathematically, we can create exact curves by defining them as such, but we can't align them exactly if their position is defined as continuously distributed and anywhere within an interval of concern.

[bdiament]

...Barry will of course point out that either they are the same, or they aren't, and if they aren't exactly the same, it is not a valid null test.

 

This is not correct.

(If words are to be attributed to me, I would be grateful if they were actually my words.)

 

I would not say "it is not a valid null test".

I would say that in such a case "a null test is not valid".

There's a difference.

 

As I've said elsewhere, either a null exists or it doesn't. There is no more "partial null" than there is partial pregnancy.

The correct way to describe what, for lack of a better term, I'll refer to as a "near null" is a "partial cancellation".

The term "null" means "nothing". There cannot be partial nothing (except perhaps with a few of our politicians).

 

Barry

Soundkeeper Recordings

Barry Diament Audio

[bdiament]

Duplicate post, deleted.

[esldude]

We have been over this ground before Barry.

 

It is something like talking about recording. Either a recording is a perfect recording of music or it isn't a recording. There are no in-between recordings. If it differs from the sound in the studio then it isn't a recording. No partial recordings. They are either recordings or not.

 

And of course by that definition no one has ever been able to do a recording. Hasn't happened.

[Speedskater]

If we are considering two analog signals that are independently sampled and then digital samples are compared. Then the two samples will most likely start at a slightly different point in the signal. The smallest fraction of a sample period will make a difference.

If we are considering only analog signals then the random background noise will make a difference.

[bdiament]

We have been over this ground before Barry...

 

Yes we have Dennis.

As I said then, if you choose to redefine what "nothing" means to include "just a tiny bit", that is your choice.

 

The folks I know who have been using null tests for a few decades would not agree though.

 

Barry

Soundkeeper Recordings

Barry Diament Audio

[esldude]

Yes, Barry, but as is the case for most English words, depending on context there is more than one exact meaning. I copied the following from dictionary.com which of course had the usual definitions of empty set, zero, nothing yada yada. Then this one as well.

 

 

5. Electronics . a point of minimum signal reception, as on a radio direction finder or other electronic meter.

 

Notice it says minimum and not zero. Another listed some places among definitions is "a difference that makes no difference".

[mitchco]

I hope this thread’s content does not devolve like so many others where it is one extreme or another…

 

Whether it is called null testing or difference testing, I could care less. What I am interested in is what it can tell us about advancing the state of the art in audio reproduction.

 

For example, here is an excellent thread on Gearslutz: The Ultimate Converter DA/AD Loopback Shootout Thread! “This test isn't about what sounds good. It's about what sounds best where accuracy and transparent conversion between digital and analog realms are concerned.”

 

While it is a whopping 67 pages long, the first 3 posts is worth a read to understand what the test is, how it was done, and what the results are as this seems to be the most exhaustive (and repeatable!) test of converters on the internet today.

 

Btw, the Lynx HILO is currently in the top spot.

[bdiament]

Dennis,

 

I can only imagine that null tests are something new for you. They've been around in the pro audio world for a few decades now. Anyone in the pro world who suggested a "partial null" would cause a great deal of laughter. Same as the old joke about adding "a little Dolby". Perhaps it is a "you had to be there".

 

As I said, define "nothing" any way you like.

 

Barry

Soundkeeper Recordings

Barry Diament Audio

[esldude]

mitchco,

 

I read the Gearslutz thread after someone pointed it out to me during my previous post on difference testing interconnects. It might even have been you.

 

Anyway, it is good info. But I don't know about Diffmaker. I used it, it claims to do sub-sample timing alignment, and my recent blogs show you need that. But that software seems easily fooled or caught out. I found it particularly was wonky if the first few seconds had silence or just low signal level. I also found you could compare the same exact two files multiple times and it would not report the same result. They were always pretty similar, but could vary by a couple db which makes me wonder about it. Further on the listed ppm of sample timing occasionally, again usually when the first few seconds were silence, it would show a timing drift when I had taken a completely digital file and altered it say by amping it a tiny bit or filtering very low frequencies. Timing obviously couldn't have been altered yet it would show that. So, I wished it did what it claimed, and often it sort of does, but I now have a jaundiced eye for Diffmaker. When Diffmaker worked it did give very close answers to those I could get doing alignment and such manually. I just came to distrust that software a bit.

[mitchco]

esldude, I understand what you are saying and I had those issues as well. But I did get to a point where I was able to get repeatable, consistent results.

 

Audio DiffMaker's measurement capabilities are extremely sensitive. Snippet from the Help file:

 

 

"Signal cancellation depth will usually vary with frequency. The sensitivity of the DiffMaker subtractive process to time and relative amplitude errors is easily analyzed mathematically (for any frequency), yielding the following results:

 

Phase or Time Sensitivity

 

The achievable depth (drop in Difference track energy, relative to the Reference track energy), at any frequency will be limited by the phase error of "theta" degrees existing between the Reference track and the Compared track at that frequency, and will be no better than

 

10*log (2-2*cos(theta)) [dB]

 

To appreciate this sensitivity to time error, consider an error of just 1/100th of a sample at 48kHz (equal to 208 nanoseconds). At a frequency of 10kHz this is equivalent to 0.75 degrees phase shift error, and it is also the time it takes sound to travel about 0.003 inch (!). From the formula you can infer that if during an acoustical recording a microphone position changes just 0.003 inch that can limit the achievable "depth" at 10kHz to 37dB. In other words, if we wanted to verify that there is no difference between tracks more than 37dB below the existing Reference track levels in a frequency band around 10kHz, microphone to loudspeaker distance should be held to within at least three thousandths of an inch over the duration of the recording of the Reference or Compared tracks"

 

I always run a calibration test to compare the exact same file to itself, resulting in a "perfect" null depth correlation of 300 dB. In one of my blog posts, I took a WAV and converted it to FLAC and re-ran the test and it would be 150 dB correlation. Still could not hear anything on the difference file, but pointing out how sensitive the measurement capabilities of the software are when all that was done was a digital file conversion.

 

You might find the signal test files that you can download from gearslutz useful for your tests. The test methodology might be useful too...

[Julf]

Barry,

 

I would not say "it is not a valid null test".

I would say that in such a case "a null test is not valid".

 

My apologies for the misquote. Thank you for the correction.

[Julf]

I can only imagine that null tests are something new for you. They've been around in the pro audio world for a few decades now.

 

While the concept of a null, in the meaning "a point of minimum signal reception", as quoted by esldude, has been around in radio and electronics for probably at least 80 years...

 

But let's get away from this arguing about words and definitions by using the term "difference testing", as stated by mitchco. That way Barry still gets to be right, and we can continue to discuss the actual test results and conclusions, as well as how to improve the tests.

 

[bdiament]

Julf,

 

...That way Barry still gets to be right, and we can continue to discuss the actual test results and conclusions, as well as how to improve the tests...

 

"Barry still gets to be right"? Look in the mirror Julf. Contentiousness for its own sake, post after post after post.

 

Thank you for that tidbit regarding the history of null testing in radio and electronics.

Did you notice where I said "in the pro audio world"? (Hint: it was in the sentences you quoted.)

I'd be very interested to learn from you just where in the pro audio world null tests were used 80 years ago. And I'd also be curious, while you're at it, about who in the audio world you know that would use the term "null" to describe a partial cancellation.

 

Folks who don't know better could get fooled into thinking your behavior is simply a matter of cultural differences but I know too many folks from the Netherlands who *do* have manners.

 

I request that you do not quote me or mention me at all in any of your future posts.

I wish you happiness.

 

Barry

Soundkeeper Recordings

Barry Diament Audio