Null test 88.2/24 and 44.1/16

[pkane2001]

On 8/17/2018 at 10:45 PM, bachish said:

Hi everyone!  First post here.

 

By way of introduction, I'm a professional cellist who has been dabbling in audio since I was a kid. I used to record my brother's band to cassette tape using radio shack microphones in the 1980s.  This eventually progressed to the point where I bought my first professional mics and mic pre and began doing my own recordings (a serious hobby since the mid 1990s).  Because I have a finite amount of funds and it gets split in three ways (cello, recording, monitoring and playback systems), I probably don't have as serious an audiophile system as many of you. My priorities are, in this order:  cello - recording equipment - playback system). But I do have some Paradigm Monitor 11s, v2 (tower speakers) that I bought back in the early 2000s, an older but transparent and detailed Harman Kardon Amp from which I bi-wire and bi-amp my Paradigms, a Grace M900 DAC, a Marantz CD player that set me back about $350 years ago, and an acoustically treated room to knock down the early reflections (makes a MASSIVE difference - highly recommended). 

 

Anyway, I posted a little null test I did over at gearslutz.com, the forum/hangout place for recording engineers.  It's not earth shattering but I thought as a first post I would share it here in a slightly different form.

 

For those who don't know, a null test is a way to see if two audio examples are identical and, if not, what the differences between them are.  All you have to do is flip the phase of one of the audio clips, make sure the levels are identical, mix them together, and see if they null.  If the audio is identical between them, the result will be a perfect phase cancellation.  No sound.  A perfect null. 

 

What I did was take an original recording I did of a jazz group at 88.2 Khz/24 bit and down sampled it to 44.1, flipped the phase, and dithered it to 16 bit. I then up-sampled back to 88.2Khz, which results in a 32 bit floating point audio (all processing results in more bits). I then truncated the 32 bit float back to 24 bits.

 

Here is the original file. Sorry the pics are so small. You may have to zoom in but you can see the extra high frequency content above 20KHz

 

 

Here is the down sampled then up-sampled version.  If you zoom in, you can see how the high frequencies are cut off in the down sampled and then re up-sampled version. The high frequencies are not regained despite the up-sampling. If you look carefully, you can see some of the added dither noise in the high frequencies as a band across the top. The other dither is hidden behind the audio.

 

 

 

I took the original 88.2/24 bit recording, loaded it onto a track in my DAW,  loaded the flipped phase version onto another track and mixed them together. The result was dead silence except at extremely high volumes during which I could hear some white noise. They nulled perfectly in the audible range (again, except for the extremely low level noise - mostly the dither noise) and so were identical in that respect. I heard zero music - not even slight residual sounds or artifacts.  Just hiss with the volume cranked full force!

 

You can see the two versions on tracks 1 and 2 and if you zoom in and look carefully, you will notice that the wave forms are mirror images of each other - they are inverted. You will also notice that the meters for the two tracks being compared are higher than the main meter to the right - so there isn't a perfect null.  But remember, the original has extra high frequency content. 

 

 

 

In a spectrogram, the nulled mix shows the dither noise (mainly shaped to the higher and lower frequencies and less in the mid-range) of the twice re-sampled version and the remaining high frequency content of the original 88.2 version - in other words, the only differences between the tracks.  All of the music from 20Hz to 20Khz nulled perfectly and so the original and re-sampled version are a perfect match in that frequency range.

 

 

 

Keep in mind that if any information was lost in the down sample to 44.1 and dithering to 16 bit, it cannot be regained by up sampling again to 88.2 and reverting to 24 bit. It's gone forever from that track.

 

This little test does not answer if the remaining high frequency content has any effect on the listener - that would necessitate another test. But it seems to me, assuming the re-sampling is done properly, that going from 88.2/24 bit to 44.1/16 bit does not change the audio between 20Hz and 20Khz - all stereo imaging, detail, transients, timbre, and anything else you can think of, remain perfectly in tact. If that were not the case, the two audio clips would not null as they did.

 

In a way it's comforting.

 

I hope this isn't seen as being too controversial as a first post!  Just a fun little experiment!

 

Here is the link at gearslutz for larger visuals and a more detailed description, if you are interested.

 

https://www.gearslutz.com/board/mastering-forum/1227563-my-null-test-88-2kh-24-bit-44-1kh-16-bit.html?posted=1#post13474865

 

Cheers

 

 

 

 

 

 

 

 

 

 

 

 

 

Can you please share the audio files you used for this analysis? I’d like to run them through my own software to see if there’s anything interesting it can find.

[pkane2001]

34 minutes ago, bachish said:

Sure, they are uploading now to the cloud.

 

Can I send the link privately in a PM? 

 

 

 

Yes, of course.

[pkane2001]

Here's what my own DeltaWave software as able to determine from about a 2.5 minutes extract from the middle of the recording.

 

The comparison is between the original 88.2Khz/24 bit recording and a downsampled version to 44.1Khz/16bit, then upsampled back to 88.2Khz/24 bit, as described by @bachish (thanks for the files!)

 

By the way, the recording sounds excellent! I also listened to the difference between the two files, which is primarily white noise. I was able to hear occasional notes come through buried in the noise, but only with software digital volume adjusted to about 70dB gain and my DAC set to 0dB (max volume).

 

The files were delivered with the upsampled version with inverted phase. Otherwise, the files match perfectly in level and phase, no phase offset or drift was found.

 

First, a comparison of the spectra of both waveforms. Original 88.2/24 is in blue:

 

 

The drop off do to downsampling/upsampling starts around 21800Hz at -92dB down. Here's the zoomed in portion where the cut-off starts. You can also see very tiny differences in level due to dither and computational error during resampling:

 

 

 

Now the actual waveforms overlaid on top each other:

 

 

Stats below show excellent correlation between the two: 76dB correlated null and -75dB difference (rms).

 

Spectrum of the difference of the two files. Well below -115dB in the audible range:

 

Spectrogram of the original 88.2/24 file:

 

And spectrogram of the downsampled/upsampled file:

 

Spectrogram of the differences of the two files:

 

Interesting results in the cepstrum analysis of the two files:

 

 

This shows that the downsampled/resampled file has some ringing/aliasing going on at a number of frequencies that are not present in the original 88.2/24 bit file. Probably the side-effect of the resampling process and filtering applied. I tried to label the main ones. Note that the vertical value (Y axis) is a correlation coefficient. It's an indication of how strong the ringing is in the measured file, while the X coordinate is the frequency at which this ringing was detected.

 

While the artifacts of the resampling process are visible in the Cepstrum plot, they are not at all noticeable in any of the measurements or in listening to the differences between the two files.

 

 

 

[pkane2001]

8 hours ago, bachish said:

Out of curiosity, do you sell your software? 

 

Not planning on it. When I'm happy with it, I'll probably make it available for free.

 

 

[pkane2001]

7 hours ago, bachish said:

In the Cepstrum plot, the y axis - I'm curious what measurement that is.  Just trying to get a handle on just how loud the ringing is. You mention the level is pretty low, I realize. 

 

Cepstrum attempts to find repeating patterns in the frequency domain. The Y axis in the plot is a correlation coefficient. All it's saying is that a relatively higher value has a relatively greater amount of ringing (i.e., the pattern is better defined). It doesn't say anything about the level of ringing.  Since it does point to specific frequencies, it might be interesting to see if the magnitude can somehow be dug up from the signal and noise in the frequency domain. Not sure that's possible, but I'll think about it some more

 

7 hours ago, bachish said:

In the 'Spectrum of the difference of the two files. Well below -115dB in the audible range' the left side of the graph looks like the noise shaping of the dither applied, which would make sense.  So essentially, that graph is showing that the difference between the two files is primarily in the dithering and the high frequency content from the original file. Am I reading that correctly?

 

 

Right. Most of the differences are well below -115dB in the frequency domain. But, that represents an average over the measured period. For a time version of the difference plot, here is what it looks like in the time domain:

 

 

You can see some peaks rising to about 0.02. In dB terms, that represents about -34dB level. But that's just a few peaks. Most of the difference is well below -100dB level. By the way, this is the waveform I listened to to hear the difference. Most of it sounds like noise, with a few very occasional notes coming through. Probably corresponding to some of the peaks in the above plot. So, there is some difference between the files, they are not a perfect match, but they are very, very close.

 

Here's that same difference plot but with all the frequencies above 20KHz removed:

[pkane2001]

51 minutes ago, Jud said:

 

If you were trying to hear noise, that's correct. But we're trying to hear music.  How much very low level musical detail is the noise masking?  (Maybe none, considering the noise level of a typical listening room, but it's the way of conceptualizing noise and masking that I'm thinking of.)

 

I turned up the volume 70dB in digital processing, and turned my DAC to 0dB setting to hear the difference file. You'd never listen to a recording like this! There is a lot of static noise with a few musical notes coming through, occasionally. So, yes, there's some information that's in the original file that's not in the resampled one, but it is at a level where it is completely buried by noise. 

[pkane2001]

3 hours ago, Jud said:

 

Following up on this -

 

Here are some recordings available from NativeDSD, and you can filter your search to come up with only those albums/tracks available in DSD256: https://www.nativedsd.com/new_browse

 

@bachish, @pkane2001, I don't know whether you're able to do the same analysis with one of these tracks, but it would be interesting to see whether a recording that presumably had experienced less processing would show any greater difference to a 44.1kHz resolution downsampled file than the 88.2kHz original you used before.

 

Sorry, Jud, no DSD support in DeltaWave  It took me a while to code WAV and FLAC file support, until I found a library that could do it all for me. No DSD support, so it would have to be converted to PCM before I can process it and that would probably negate any analysis of the very low level details. Why do you think that DSD captures would be subject to less processing than PCM that @bachish captured and processed himself?

 

 

[pkane2001]

4 hours ago, crenca said:

Interesting.  When the OP transmitted these files to Paul over the Internet, was the EM/RF "noise" of your computer, your ethernet cables and switches/routers, etc. taken into account?  Would "audiophile" networking equipment made any difference to any computational analysis? 

 

@crenca, I didn't realize your name was Alex??? 

[pkane2001]

7 hours ago, cookiemarenco said:

 

Hello Bachish, I did read your thread from the beginning.  I thought it was quite charming and adventurous for a first post.  I was familiar with the null test from years gone by.  I'm not sure how much has been deleted from this forum, but there have been some very interesting responses over the years to issues you brought up about the math.  I find no reason to argue any more.  I understand we all have different opinions and different experiences in life.

 

I'd much rather argue about math, as the differences there can be resolved between knowledgeable people. Unlike listening tests of the type typically practiced by most here on CA, math is not a matter of opinion.

 

Having just gone through the implementation of a software null-test, I'd love to hear about some of the issues that have been brought up. Certainly a null test by itself says nothing about audibility of the difference, although you can play it back and listen for yourself. But, since I also spent some time coming up with (what I think are) some better analysis tools of both, the original files and their difference, I'd like to know what I might be missing.

[pkane2001]

1 hour ago, Jud said:

First things first, I'm hoping in the midst of our more usual discussions that it will be possible to go a little further along with @bachish's and @pkane2001's exploration of the resolution question to see what if any difference might be obtained in a comparison between Cookie's files  and a 44.1 version, or a 2L DXD file and the 44.1 version.

 

Jud, you had to ask, didn't you?  

 

Now I'll have to look into a way to compare DSD captures, as that raises my curiosity, as well. May take some time, as I have to come up to speed with delta-sigma modulator and proper processing for DSD encoding.

[pkane2001]

20 minutes ago, Jud said:

For example, I recall Miska saying many (most?) of the filters used ring.  

 

All do, this is the nature of band-limited digital filtering. Sharper filters ring more. I know Miska measures DACs well into MHz, so yes, you can see all the ugly artifacts in the inaudible range. But do they matter?

 

In testing DeltaWave, I found some recordings that had noise pushed out to well above 88KHz range by noise shaping, at nearly the same level as the original (audible) signal. Perhaps that's an example of a poorly designed filter? And yet, it's still ultrasonic energy that most likely doesn't affect the audible range... May destroy a tweeter or two, if using a high-bandwidth amp, though

[pkane2001]

33 minutes ago, Jud said:

 

I'm curious as to what recordings.   

 

I'll let you when I come across it again. I've run through a lot of various null test files for various DACs/ADCs in the last few months.

 

[pkane2001]

6 minutes ago, crenca said:

 

 

Well stated PeterSt.

 

You see pkane2001 and bachish, the subject, the math, is moot.  There is an audio reality in between the math, the digital signal, and it is this in between reality that really matters.  

 

Yes, that's absolutely true. Math, logic, measurements, science, engineering are all moot when it comes to audio and healing bracelets.

[pkane2001]

On 8/23/2018 at 10:51 AM, Jud said:

What I'm curious about, for anyone who feels like being helpful, is whether and to what extent music in 44.1k from the "majors" has had different filtering used in its creation than the 44.1k examples 2L and Cookie have put up, and if so, what if any differences might be seen in the types of comparison measurements @bachish and @pkane2001 are doing.

 

Took me longer than I expected to add DSD support to DeltaWave. Turned out it was not straightforward since I had to demodulate and resample, but I think I have it mostly working.

 

I downloaded the DSD and PCM free samples from Blue Coast Music. Here's a comparison between 24/96KHz PCM and DSD 256 version of a Carmen Gomez track:

 

First, the match quality between PCM and DSD tracks. There's some variability possible in the DSD integrator/demodulator and filtering, but this appears to produce a good null depth of about 70dB:

 

Spectrum comparison reveals something interesting:

 

There's obviously a fairly sharp filter being used in DSD processing compared to PCM. This is not done in DeltaWave, and the same filter appears in all DSD versions of this track.

 

Spectrum of the difference track created by subtracting PCM from DSD shows fairly high ultrasonic differences, as one would expect based on the previous result:

 

Here's the audible range part of the delta spectrum, showing the differences are mostly below -105dB:

 

Spectrogram shows the same differences in ultrasonic range:

 

First, the DSD spectrogram:

 

And now the PCM one:

 

Cepstrum doesn't reveal any major differences in ringing in either track:

 

In the audible range, there appears to be a very good match between DSD256 and PCM 24/96Khz versions of the track. Ultrasonic range is very, very different due to processing in creating these files. PCM version, in particular, seems to contain a lot of ultrasonic energy due to what I believe to be noise shaping, while DSD version has a sharp roll-off filter around 24KHz.

 

 

 

[pkane2001]

7 minutes ago, mansr said:

DSD can be analysed directly if you simply translate the 1 and 0 bits into numerical values of 1 and -1. You can run an FFT directly on this, though to get the same frequency resolution as with normal PCM, you obviously need to scale the transform size up accordingly. If you're only interested in frequencies below, say, 100 kHz it is more efficient to resample to a lower rate using a good anti-aliasing filter before doing the FFT. Besides being faster, this also avoids potential numerical precision issues.

 

That's from Sound Liaison. It was recorded in 352.8 kHz PCM. For the DSD conversions, they play the PCM recording on an unspecified DAC and record the analogue output with a DSD ADC, also unspecified.

 

The drop-off there is suspiciously close to 24 kHz. I wonder if the DAC they used during the DSD conversion was running at 48 kHz.

 

Since there's quite a bit more than a simple FFT that's needed in DeltaWave calculations, I don't see another way but to demodulate DSD. It's not a complicated process, but took me a while to find the relevant papers that I could convert into code.

 

Sound Liaison? Very possibly. I downloaded this a couple of weeks ago, and by now don't remember where. Old age is kicking in 

 

[pkane2001]

2 hours ago, mansr said:

That's from Sound Liaison. It was recorded in 352.8 kHz PCM. For the DSD conversions, they play the PCM recording on an unspecified DAC and record the analogue output with a DSD ADC, also unspecified.

 

Went back to Blue Coast Music downloads this time, and the picture is a lot better.

 

Again, DSD256 format compared to PCM 24/96:

 

Spectrum of the differences in audible range:

 

Null depth is much better, as well at 85dB

 

 

[pkane2001]

1 minute ago, mansr said:

Demodulate? It's just a standard low-pass filter.

 

CIC integrator is what I'm using.

[pkane2001]

2 minutes ago, mansr said:

Why? Those are computationally cheap but otherwise rather limited. A regular linear phase FIR filter with 10k or so taps would give better results. If you don't have enough stopband rejection, all that modulator noise (and there's a lot of it) will alias and cause problems.

 

May try it when I figure out a way to code it efficiently. I tried smaller FIR filters with poor results, so went with CIC as it was simple enough to code from scratch.

 

 

[pkane2001]

8 hours ago, bachish said:

 

These are quite cool and informative. Thanks, pkane2001.

 

These results are very interesting but I am a bit confused. I'm wondering why use DSD when there is such a steep filter?  Excuse my ignorance here but wasn't the point of DSD to avoid having a filter?  That is why the sample rate is over 2 million, so all artifacts are out of the audible range.  

 

And that they null as well as they do sems to demonstrate that the idea of 1 bit sampling - i.e. following the curve of the analog wave  better than PCM - doesn't hold water so well.

 

Having said that, I'm not against DSD. It sounds great when done correctly but it doesn't seem to be worth all the extra inconvenience when recording and in post.

 

I think that’s true, bachish, but I’m not sure I proved the general care here, just a couple of recordings.

 

The first was obviously run through a sharp low pass filter around 24KHz, so there’s almost no useful information above that point. Whether information above 24KHz is ever useful is also certainly worth considering

 

DSD quantization noise is huge in the audible range, so it needs to be shifted to higher frequencies through noise shaping. Once there, I think it’s still a very good idea to filter it, otherwise there will be a lot ultrasonic noise that can interact with the components in your system, and some say, even intermodulate into the audible range. So, if filtering is necessary, the next question is where to do it.

 

24 kHz seems reasonable enough, although I don’t see the point of using such a sharp filter as in the recording from Sound Liaison. As Mansr said, perhaps they are using some equipment that has this filter without realizing it’s there. I’d move it further out and make it a more gradual filter. At least that might allow some possible benefits of DSD to come through.

 

As to the comparison with 24/96 PCM recording,  I take DSD256 and then convert it to 24/96 for analysis. Possibly whatever benefit there exists in DSD due to less aggressive filtering and potentially higher resolution is lost in this process? Not sure. I’ll try to convert to higher sampling rates to see if it makes a difference.

 

Personally, I’ve stopped listening to DSD primarily because it (subjectively) sounds a bit wrong to me on my DACs. To my ears it adds this tiny level of brittleness to the sound that makes human voice and strings sound  bit more natural, but makes a lot of instruments sound less ‘whole’ — don’t know if I can explain this better. Very minor effect, but enough for me to notice. Of course, on the objective side, the lack of processing tools in DSD and large file size for high DSD rates is also a problem for that format.

[pkane2001]

1 minute ago, Jud said:

 

Although maybe you kind of haven't, since your DACs very likely do sigma-delta modulation internally.   

 

The ESS DAC, yes, but not the R2R one

 

 

[pkane2001]

25 minutes ago, Jud said:

 

Right, but you never played DSD on that one anyway.  

 

Why? Been playing DSD on both DACs for at least 6 months straight (both, PCM and DSD content) until I tried switching back to PCM. The R2R DAC I have is using a resistor ladder for both, PCM and DSD playback. Mansr's point that nearly all PCM is captured using sigma-delta ADC process is valid, though

 

 

[pkane2001]

2 hours ago, pkane2001 said:

As to the comparison with 24/96 PCM recording,  I take DSD256 and then convert it to 24/96 for analysis. Possibly whatever benefit there exists in DSD due to less aggressive filtering and potentially higher resolution is lost in this process? Not sure. I’ll try to convert to higher sampling rates to see if it makes a difference.

 

So I upped the DSD256 conversion rate to 192KHz for comparison. The PCM track was also resampled to the same rate since the original was at 96Khz. Here's the result:

 

Here's the original 96KHz sample-rate comparison, for reference:

 

[pkane2001]

1 minute ago, mansr said:

That doesn't make sense. Which DAC are we talking about?

Holo Spring.

[pkane2001]

7 hours ago, pkane2001 said:

 

So I upped the DSD256 conversion rate to 192KHz for comparison. The PCM track was also resampled to the same rate since the original was at 96Khz. Here's the result:

 

 

Here's the original 96KHz sample-rate comparison, for reference:

 

 

 

And if anyone is still watching... Here's the Blue Coast Music track, DSD256 converted to 24/192 (blue) compared to 24/96 converted to 24/192. The falloff at 48KHz in the PCM track is due to the Nyquist frequency  filter applied as part of the upsampling process:

 

 

[pkane2001]

Just now, mansr said:

Could you please make the frequency axis tick marks a little less random?

 

Not easily. The charting package is selecting these for me, something I've struggled with for a while (oxyplot). I can zoom in on any area of the chart, and it'll fill them in a bit better