Bit-identical playback CAN sound different

[manisandher]

I've noticed that there have been a few references recently to my 'Blue or red pill?' thread:

 

I really have no interest in repeating the test. If you believe that all bit-identical playback sounds the same, then more power to you. You're delluded I'm afraid, but at least you've saved yourself a lot of time and effort in optimising your playback chain.

 

But recent mention of the test prompted me to do a quick follow-up. I captured the output from the speakers in my office, using the two exact same bit-identical settings we used in the test. Here's the replay/record chain I used:

 

XXHighEnd -> USB -> RME ADI-2 Pro FS R (as DAC) -> balanced XLR -> First Watt F4 mono amps -> Tune Audio Marvel speakers -> Rode NT4 stereo mic (~3m from speakers) -> MOTU M2 ADC -> Audacity (24/48)

 

And here are the two files:

 

Take 1: https://drive.google.com/file/d/1pA6Wq67q8Nh7csC-iuA2lN4k-JZR0wT_/view?usp=sharing

Take 2: https://drive.google.com/file/d/17UXfLQtKLqunanHsE9NTlIDIVl9ZppXE/view?usp=sharing

 

If you can't hear a difference... sleep well at night. If you can... you've swallowed the red pill 😉.

 

Mani.

[manisandher]

DeltaWave shows the main difference between the two mic captures as lying between 16-20kHz:

 

 

This is just one data point of course. At some point, I'll take a number of captures at both settings to see if there is a consistent difference.

 

Mani.

[manisandher]

I was just replying... but now no need to 🙂.

 

Let me know if there's anything else I can help with.

[manisandher]

I'll try to take more mic captures when the boys are asleep... and the wife's not blasting the TV in the living room 😉.

[manisandher]

6 hours ago, sandyk said:

... however the conversions are no longer Bit Perfect.

 

Sorry Alex, I don't understand what you mean.

 

Mani.

[manisandher]

1 hour ago, sandyk said:

By that, I mean that because the captured microphone recordings are not bit perfect, this means that you should also also include the  original captured identical .wav files for transparency. 

 

Alex, I don't think bringing up the term 'bit-perfect' is at all useful here.

 

The two microphone captures are of a file being played back in two bit-identical ways. With just a simple listen, one should be able to determine whether they sound different or the same. If they sound different, then we can start asking why that might be. The DAC is a well-regarded professional unit with excellent measurements. And yet, without changing the bits reaching it at all, we can change its sound.

 

Nice and simple.

 

Mani.

[manisandher]

Thanks Paul

 

32 minutes ago, pkane2001 said:

The thing that's interesting above is the difference below 75Hz or so.

 

I wouldn't be inclined to read too much into this. Our front drive is a few meters away from a main road and it may well be that a lorry just happened to have passed by during one of the takes.

 

32 minutes ago, pkane2001 said:

Also, not that it should be audible (maybe through IMD?), but there's some noise above 19kHz:

 

This is what caught my eye immediately. The noise shows up on the capture with XXHighEnd SFS=0.69. Generally, the lower the SFS setting the 'sharper' the sound. Perhaps due to noise?

 

32 minutes ago, pkane2001 said:

PK Metric also shows some areas of concern:

 

What exactly does the 'PK Metric' show?

 

I obviously need to take more measurements to eliminate random occurrences...

 

Mani.

[manisandher]

9 minutes ago, yamamoto2002 said:

For control experiment, it is nice to have two recorded data of the same file

 

Yep. And I'll get around to doing more recordings.

 

Mani.

[manisandher]

28 minutes ago, pkane2001 said:

Yeah, the noise above 19kHz is persistent through the whole recording, and it's about 10dB in magnitude, so not tiny.

 

That doesn't sound too promising to me. The MOTU M2 is connected to the regular work PC on my desk. Although passively cooled for the most part, it does have a couple of fans in its graphics card which are just about audible when the room is otherwise silent. I wonder if the HF noise comes from these fans - maybe they ramped up during one of the takes? Easily checked - I'll take a few recordings of the ambient room noise.

 

If necessary, I can plug the M2 into my totally passive laptop for further mic captures.

 

35 minutes ago, pkane2001 said:

PK Metric attempts to apply perceptual weights to the difference signal. It takes into account equal loudness curves, so that both, the level and the frequency of the error drive the magnitude, and then also uses ERB smoothing to compensate for errors that occur close enough in frequency to be masked by the main signal. In effect, it helps with audibility determination, as well as, shows where in the file the error is likely to be more audible (peaks, like the one around 8-9 second mark and then above 11 secs). 

 

Thanks.

 

Playing either track back through DeltaWave, two things occur to me:

 

1. the leading edges of the guitar plucks coincide perfectly with the peaks in the PK Metric curve

2. the leading edges of the guitar plucks clearly sound softer on one of the tracks than the other (in exactly the way any XXHighEnd user would expect)

 

As I've said, I'll take more captures with the mic...

 

Mani.

[manisandher]

9 minutes ago, danadam said:

Can't say I hear anything different at 8-9 or 11 seconds.

 

Listen for the following differences between the two tracks:

1. the leading edges of the guitar plucks

2. the sibilance in his voice

 

One track is slightly softer (easier to listen to) in both cases... to my ears.

 

Mani.

[manisandher]

10 minutes ago, yamamoto2002 said:

That 19kHz noise may be inductor whine, move microphone to close to any switching power supply or DC-DC converter (PC, network switch, monitor display, graphics board etc) and see what device emit the noise :)

 

The mic remained totally static for both captures. Why would there be more noise in one than the other, the second recorded literally seconds after the first finished?

 

Mani.

[manisandher]

34 minutes ago, yamamoto2002 said:

network traffic, background tasks, power state vary in time. computer display noise varies when drawing content is different.

 

4 minutes ago, PeterSt said:

You realize that it is all about exactly that, right ?

(but the taming of it by various means and in various base states (statuses))

 

We need to distinguish between the optimised audio PC connected to the DAC and the regular desktop PC connected to the ADC. I think Peter is referring to the former and how it might affect the output of the DAC (and therefore indirectly the output from the speakers), and yamamoto2002 is referring to the latter and how it might affect what the mic is picking up directly.

 

40 minutes ago, yamamoto2002 said:

turn off room lights and computer monitor (set 1 minute timeout), and disconnect network, and use battery powered laptop. Insert several minutes of silence track before test signal and play

 

Yep, will do.

 

Mani

[manisandher]

1 hour ago, pkane2001 said:

What is interesting, as well, is if you listen to the difference between the two. It sounds like a very quiet reverb/reflection rather than the original sound, almost like an echo. But it is true that it's more of the high frequency sound (much lower than 19kHz, but like you describe, the leading edges are most pronounced):

 

https://app.box.com/s/had70geer8z8jkkzd5idijah1ddjrrh0

 

Thanks Paul. Very interesting.

 

1 hour ago, pkane2001 said:

Mani, are you sure you didn't move between the two takes? Even a small repositioning could change the reflections when they reach the mic, especially if you were not far away.

 

I was seated at my desk ~1m immediately to the left of the mic and remained as still as I could throughout. Only my hand/arm really moved to control the mouse/keyboard.

 

1 hour ago, pkane2001 said:

The sound at 32 seconds is something dropping or door slamming, I doubt this came from the speakers.

 

No doubt my wife or one of the boys.

 

I'll try to do any retakes when they're all asleep 🙂.

 

Mani.

[manisandher]

2 hours ago, pkane2001 said:

Note: I've increased the level by 15dB to make this easier to hear...

 

I actually find it more instructive to listen to it without the 15dB increase. Playing it against the two original files, the difference between them becomes immediately apparent. As I've been saying, listen to difference in the leading edges of the guitar plucks and the sibilance in his voice.

 

Mani.

[manisandher]

6 hours ago, March Audio said:

You will be lucky to get a 60dB signal to noise ratio when you need to be looking more like 120dB+.  The room has far too much (and inconsistent noise).  Your speaker and microphone add distortion.

 

Agreed. And this will probably come to nothing as a result.

 

6 hours ago, March Audio said:

A far better way would be to just use the audio interface in loopback feeding the output back to the input and recording that.

 

Have done this many times already, but cannot find any consistent differences between recordings. And yet, I hear differences during playback (9/10 in blind ABX, etc.). That's why I thought I'd try a mic, just to see what comes up...

 

Mani.

[manisandher]

3 hours ago, March Audio said:

So I am surprised that you wont participate in a listening test.

 

Did you take a listen to the two files I posted in the original post? If so, could you hear a difference in sound quality (and not the obvious differences in background noise) between them?

 

Mani.

[manisandher]

20 hours ago, PeterSt said:

The gist of the thread is also about being under 70dB loud kitchen hoovers, and still being able to pick 10 out of 10 the differences.

That easy it is.

 

I find it absolutely amazing that differences between bit-identical playback can be heard easily, almost irrespective of the level of the room's (non-correlated) ambient noise. And yet, as hard as I've tried, using the best DACs and ADCs available, I can't measure any differences at the output of the DAC.

 

If you haven't already done so, just take a casual listen to the two files I posted originally (an ABX won't work because there are obvious differences in ambient noise, and in any case, isn't at all necessary).

 

It seems to me that the human ear must be incredibly sensitive to the leading edges of sounds. And that whatever mechanism is at play isn't limited by hearing bandwidth.

 

Anyway, I plan on taking a few more mic captures at some point today. I'm not really hopeful of anything coming from this (too many past disappointments), but let's see...

 

Mani.

[manisandher]

20 minutes ago, PeterSt said:

3. Capture the sound of the two playbacks by microphone. Look at the diff files of it, might you be able to do something with that.

 

Not a lot from what I'm seeing right now. Listening to some new mic recordings, bit-identical SQ differences are audible. But all the extra noise in the recordings (ambient room, mic preamps, bloody cars passing by, etc.) makes it pretty much impossible to get consistent nulls in DeltaWave.

 

I'll link the new recordings in a short while, for anyone who might be interested in playing with them...

 

Mani.

[manisandher]

3 minutes ago, March Audio said:

It's a very insensitive and inconsistent test.

 

I agree. And yet the differences are easily audible... to my ears at least.

 

Mani.

[manisandher]

I took 6 new mic captures, with two different bit-identical settings, A and B. The recording sequence was A, B, A, B, A, B. I then reduced each recording down to a 2-second snippet with a strong guitar pluck. So, we have 15 differences: three A vs. A, three B vs. B and nine A vs. B.

 

There's not a lot to take away from this, but here are the results from DeltaWave for completeness:

 

A vs. A and B vs. B (6 samples)

Average correlated null depth = 37.93dB (SD=8.40dB)

Average PK Metric (max) = -38.20dB (SD=3.78dB)

 

A vs. B (9 samples)

Average correlated null depth = 35.26dB (SD=2.45dB)

Average PK Metric (max) = -38.03dB (SD=5.20dB)

 

As I and many others have already speculated, there are just too many uncontrolled variables to get any meaningful results. And I would have to take a whole load more mic captures to get more accurate SDs.

 

I haven't totally given up... yet. I've been using my office system to date, because all the recording equipment is already set up for virtual meetings - all I have to do is swing the mic boom into position and hit record and play. But my home office looks out onto a main and usually noisy road. So, I'll switch to using my main system going forward... when I muster enough enthusiasm to do so 🙂.

 

Mani.

[manisandher]

1 hour ago, PeterSt said:

Mani, you won't be able to get nulls anyway because of the inherent noisy nature of molecules and microphones etc. etc.

 

Agreed... for the time being at least 😉.

 

Mani.

[manisandher]

4 minutes ago, March Audio said:

So if you agree the test is inconsistent and insensitive why draw the conclusion the differences you perceive are due to the player?

 

Because I've been hearing differences between bit-identical playback for decades now, along with a whole bunch of others who have too. That's why I set up the blind ABX with @mansr 3 years ago. My 9/10 in the ABX wasn't a fluke - it was because I really could hear differences... even under the stress of a blind ABX, the results of which I knew would be posted for all to see.

 

4 minutes ago, March Audio said:

This conclusion implies the differences are gross and yet we can measure no differences.

 

I'm not sure about "gross", but the differences are certainly large enough to be audible. I think the tracks I posted give some sense of this (though not nearly as much as listening directly) - perhaps you should take a listen and decide for yourself? If the differences don't show up in the measurements, then it must be that the measurements aren't up to the job. And I think we're both agreed on this final point.

 

Mani.

[manisandher]

4 hours ago, John Dyson said:

What is the drive to 'tweak' or be worried about impairments that are a small portion of the actually controllable damage to the recordings?   Is it from frustration, and starting an OCD type process because of deep frustration about the quality?

Even if people don't directly perceive the impairments in the recordings, even with accommodation, there should be something 'subconscious' that hears the damage against the sound...

 

There seem to be mechanisms at play, that we don't really understand, that change the sound of bit-identical playback. That those of us who hear these changes have to rely on trial and error rather than a systematic approach frustrates the hell out of me.

 

Why is there no systematic approach? Well, because many (most?) digital audio engineers are entrenched in the belief that as long as the bits remain identical, the sound will remain identical, provided jitter remains below a threshold of a few ns or so. So, a "as long as the DAC is well-designed the sound cannot change" sort of argument pervades engineering circles. But I hear a change in the sound of my RME AD-2 Pro FS R when I make a (cost-free) bit-identical change in the software player. Or when I change the USB cable. Believe me, I really wish I didn't. But I do.

 

I don't consider such changes in sound a "small portion". For example, I can change from one software player setting to another (both bit-identical) and just lose interest in listening to music. When I capture the two analogue outputs from the DAC with a high-performing ADC, they null to below 105dB. I have no bias for one software setting over the other - I really couldn't care less. But I do care very much about the enjoyment I get from listening to music.

 

I've been experiencing these sorts of things for over two decades now - change this, the sound changes; change that, the sound changes. 3 years ago, I 'proved' it with my 9/10 score in a blind ABX. And I'm not the only one - there must be literally thousands of people who hear bit-identical changes as sounding different, mostly through different USB cables I suspect.

 

I totally agree that this opens up the opportunity for profiteers to take advantage of those of us who are on the OCD spectrum (all audiophiles perhaps?). This is where one's judgment comes in. Who's selling what? How credible are they? How trustworthy? Etc.

 

It seems to me that there's something fundamental at play here that is not well-established and accepted. I believe that if we can explore it with an open mind, there may be something really fruitful at the other end.

 

Mani.

[manisandher]

Thanks Peter.

 

2 hours ago, PeterSt said:

So while this is a discrete / qualifyable - even including patterns - plot, the diff files working at the "dB" level are not at all. They show differences alright, but we can't work wit hit.

 

Hypothetically speaking... At what level of resolution would the diff files have to be in order to be useful? -120dB? -150dB? -200dB? Or lower still?

 

Mani.

[manisandher]

23 minutes ago, PeterSt said:

Ah, I planned for that myself. In that case I will also have one by the end of the week. :-)

 

I have one too 🙂.

 

Mani.