red or blue pill - Part II?

[manisandher]

Many of you will be familiar with the test that @mansr and I conducted almost two years ago:

 

Over a long period of listening, I became certain that I could hear consistent differences between various bit-identical playback means (streaming vs. local playback, various bit-identical settings in playback software, etc.), and was confident that I could demonstrate this. So, I invited Mans up to my place to help me test my hypothesis.

 

I scored 9/10 in the blind ABX (1% probability of achieving this result through guessing alone), proving my hypothesis correct... or so I thought.

 

Many objectivists here remained sceptical, some even suggesting that I had achieved the result through pure luck!

 

So...

 

If there were to be a retest, what would need to be done differently for those of you who remained sceptical to have more confidence in the results?

 

Mani.

[manisandher]

Hi Paul,

 

14 hours ago, pkane2001 said:

1. Test something that's more of a common concern. A setting in Phasure XXX software is too obscure and not of much interest to anyone who doesn't use this software (I'd say the majority here).

 

Yes, any retest would use streaming vs. local playback using Roon.

 

14 hours ago, pkane2001 said:

2. Run more tests. You had a few low-scoring attempts before getting 9/10 (I'm aware of the reasons you gave, but an explanation of why a test failed is usually not acceptable in lieu of a positive result). Two consecutive 9/10 tests or 8/10 would be much more convincing. If you do run multiple tests, the results of all of those tests should be included in the calculation, good or bad.

 

I reckon a 10-run ABX would take around 10-12 minutes. I'd go for 3 of these in total, with a small break between each. This would give a total sample size of 30. Would you agree that a score of >25/30 could be taken as a positive?

 

14 hours ago, pkane2001 said:

3. Use better-vetted equipment to record digital and analog feeds -- I was confused by the provided files when trying to analyze them, there were some issues with one ADC and different issues with another, etc. It wasn't clear to me which recordings were accurate representation of what was tested (if any).

 

There was no issue with the digital feed - it proved that the DAC received bit-identical signals during the ABX, where I scored 9/10.

 

I used a Tascam DA-3000 ADC for the initial analysis of the analogue output of the DAC. Here are its specs:

How much better would the ADC need to be to be acceptable?

 

14 hours ago, pkane2001 said:

4. Document and review the test procedure with us before the test, not after. 

 

Absolutely, and hence this thread.

 

14 hours ago, pkane2001 said:

5. Some photos/video recording of the session might be helpful in reviewing.

 

I'd video the entire test in both the listening and control rooms. (Capturing the sound with a decent microphone in the listening room might prove useful during analysis too.)

 

Mani.

[manisandher]

13 hours ago, mansr said:

... I was initially told the comparison would be between playback of the same file from local vs network storage and/or Tidal. Only on arrival at Mani's house did I find out that he had changed his mind.

 

You don't half talk bollocks sometimes.

 

Over 3 weeks before you came up:

 

On 3/2/2018 at 11:48 AM, mansr said:

The hypothesis is that identical files played from different storage media sound different. Presumably, the 5-10 seconds is the time it takes to stop the software player and start playing a different file.

 

On 3/2/2018 at 1:06 PM, manisandher said:

That's a specific case of the more general hypothesis: a file played* back bit-identically can sound different.

 

Specific cases then include:

- different storage media

- different digital cables (spdif, USB, etc)

- different software player configurations (buffers, etc)

 

I'm considering which of these would be best for the ABX, and am leaning towards different software player configs.

 

And then, the day before you came up, I sent you this PM:

 

"Hi Mans,

 

This is the procedure I'd like to use tomorrow:

 

1. take a quick listen together

- I'd like to demonstrate a few things to you and get your initial thoughts

 

2. conduct the A/B/X

- I've chosen the track and the bit-identical changes we'll use in the playback software

- you'll be sitting in my office, controlling playback from there, and I'll be sitting in the listening room

- we'll have the Tascam set to auto-record, sitting in the basement next to the audio PC and DAC, capturing the digital output of the audio PC in real time

3. ensure that the digital captures are identical

- I have Audacity and MusicScope here

- if you have other software you'd rather use feel free to bring your laptop along with you

 

4. capture analogue outputs (test track plus tones)

a. directly from the DAC, using the Tascam

b. from the speakers, using my Earthworks microphone and portable Korg recorder

 

5. analyse analogue outputs

- we could either attempt to do this here or you could take the files away with you

 

Obviously, there's no need to go on to step 3 unless you're convinced there really are audible differences (either because you hear them too, or because I manage to demonstrate that this is the case in the A/B/X).

 

It'd be great if we can get through all of this tomorrow, but really only need to get through 1 and 2 as a must.

 

See you tomorrow!

 

Cheers,

Mani."

 

Your reply:

 

"Sounds like a plan."

 

Mani.

[manisandher]

12 hours ago, Archimago said:

Obviously there were enough major concerns and uncontrolled variables that a more thorough investigation seems fair, right?

 

I find it useful to consider this in two parts:

 

1. the ABX listening test itself

 

The specific DAC, amps and speakers used for the ABX, and where they're situated, etc. are all totally irrelevant. It's a red herring on Mans's part. The only thing that matters is showing that the DAC received bit-identical signals when replaying A and B. The digital feed to the DAC was captured throughout the ABX, and it was proven that the DAC had indeed received bit-identical signals throughout. And yet I heard consistent differences between A and B, as shown by my 9/10 score.

 

2. the analysis of the analogue output of the DAC

 

This proved difficult. I've posted the specs of the ADC used in the original analysis, which show nothing untoward. But on analysing the analogue captures, the ADC proved unsatisfactory.

 

Mani.

[manisandher]

8 minutes ago, Arpiben said:

In principle, ABX testing and analogue recording (ADC) should be performed simultaneously.

 

Hi,

 

Yes, I can see why this would beneficial. But during the previous ABX, I only had one capturing device to hand. We could have used it to either capture 1) the digital input into the DAC or 2) the analogue output from the DAC, in real-time during the ABX. It was absolutely critical to determine that the digital input remained bit-identical for both A and B (else the ABX would have been invalid), so we prioritised 1) over 2).

 

In a retest, I think it would be great if we could use two capturing devices, one for digital and one for analogue captures in real-time during the ABX. Even better if I could use a single clock to synchronise them.

 

So, I need two ADCs with great specs and wordclock inputs/outputs...

 

Mani.

[manisandher]

12 minutes ago, pkane2001 said:

Mani, there are some things that need to be considered in (1) as well. Bit-perfect transmission is not sufficient if the interface (SPDIF) also carries the clock signal used to drive the DAC output. Large amount of jitter on the interface can cause timing errors. For this reason, using a USB or another asynchronous connection to the DAC would be a much better choice, IMO.

 

So, use a USB DAC with an spdif output, and capture spdif output in real-time to test that A and B are bit-identical?

 

Any way of checking A and B are bit-identical using a USB DAC without an spdif output?

 

Mani.

[manisandher]

9 minutes ago, semente said:

Did you get torture-endurance training in the special forces?

 

The previous 10-run ABX took nearly 20 minutes to complete. Each A and B sample lasted for about 20 seconds, and then another 20 seconds for the X. So you're concentrating hard for over a minute for each run. I can tell you, doing 10 runs back-to-back was exhausting.

 

Probably why I got #9 wrong .

 

Mani.

[manisandher]

44 minutes ago, pkane2001 said:

With DeltaWave software, I've compared now a large number of DAC/ADC loop-back recordings from various sources. The better equipment produces a null of -90dB or better. The recordings your sessions produced generated, at best, a -50dB null. That's a really poor conversion quality compared to some of the better equipment out there.

 

I've got the following ADCs to hand:

- RME Fireface 800 (need to repair the SMPS first)

- MOTU 896HD (mkI)

- Prism AD124 (44.1/48 only)

- Tascam DA-3000

 

(Shame I don't still have my Pacific Microsonics Model Two .)

 

The Tascam should be fine for the digital captures, as there's no actual conversion going on.

 

I'll do some DA/AD loops with my USB DAC and the other converters to figure out how they fare. Would a simple sine wave suffice, or would real music content better?

 

Mani.

[manisandher]

15 hours ago, fas42 said:

But to do a serious test, well, you need to be at complete ease with the situation.

 

Agree 100%.

 

Mani.

[manisandher]

Thanks for your thoughts Peter.

 

7 hours ago, PeterSt said:

Can someone please repeat what's on then test bench here ? apart from a nice discussion, I personally lost the subject.

 

At this point, just a "nice discussion".

 

7 hours ago, PeterSt said:

Mani, you can't just let yourself drive into a corner and remain to be in the position that you are able to hear differences between "anything", which is what it seems to be about by now. I mean, up to listening to 1KHz sines, as it seems. 😮

You never clearly stated what this listening test will be about, according to you and your capabilities. You seem to allow to let others determine what you will be able to discern. Well, that most certainly isn't going to work ...

 

No, I'm not going to get myself into any corner(s). At this point, I just want to gather some ideas and get a sense of how we could improve on the previous listening test. That's all. I'm absolutely not committing to anything.

 

And I certainly wouldn't be listening to sine waves! That discussion with Paul was purely about testing the capability of an ADC, using a loop-back, and comparing against the source. We'd be looking for a difference of below -90dB or so between the resulting DA/AD and the source file. I just wanted to know what source would be best for the loop-back test. No listening involved whatsoever.

 

7 hours ago, PeterSt said:

Like you, I too can easily state what I can (and what I can't) discern, but that doesn't mean that I would be able to technically prove it. Scientifically, yes (statistics), but technically not. I'd even give you or anyone the guarantee.

 

OK, this gets to the crux of things. And it's why I think it's helpful to separate the methodology into two:

 

1) Proving that bit-identical replay can sound audibly different. ("Statistically")

 

Can be done via a controlled blind ABX, and showing that A and B remain bit-identical. This was exactly what we did in the previous ABX. I'm confident that I could at least match my previous result in a new ABX, using streaming vs. local replay, say.

 

But this shouldn't really be that controversial. As Paul stated earlier, jitter on the interface can cause timing errors. So, bit-identical replay certainly can sound different if something changes the amount of jitter reaching the DAC chip(s). I remember buying my first CD green marker pen, which certainly changed the sound... but unfortunately for the worst. I tried on on a couple of of CDs before cleaning their sides again and throwing the pen away. The green marker changed the sound, without changing any bits whatsoever.

 

People seem to think that modern USB DACs are immune to these sorts of influences. I don't think so - based on my experience.

 

2) Understanding the mechanisms that cause bit-identical replay to sound audibly different. ("Technically")

 

The biggest issue here by far is actual measurement.

 

7 hours ago, PeterSt said:

Let me remind people that quite in the beginning of the Red and Blue Pill I, I indicated that no technical test means would exist to prove it. People, mansr too, ignored that too easily. This, while it is (and appeared to be) the most crucial. Thus, someone scoring 99% probability is one, but trying to prove by technical means how that was done, is two - and impossible. One simply can-not prove the rightness of it by means of using too weak test means. So better have consensus about that first. And that can't happen by means that I know of, so we should ditch that part.

 

I totally agree that before any repeat ABX, we need to get a means of measuring the analogue output of the DAC that changes consistently with bit-identical changes upstream, e.g. streaming vs. local playback. You say it's impossible. I'd like to at least explore things further...

 

Peter, a question for you: If the ear can detect the difference (assuming proven by the ABX), why can't we measure it, either at the output of the DAC, or in the listening room with a microphone? Is it that the measurement device (in this case the ADC) simply isn't resolving enough (THD+N, clock stability, etc), or is it that the measurement device is simply measuring the wrong thing, and therefore not fit for purpose? If the latter is the case, then my earlier suggested loop-back tests will prove pointless - I could get the best-measuring ADC on the planet, and it still wouldn't find any difference between the analogue outputs.

 

7 hours ago, PeterSt said:

What thus, IMHO, remains, is a fairly simple test depicted by Mani (he knows what his capabilities are and with what means), which definitely needs real time checking for bit perfectness. And that is easy enough. Still a note of warning: whatever the means, it will influence SQ and Mani can't just let it happen that any random means is injected in his chain which will obfuscate. So whatever the means for testing bit perfectness, *or* #1 Mani must have applied that himself for approval by those directing the later test, *or* #2 those who direct the test must apply it, for Mani to check the usability for him. Probably for quite some days of easy listening (not under stress of the day itself). Next he should be allowed to reject the means because it does not work for him.

If I may give an advice on this one: try to accept #1 because else the lot will be a too tedious procedure (#2 may even imply several iterations).

 

I would only ever do an ABX if I were confident that I could still hear audible differences with the final setup. For example, before the previous ABX, I tried putting a preamp (a Pass Labs X1) into the chain to give us the option of taking analogue captures in real time. But it totally screwed up the sound, so I insisted on a direct DAC -> power amps path for the ABX.

 

Mani.

[manisandher]

On 1/13/2020 at 4:17 PM, pkane2001 said:

Our ears are not any more sensitive than precision electronic equipment. We may not be measuring the right things or using the right tools to measure in some cases, but there is absolutely no evidence that the ear can hear something that the measurement instruments cannot detect. 

 

Paul, as you know, I've been exploring getting hold of an ADC that would be up to the job of measuring the bit-identical differences I'm hearing. Here are the loopback measurements (most taken from the Gearslutz DA_AD thread) of the ADCs I currently have to hand:

(* Potential future purchase.)

 

I suspect that in the majority of cases it's the ADC that's the limiting factor in these measurements, and not the DAC. So, it seems to me that none of these are really be up to the job at hand... and the RME is considered one of the best currently available!

 

So where to now?

 

Mani.

[manisandher]

8 minutes ago, pkane2001 said:

Let me run some of these through DW (assuming the loopback files are available) to get proper results.

 

That'd be great. Thanks.

 

Mani.

[manisandher]

30 minutes ago, pkane2001 said:

 

 

 

Thanks Paul, that's really, really helpful. I might hang on to my new M2 after all .

 

I've got a Fireface 800, so will do some loopbacks (once I've fixed its SMPS - second one to fail on me on an FF800!).

 

The ADI-2 Pro FS is supposed to be the king of modern 'affordable pro' ADCs. Any idea why it performs so badly in the loopback?

 

Mani.

[manisandher]

7 minutes ago, pkane2001 said:

 

Hold the presses! Found another loopback recording. This one is described as using ADI-2 Pro FS with sharp filter setting for both, DAC and ADC. A much better result (see last item):

 

 

Great! I'll go ahead and order an ADI-2 Pro FS.

 

(The M2 seems pretty good for the price though.)

 

Mani.