Expectation Bias

[manisandher]

2 minutes ago, firedog said:

Sort of. It's a necessary but not sufficient condition.

People who are totally unscientific also have intuition and curiosity.

 

A hypothesis is based on intuition, almost by definition. Curiosity then leads you to test it.

 

3 minutes ago, firedog said:

Science is a method for finding out repeatable objective results and producing hypotheses that get us closer to the truth about the natural world. 

That's why eliminating all bias from testing and discovering something that's repeatable is important.

Your "experiment" is interesting, but is only a first step.

But it shows why this stuff is almost never done in audio. The comments about experiment design and the need to do more are spot on. It's difficult and expensive  to do right, and the incentive really isn't there. There's little academic interest, and commercial interest is mostly against it. 

 

 

Agreed.

[pkane2001]

5 hours ago, manisandher said:

 

Again, you're conflating two things. The listening test was only to determine if differences in bit-identical playback were audible. The digital recording of the spdif signal was necessary to show that the differences (in this case buffer settings) remained bit-identical at all times.

 

 

No, I meant I can easily capture the digital input into a USB DAC in real time, to show that things remain bit-identical (because I have DACs with USB inputs and spdif outputs). So, repeating the test using USB instead of spdif would now be possible.

 

I have a bunch of USB cables here, and am confident I hear differences between them. However, I would say that there is zero correlation between price and SQ. One of my favourite-sounding ones is a 1.5m USB-certified cable that I paid a few pounds for many years ago. The most expensive I have here I paid a stupid amount for (in my more gullible days), and sounds terrible... IMO.

 

Now, how's that for expectation bias?

 

 

I have a number of modern, well-measuring DACs here (Okto dac8 PRO, SMSL DO200, RME ADI-2 Pro, MOTU UltraLite-mk5). I'm certain I hear differences between USB cables with them all.

 

 

Yep.

 

I haven't stopped thinking about these things over these last few years. I've come up with a method that I'm hoping will show differences at the analogue output of the DAC with bit-identical changes upstream. But it's going to take a lot of time and effort to do, which I just haven't been able to find to date.

 

If the method works and differences are indeed detectable, I'll then like to start exploring possible mechanisms at play, and perhaps setting up more listening tests.

 

I'm not a manufacturer, a dealer or a reviewer. I have nothing to gain from this endeavour... other than to satisfy my curiosity 😉.

 

Well, I guess I misunderstood the purpose of the test, then, when we first discussed it. SPDIF and USB handle clocks very differently and the amount of transmission (or source) jitter in SPDIF can have a large effect on the DAC analog output, depending on the clock recovery circuit. USB isochronous protocol is mostly immune to these issues. What's more, the setting that was varied in the test, at least as described by PeterSt, appears to be a very likely source of jitter. Based on this, I'd be measuring jitter performance of the system, first, before trying to look for any other causes. Ground loop(s) would be my next check. 

 

Oh, and I'm not a manufacturer or dealer, either. All I do is out of curiosity ;)

[botrytis]

5 hours ago, manisandher said:

 

A hypothesis is based on intuition, almost by definition. Curiosity then leads you to test it.

 

 

Agreed.

 

NO - a hypothesis is based on previous data, similar type of experiments, etc. It is not intuition; it many times is extrapolation.

[The Computer Audiophile]

46 minutes ago, pkane2001 said:

 

Well, I guess I misunderstood the purpose of the test, then, when we first discussed it. SPDIF and USB handle clocks very differently and the amount of transmission (or source) jitter in SPDIF can have a large effect on the DAC analog output, depending on the clock recovery circuit. USB isochronous protocol is mostly immune to these issues. What's more, the setting that was varied in the test, at least as described by PeterSt, appears to be a very likely source of jitter. Based on this, I'd be measuring jitter performance of the system, first, before trying to look for any other causes. Ground loop(s) would be my next check. 

 

Oh, and I'm not a manufacturer or dealer, either. All I do is out of curiosity ;)

Do you think the setting could change ground loops or change how a system responds to ground loops? 

[pkane2001]

4 minutes ago, The Computer Audiophile said:

Do you think the setting could change ground loops or change how a system responds to ground loops? 

 

No, what it changes is the PC-generated noise patterns based on this setting. These can leak into the DAC analog circuit and the sound differences can be recognizable.

 

[The Computer Audiophile]

Just now, pkane2001 said:

 

No, what it changes is the PC-generated noise patterns based on this setting. These can leak into the DAC analog circuit and the sound differences can be recognizable.

 

That would be great stuff to measure and have more information about. 

[Qhwoeprktiyns]

7 minutes ago, The Computer Audiophile said:

That would be great stuff to measure and have more information about. 

 

Its been discussed over and over again. Here is a pretty good recap from our friends at ECD, explaining what they think are at play:

 

https://www.diyaudio.com/community/threads/is-jitter-an-issue-with-usb-signals.290973/post-4722474

 

and 

 

https://www.diyaudio.com/community/threads/is-jitter-an-issue-with-usb-signals.290973/post-4724351

 

Quoting a passage on "measurements" (in the context of jitter): "Graphs won’t tell you what audible advantages or disadvantages to expect, because these jitter graphs are based on single or dual test tones (fundamentals). Music is infinitely more complex, it has a fast changing dynamic spectrum that creates the biggest problems in any DAC. The jitter graph won’t cover this because it only shows the DAC jitter response with these test signals only, it doesn’t show what happens when the data content and spectrum are infinitely more complex."

 

Proper measurements would require comparing a DAC's input and output (after conversion back to digital), using something else than a single test tone. Good luck with that !

 

 

 

[firedog]

1 hour ago, botrytis said:

 

NO - a hypothesis is based on previous data, similar type of experiments, etc. It is not intuition; it many times is extrapolation.

Agreed. The scientific use of the word is different than the lay use. A scientific hypothesis is supposed to be a proposed explanation of previously known data, etc. And it needs to be something that can be tested. 

[botrytis]

37 minutes ago, firedog said:

Agreed. The scientific use of the word is different than the lay use. A scientific hypothesis is supposed to be a proposed explanation of previously known data, etc. And it needs to be something that can be tested. 

 

In layman's terms theory = hypothesis. In Science Theory = actual proven. 

[pkane2001]

17 minutes ago, hopkins said:

The jitter graph won’t cover this because it only shows the DAC jitter response with these test signals only, it doesn’t show what happens when the data content and spectrum are infinitely more complex."

 

Proper measurements would require comparing a DAC's input and output (after conversion back to digital), using something else than a single test tone. Good luck with that !

 

Signal-correlated jitter is what changes based on content. This is possible with SPDIF because the clock is carried embedded in the digital signal stream. With USB the clock is not carried by the transmission (it's an on-board DAC clock), and the signal, as such, is carried in micro packets at a fixed frequency, so much less of a possibility of signal-correlated jitter being caused by the transmission or the PC -- which is what we're discussing here.

 

Also, I've been measuring and testing devices using full orchestral music pieces for a good part of the last 3-4 years, so I don't know if I need their "good luck" wishes ;)

 

[Qhwoeprktiyns]

24 minutes ago, pkane2001 said:

Signal-correlated jitter is what changes based on content. This is possible with SPDIF because the clock is carried embedded in the digital signal stream. With USB the clock is not carried by the transmission (it's an on-board DAC clock), and the signal, as such, is carried in micro packets at a fixed frequency, so much less of a possibility of signal-correlated jitter being caused by the transmission or the PC -- which is what we're discussing here.

 

Not really. It's pretty clear from his explanations and measurements that  clock performance can be degraded even with an on-board clock. USB does not change anything in that respect. 

[Qhwoeprktiyns]

23 minutes ago, pkane2001 said:

Also, I've been measuring and testing devices using full orchestral music pieces for a good part of the last 3-4 years, so I don't know if I need their "good luck" wishes ;)

 

I'd be curious to know more of what you are doing. 

[manisandher]

2 hours ago, botrytis said:

 

NO - a hypothesis is based on previous data, similar type of experiments, etc. It is not intuition; it many times is extrapolation.

 

1 hour ago, firedog said:

Agreed. The scientific use of the word is different than the lay use. A scientific hypothesis is supposed to be a proposed explanation of previously known data, etc. And it needs to be something that can be tested. 

 

I can't be bothered to argue with you. Read McGilchrist, or the likes.

[pkane2001]

17 minutes ago, hopkins said:

 

Not really. It's pretty clear from his explanations and measurements that  clock performance can be degraded even with an on-board clock. USB does not change anything in that respect. 

 

The chance of that is small with any decent DAC and can be easily seen using even simple jitter tests. More advanced tests are available using AP analyzers, as an example. Heroic efforts to measure this are not required. This also would not change with a buffer setting in the PC player, like the one that was used in Mani's testing. What USB connection changes is, like I said, transmission and source-related jitter rejection compared to SPDIF.

 

16 minutes ago, hopkins said:

I'd be curious to know more of what you are doing. 

 

https://deltaw.org/

[Qhwoeprktiyns]

2 hours ago, pkane2001 said:

 

The chance of that is small with any decent DAC and can be easily seen using even simple jitter tests. More advanced tests are available using AP analyzers, as an example. Heroic efforts to measure this are not required. This also would not change with a buffer setting in the PC player, like the one that was used in Mani's testing. What USB connection changes is, like I said, transmission and source-related jitter rejection compared to SPDIF.

 

 

https://deltaw.org/

 

Jitter tests, AP analyzers work with test signals, so no chance of measuring anything significant. For the limits of USB, read over the post I linked. 

[musicjunkie917]

3 hours ago, botrytis said:

In Science Theory = actual proven.

 

No! A scientific theory is an explanation that has been tested multiple times and accepted as valid. That is different than "proven". It is not fact.

 

https://www.forbes.com/sites/startswithabang/2017/11/22/scientific-proof-is-a-myth/?sh=7eac76842fb1

[Qhwoeprktiyns]

2 hours ago, pkane2001 said:

 

 

https://deltaw.org/

 

Have you used this to compare an original WAV file with it's recorded playback from a DAC (using ADC) ? If so, what does it tell you? 

[botrytis]

28 minutes ago, musicjunkie917 said:

 

No! A scientific theory is an explanation that has been tested multiple times and accepted as valid. That is different than "proven". It is not fact.

 

https://www.forbes.com/sites/startswithabang/2017/11/22/scientific-proof-is-a-myth/?sh=7eac76842fb1

 

Well, then how do you explain the Theory of Relativity - It has only been proven partly and not when it was proposed? String theory is another, etc.

 

 

[pkane2001]

38 minutes ago, hopkins said:

 

Have you used this to compare an original WAV file with it's recorded playback from a DAC (using ADC) ? If so, what does it tell you? 

 

That's what DeltaWave is designed to do. I and others have done the analysis with at least a few hundred DACs, ADCs, and audio interfaces. The results include time domain and frequency domain analysis, filter comparisons, linearity, jitter, and audibility analysis, and a whole lot more. The tool is designed for you to explore the differences through zoomable graphs and computed statistics, but also to listen to the differences.

 

[Qhwoeprktiyns]

21 minutes ago, pkane2001 said:

 

That's what DeltaWave is designed to do. I and others have done the analysis with at least a few hundred DACs, ADCs, and audio interfaces. The results include time domain and frequency domain analysis, filter comparisons, linearity, jitter, and audibility analysis, and a whole lot more. The tool is designed for you to explore the differences through zoomable graphs and computed statistics, but also to listen to the differences.

 

 

OK, but what does it tell you? How do you account for differences with the original WAV files - is it the DAC or the ADC that are generating differences ? How do you interpret the significance of differences in terms of SQ?

 

If you  listen to the results, it's even worse because then you are back to simply comparing DACs / systems without knowing how accurate any of them are. 

 

There is no '"reference" against which you can evaluate results.

[musicjunkie917]

43 minutes ago, botrytis said:

 

Well, then how do you explain the Theory of Relativity - It has only been proven partly and not when it was proposed? String theory is another, etc.

 

 

 

Neither have been "proven". 

[pkane2001]

21 minutes ago, hopkins said:

 

OK, but what does it tell you? How do you account for differences with the original WAV files - is it the DAC or the ADC that are generating differences ? How do you interpret the significance of differences in terms of SQ?

 

If you  listen to the results, it's even worse because then you are back to simply comparing DACs / systems without knowing how accurate any of them are. 

 

There is no '"reference" against which you can evaluate results.

 

??? The reference is the original file.

 

[Qhwoeprktiyns]

Just now, pkane2001 said:

 

??? The reference is the original file.

 

 

I meant when you listen... 

 

[pkane2001]

16 minutes ago, hopkins said:

 

I meant when you listen... 

 

 

The original file is the reference. What you listen to is the difference between the two.

 

[Qhwoeprktiyns]

13 minutes ago, pkane2001 said:

 

The original file is the reference.

 

 

The original file can only be listened to through a system... 

 

Let's say you play a music file through a DAC and record it using an ADC. Then you listen to the resulting file, and compare it by listening to the original music file. You are listening through a system, and cannot make any assumptions on any subjective differences (or the lack thereof). 

 

It's probably a fun tool to play with, but I don't see what conclusions you can come to in terms of DAC performance and subjective results. Especially not the ones you  made above (no jitter issue with USB, or no jitter issue with an on-board clock).