Expectation Bias

[pkane2001]

53 minutes ago, botrytis said:

We have to live in reality, or we can join Frank.

[pkane2001]

24 minutes ago, pkane2001 said:

 

OMG! @botrytis, how can you disagree with the Matrix??? What's the world coming to...  😎

[pkane2001]

29 minutes ago, The Computer Audiophile said:

If this is the DBT you want to talk about, then we need to understand that it has nothing to do with anything ever conducted in audio. You guys talk about DBT as if it's the gold standard in audio and as if it's been used and works great. NOBODY does this in audio. NOBODY does this outside of a scientific study.

 

You guys are using a scientific straw man. 

 

NOBODY? Talk about a strawman argument... Of course, nobody who isn't interested in learning the truth does DBTs. That's a given.

 

[pkane2001]

5 minutes ago, The Computer Audiophile said:

Show me some true DBTs done in the context of consumer audio. A sample of one doesn't count. 

 

Why not sample of one? But sure, Harman has done a large number of DBTs on speakers and headphones.

 

[pkane2001]

1 minute ago, The Computer Audiophile said:

A sample of one would never be accepted by you, for any subjective endeavor. It shouldn't be accepted for an objective endeavor. 

 

I've accepted a sample of one in DBTs for many years. As I said above, to me the value of DBTs is in proving things to yourself, not to someone else.

 

[pkane2001]

Just now, The Computer Audiophile said:

I'm still searching for those true DBTs and the results. Can't find any. 

 

Are you looking for my DBT results? I've mentioned them online a number of times over the years, but, as I said, I don't do these to prove something to others. I've done hundreds of DBTs, everything from DACs, to speakers, to digital filters and DSP and room correction curves. 

 

[pkane2001]

1 minute ago, The Computer Audiophile said:

I'm really interested in your methodology for items like DACs and speakers. this seems nearly impossible to conduct without a team of people and a rigorous process. 

 

The main test design considerations for me are to:

 

1. Ensure the same output level, or as close as possible between devices under test (DUTs)

2. Randomize switching between the devices for each trial, so that I don't know which device is in circuit. This also means that I don't know if a device was really changed or not between trials. If done by computer, this is automatically a DBT. If done by a friend or family member, you need to make sure they are not providing any clues to what they are doing when switching devices and use a random number generator to pick which device to switch (or not to switch).

3. Allow for as long as necessary sighted comparison to try to learn the differences between devices before running the test

4. Perform at least 10 (usually 15) trials with random switching, have the software keep track of correct/incorrect results

5. I usually perform fast switching between devices during testing, but sometimes I spend a lot longer after each switch to try to let my "subconscious" detect differences that I'm not catching on the conscious level.

6. If I get a 5% p-value result or better, I assume there is a difference. Anything worse than that, I might rerun the test if I still feel that I can hear the differences. 

 

DACs are fairly easy. ABX or similar methodology can be used to switch between DAC recordings. I've developed my own testing software, but Foobar and a few others do this, mostly automatically. A more complex design would involve switching through a preamp or a line switcher. If this can't be controlled by a computer (say through a remote control software), then a family member or a friend can do this, as long as there's no visual or audible communication possible (or any other tells, like the time it takes to switch) between them and me at the listening position.

 

Speakers are a more complicated device to test, as switching between speakers isn't easy, and hiding speakers requires a little ingenuity, level matching is harder, and speaker positioning can be a problem, especially if these are different size/driver position. I've only done this a few times at a local dealer where the speakers were not visible to me, covered up by fabric, and someone outside of my field of view did the switching based on a random number generator. This was a preference test, it wasn't to detect if the two sets of speakers sounded different. Harman has done extensive speaker testing and built an automated facility for doing so. Of course, I don't have the same resources available to me 🤨

 

 

[pkane2001]

3 minutes ago, The Computer Audiophile said:

 

How do you control for bias? If you run 100 tests of DACs and they all measure "perfect" or at least without any issues that you believe can be heard, testing the 101st DAC will be impossible to do without having a bias that they are all the same and without sonic differences. If you expect no difference, you likely won't hear a difference. 

 

Random selection controls for bias. If I can't hear a difference between DACs in a sighted a test, I don't do a blind test, to me there's no point. If I do hear differences in a sighted test, then I validate my findings using a blind test. In most cases, I find there are no audible differences, but I have found some DACs (including in some online tests) that I can detect in a blind test, so I guess I'm not all that biased against DACs ;)

 

[pkane2001]

11 minutes ago, The Computer Audiophile said:

 

Thanks for the follow up. I still don't see how random selection can control for bias. If one's expectation bias, which isn't a conscious process, tells him there can't be a difference, he won't hear a difference.

 

It really comes down to trusting oneself to do the test without bias, but as you said ...

 

If you hear an obvious difference in a sighted test that disappears once you don't know the identity of the device under test, then you should seriously question your initial results.

 

[pkane2001]

20 minutes ago, The Computer Audiophile said:

Help me with the logic here. Isn't this the same logical error, but in reverse, of accepting that you do here are difference while sighted? It's sighted, so that effects you and your bias is not conscious. 

 

I can see this being an issue with Mk1 and Mk2 of the same DAC. Same external look, and only marketing claims from the manufacture that there is a difference. It isn't possible to remove one's bias.

 

However, I totally understand that you aren't conducting DBTs for a research facility and it's your own methodology to satisfy you. Completely understandable. But, it all seems very subjective and selective. 

 

 

You're still stuck on me trying to prove something to others. I'm only interested in things that make an audible difference to me. If I can't hear it sighted, I don't care about testing it further. 

 

[pkane2001]

5 minutes ago, The Computer Audiophile said:

To a certain extent I hoped your information could be helpful to others as well as me. 
 

I think DBTs should be considered the bronze standard. Pretty good, but far from perfect. Touting them as evidence of anything in consumer audio is a bit preposterous. 

 

Not quite. Properly done DBTs are the gold standard for bias-controlled testing and is the main thing that is used as evidence of anything related to perception. In audio or any other field where perception is involved, "sighted" testing just doesn't even rise to the bronze level. Maybe wood :) The fact that most audiophile companies don't do DBTs doesn't change their importance or value, in fact, maybe it points to where the industry has room to improve.

 

Audio enthusiast-done DBTs are not going to rise to the level of a scientific study. That's not surprising or unexpected. Once again, the value here is in proving things to yourself rather than to someone else.

[pkane2001]

5 minutes ago, The Computer Audiophile said:

Your enthusiasm for DBTs is very evident and honest. I certainly can’t fault you for that. I just disagree with you about their usefulness in consumer audio. I get hung up on, what seems to me, is a lack of controlling for all bias. Perhaps it’s irrational to expect all bias to be controlled. 

 

I see a huge benefit to consumers to have manufacturers produce equipment that actually delivers what they claim it does.

 

The type of testing needed for this exists and is well documented in perception studies. There is no other scientifically valid way to evaluate perception accuracy without using DBTs. Sighted testing is notoriously bad (provably so) and the results cannot be trusted except in the cases where the differences are very large and obvious, such as background noise or very large levels of distortion.

 

When doing large scale perception studies, the protocol should be designed to eliminate all possible sources of bias.  There are standard testing protocols that control for example, for negative bias "nocebo" (the one you seem to be concerned about) that are well documented in audio testing. For example, a standard protocol called MUSHRA incorporates controls for both types of bias and even includes pre-screening of test subjects to eliminate those with hearing deficiencies or other disabilities that might have an impact the result of the test.

 

Again, I don't expect anyone individual to do this at home. But that doesn't make the test any less valuable to the consumer.

 

 

[pkane2001]

7 minutes ago, musicjunkie917 said:

This is hilarious and massively naive!

 

Is it naive to expect some validity to the claims made about often very expensive pieces of equipment?

 

[pkane2001]

1 minute ago, manisandher said:

 

Yes, but when they give 'inconvenient' results, they're ignored.

 

Remember the red/blue pill thread? As a reminder:

 

1. I was convinced that I could hear differences between bit-identical replay (two different buffer settings in the playback software)

2. I was confident that I could demonstrate what I was hearing

3. I set up a blind test (essentially double blind) - Mans controlled the replay from my home study, two doors and a corridor away from the listening room, where I sat

 

We did 3 tests in total - the first two were non-ABX, and the third ABX. Why were the first two non-ABX? Because I was so confident that I'd be able to correctly identify A and B, that I wouldn't need an AB reference before the X. I was totally wrong.

 

We captured the digital input into the DAC in real time during the test. All 54 samples (about 15 seconds each in length) were shown to be bit-identical.

 

In any event, here are the results:

 

 

The first non-ABX was a disaster - I'd never done any listening tests before. The second was better - I was getting used to the test and how to differentiate between A and B. The ABX was stellar - 1% probability of guessing (way beyond the generally accepted 5% threshold).

 

I think any reasonable person would conclude that there's definitely something going on here. Or at the very least, that there's something worth exploring further. But all the hardcore objectivists could come up with was "the ABX result must have been a fluke". Yeah, right.

 

I set up a (double) blind listening test having never been involved in one before. I agreed that the results would be published on this site for everyone to see, irrespective of the outcome. I essentially put my neck on the line. I achieved 9/10 in the ABX (1% probability by guessing alone), and was told that it must have been a fluke.

 

And the objectivists wonder why audiophiles are reluctant to undertake DBTs to demonstrate that they really do hear the things they say they do???

 

Mani.

 

Mani, we've had this discussion repeated many times over the years. You didn't have a protocol designed or agreed to. Between you and Mans, you didn't work out the details of what was being tested, why, or how or what result would be acceptable as "proof". These are rookie mistakes, and I understand how it all happened and don't blame you or Mans -- you're not professional scientists doing perception studies.

 

But you, like some others in this thread, seem to happily ignore my actual statements. So, I'll repeat (and this applies to the test you conducted by you and Mans):

 

Quote

Audio enthusiast-done DBTs are not going to rise to the level of a scientific study. That's not surprising or unexpected. Once again, the value here is in proving things to yourself rather than to someone else.

 

[pkane2001]

2 minutes ago, musicjunkie917 said:

What are these claims you think they are making? Are these claims invalidated by measurements? Otherwise, it is all subjective.

 

But of course! Personally I do both, measurements and DBTs. This allows me to correlate my preferences and audibility thresholds with device measurements. 

 

[pkane2001]

4 minutes ago, musicjunkie917 said:

Except the things that we measure have very little correlation to sound quality. Again, you are being naive.....

 

Curious how you've determined this. Please present your findings.

 

[pkane2001]

5 minutes ago, The Computer Audiophile said:

@pkane2001 says a sample size of one is good 🙂

 

It's a perfect sample size for the audience of one.

 

[pkane2001]

9 minutes ago, manisandher said:

You know, what really strikes me is the total lack of curiosity. I really expected people to be interested in exploring the results further, but they just weren't.

 

To my mind, there's nothing particulary strange going on here. We were feeding an spdif signal into a crappy DAC. It seems perfectly reasonable to me that changing the buffer setting in the software player might be changing the noise profile reaching the DAC down the spdif cable. It might have been interesting to explore this hypothesis further, but we never even got to the hypothesis stage... it was all just a fluke.

 

The whole of science relies on intuition and curiousity, which seems to be seriously lacking in the objectivist audio community.

 

Mani.

It really was a poorly designed test, Mani. It was not clear until weeks after what was being tested, and even then, the buffer setting was not documented or well defined by PeterSt as to what it controlled or what effect it had on the digital signal. As you may recall, I actually spent quite a bit of time on the analysis at that time, but there were problems with recordings and different versions were generated after the test was completed.

[pkane2001]

7 minutes ago, musicjunkie917 said:

 

What measurements do manufacturers provide that readily correspond to sound quality??? Do you think that an amp that measures at .0001 THD automatically sounds better than one that measures with .01 THD?

I actually know this answer for various distortion levels and signals for myself, because I spent the time testing and studying this. Do you?

[pkane2001]

2 minutes ago, manisandher said:

 

Paul, you're conflating two things here.

 

The first is the listening test. All we were testing for was whether two bit-identical playback means could sound audibly different. The mechanism at play was irrelevant. It could have been anything, as long as we could show that things remained bit-identical at all times, which we did.

 

The second is the mechanism at play. Here I agree with you. It was difficult to figure out exactly what was happening. But that's exactly why it would have been interesting to explore things further. I even stated that I would have been prepared to repeat the test, but this time with a clear hypothesis. And I would have done this... if people had been more curious about the result of the initial test, and not dismissed it out of hand as a fluke.

 

Mani.

The test hypothesis going in to the test was that USB bit-identical signals will be indistinguishable. The test was for something completely different, where transmission jitter could have easily been the issue. Which is why a proper protocol design is important.
 

I’m personally not very curious about audibility of jitter. I know at what levels it becomes audible for me. But since jitter was not measured in your test, there is no way to confirm or deny this after the test.

[pkane2001]

29 minutes ago, musicjunkie917 said:

Of course I know the answer. The answer is that you cannot tell how good an amp sounds based on THD. There are amps that measure great and sound terrible and there are amps that measure not nearly as good that sound great. Humans can't hear distortion much below 1%. So there is no way you can hear the difference between .0001, .001, and .01 THD.

 

How did you come by this answer, was my question. You make some pretty strong claims, but I doubt you understand them. THD is a very simple measure that is quick to look at but not enough to judge anything about the "sound of an amp", if there is any such thing.

 

[pkane2001]

32 minutes ago, manisandher said:

 

No.

 

I had listed a number of things that we could try, including different USB cables. Mans wanted to capture the digital input into the DAC in real time during the test (correctly, in my opinion). I only had the means to do this with spdif, so we went with software buffer settings instead. This was agreed upon a couple of days before the test.

 

Today, I'd be able to do it with USB easily.

 

 

Agreed. But never followed up on because the results of the listening test were not accepted.

 

Certainly all the discussion on the forums prior to your tests was all about USB. The test design would, by necessity, be different for SPDIF. Neither you nor Mans were thinking about how to test for this properly. A digital recording of SPDIF signal wouldn't reveal jitter and so wouldn't be very useful if that was a likely source of error.

 

As to you doing it with USB easily, I doubt it. There can be some pathological cases where this might happen due to various tells (clicks, delays on start up, initial distortion, actual bit losses in transmission, etc.) and possibly due to poor USB interface design causing ground loops carrying PC-generated noise into the DAC. But otherwise, I really don't think so.

 

That your test results were accepted by others or not isn't what's important. You trying to figure out what you heard and why is much more interesting. Assuming you are curious, you'd try to find the answer by doing more tests, coming up with hypothesis on what caused the difference, and then testing for it. I know I would.

[pkane2001]

5 hours ago, manisandher said:

 

Again, you're conflating two things. The listening test was only to determine if differences in bit-identical playback were audible. The digital recording of the spdif signal was necessary to show that the differences (in this case buffer settings) remained bit-identical at all times.

 

 

No, I meant I can easily capture the digital input into a USB DAC in real time, to show that things remain bit-identical (because I have DACs with USB inputs and spdif outputs). So, repeating the test using USB instead of spdif would now be possible.

 

I have a bunch of USB cables here, and am confident I hear differences between them. However, I would say that there is zero correlation between price and SQ. One of my favourite-sounding ones is a 1.5m USB-certified cable that I paid a few pounds for many years ago. The most expensive I have here I paid a stupid amount for (in my more gullible days), and sounds terrible... IMO.

 

Now, how's that for expectation bias?

 

 

I have a number of modern, well-measuring DACs here (Okto dac8 PRO, SMSL DO200, RME ADI-2 Pro, MOTU UltraLite-mk5). I'm certain I hear differences between USB cables with them all.

 

 

Yep.

 

I haven't stopped thinking about these things over these last few years. I've come up with a method that I'm hoping will show differences at the analogue output of the DAC with bit-identical changes upstream. But it's going to take a lot of time and effort to do, which I just haven't been able to find to date.

 

If the method works and differences are indeed detectable, I'll then like to start exploring possible mechanisms at play, and perhaps setting up more listening tests.

 

I'm not a manufacturer, a dealer or a reviewer. I have nothing to gain from this endeavour... other than to satisfy my curiosity 😉.

 

Well, I guess I misunderstood the purpose of the test, then, when we first discussed it. SPDIF and USB handle clocks very differently and the amount of transmission (or source) jitter in SPDIF can have a large effect on the DAC analog output, depending on the clock recovery circuit. USB isochronous protocol is mostly immune to these issues. What's more, the setting that was varied in the test, at least as described by PeterSt, appears to be a very likely source of jitter. Based on this, I'd be measuring jitter performance of the system, first, before trying to look for any other causes. Ground loop(s) would be my next check. 

 

Oh, and I'm not a manufacturer or dealer, either. All I do is out of curiosity ;)

[pkane2001]

4 minutes ago, The Computer Audiophile said:

Do you think the setting could change ground loops or change how a system responds to ground loops? 

 

No, what it changes is the PC-generated noise patterns based on this setting. These can leak into the DAC analog circuit and the sound differences can be recognizable.

 

[pkane2001]

17 minutes ago, hopkins said:

The jitter graph won’t cover this because it only shows the DAC jitter response with these test signals only, it doesn’t show what happens when the data content and spectrum are infinitely more complex."

 

Proper measurements would require comparing a DAC's input and output (after conversion back to digital), using something else than a single test tone. Good luck with that !

 

Signal-correlated jitter is what changes based on content. This is possible with SPDIF because the clock is carried embedded in the digital signal stream. With USB the clock is not carried by the transmission (it's an on-board DAC clock), and the signal, as such, is carried in micro packets at a fixed frequency, so much less of a possibility of signal-correlated jitter being caused by the transmission or the PC -- which is what we're discussing here.

 

Also, I've been measuring and testing devices using full orchestral music pieces for a good part of the last 3-4 years, so I don't know if I need their "good luck" wishes ;)