Expectation Bias

[Iving]

Subjective reality and Objective reality are two equally Great realities.

 

If someone believes they are Julius Caesar, who can argue.

 

If someone measures something reliably, who can argue.
______________

Expectation Bias, Cognitive Dissonance etc are recognised phenomena: experimental procedures detect that a Subject rates the favourability of something differently simply because they have been forced to choose it as a prize in a contest.

 

In audio, the usual argument is that we will like something because we have spent money on it.

 

In both cases, we are supposedly justifying cognitively a choice we have made - in order to mitigate emotional tension - the tension that would accompany, "I could have taken the other prize" or "My money was better spent feeding the kids".
______________

For the most part DBT is not helpful.

 

Use type #1: Objectivists denying Subjective reality will claim A and B cannot be discriminated, but forget about Type 2 error - and there is always a sensitivity doubt remaining after a discrimination failure.

 

Use type #2: DBT can be used to illustrate discrimination - often the result of training - which is a class of sensitivity-enhancement.
______________

DBT could demonstrate - under lab conditions - that an audiophool cannot discriminate A (cheaper amp) from B (expensive amp they just bought) even though they claimed on a Forum that they could (thus, arguably, Expectation Bias); however:

 

a) Test conditions might explain the discrimination failure - the sucker feels foolish in the DBT lab - but goes home and in the privacy of her or his listening room believes, with just the same conviction as before, that the expensive amp is better. They could be right even against untested objective criteria.

 

b) What a person hears is no less "truth" than what a technician measures. If I can create a reliable hedonic perception - nobody can prove it is not real. It may be an illusion by consensus - but it is perceptually real to the listener. Who knows what parameters contribute to that hedonic perception, itself the product of what the brain does. Nobody on the planet understands or explains how brain produces music from sound waves with any degree of completeness.

 

c) Who cares enough anyway to sledgehammer audiophiles with DBTs except Objectivists with an "I am right" agenda?
______________

If you wanted to demonstrate to someone that a given audio claim was explained by Expectation Bias or Cognitive Dissonance, the strongest card you could play would be presentation of a bespoke experimental proposal. You wouldn't have to go so far as to carry it out. You would have a proposition, a proposed method and procedure - and you would be able to argue a logical track from proposal through to interpretation of results (no matter which way they went).

 

Several listeners claiming something about e.g. a particular DAC/amp etc - let's say suspiciously - because of a great review - presents certain kinds of challenges and not others.

 

To demonstrate the delusion of an individual, you'd resort to something a bit more sophisticated than a DBT to drive the point home. Perhaps you'd demonstrate suggestibility in another matter - hoping the gullible individual would put 2 and 2 together. But I doubt you'd ever convert someone. In the end I think that all these stances have to do mostly with human personality. That's how we end up in tribes/echo chambers on the internet. "Birds of a feather flock together" believing that "Great minds think alike". But we also know that "fools seldom differ".

[The Computer Audiophile]

10 minutes ago, pkane2001 said:

 

NOBODY? Talk about a strawman argument... Of course, nobody who isn't interested in learning the truth does DBTs. That's a given.

 

Show me some true DBTs done in the context of consumer audio. A sample of one doesn't count. 

[pkane2001]

5 minutes ago, The Computer Audiophile said:

Show me some true DBTs done in the context of consumer audio. A sample of one doesn't count. 

 

Why not sample of one? But sure, Harman has done a large number of DBTs on speakers and headphones.

 

[The Computer Audiophile]

1 minute ago, pkane2001 said:

 

Why not sample of one? But sure, Harman has done a large number of DBTs on speakers and headphones.

 

Ha! Then advertised it in marketing / PR campaigns. If you think it was text book DBT, and you take Harman's word for it, your bias is showing.

[The Computer Audiophile]

3 minutes ago, pkane2001 said:

Why not sample of one?

A sample of one would never be accepted by you, for any subjective endeavor. It shouldn't be accepted for an objective endeavor. 

[pkane2001]

1 minute ago, The Computer Audiophile said:

A sample of one would never be accepted by you, for any subjective endeavor. It shouldn't be accepted for an objective endeavor. 

 

I've accepted a sample of one in DBTs for many years. As I said above, to me the value of DBTs is in proving things to yourself, not to someone else.

 

[The Computer Audiophile]

6 minutes ago, pkane2001 said:

 

I've accepted a sample of one in DBTs for many years. As I said above, to me the value of DBTs is in proving things to yourself, not to someone else.

 

I'm still searching for those true DBTs and the results. Can't find any. 

[pkane2001]

Just now, The Computer Audiophile said:

I'm still searching for those true DBTs and the results. Can't find any. 

 

Are you looking for my DBT results? I've mentioned them online a number of times over the years, but, as I said, I don't do these to prove something to others. I've done hundreds of DBTs, everything from DACs, to speakers, to digital filters and DSP and room correction curves. 

 

[kumakuma]

15 minutes ago, The Computer Audiophile said:

I'm still searching for those true DBTs and the results. Can't find any. 

 

Perhaps you could share with us your definition of a "true DBT".

 

To me, it simply means a test in which both parties are unaware of which equipment they are listening to during the test.

 

I'm pretty sure that the tests Fred Toole designed for Harman met this criteria.

[The Computer Audiophile]

3 minutes ago, pkane2001 said:

 

Are you looking for my DBT results? I've mentioned them online a number of times over the years, but, as I said, I don't do these to prove something to others. I've done hundreds of DBTs, everything from DACs, to speakers, to digital filters and DSP and room correction curves. 

 

I'm really interested in your methodology for items like DACs and speakers. this seems nearly impossible to conduct without a team of people and a rigorous process. 

[The Computer Audiophile]

2 hours ago, botrytis said:

The point is with a DBT, is you don't know what you are listening to. If you don't know, then HOW can you have expectations?

 

This is s popular belief for those with a more objective mindset. Many of those people, not you specifically, believe all DACs sound the same, all amps sound the same, all bit perfect audio sounds the same. If everything sounds the same, it's pretty easy to see how one's expectations will dictate the outcome. 

 

 

 

8 minutes ago, kumakuma said:

Perhaps you could share with us your definition of a "true DBT".

 

To me, it simply means a test in which both parties are unaware of which equipment they are listening to during the test.

 

I'm pretty sure that the tests Fred Toole designed for Harman met this criteria.

 

I'm no expert in DBTs and I don't think anyone here is. Conducting DBTs is a profession in and of itself. I think a true DBT would mean that nobody in the "room" knows what is being tested. They don't know if it's a codec, amp, DAC, speaker, or even the softness of the chair on which they are sitting, that is being tested. It seems like that would be the only way to do a DBT, but I still think it's a bit crazy to expect usable results in a consumer audio application.

[pkane2001]

1 minute ago, The Computer Audiophile said:

I'm really interested in your methodology for items like DACs and speakers. this seems nearly impossible to conduct without a team of people and a rigorous process. 

 

The main test design considerations for me are to:

 

1. Ensure the same output level, or as close as possible between devices under test (DUTs)

2. Randomize switching between the devices for each trial, so that I don't know which device is in circuit. This also means that I don't know if a device was really changed or not between trials. If done by computer, this is automatically a DBT. If done by a friend or family member, you need to make sure they are not providing any clues to what they are doing when switching devices and use a random number generator to pick which device to switch (or not to switch).

3. Allow for as long as necessary sighted comparison to try to learn the differences between devices before running the test

4. Perform at least 10 (usually 15) trials with random switching, have the software keep track of correct/incorrect results

5. I usually perform fast switching between devices during testing, but sometimes I spend a lot longer after each switch to try to let my "subconscious" detect differences that I'm not catching on the conscious level.

6. If I get a 5% p-value result or better, I assume there is a difference. Anything worse than that, I might rerun the test if I still feel that I can hear the differences. 

 

DACs are fairly easy. ABX or similar methodology can be used to switch between DAC recordings. I've developed my own testing software, but Foobar and a few others do this, mostly automatically. A more complex design would involve switching through a preamp or a line switcher. If this can't be controlled by a computer (say through a remote control software), then a family member or a friend can do this, as long as there's no visual or audible communication possible (or any other tells, like the time it takes to switch) between them and me at the listening position.

 

Speakers are a more complicated device to test, as switching between speakers isn't easy, and hiding speakers requires a little ingenuity, level matching is harder, and speaker positioning can be a problem, especially if these are different size/driver position. I've only done this a few times at a local dealer where the speakers were not visible to me, covered up by fabric, and someone outside of my field of view did the switching based on a random number generator. This was a preference test, it wasn't to detect if the two sets of speakers sounded different. Harman has done extensive speaker testing and built an automated facility for doing so. Of course, I don't have the same resources available to me 🤨

 

 

[The Computer Audiophile]

5 minutes ago, pkane2001 said:

 

The main test design considerations for me are to:

 

1. Ensure the same output level, or as close as possible between devices under test (DUTs)

2. Randomize switching between the devices for each trial, so that I don't know which device is in circuit. This also means that I don't know if a device was really changed or not between trials. If done by computer, this is automatically a DBT. If done by a friend or family member, you need to make sure they are not providing any clues to what they are doing when switching devices and use a random number generator to pick which device to switch (or not to switch).

3. Allow for as long as necessary sighted comparison to try to learn the differences between devices before running the test

4. Perform at least 10 (usually 15) trials with random switching, have the software keep track of correct/incorrect results

5. I usually perform fast switching between devices during testing, but sometimes I spend a lot longer after each switch to try to let my "subconscious" detect differences that I'm not catching on the conscious level.

6. If I get a 5% p-value result or better, I assume there is a difference. Anything worse than that, I might rerun the test if I still feel that I can hear the differences. 

 

DACs are fairly easy. ABX or similar methodology can be used to switch between DAC recordings. I've developed my own testing software, but Foobar and a few others do this, mostly automatically. A more complex design would involve switching through a preamp or a line switcher. If this can't be controlled by a computer (say through a remote control software), then a family member or a friend can do this, as long as there's no visual or audible communication possible (or any other tells, like the time it takes to switch) between them and me at the listening position.

 

Speakers are a more complicated device to test, as switching between speakers isn't easy, and hiding speakers requires a little ingenuity, level matching is harder, and speaker positioning can be a problem, especially if these are different size/driver position. I've only done this a few times at a local dealer where the speakers were not visible to me, covered up by fabric, and someone outside of my field of view did the switching based on a random number generator. This was a preference test, it wasn't to detect if the two sets of speakers sounded different. Harman has done extensive speaker testing and built an automated facility for doing so. Of course, I don't have the same resources available to me 🤨

 

 

 

How do you control for bias? If you run 100 tests of DACs and they all measure "perfect" or at least without any issues that you believe can be heard, testing the 101st DAC will be impossible to do without having a bias that they are all the same and without sonic differences. If you expect no difference, you likely won't hear a difference. 

[pkane2001]

3 minutes ago, The Computer Audiophile said:

 

How do you control for bias? If you run 100 tests of DACs and they all measure "perfect" or at least without any issues that you believe can be heard, testing the 101st DAC will be impossible to do without having a bias that they are all the same and without sonic differences. If you expect no difference, you likely won't hear a difference. 

 

Random selection controls for bias. If I can't hear a difference between DACs in a sighted a test, I don't do a blind test, to me there's no point. If I do hear differences in a sighted test, then I validate my findings using a blind test. In most cases, I find there are no audible differences, but I have found some DACs (including in some online tests) that I can detect in a blind test, so I guess I'm not all that biased against DACs ;)

 

[The Computer Audiophile]

11 minutes ago, pkane2001 said:

 

Random selection controls for bias. If I can't hear a difference between DACs in a sighted a test, I don't do a blind test, to me there's no point. If I do hear differences in a sighted test, then I validate my findings using a blind test. In most cases, I find there are no audible differences, but I have found some DACs (including in some online tests) that I can detect in a blind test, so I guess I'm not all that biased against DACs ;)

 

 

Thanks for the follow up. I still don't see how random selection can control for bias. If one's expectation bias, which isn't a conscious process, tells him there can't be a difference, he won't hear a difference.

 

It really comes down to trusting oneself to do the test without bias, but as you said ...

 

On 9/20/2022 at 6:25 AM, pkane2001 said:

Expectation bias is not a conscious process. 

 

 

[pkane2001]

11 minutes ago, The Computer Audiophile said:

 

Thanks for the follow up. I still don't see how random selection can control for bias. If one's expectation bias, which isn't a conscious process, tells him there can't be a difference, he won't hear a difference.

 

It really comes down to trusting oneself to do the test without bias, but as you said ...

 

If you hear an obvious difference in a sighted test that disappears once you don't know the identity of the device under test, then you should seriously question your initial results.

 

[The Computer Audiophile]

25 minutes ago, pkane2001 said:

If I can't hear a difference between DACs in a sighted a test, I don't do a blind test, to me there's no point.

Help me with the logic here. Isn't this the same logical error, but in reverse, of accepting that you do here are difference while sighted? It's sighted, so that effects you and your bias is not conscious. 

 

I can see this being an issue with Mk1 and Mk2 of the same DAC. Same external look, and only marketing claims from the manufacture that there is a difference. It isn't possible to remove one's bias.

 

However, I totally understand that you aren't conducting DBTs for a research facility and it's your own methodology to satisfy you. Completely understandable. But, it all seems very subjective and selective. 

 

[pkane2001]

20 minutes ago, The Computer Audiophile said:

Help me with the logic here. Isn't this the same logical error, but in reverse, of accepting that you do here are difference while sighted? It's sighted, so that effects you and your bias is not conscious. 

 

I can see this being an issue with Mk1 and Mk2 of the same DAC. Same external look, and only marketing claims from the manufacture that there is a difference. It isn't possible to remove one's bias.

 

However, I totally understand that you aren't conducting DBTs for a research facility and it's your own methodology to satisfy you. Completely understandable. But, it all seems very subjective and selective. 

 

 

You're still stuck on me trying to prove something to others. I'm only interested in things that make an audible difference to me. If I can't hear it sighted, I don't care about testing it further. 

 

[The Computer Audiophile]

19 minutes ago, pkane2001 said:

 

You're still stuck on me trying to prove something to others. I'm only interested in things that make an audible difference to me. If I can't hear it sighted, I don't care about testing it further. 

 

To a certain extent I hoped your information could be helpful to others as well as me. 
 

I think DBTs should be considered the bronze standard. Pretty good, but far from perfect. Touting them as evidence of anything in consumer audio is a bit preposterous. 

[pkane2001]

5 minutes ago, The Computer Audiophile said:

To a certain extent I hoped your information could be helpful to others as well as me. 
 

I think DBTs should be considered the bronze standard. Pretty good, but far from perfect. Touting them as evidence of anything in consumer audio is a bit preposterous. 

 

Not quite. Properly done DBTs are the gold standard for bias-controlled testing and is the main thing that is used as evidence of anything related to perception. In audio or any other field where perception is involved, "sighted" testing just doesn't even rise to the bronze level. Maybe wood :) The fact that most audiophile companies don't do DBTs doesn't change their importance or value, in fact, maybe it points to where the industry has room to improve.

 

Audio enthusiast-done DBTs are not going to rise to the level of a scientific study. That's not surprising or unexpected. Once again, the value here is in proving things to yourself rather than to someone else.

[The Computer Audiophile]

11 minutes ago, pkane2001 said:

 

Not quite. Properly done DBTs are the gold standard for bias-controlled testing and is the main thing that is used as evidence of anything related to perception. In audio or any other field where perception is involved, "sighted" testing just doesn't even rise to the bronze level. Maybe wood :) The fact that most audiophile companies don't do DBTs doesn't change their importance or value, in fact, maybe it points to where the industry has room to improve.

 

Audio enthusiast-done DBTs are not going to rise to the level of a scientific study. That's not surprising or unexpected. Once again, the value here is in proving things to yourself rather than to someone else.

Your enthusiasm for DBTs is very evident and honest. I certainly can’t fault you for that. I just disagree with you about their usefulness in consumer audio. I get hung up on, what seems to me, is a lack of controlling for all bias. Perhaps it’s irrational to expect all bias to be controlled. 

[pkane2001]

5 minutes ago, The Computer Audiophile said:

Your enthusiasm for DBTs is very evident and honest. I certainly can’t fault you for that. I just disagree with you about their usefulness in consumer audio. I get hung up on, what seems to me, is a lack of controlling for all bias. Perhaps it’s irrational to expect all bias to be controlled. 

 

I see a huge benefit to consumers to have manufacturers produce equipment that actually delivers what they claim it does.

 

The type of testing needed for this exists and is well documented in perception studies. There is no other scientifically valid way to evaluate perception accuracy without using DBTs. Sighted testing is notoriously bad (provably so) and the results cannot be trusted except in the cases where the differences are very large and obvious, such as background noise or very large levels of distortion.

 

When doing large scale perception studies, the protocol should be designed to eliminate all possible sources of bias.  There are standard testing protocols that control for example, for negative bias "nocebo" (the one you seem to be concerned about) that are well documented in audio testing. For example, a standard protocol called MUSHRA incorporates controls for both types of bias and even includes pre-screening of test subjects to eliminate those with hearing deficiencies or other disabilities that might have an impact the result of the test.

 

Again, I don't expect anyone individual to do this at home. But that doesn't make the test any less valuable to the consumer.

 

 

[musicjunkie917]

20 minutes ago, pkane2001 said:

I see a huge benefit to consumers to have [audio] manufacturers produce equipment that actually delivers what they claim it does.

 

This is hilarious and massively naive!

[pkane2001]

7 minutes ago, musicjunkie917 said:

This is hilarious and massively naive!

 

Is it naive to expect some validity to the claims made about often very expensive pieces of equipment?

 

[kumakuma]

2 hours ago, The Computer Audiophile said:

 

Thanks for the follow up. I still don't see how random selection can control for bias. If one's expectation bias, which isn't a conscious process, tells him there can't be a difference, he won't hear a difference.

 

My feeling is that any difference so slight that it disappears due to expectation bias isn't moving the dial much from an SQ standpoint.