Does BIAS affect audio test results?

[pkane2001]

An article by Brent Butterworth (SoundStage!) about blind testing and a headphone/DAC blind test he conducted with attendees at 2017 RMAF:

 

https://www.soundstagexperience.com/index.php/features-menu/pulse-menu/834-the-problem-with-blind-testing

 

The quote from Paul Barton (PSB fame) was interesting to me, although the whole article is worth reading. Brent writes:

 

Quote

One comment, from PSB Speakers founder and chief engineer Paul Barton, especially stuck with me. As best I can recall, he said, “Once you go to blind testing, where the listeners can’t see the identity of the products, everything changes,” and he punctuated it with a wave of both arms.

 

That is so true in my experience. Once you try to hear the obvious, night-and-day differences in a test where you don't know the identity of the device, everything changes. It's easy to tell what you like when you see it and can identify it with your eyes. Try using ears only, without sighted bias, and things change dramatically.

 

To reveal any possible personal bias , I do have PSB Stratus Gold speakers in my second system. These were purchased new in the mid 90's.

[The Computer Audiophile]

9 minutes ago, pkane2001 said:

I do have PSB Stratus Gold speakers in my second system.

Of course you ran blind listening tests versus the Stratus Silver and Bronze before purchasing :~)

[pkane2001]

1 minute ago, The Computer Audiophile said:

Of course you ran blind listening tests versus the Stratus Silver and Bronze before purchasing :~)

 

I did. Not against those specific ones, but my local dealer helped us set up a blind test between the few models that I and my audiophile buddy were evaluating. 

[The Computer Audiophile]

This may not be within the scope of your thread, but it would be good to read about what all this means, what's it for, or the "so now what" bottom line type of thing. 

[pkane2001]

12 minutes ago, The Computer Audiophile said:

This may not be within the scope of your thread, but it would be good to read about what all this means, what's it for, or the "so now what" bottom line type of thing. 

 

Probably should be another thread...

 

The goal of this thread was to provide some references, experiments and material to expose and document various biases, both in listeners and in experimenters. If you read any of the studies I shared so far, you'll find that sighted listening is fraught with errors. Listener preferences are severely affected by sighted bias, even those who are aware of it, even those who are trained listeners, even those with a lot of experience. The effect of bias is often greater than some very significant errors and distortions in the playback system. That's why any objective study of listener preferences must be done blind, and this is borne out by the cited papers and experiments.

 

There are additional biases that must be addressed in any scientific study. These include equipment and material selection, proper set up, training, proper scaling of preferences, proper statistical analysis, sufficient number of tests, and controlling for other variables. This is not magic. This is part of a proper scientific study procedure and you can see it documented in the studies I cited. That was another reason for this thread.

 

My conclusion as it applies to our hobby? If you want to know what really sounds better, close your eyes (or hide the identity of the device) and try to really hear the differences. It takes some courage to try this. Once you do, it also takes courage to admit that you may not be able to hear the night-and-day differences you hear sighted because they are either very tiny, or not even there. That's where I fully agree with the quote from Paul Barton: “Once you go to blind testing, where the listeners can’t see the identity of the products, everything changes.” 

 

[The Computer Audiophile]

1 hour ago, pkane2001 said:

Probably should be another thread...

Understood.

[pkane2001]

This presentation is a little different, as it doesn't apply specifically to audio. But, since hearing is one of the senses, this should have some relevance (pdf from a course at Washington State University)

 

Experimental Design and Sensory Analysis

 

In addition to describing discrimination and affective (preference) tests, this also covers an "attribute test". This one I see less often in audio space, perhaps because the attributes of audio quality are not as well defined. The idea of an attribute test is to let the panel rank each DUT by some number of attributes, rather than a simple better/worse, or different/same. It's still a blind test, since in most cases the tester would want to eliminate the influence of other senses in the result.

 

An attribute test should probably follow a discrimination test, since if the subjects can't tell the difference between DUTs, any attributes they rank for each one are probably not valid. But the real question in my mind, what are the attributes that would be useful to rank in an audio attribute test? Here are some ideas, just based on terms frequently thrown around in audiophile circles. Some of these need to be defined, first, IMHO, and yet I think this would make for an interesting test. I'm listing them in no particular order, and mixing well-defined attributes with those that may be completely undefined:

• Treble response (brightness)
• Bass response
• Midrange, vocals
• Full sound
• Impact/Slam/Dynamics
• Microdynamics
• Upfront or laid back presentation
• Depth of sound stage
• Width of sound stage
• Sound source separation
• Air
• Transparent/Veiled
• Ambiance
• Realism
• ???? Any others ????

Example from the presentation showing various attributes (in a taste test):

 

 

 

[fas42]

10 hours ago, pkane2001 said:

An attribute test should probably follow a discrimination test, since if the subjects can't tell the difference between DUTs, any attributes they rank for each one are probably not valid. But the real question in my mind, what are the attributes that would be useful to rank in an audio attribute test? Here are some ideas, just based on terms frequently thrown around in audiophile circles. Some of these need to be defined, first, IMHO, and yet I think this would make for an interesting test. I'm listing them in no particular order, and mixing well-defined attributes with those that may be completely undefined:

• Treble response (brightness)
• Bass response
• Midrange, vocals
• Full sound
• Impact/Slam/Dynamics
• Microdynamics
• Upfront or laid back presentation
• Depth of sound stage
• Width of sound stage
• Sound source separation
• Air
• Transparent/Veiled
• Ambiance
• Realism
• ???? Any others ????

 

 

 

 

 

 

Nice one, Paul ... 🙂

 

I would rank the value of them for assessment, in this order:

 

1. Realism
2. Full sound
3. Impact/Slam/Dynamics
4. Transparent/Veiled
5. Sound source separation
6. Air
7. Ambiance
8. Depth of sound stage
9. Width of sound stage
10. Microdynamics
11. Upfront or laid back presentation
12. Treble response (brightness)
13. Midrange, vocals
14. Bass response

[pkane2001]

2 hours ago, fas42 said:

 

Nice one, Paul ... 🙂

 

I would rank the value of them for assessment, in this order:

 

1. Realism
2. Full sound
3. Impact/Slam/Dynamics
4. Transparent/Veiled
5. Sound source separation
6. Air
7. Ambiance
8. Depth of sound stage
9. Width of sound stage
10. Microdynamics
11. Upfront or laid back presentation
12. Treble response (brightness)
13. Midrange, vocals
14. Bass response

 

That's true, Frank. While an attribute study assigns a scale, say 1-10 for each one of these attributes, it must be that the attributes themselves have a relative scale of importance, similar to your list.

 

Interesting that your last item, bass response, appears at or near the top in a number of formal studies. Realism is top for most audiophiles, I would assume. Although that's probably a composite consisting of a number of others attributes below it. For a proper evaluation/study, it would be important to describe and document what each attributes really means. Maybe that would allow this list to be reduced to a smaller set of attributes. 

 

[fas42]

I ranked bass last, because nothing I have come across has flagged that getting spectacular bass response flatness has anything to do with the item at the other end, realism. Subjectively intense, powerful bass lines automatically emerge when the other attributes in the above list are at a high level, IME - a system can sound like "it has no bass!"; or knock one over with the gutsiness of what's happening in the bottom end - but the FR characteristics do not change one iota between those two states ...

[pkane2001]

On 6/4/2020 at 12:04 AM, fas42 said:

I ranked bass last, because nothing I have come across has flagged that getting spectacular bass response flatness has anything to do with the item at the other end, realism. Subjectively intense, powerful bass lines automatically emerge when the other attributes in the above list are at a high level, IME - a system can sound like "it has no bass!"; or knock one over with the gutsiness of what's happening in the bottom end - but the FR characteristics do not change one iota between those two states ...

 

Anecdotally, from studies (including those by Floyd Toole), and in my own, personal testing, lack of bass leads to a less satisfying and  less realistic presentation. Perhaps it's not true bass, but more the mid-bass that's important. For example, a recording of a piano sounds unrealistic to me if I don't hear the resonance of the lower frequencies produced by a concert grand. A good piano is like a sports car engine: if it sounds thin, it isn't satisfying  Your second most important attribute is 'full sound', which to me, does seem to require a sufficiently competent lower-frequency presentation.

 

I think it's important to consider the relative scale that each of us assigns to various attributes, and I'm certain the scale is not the same for everyone. For me the tonal balance is more important than soundstage depth/width/separation, for example. As a long-time audiophile, I do enjoy 'spatial fireworks' in audio reproduction, but these are more often a curiosity, a nice-to-have. I notice them when they are present, but don't miss them that much when listening to good music. Proper tonal balance is something I simply couldn't live without, on the other hand. 

[pkane2001]

A follow-up to attribute testing and quality/preference scaling in audio reproduction, here's a recent study comparing quality of a recording and its like-ability:

 

http://www.aes.org/e-lib/browse.cfm?elib=18102

 

Here's a table summarizing some attributes that were used by study participants to associate with the perceived 'quality' of a recording (Distorted is mostly associated with the lowest quality rating of 1, for example, while Punchy is associated with quality near the top, 4):

 

 

Another interesting result from that same study shows correlation between various objectively measured sound features and their associated subjective rating broken-down between Quality and Like:

 

 

The arrows next to the R² values indicate positive correlation (up-arrow) or negative correlation (down-arrow).

 

[pkane2001]

And a follow-up to the follow-up on audio quality perception and related attributes.

 

Wilson, Alex & Fazenda, Bruno. (2013). Perception & Evaluation of Audio Quality in Music Production

https://www.researchgate.net/publication/286018981_Perception_Evaluation_of_Audio_Quality_in_Music_Production

 

This explains some of the objective terms used in the previous study and highlights in more detail the relationship between the objective and subjective recording quality evaluations.

[fas42]

11 hours ago, pkane2001 said:

 

Anecdotally, from studies (including those by Floyd Toole), and in my own, personal testing, lack of bass leads to a less satisfying and  less realistic presentation. Perhaps it's not true bass, but more the mid-bass that's important. For example, a recording of a piano sounds unrealistic to me if I don't hear the resonance of the lower frequencies produced by a concert grand. A good piano is like a sports car engine: if it sounds thin, it isn't satisfying  Your second most important attribute is 'full sound', which to me, does seem to require a sufficiently competent lower-frequency presentation.

 

Having listened, somewhat recently, to a rig which used DEQX DSP, active amplification, and 2 extremely well built subwoofers to guarantee that the FR was as flat as, all the way to 20Hz - a frequency sweep showed that the response to the inaudible regions was definitely there - this didn't do an ounce of good for bringing forth the richness of a pipe organ CD I brought. The fact all the bass frequencies were there didn't stop the overall sound being "thin", and unsatisfying ... QED.

 

Quote

 

I think it's important to consider the relative scale that each of us assigns to various attributes, and I'm certain the scale is not the same for everyone. For me the tonal balance is more important than soundstage depth/width/separation, for example. As a long-time audiophile, I do enjoy 'spatial fireworks' in audio reproduction, but these are more often a curiosity, a nice-to-have. I notice them when they are present, but don't miss them that much when listening to good music. Proper tonal balance is something I simply couldn't live without, on the other hand. 

 

Agree with all of that, Paul 🙂.

[pkane2001]

Today, a slightly different bias study. As most of the studies I posted here so far indicate, there's a significant visual bias that affects a person's ability to judge audio. This study turns the question on its head: does better sound help improve visual perception? 😃 To save you the suspense, it appears that this effect is minor, but does exist. In effect, both senses, hearing and eyesight seem to bias each other, but vision appears to be the stronger "biasing" agent of the two (this is my conclusion, not part of this particular study):

 

https://www.semanticscholar.org/paper/Visual-bias-in-subjective-assessments-of-automotive-Ellermeier-Legarth/3cebe9cfb53e015de60924903580689fe8962761

 

Visual bias in subjective assessments of automotive sounds

• Wolfgang Ellermeier, Søren Vase Legarth
• Published 2006
• Psychology

In order to evaluate how strong the influence of visual input on sound quality evaluation may be, a naive sample of 20 participants was asked to judge interior automotive sound recordings while simultaneously being exposed to pictures of cars.

[pkane2001]

Unrelated to audio, but still an interesting list of 50 Cognitive and Affective Biases in Medicine:

 

http://sjrhem.ca/wp-content/uploads/2015/11/CriticaThinking-Listof50-biases.pdf

 

 

[pkane2001]

Fun and quick introduction to 12 cognitive biases with simple examples:

 

 

 

[Summit]

Big Sound 2015 blind listning test with some participants that you may know. 

 

https://www.innerfidelity.com/category/big-sound-2015

https://www.innerfidelity.com/content/big-sound-2015-wrap-what-i-learned

https://www.innerfidelity.com/content/big-sound-2015-roy-romaz-nails-it

https://www.head-fi.org/threads/schiit-happened-the-story-of-the-worlds-most-improbable-start-up.701900/page-516#post-11921090

[Ajax]

Hi Paul,

 

Thanks for the effort you have put into this, I believe it is incredibly relevant to how we choose audio equipment. I'm really at a loss to understand why more audio enthusiasts don't recognise (or accept) the negative effect of bias and the need for DBT. Maybe it is just too hard and will spoil their fun but with 30 day return offers I'm not sure why audio clubs for example do not promote them at their regular get togethers. I would have thought manufactures would jump at the chance of having 20-30 of their target market listen to their gear.

 

FYI they just completed a controlled test on the effect of aspirin on bowel cancer in Aus. They were so concerned about the placebo effect (as recipients had to be told what they were being tested for) that the randomised-controlled trial monitored 427 people that given aspirin and 434 people that were given a placebo, for between 10 to 20 years. All of them had Lynch syndrome, an inherited disorder which puts them at higher risk of a range of cancers, including bowel cancer.

 

Of interest they found that two aspirins a day, for an average of 2½ years, reduced the rate of bowel cancer by about 50 per cent.

 

https://www.smh.com.au/national/an-aspirin-a-day-helps-keep-bowel-cancer-away-study-finds-20200612-p5524r.html

 

Anyway my point is that if medical scientists, who are testing for life and death cures, are so concerned about bias why aren't we? Why do audiophiles resist DBT so adamantly. Why do advertises pay celebrities exorbitant amounts of money to promote their products if they don't think they will influence our thinking?

 

Apologies if off your specific topic, just wanted to say a very sincere thanks for your efforts.

 

[The Computer Audiophile]

17 minutes ago, Ajax said:

Hi Paul,

 

Thanks for the effort you have put into this, I believe it is incredibly relevant to how we choose audio equipment. I'm really at a loss to understand why more audio enthusiasts don't recognise (or accept) the negative effect of bias and the need for DBT. Maybe it is just too hard and will spoil their fun but with 30 day return offers I'm not sure why audio clubs for example do not promote them at their regular get togethers. I would have thought manufactures would jump at the chance of having 20-30 of their target market listen to their gear.

 

FYI they just completed a controlled test on the effect of aspirin on bowel cancer in Aus. They were so concerned about the placebo effect (as recipients had to be told what they were being tested for) that the randomised-controlled trial monitored 427 people that given aspirin and 434 people that were given a placebo, for between 10 to 20 years. All of them had Lynch syndrome, an inherited disorder which puts them at higher risk of a range of cancers, including bowel cancer.

 

Of interest they found that two aspirins a day, for an average of 2½ years, reduced the rate of bowel cancer by about 50 per cent.

 

https://www.smh.com.au/national/an-aspirin-a-day-helps-keep-bowel-cancer-away-study-finds-20200612-p5524r.html

 

Anyway my point is that if medical scientists, who are testing for life and death cures, are so concerned about bias why aren't we? Why do audiophiles resist DBT so adamantly. Why do advertises pay celebrities exorbitant amounts of money to promote their products if they don't think they will influence our thinking?

 

Apologies if off your specific topic, just wanted to say a very sincere thanks for your efforts.

 

All great points,  but probably off topic for this thread. 

[pkane2001]

12 hours ago, Summit said:

Big Sound 2015 blind listning test with some participants that you may know. 

 

https://www.innerfidelity.com/category/big-sound-2015

https://www.innerfidelity.com/content/big-sound-2015-wrap-what-i-learned

https://www.innerfidelity.com/content/big-sound-2015-roy-romaz-nails-it

https://www.head-fi.org/threads/schiit-happened-the-story-of-the-worlds-most-improbable-start-up.701900/page-516#post-11921090

 

Headphones and speakers often have a very obvious, easily measurable distortion profile. A blind test can reveal these fairly easily, as has been done in the studies quoted above.

 

Tyll's observations don't rise to the level of a study. When he says that blind tests trade one set of biases for another, and that somehow blind tests make it difficult to distinguish minor differences, while sighted testing avoids these is ... shall we say, a bit unsubstantiated? Time and time again, sighted listening has revealed obvious, large biases unrelated to actual audio quality. Doing the tests blind eliminated these and produced much more consistent results. Absolutely, some training is required for a test taker to feel comfortable with the test. That's also described in nearly every study above. Instructions, as well as a time to familiarize and to do dry runs are important.

 

As to Mike Moffat's statement about longer blind tests? Absolutely, do the tests whichever way works for you, as long as it's blind. I don't believe minor differences are easy to tell in an hour-long listening session, at least for me. If that works for you, then great! I'm all for seeing the results of such blind tests. Yet, none of the studies I've seen provide evidence that longer listening sessions allow for better discrimination testing. If you have some well-documented or published studies that do, I'd love to see them.

[pkane2001]

1 hour ago, Ajax said:

Hi Paul,

 

Thanks for the effort you have put into this, I believe it is incredibly relevant to how we choose audio equipment. I'm really at a loss to understand why more audio enthusiasts don't recognise (or accept) the negative effect of bias and the need for DBT. Maybe it is just too hard and will spoil their fun but with 30 day return offers I'm not sure why audio clubs for example do not promote them at their regular get togethers. I would have thought manufactures would jump at the chance of having 20-30 of their target market listen to their gear.

 

This is my personal opinion, but it's hard to go from 'huge, obvious, night-and-day differences' to 'minor, often impossible to tell'. And yet, that's what blind tests often reveal: the differences are either extremely minor, or impossible to detect. Accepting the results that so clearly clash with person's life-long experience and many years of hobby pursuit is hard and sometimes painful.

 

And yet, as a lot of the cited studies above show, visual bias is just one of many that affect listening evaluations, even though it has a very large and easy to demonstrate effect on the outcome.

[semente]

ON THE DESIGN OF LOUDSPEAKERS FOR BROADCAST MONITORING

BBC Research Department Report - December 1988

C.D. Mathers, M.Sc., C.Eng., M.I.E.E., M.I.O.A.

http://downloads.bbc.co.uk/rd/pubs/reports/1988-14.pdf

 

9 LOUDSPEAKER EVALUATION

9.1 introduction
The obvious and definitive means of evaluating a loudspeaker is of course by listening to it.
An expert listener auditioning known programme material can learn a great deal from a listening test.
If all of the sound balancers who use a particular loudspeaker declare it to be excellent, then by definition it is excellent.
In the author's experience at least, such universal approbation is rare.
Although a group of users in an organisation like the BBC usually show remarkable accord in their evaluations, they tend to use adjectives like 'woolly, ‘hard’, or ‘chesty’, and nouns like 'honk', 'quack', or ‘lisp’.
One can often hear what they refer to, but such quirks can rarely be identified by objective measurement, and are very poor guides indeed to any design modifications that might effect significant tonal improvements.
(Very rarely, complimentary expressions like 'clean' or 'uncoloured' are applied; perhaps one reason for the rarity of these is that a perfect loudspeaker should presumably have no perceptible characteristics of its own.)

What is required, of course, is a well-defined relationship between subjective peculiarities, measurable deviations from 'ideal' acoustic output, and oddities in physical behaviour.
A 'dreadful quack at 800 Hz' should be confirmed by a disturbance in the otherwise serene acoustic time-frequency-acceptability plot, and by an agonised writhing at 800 Hz to disturb the otherwise exemplary piston-like movement of the diaphragm.

Reality is otherwise.
'Good' loudspeaker drive units appear to exhibit just as complex mechanical and acoustic behaviour as 'bad' ones.
The author is currently engaged in a project to try to find some relationship between the subjective, acoustic, and mechanical facets of loudspeaker behaviour.
This has been undertaken in the knowledge that previous attempts during four decades have not yielded a final solution.
Results (positive or negative) will be published in due course.
Two reference works only are listed relating to this subject, each includes an extensive bibliography.

 

9.2 Subjective evaluation
Experience shows that comparative judgements of loudspeaker quality can be made more consistently than absolute ones.
An absolute assessment of a new design is something which emerges gradually out of weeks or months of use in control rooms.
Often, a pair of new loudspeakers sent out for 'field trial' will be received with cautious approval, yet returned after a month or two with a list of criticisms detailing points that have emerged only gradually from continuous use.
For comparative tests, a reference loudspeaker is of course needed.
This is provisionally selected during the early stages of commercial production as being a typical unit of acceptable quality; once production is well established, a new reference may be adopted as a clearer picture emerges of what is 'typical'.
In fact, at least three such units are selected in normal BBC practice, to provide a working standard for acceptance testing: a spare (which is carefully stored): and a standard by which the manufacturers can assess the consistency of their output, whether by listening or by measurement.
An established standard is also of course the only reasonable reference available in appraising a new design.

In listening tests, it is important that the listener should begin with as few preconceived ideas as possible.
For example, a look at a response plot may cause him, consciously or otherwise, to listen for some expected peculiarities. Normally, an A/B switch is provided, and the loudspeaker to be used as reference is indicated.
The loudspeakers are placed behind an acoustically transparent but optically opaque curtain, especially if any aspect of the units under test might be visually identifiable.
To help eliminate room effects, the test may be repeated with the loudspeaker positions interchanged.
If several units are to be tested, it is useful to include one twice — anonymously — to test the listener's consistency.
(Experienced listeners expect this.)

Finally, it is essential that the listener delivers his judgement before any additional information is given to him; not (one would trust) that he might 'cheat', but rather that he might re-interpret what he thought he had heard in the light of further knowledge.
Subsequent discussion may well prove valuable, but must be subsequent.

Formal tests involving a number of listeners may need further care, particularly if, as is likely, they permit less in the way of personal communication between subjects and test organiser.
Past experience suggests that a particular hazard is the use of descriptive terms whose meaning seems obvious to everyone, but which can actually mean different things to different people.

 

[pkane2001]

8 minutes ago, semente said:

ON THE DESIGN OF LOUDSPEAKERS FOR BROADCAST MONITORING

BBC Research Department Report - December 1988

C.D. Mathers, M.Sc., C.Eng., M.I.E.E., M.I.O.A.

http://downloads.bbc.co.uk/rd/pubs/reports/1988-14.pdf

 

9 LOUDSPEAKER EVALUATION

9.1 introduction
The obvious and definitive means of evaluating a loudspeaker is of course by listening to it.
An expert listener auditioning known programme material can learn a great deal from a listening test.
If all of the sound balancers who use a particular loudspeaker declare it to be excellent, then by definition it is excellent.
In the author's experience at least, such universal approbation is rare.
Although a group of users in an organisation like the BBC usually show remarkable accord in their evaluations, they tend to use adjectives like 'woolly, ‘hard’, or ‘chesty’, and nouns like 'honk', 'quack', or ‘lisp’.
One can often hear what they refer to, but such quirks can rarely be identified by objective measurement, and are very poor guides indeed to any design modifications that might effect significant tonal improvements.
(Very rarely, complimentary expressions like 'clean' or 'uncoloured' are applied; perhaps one reason for the rarity of these is that a perfect loudspeaker should presumably have no perceptible characteristics of its own.)

What is required, of course, is a well-defined relationship between subjective peculiarities, measurable deviations from 'ideal' acoustic output, and oddities in physical behaviour.
A 'dreadful quack at 800 Hz' should be confirmed by a disturbance in the otherwise serene acoustic time-frequency-acceptability plot, and by an agonised writhing at 800 Hz to disturb the otherwise exemplary piston-like movement of the diaphragm.

Reality is otherwise.
'Good' loudspeaker drive units appear to exhibit just as complex mechanical and acoustic behaviour as 'bad' ones.
The author is currently engaged in a project to try to find some relationship between the subjective, acoustic, and mechanical facets of loudspeaker behaviour.
This has been undertaken in the knowledge that previous attempts during four decades have not yielded a final solution.
Results (positive or negative) will be published in due course.
Two reference works only are listed relating to this subject, each includes an extensive bibliography.

 

9.2 Subjective evaluation
Experience shows that comparative judgements of loudspeaker quality can be made more consistently than absolute ones.
An absolute assessment of a new design is something which emerges gradually out of weeks or months of use in control rooms.
Often, a pair of new loudspeakers sent out for 'field trial' will be received with cautious approval, yet returned after a month or two with a list of criticisms detailing points that have emerged only gradually from continuous use.
For comparative tests, a reference loudspeaker is of course needed.
This is provisionally selected during the early stages of commercial production as being a typical unit of acceptable quality; once production is well established, a new reference may be adopted as a clearer picture emerges of what is 'typical'.
In fact, at least three such units are selected in normal BBC practice, to provide a working standard for acceptance testing: a spare (which is carefully stored): and a standard by which the manufacturers can assess the consistency of their output, whether by listening or by measurement.
An established standard is also of course the only reasonable reference available in appraising a new design.

In listening tests, it is important that the listener should begin with as few preconceived ideas as possible.
For example, a look at a response plot may cause him, consciously or otherwise, to listen for some expected peculiarities. Normally, an A/B switch is provided, and the loudspeaker to be used as reference is indicated.
The loudspeakers are placed behind an acoustically transparent but optically opaque curtain, especially if any aspect of the units under test might be visually identifiable.
To help eliminate room effects, the test may be repeated with the loudspeaker positions interchanged.
If several units are to be tested, it is useful to include one twice — anonymously — to test the listener's consistency.
(Experienced listeners expect this.)

Finally, it is essential that the listener delivers his judgement before any additional information is given to him; not (one would trust) that he might 'cheat', but rather that he might re-interpret what he thought he had heard in the light of further knowledge.
Subsequent discussion may well prove valuable, but must be subsequent.

Formal tests involving a number of listeners may need further care, particularly if, as is likely, they permit less in the way of personal communication between subjects and test organiser.
Past experience suggests that a particular hazard is the use of descriptive terms whose meaning seems obvious to everyone, but which can actually mean different things to different people.

 

 

It's an oldie, from 1988, but still very interesting! Thanks for sharing. 

 

It's impressive how much research BBC has produced in audio space over the years, from research studies to publishing industry standards, including testing methodologies. Floyd Toole followed up with a much more detailed and specific research on the subject of subjective speaker evaluation.

[fas42]

On 6/14/2020 at 8:38 AM, pkane2001 said:

 

This is my personal opinion, but it's hard to go from 'huge, obvious, night-and-day differences' to 'minor, often impossible to tell'. And yet, that's what blind tests often reveal: the differences are either extremely minor, or impossible to detect. Accepting the results that so clearly clash with person's life-long experience and many years of hobby pursuit is hard and sometimes painful.

 

My experience has shown that the circumstances that blind tests are conducted in will be enough to completely undermine the optimisation that the hobbyist has applied to his particular situation. IOW, the blind tests never actually test what they purportedly are attempting to reconcile - the abyss in understanding between the two camps remains just as wide and deep as it ever has; and will remain so, until greater overall understanding evolves ...