Does BIAS affect audio test results?

[semente]

ON THE DESIGN OF LOUDSPEAKERS FOR BROADCAST MONITORING

BBC Research Department Report - December 1988

C.D. Mathers, M.Sc., C.Eng., M.I.E.E., M.I.O.A.

http://downloads.bbc.co.uk/rd/pubs/reports/1988-14.pdf

 

9 LOUDSPEAKER EVALUATION

9.1 introduction
The obvious and definitive means of evaluating a loudspeaker is of course by listening to it.
An expert listener auditioning known programme material can learn a great deal from a listening test.
If all of the sound balancers who use a particular loudspeaker declare it to be excellent, then by definition it is excellent.
In the author's experience at least, such universal approbation is rare.
Although a group of users in an organisation like the BBC usually show remarkable accord in their evaluations, they tend to use adjectives like 'woolly, ‘hard’, or ‘chesty’, and nouns like 'honk', 'quack', or ‘lisp’.
One can often hear what they refer to, but such quirks can rarely be identified by objective measurement, and are very poor guides indeed to any design modifications that might effect significant tonal improvements.
(Very rarely, complimentary expressions like 'clean' or 'uncoloured' are applied; perhaps one reason for the rarity of these is that a perfect loudspeaker should presumably have no perceptible characteristics of its own.)

What is required, of course, is a well-defined relationship between subjective peculiarities, measurable deviations from 'ideal' acoustic output, and oddities in physical behaviour.
A 'dreadful quack at 800 Hz' should be confirmed by a disturbance in the otherwise serene acoustic time-frequency-acceptability plot, and by an agonised writhing at 800 Hz to disturb the otherwise exemplary piston-like movement of the diaphragm.

Reality is otherwise.
'Good' loudspeaker drive units appear to exhibit just as complex mechanical and acoustic behaviour as 'bad' ones.
The author is currently engaged in a project to try to find some relationship between the subjective, acoustic, and mechanical facets of loudspeaker behaviour.
This has been undertaken in the knowledge that previous attempts during four decades have not yielded a final solution.
Results (positive or negative) will be published in due course.
Two reference works only are listed relating to this subject, each includes an extensive bibliography.

 

9.2 Subjective evaluation
Experience shows that comparative judgements of loudspeaker quality can be made more consistently than absolute ones.
An absolute assessment of a new design is something which emerges gradually out of weeks or months of use in control rooms.
Often, a pair of new loudspeakers sent out for 'field trial' will be received with cautious approval, yet returned after a month or two with a list of criticisms detailing points that have emerged only gradually from continuous use.
For comparative tests, a reference loudspeaker is of course needed.
This is provisionally selected during the early stages of commercial production as being a typical unit of acceptable quality; once production is well established, a new reference may be adopted as a clearer picture emerges of what is 'typical'.
In fact, at least three such units are selected in normal BBC practice, to provide a working standard for acceptance testing: a spare (which is carefully stored): and a standard by which the manufacturers can assess the consistency of their output, whether by listening or by measurement.
An established standard is also of course the only reasonable reference available in appraising a new design.

In listening tests, it is important that the listener should begin with as few preconceived ideas as possible.
For example, a look at a response plot may cause him, consciously or otherwise, to listen for some expected peculiarities. Normally, an A/B switch is provided, and the loudspeaker to be used as reference is indicated.
The loudspeakers are placed behind an acoustically transparent but optically opaque curtain, especially if any aspect of the units under test might be visually identifiable.
To help eliminate room effects, the test may be repeated with the loudspeaker positions interchanged.
If several units are to be tested, it is useful to include one twice — anonymously — to test the listener's consistency.
(Experienced listeners expect this.)

Finally, it is essential that the listener delivers his judgement before any additional information is given to him; not (one would trust) that he might 'cheat', but rather that he might re-interpret what he thought he had heard in the light of further knowledge.
Subsequent discussion may well prove valuable, but must be subsequent.

Formal tests involving a number of listeners may need further care, particularly if, as is likely, they permit less in the way of personal communication between subjects and test organiser.
Past experience suggests that a particular hazard is the use of descriptive terms whose meaning seems obvious to everyone, but which can actually mean different things to different people.