A/B testing favors B over A

[Jud]

31 minutes ago, pkane2001 said:

 

Assuming this bias towards the second test is true, it's a flaw only if you do a small number of A/B tests. It in no way invalidates blind testing as a methodology. 

 

I usually switch between components for days while trying to evaluate differences. I am very well aware of my attention changing and often revealing differences that are not there. That's why I always try to confirm what I hear with repeated tests. I try to focus on something very specific in a familiar sound track, played for a very short time (less than a minute) before switching. I have a few favorite recordings I know very well, and I use specific portions of those recordings to test for different sound qualities. I do this blind, if I can set up such a test (and I always try to do it this way, if at all possible).

 

A few thoughts here:

 

- The "less than a minute" time frame isn't nearly short enough.  Scientific research shows echoic memory for everything except loudness lasts maybe 4-10 seconds.

 

- "[V]ery specific in a familiar sound track" is a really good thing.  It means you may not actually be trying to compare your memories of the two sounds (see the first point; after a few seconds this is fruitless, *especially* if there's intervening music - I can give you a web link if you'd like), but rather doing "pattern matching," at which people are very, very good.  (In fact we're so good we can sense patterns where none exist, which is responsible for optical and aural illusions).

 

- Regarding loudness, we are *really* good at detecting and remembering this, but really bad (as I mentioned before) at remembering other acoustic qualities.  So very often, when we think we are comparing two musical passages as a whole, I think it is very possible what we are in fact doing is comparing loudness of the end of passage A with the beginning of passage B.  We like louder (thus the loudness wars).  I think if the end of music sample A is softer than the beginning of music sample B, that alone might easily account for a preference of B over A.

[Jud]

47 minutes ago, jabbr said:

 

Yes but... when I compare two sounds, be they components or instruments or recording I "feature extract" that is commit my impression to memory be that smoothness of a string in a certain octave or extension of bass or position of instruments on soundstage, and these features can be remembered longer than 10sec

 

How does this feature extraction capability relate to the results of this experiment?

 

http://deutsch.ucsd.edu/psychology/pages.php?i=209

[Jud]

3 minutes ago, Daudio said:

 

I think this whole 'short echoic memory' thing is a red herring. Obviously we remember sounds for far longer then a few seconds. How long have you known the sound of your mother voice, in all it's different volumes, moods, health, age, emotion, inflection, etc. ? How can one hear the differences in a new remaster of a familiar album, if one can't recall sound beyond a few seconds ? How can a mother pick out the sound of only her baby from a group ?

 

I think that this 'echoic memory' thing is a myth build up from agenda driven extrapolation of the only formal study of anything having to do with audio 'memory. But it just doesn't pass the simplest smell test, of course we remember sounds for long periods, some most of our lives !

 

 

 

Another thing that bothers me about this 'short A/B test' thing involves the 'test signals', which I find extremely limited, and possibly of little to no use in showing any difference between UUTs. What I mean is that in the other, longer term, types of testing we can expose our ears to many different stimuli (albums, tracks, clips) to compare lots of effects for differences, which may only be exhibited in a small set of recorded musical moments. Dynamic range, dynamics across frequency, noise levels and character, the multiple aspects of 'soundstage', subtle musical cues, details of breath, bowing, plucking, etc. as well as specific cuts that may help some uncover subtle points of tonality.

 

The point is that musical test signals that can show differences are not particularly common, or even known in advance, and are quite diverse, so how can one, or a few, preselected, very short clips, be expected to excite the differences we are interested in teasing out of our testing ?

 

Doesn't make any sense to me

 

 

 

I think you're half right.

 

The short duration of echoic memory is an experimentally verified reality.  The things you are talking about (sound of mother's or child's voices) fall into the category of "pattern matching."  The fact that humans do this sensory "pattern matching" is another known, experimentally verified reality.

 

With regard to the "race" between echoic memory and the types of sonic phenomena we are thinking of when we talk about high end audio - fine discrimination of timbre, soundstaging, etc. - I completely agree.  It's why I ran a little test of timbre discrimination on the forum (in which 50+ people participated) that did not depend on echoic memory, but did test for ability to make that timbre discrimination over a short versus a longer period of time:

 

https://www.computeraudiophile.com/forums/topic/22709-heres-a-little-test-you-can-all-join-in/#comment-390666

[Jud]

2 hours ago, Daudio said:

Yes I recall that thread, but I don't recall a resolution ! And I didn't find one by going to the last page of the thread either, just promises. So the next time you want to link to that idea, perhaps you could link to where you summarize the results of your test, so there might be something worthwhile to discuss about it.

 

P.S (to be fair, I'm not too good about finishing things either)

 

 

You're correct, so I've attached an Excel spreadsheet with a statistical analysis of the results.  (I sure as heck didn't do the analysis - statistics is one of the many subjects I regret not learning much of anything about in my life.)  First a very brief recapitulation of the test, then an equally brief summary of what little I gathered from the attached statistical analysis.

 

The test: I showed photos of two different acoustic guitars and asked people to tell me which was primarily in the left and which primarily in the right channel in a musical selection.  One selection was 30 seconds of a track where these two guitars were being played.  The other was a 2 second selection from those 30 seconds, then two seconds of silence, then the 2-second selection, then two seconds of silence, etc., for a total of 30 seconds.  (So: On (2 sec), Off (2 sec) On (2 sec), until 30 seconds has elapsed.)  This was intended to simulate the timbre information about the guitars that you might get in a rapid switching A/B test.  Anyone who said he was an objectivist got the 2-second sequence, and anyone who said he was a subjectivist got the 30-second passage.  This was so subjectivists wouldn't give up on the 2-second sequence without really trying and screw up the results.

 

I counted an answer as correct if someone could clearly describe to me the sonic differences between the two guitars (the Epiphone archtop with the f-holes sounded very nearly like an electric guitar with much less body resonance than the classic Gibson with the round sound hole), even if they got the channels wrong (i.e., if they had their "wires crossed").  One person did get the 30-second passage wrong.  He said using his iPhone and earbuds that both guitars sounded like they were somewhere in the middle.  I counted it as an incorrect answer (the only one) for the 30-second passage.

 

The statistics:  While 1 person of 40 got the 30-second passage wrong, 3 of 14 got the 2-second sequence wrong.  The  attached statistical analysis as I understand it says this equates to a non-chance probability of about 94% that people were better able to identify the separate timbres of the two guitars with the longer passage.

Survey results analysis for Jud.xlsx

[Jud]

6 minutes ago, mmerrill99 said:

Thanks, Jud, maybe we should take discussion of those results back to the test thread although they are relevant here too?

 

I don’t mind either or both, but whether we discuss it further in this thread is up to the OP.

[Jud]

19 minutes ago, Daudio said:

 

Thanks, but I don't follow exactly. If two instruments are playing, is that together, separate tracks, separate channels, or what ???

 

Same song, two acoustic guitars playing at the same time, one primarily (close to exclusively) in the left channel, one primarily (close to exclusively) in the right channel.

 

Quote

 

Were these 'objective' & 'subjective' subjects self classified ? Why the 40 vs 14 subject tests ?

 

 

Self classified.  That's how many people who took the test called themselves objectivists (14) and subjectivists (40).

 

Quote

 

That prompts me to ask why then are you still pushing the whole short echoic memory thing ?

 

And makes me think that your test could be extended to longer time periods 

 

This doesn't test echoic memory.  The two guitars are playing at the same time.  The test is to distinguish them from each other by timbre.

 

For a test of echoic memory, see this: http://deutsch.ucsd.edu/psychology/pages.php?i=209

[Jud]

16 minutes ago, Daudio said:

I think that what you are actually testing is how long it takes to make an accurate discrimination

 

Of at least this degree of timbre variation, yes.

 

If we were talking about a loudness difference of say 1dB, the time needed to make an accurate discrimination would I am guessing be less.

 

A smaller timbre difference might take longer to discriminate.  That's the point about A/B comparisons as they are usually done, in sequence: You are in a race between echoic memory and the time it takes to discriminate what might be fairly small sonic differences.  (Yes, it's the same link, but that link summarizes one of a very large number of experiments by various people over decades all coming to the conclusion that echoic memory lasts perhaps 4-10 seconds; and hearing more notes, as you do when you play two pieces of music in sequence, unavoidably degrades that memory.)

[Jud]

32 minutes ago, Bob Stern said:

 

I hereby relinquish my prerogatives and status as OP.

 

OK, well I don’t care either, so whatever anyone else prefers.

[Jud]

11 minutes ago, mmerrill99 said:

Seems like continuing here makes sense?

 

Sure.

[Jud]

I would like to stay with the specific rather than the general here.  There are many potential problems with blind tests, as with any kind of test, and various ways of dealing with these potential problems.  We can (and I sometimes have) had discussions about some of these problems.  But the discussion up to now has been about one particular potential problem, preference for a second sample.

 

Randomization has been suggested as one of several possible ways of dealing with the potential problem, as has throwing out the first two samples.  One specific objection that has been raised is that this does not account for other potential problems and is therefore not a useful solution.

 

It seems to me obvious that (1) if I fix my car it will not deal with any potential problems with the dishwasher; and (2) this isn’t a reason not to fix the car.

 

 If there are yet other potential problems with a given test, they of course should also be minimized insofar as possible.  From my reading in the scientific literature so far, there are a few of these that I think might be difficult to resolve with sequential A/B/X testing.  But I would suppose if good solutions exist, the way I might find them is by further reading, not performing “thought experiments” on the possibly faulty basis of my current knowledge.