Who's afraid of DBTs

[PeterSt]

but certainly not to act like an annoying know-it-all, when it is obvious you are not.

 

Dangerous again. Some act like fools while actually they are not.

 

Is my avatar gaining some points by now ?

prrrrt

[trithio]

Wait. This isn't funny. trithio talked about black wholes earlier on.

Now you've done it.

 

At least I do read.

 

Dont remember that but feel free to improve my autocorrection keyboard. In its defense, it did things much funnier than that

[trithio]

No, you are minimising what I'm saying - it's a tool that should only be used for perceptual testing by research labs & those who have expertise in that area. What you are failing to acknowledge is that 99.9999% of the time it is used incorrectly in home based DBTs

 

Many things were said here, hard to remember all. But I am quite sure I never said that any fool can use any tool. And also that I agreed with a previous post of yours which made the same point about the proper usage of DBTs being very hard and not for everyone.

Talking about non-issues like this is a waste of time. But I guess we already passed that point. Long ago

[PeterSt]

Sorry that was truly just lazy thinking out loud from my side. Wont go into an argument because I consider the burnin/revert scenario a rational black whole. Logic and reason simply dissapear if you go in there.

 

[mmerrill99]

It's probably best to talk with real test results, rather than talk in the abstract - so Arny Kreuger's results are a good example of the need for controls to catch problems. Here's the results again but with the problem highlighted

21:04:53 : Test started.

21:05:18 : 00/01

21:05:39 : 01/02 --- 21 seconds

21:06:39 : 02/03 --- 60 seconds

21:06:45 : 03/04 --- 6 seconds

21:06:47 : 04/05 --- 2 seconds!!!

21:06:50 : 04/06 --- 3 seconds!!!

21:06:54 : 04/07 --- 4 seconds!!!

21:06:56 : 05/08 --- 2 seconds!!!

21:06:58 : 06/09 --- 2 seconds!!!

21:06:59 : 07/10 --- 1 second!!!!

21:07:01 : 07/11 --- 2 seconds!!!

21:07:04 : 07/12 --- 3 seconds!!!

21:07:05 : 08/13 --- 1 second!!!

21:07:08 : 08/14 --- 3 seconds!!!

21:07:10 : 08/15 --- 2 seconds!!!

21:07:31 : 08/16 --- 21 seconds

21:07:31 : Test finished.

 

----------

Total: 8/16

Probability that you were guessing: 59.8%

 

So what we see demonstrated here is somebody not trying to hear any difference & just guessing randomly for each trial - which of course gives a NULL result - which most interpret to mean that there is no difference between the two audio samples A & B (in this case A was an MP3 & B the WAV version)

 

For those not familiar with this tool ABX is a plugin utility for Foobar You select two files that you want to check blind if you can hear differences between them. It hides the names of the 2 audio files & just gives them letter designations A or B & X or Y. Your job is to listen to A or B & choose if it is the same as X or Y. So you click on A or B to listen & then click on X or Y to make your choice. That's one trial done. It randomises the files again & you do another trial.

 

So what Arny's test results show is the timestamp that he took for each trial. You can see in 2 trials he only took 1 second to do the listening & choosing. In 4 trials he took 2 seconds & so on.

 

He wasn't listening, he was guessing randomly from the 5th trial to the 15th trial.Of course the results were NULL

 

Now what if we didn't have these timings - what if he took longer between trials but still guessed or lost focus or was tired (many reasons that he might not actually be listening) - we would have no way of knowing this - it would look like the trial was done & no difference was heard.

 

That's what the false negative controls are for

[mmerrill99]

Many things were said here, hard to remember all. But I am quite sure I never said that any fool can use any tool. And also that I agreed with a previous post of yours which made the same point about the proper usage of DBTs being very hard and not for everyone.

Talking about non-issues like this is a waste of time. But I guess we already passed that point. Long ago

 

Ok, so let me ask you - do you think DBTs should NOT be used for perceptual testing by anyone, other than trained professionals?

[Jud]

Not sure where you see the problem with controlled tests but I am not aware of any other than funny smear campaigns. We do not need to establish anything here, DBTs are just another scientific tool which works in audio same as everywhere. That is, it works good. Or as good as you set it up.

Also not sure where is this stuff about proving things 'for audio' coming from. No one needs to do that cause there is nothing special about audio. And science and its metods work everywhere the same anyway. That's the beauty of it. Not many people see it but nobody will convince those.

Other than that thanks for the few nice and detailed posts.

 

There's obviously great difficulty in understanding why DBT would not necessarily be scientific in audio (or, as mmerrill says, in perceptual testing in general). I've mentioned the problem of false negatives before. mmerrill says there are DBT procedures set up to uncover false negatives (or actually, to try to control for them). Let me get into what I didn't have time for earlier, possible experimental protocols to attempt to determine a false negative rate for even well controlled DBTs.

 

First protocol: Let the subjects play file A, then file A again, though labeled as file B. Allow them to repeat as many times and in whatever order they like. Ask your experimental population to indicate whether they prefer A, prefer B, or have no preference. This will give baseline "prefer A," "prefer B," and "no preference" numbers when no actual difference exists. Then give one of the two files some level of simulated jitter, let us say, and again ask for preferences. Increase the jitter until there is a repeatable difference from the baseline significantly greater than chance (can be favorable or unfavorable, as there is anecdotal information that some people like the sound of increased jitter, or other forms of distortion); or if the first introduction of simulated jitter results in repeatable differences greater than chance, reduce the level of jitter until the results don't show those differences any longer. Notice there is no "right" or "wrong" answer here, and no explicit testing of ability to identify any particular file or phenomenon, just preference (or lack of one).

 

Now DBT with simulated jitter, and see what the minimum level is at which subjects can identify the file with jitter at greater than chance rates.

 

If the level of jitter at which there's a change in preferences for the first experimental protocol is lower than that where jitter can be identified with DBT, then you've got a potential false negative rate for DBT (or at least the relative rate of false negatives versus the first protocol).

 

That's the type of thing I'm after, and the type of thing that would have to be the subject of a controlled scientific study before use of DBT could be said to be scientifically valid for audio testing. Obviously, in order for a test to be scientifically valid, you have to determine that it is effective at showing the existence or absence of a phenomenon (say, audible jitter) when and where it is used. You can't assume effectiveness, at least not if you want to be doing science. To repeat: I am not yet aware of any study showing the effectiveness of DBT (with particular attention to false negative rates) to determine what levels of particular phenomena are generally audible, or audible to some designated percentage of the general population. Thus we can't yet say any DBT results are scientifically valid, any more than we could say the results of any other uncalibrated measuring tool are scientifically valid.

[trithio]

 

I surely seen worse

'Black whole' isnt even so much different that 'black hole'. Somehow it sounds appropiate to me. Like more interesting and misterious and closer to how I think about the underlying phenomenon.

But then, I'm not an english native and may be super wrong.

[PeterSt]

He wasn't listening, he was guessing randomly from the 5th trial to the 15th trial.Of course the results were NULL

 

OK, let's say that you don't "admit" that my idea about this made some sense. Then I can turn all easily around :

I listen for some longer for two times, next know what to listen for and know it occurs in the first second, and go along with that. No idea why I would listen for longer (21 seconds) to the last track - I guess I liked the track to that point (21 seconds); had one last chance to listen to that beautiful lady.

 

You know I ain't making up things, because I told about doing it like that (Julf's test). Only that was 10 seconds which of course would not have turned up in red in your last list. But this is non-sense.

 

So you see, now someone comes up with a nice example, and it can be debunked one way or the other. I'll just change my story.

But then I learned this from this thread. Ahum.

[PeterSt]

But then, I'm not an english native and may be super wrong.

 

AHA ! That's out. So one day we will know who you are, prot.

[Jud]

Jud, I mostly agree with you but would actual phrase it differently - in well run DBTs there are internal hidden controls to uncover false negatives, it's just that most DBTs are not well run.

 

Here's extracts from "Methods for the subjective assessment of small impairments in audio systems" ITU-R BS.1116-2 (2014)

 

So false negatives are a part of the recommended procedures for this sort of assessment.

 

Why are such negative controls needed - because perceptual testing is a very difficult thing to get right & it is recognised that fatigue & loss of focus quickly set in when one is repeatedly listening to the same piece/snippet of audio through many repetitions:

 

This is recognised again in the recommendations:

 

 

mmerrill, I would characterize the bits you quoted and I left out as controls designed to minimize false negatives in DBT, but *not* measurement of any false negative rate inherent in even well controlled DBT. Please refer to my prior comment regarding possible protocols to attempt to determine such a "baseline" false negative rate.

[mmerrill99]

OK, let's say that you don't "admit" that my idea about this made some sense. Then I can turn all easily around :

I listen for some longer for two times, next know what to listen for and know it occurs in the first second, and go along with that. No idea why I would listen for longer (21 seconds) to the last track - I guess I liked the track to that point (21 seconds); had one last chance to listen to that beautiful lady.

 

You know I ain't making up things, because I told about doing it like that (Julf's test). Only that was 10 seconds which of course would not have turned up in red in your last list. But this is non-sense.

 

So you see, now someone comes up with a nice example, and it can be debunked one way or the other. I'll just change my story.

But then I learned this from this thread. Ahum.

Peter, it's not possible to click on a button A or B, listen for a short snippet, then choose & select another button X is A or X is B in 1 second - it's not possible in 2 seconds - its' not possible to do it in 4 secs - try it

[Daudio]

I think short auditory echoic memory is something like 5 secs? Whatever, it is short & is the premise that underlies A/B testing.

 

Last night I was listening/watching a Pink Floyd 1994 live concert DVD. While listening to some of the songs from the 'Dark Side of the moon' album from 20 years previous, I was continually struck by the little differences that I perceived between what I heard from the live performance, and what I remembered from all those hundreds(?) of exposures to the 1973 album version.

 

I can't remember anything about the songs to hum them, or play them(Ha !), but I had an expectation (memory) of almost every note, timbre and intensity, as I remembered it, which resulted in so many instances of unbidden perceptions of 'this sounds different then the album version'. Timing, emphasis, timbre, style, orchestration, whatever; all these things and more popped out as sounding different !

 

I'll bet there are a big proportion of CA members who not only remember the sound of some, or all of those songs, but also remember how they sounded on different audio systems ! Separate memory's ? Or associated memory's, like the way we store basic concepts. Again, part of the same long term, detailed, auditory memory.

 

Anybody who still thinks we don't have deep, accurate, 'auditory memory' desperately needs some re-education !!

 

I have previous commented on the fact of deep auditory memory of our mothers(father, wife, children, friends, etc.) voices. Again, proof of human auditory memory. And yes, it is long term, not short term memory I am talking about. While echoic memory might be appropriate for very short term tests, why bother ? It seems more problematic, nowhere as rich as long term auditory memory, and thus not as useful for the detection of subtle phenomena. Which is, after all, what the audiophile is interested in

 

If one wants to limit their audio testing 'samples' to 2.5, or 4 seconds, that is their right. But I think that wastes most of the power of the human auditory perception system.

 

And the idea of using the tiny 'samples' of echoic memory to make statements about how audiophiles hear, is, to me, like taking the analysis of a few registers in a computer processor, and trying to imply the behavior of an entire Operating System from it. Fools errand

 

Cheers,

[Jud]

AHA ! That's out. So one day we will know who you are, prot.

 

Not prot, definitely. prot was, I'm quite sure, a native English speaker, because only a native could do such violence to the mother tongue. (Analogous to large companies using non-English speakers for transcription of English-language legal case reports, because native speakers import their own errors of spelling, etc.)

[trithio]

AHA ! That's out. So one day we will know who you are, prot.

 

That non-native thing should have been quite obvious after my first 3 posts . Although I do know at least a few natives who write worse than me.

As about the prot thing, keep beating it if you enjoy. Not me. The only thing I'm worried about in that area, is that I cannot really explain how senorx sometimes posts exactly my words, down to commas and quotes. Funny stuff.

[Daudio]

Originally Posted by Daudio

Yeah, science is so cool that it works the same on chemistry and cognition, gravity and electromagnetics, quarks and black holes.

 

 

Exactly. And glad we finally 'agree'

 

Ha, ha! We do Not agree, because what I said is totally incorrect, those separate sciences do NOT work the same. Which you would know, if you knew the least little thing about science.

 

Mr. Miss-the-Point *

 

 

*(a little nastygram borrowed from wscott)

[trithio]

Anybody who still thinks we don't have deep, accurate, 'auditory memory' desperately needs some re-education !!

,

 

The wiki article on echoic memory contains lots of references, papers and such. With names. You can easily locate them and go reeducate them. I hear baseball bats are the perfect tools for that.

That's not gonna change hearing physiology, the truth, the science or the wiki but that way you can finally be happy about it too

[trithio]

.

 

Ha, ha! We do Not agree, because what I said is totally incorrect, those separate sciences do NOT work the same. Which you would know, if you knew the least little thing about science.

 

Mr. Miss-the-Point *

 

 

*(a little nastygram borrowed from wscott)

 

Not sure whether you are Mr or Mrs Obvious but thanks for 'clarifying' that

[mmerrill99]

mmerrill, I would characterize the bits you quoted and I left out as controls designed to minimize false negatives in DBT, but *not* measurement of any false negative rate inherent in even well controlled DBT. Please refer to my prior comment regarding possible protocols to attempt to determine such a "baseline" false negative rate.

Yes, I agree but I think we are using the wrong terminology & talking about two different issues:

- one issue is the one I just reported for ArnyK - real listening isn't done - just a random selection & the final results show a NULL result.

- the issue you are talking about is a calibration of the sensitivity of the test - just how small a difference can the test setup, equipment, test subjects, differentiate

 

Both may be able to be addressed by controls but it's difficult to do.

However, without them, the reliability of the results is unkown

[PeterSt]

I'll bet there are a big proportion of CA members who not only remember the sound of some, or all of those songs, but also remember how they sounded on different audio systems ! Separate memory's ? Or associated memory's, like the way we store basic concepts. Again, part of the same long term, detailed, auditory memory.

 

Well, nice, because I don't think this subject passed yet in this thread. And I didn't dare to bring it up.

But Dave, I wouldn't count on too many.

 

I too can do it, but it goes somewhat more explicit. I mean, I still know how a track sounded 2 years ago in a certain situation. Hard to tell in advance exactly how (think relative to how it sounds today), but when I hear that particular situation back, I can tell the situation. Like "oh, that was on Windows 7 Native".

It is not much useful to tell about it, because I know of no one who can do this. Similar as how for many it is very hard to judge from tracks they never heard. I can do that just the same. I guess this is because I tend to tear down all I hear into elements.

Btw, I spent a post on this a couple of months ago, about how "we" can learn from a 100 distorting versions of a violin, how a violin can become the real thing in your head; never heard one for real, but you can synthesize it.

blabla

 

In mere practical appliance, it works far more easy if you can do a bit of this. Here's an easy example, though you guys have to trust me on the truth of it :

Maybe beginning of last year I listened for the last time to Tusk (the whole album). Since then, there was an NOS1 upgrade, plus some new reasoned out interlink and from both together it is known that it does a lot (to SQ). Nice.

But two days ago I listened to Tusk again, and my ears fell off because the cymbals now sounded for a sheer 10 seconds long.

On estimate, before this was 3 or 4.

 

If you can't hear such a difference, then your ears were off to begin with. One thing : you'd need to focus on the length of cymbals say always. This is also easy, once you have a drum kit in the house, and know they sound for 15 seconds. All it further needs is your "obsession" to make them the same through loudspeakers. Ok, they really do from a recording of that same drum kit, and now it only bothers infinitely more because why the heck is that ? Fairly difficult to get that out of your mind while actually working on improvement.

 

If this was readable and you payed attention, then you can easily see how reasons can exist to remember really everything. And I have those reasons.

Don't ask me what's FiFo'd out, because I guess not everything can be stored in there. If this was my last post I forgot my password.

[mmerrill99]

Last night I was listening/watching a Pink Floyd 1994 live concert DVD. While listening to some of the songs from the 'Dark Side of the moon' album from 20 years previous, I was continually struck by the little differences that I perceived between what I heard from the live performance, and what I remembered from all those hundreds(?) of exposures to the 1973 album version.

 

I can't remember anything about the songs to hum them, or play them(Ha !), but I had an expectation (memory) of almost every note, timbre and intensity, as I remembered it, which resulted in so many instances of unbidden perceptions of 'this sounds different then the album version'. Timing, emphasis, timbre, style, orchestration, whatever; all these things and more popped out as sounding different !

 

I'll bet there are a big proportion of CA members who not only remember the sound of some, or all of those songs, but also remember how they sounded on different audio systems ! Separate memory's ? Or associated memory's, like the way we store basic concepts. Again, part of the same long term, detailed, auditory memory.

 

Anybody who still thinks we don't have deep, accurate, 'auditory memory' desperately needs some re-education !!

 

I have previous commented on the fact of deep auditory memory of our mothers(father, wife, children, friends, etc.) voices. Again, proof of human auditory memory. And yes, it is long term, not short term memory I am talking about. While echoic memory might be appropriate for very short term tests, why bother ? It seems more problematic, nowhere as rich as long term auditory memory, and thus not as useful for the detection of subtle phenomena. Which is, after all, what the audiophile is interested in

 

If one wants to limit their audio testing 'samples' to 2.5, or 4 seconds, that is their right. But I think that wastes most of the power of the human auditory perception system.

 

And the idea of using the tiny 'samples' of echoic memory to make statements about how audiophiles hear, is, to me, like taking the analysis of a few registers in a computer processor, and trying to imply the behavior of an entire Operating System from it. Fools errand

 

Cheers,

I know Dave, I'm not defending this - I'm just saying what the premise is underlying A/B testing

[PeterSt]

As about the prot thing, keep beating it if you enjoy. Not me. The only thing I'm worried about in that area, is that I cannot really explain how senorx sometimes posts exactly my words, down to commas and quotes. Funny stuff.

 

Yes Sir. And if your IQ is not above 250 then I will believe senorx is not again prot, just because you come up with it now. But if you hadn't ...

This still doesn't say that you are not him. Just saying.

 

But it is worse ...

Really worse.

 

It is out of all you, mister, who writes exactly the same as I do myself. Not the words, but the how. And that includes he commas the quotes and the ... Do notice though that I don't write like this in this forum. But look at my own.

Now I can tell you that if I examine all your stoopid writings, all I see is myself. Now THAT is worrying, prot !

;)

 

PS: If you are not prot, what's the difference. I myself only see two small differences and they are easily faked. And if not, prot by now is a phenomenon. THE PROT IS BACK !

but then he was from day 1.

[Jud]

- the issue you are talking about is a calibration of the sensitivity of the test - just how small a difference can the test setup, equipment, test subjects, differentiate

 

Both may be able to be addressed by controls but it's difficult to do.

However, without them, the reliability of the results is unkown

 

Mostly agree, except for a little clarification about the "addressed by controls" thing.

 

Let me use an illustration (always a fraught thing, since my example may be more or less apt), which should at least give you a notion of what I'm thinking:

 

You've got a clock, and you want to use it as a timer for an experiment. There are "controls" you can use (get the clock tuned up by a horologist, for example), which should minimize any error. But in order to determine what the baseline timing error may be for such a clock in excellent working order, there is nothing you can do with that clock alone; you need a different timer to compare it to. So consider DBT the clock and a different type of experimental protocol (such as the one I proposed, or any other that will work) as the different timer.

[Jud]

It so happens that we have the rare opportunity of hearing from the person who pretty well invented the DAC as a piece of equipment, Mike Moffat, regarding the subject of this thread. This was posted over on Head-Fi a few days ago:

 

OK everybody – I am in the middle of a lot right now but am driven to comment on “warm” “cold” or whatever adjective anyone chooses to subjectively describe what they hear. Everyone is entitled to their subjective universe. They can be expected to describe their experiences in a way that is real to them. What is right for them may well be BS to another. Also, we are all entitled to our own constructs to argue our own subjective cases, for example electrons taking more time to transverse a coil than a straight piece of wire. This is not wrong to the author; an objective environment, however is far different.

 

Much of this subjectivity overlaid on a discussion/debate of analog (turntables) vs. digital recording playback systems. This is complicated by case to case differences. Is the record (vinyl) a digital recording? Is the digital track or file been converted to digital from analog? Etc., etc., etc.............??? Too many permutations and combinations to arrive at any generalizations.

 

What are the main important differences? Analog is a continuous recording over time, and digital is a sampled, dot to dot recording, which requires reconstruction (no such reconstruction required in analog) upon playback, by information theory rules which primarily live in the frequency domain (emphasis mine). That is why I also utilize time domain (positional) reconstruction methods in the Yggy. To give digital its significant due, there is a huge (at least 20+db) advantage in signal to noise ratio in digital recording and playback devices.

 

Now that is not to say that the frequency domain (response) is not important. A friend of mine back in the 1970's, John Koval, used to argue that the only audible differences between any two pieces of audio gear was due to frequency response (domain). He is the exact reason that the Mani has an extremely accurate RIAA response in 2015. It was his position that all amplifiers essentially sounded alike because of their relatively flat frequency responses. John constructed an A/B sound comparison box that matched levels to blindly compare two amplifiers.

 

Now it was very difficult for me to tell the difference between two amplifiers when they were switched back and forth, particularly when they were precisely level matched (even tube vs. solid state). This was really messing with my manufacturing better sounding amps and preamps. In fact, if all of this Schiit really sounded the same, why was I in the audio biz?? Why was there a market for audio gear, audio shows, etc., etc?? Are audio industry skeptics like Douglas Self really right? Is a 5534 op-amp really unbeatable?

 

So I built one of those A/B boxes with a remote push-button to switch between two amps, preamps, whatever. No matter what I did, I couldn't tell any differences switching back and forth. Quickly I learned that I was not comfortably listening in an interval based switcher. If I listened to an entire LP side or analog series of tracks (an early playlist) on one amp and then the same stuff on the next amp............. I could easily and consistently tell the difference!! I demonstrated it to John and he was amazed. I won plenty of bets with other audiophiles. So it seems that audio listening is an integral, as opposed to differential process.

 

Now this is not to say that our industry has not been blighted by grifters. I will not be in line to buy dog spooge loaded equipment racks due to the amazing hearing ability of canines. But I will add parts per million as opposed to parts per thousand voltage coefficient resistors to my better products because I, for one believe they can sound better. One day, perhaps we will know the precise threshold of the human ear/brain's sensitivity, whether we call it “warm” “flat” or even “curly”.

[Daudio]

I know Dave, I'm not defending this - I'm just saying what the premise is underlying A/B testing

 

mm,

 

Of course. I understood your position, but used your post as a trigger, to post these thoughts, as the mistaken(IMHO) importance of echoic memory to audio has come up quite a bit lately.

 

Cheers,