Why I never got that perfect null in difference testing

[Julf]

Barry,

 

In accordance with your wishes I won't quote any parts of your message, despite that being the normal way of communication on a forum like this.

 

My comment about nulls in radio and electronics was in response to your possibly somewhat condescending comment to Dennis about null tests being new to him. My point is exactly the point you are making - that pro audio, a small subset of the application space of electronics, is a relative newcomer to the field of "nulls", while the larger world of electronics and radio is very much familiar with the concepts and ideas.

 

In order to be fair to the Dutch, I want to point out that I am not Dutch, but I give the Dutch communication culture a lot of credit for being direct and fair, with a preference for facts and logic over prejudice and recourse to perceived authority.

 

On that note, I think it is fair to declare this particular sub-thread more than done, but, of course, leave the door open for someone else to have the last word.

[Jud]

Dennis, you said you were looking forward to comments on part 3, and I wanted to leave one, but found no way of doing so. Probably just me being thick, but just in case, thought I'd bring it to your attention.

 

Anyhow, the comment I wanted to leave was this: What I would very much like to see, and haven't, is assumption testing: A real-world proof of the proposition that audible differences between two components of an audio system must show up in null testing of the kind you are doing. What I am thinking of is an experiment taking two components (cables, amps, pre-amps, bit perfect software players, or what-have-you) between which you can detect audible differences in blind tests, and running your null test on them.

 

While it is not the sort of experiment I suggest above, I do find it interesting that you saw the least difference between two different cables, rather than between two different "runs" on the same cable. Was this consistent over repeated runs?

[esldude]

Dennis, you said you were looking forward to comments on part 3, and I wanted to leave one, but found no way of doing so. Probably just me being thick, but just in case, thought I'd bring it to your attention.

 

Anyhow, the comment I wanted to leave was this: What I would very much like to see, and haven't, is assumption testing: A real-world proof of the proposition that audible differences between two components of an audio system must show up in null testing of the kind you are doing. What I am thinking of is an experiment taking two components (cables, amps, pre-amps, bit perfect software players, or what-have-you) between which you can detect audible differences in blind tests, and running your null test on them.

 

While it is not the sort of experiment I suggest above, I do find it interesting that you saw the least difference between two different cables, rather than between two different "runs" on the same cable. Was this consistent over repeated runs?

 

No it wasn't consistent with repeated runs. It just happened that of the dozens of files I have run the pair that null out the best were two different cables. And it appears the reason this pair of files nulls out best is they just randomly had the smallest timing difference. Other runs between this same pair of cables have a larger difference which also seems to be a randomly worse timing match.

 

At least with two and three tone test signals I have been able to match level to the nearest .000001 db. With music that is harder to do though I have taken some as close as I can get and then alter them by tiny amounts first one way and then the other to see which nulls better. Using this tedious approach I could get within .00001 db or a bit less. So with test tones I can get levels right. They can be gotten right with all the cables I have tested. So at least with general level matching the cables don't appear to matter. Two signals different by .000001 db will null to -137 db from full scale.

 

When I match levels there is still a difference signal with a null of -110-115 db generally. On the best one I had it was better than -125 db and as the original signal was -9db that put the result around -134db which is about -3db above the noise floor. When you still have higher residuals with matched levels it is due to phase shift between the compared files as a result of a timing difference. The tip off is when the residual level differs by 6 db when the signals are an octave apart which is what I have seen.

 

So to recap, with music or test signals I typically get a null between -95 and 105 db. The gain of the equipment varies a bit over time a worst difference of .0003 db though usually less than half that. When I correct for gain differences the nulls drop to -110 db to -115 db with a now and then result above -120 db. It appears what is left is a timing difference which best I can tell varies generally between 200 picoseconds and a nanosecond. The amount of timing difference seems random whether with different cables or the same cable. Which is why it looks like the cable isn't a contributing factor at levels higher than timing or gain shift.

 

So thus far, doing IMD tests, the cables don't matter, doing frequency response they don't matter, doing these nulls tests they don't matter.........so what else is there? Timing, frequency, and level are what you have digitally. Even going through an analog pass it looks like minor level shifts not due to the cable show up, ditto for timing, frequency seems fine.

 

Now the only thing I think left for cables is some people who find them abundantly obvious to do a well organized blind test in a format they are comfortable with. Regardless of any technical stuff, if people can differentiate cables something is happening.

 

Now I suppose what you are asking for is confirmation that difference testing works. Not sure what would satisfy you. I mean a relatively small droop in frequency response has been proven audible blind. Such a thing is abundantly clear in a difference test. Though I haven't done such testing, THD of tones has been shown audible in blinded listening at around the .1 % level and above. I just created that here then took a difference and you can hear the difference result clearly even over small computer speakers on my desktop. It is generally accepted that a pure level difference has to be a bit above .1 db to be audible blind. I just created and differenced two files by .01 db and the difference result is low in level, but clearly there. A .1 db difference is not even hard to hear as a difference file.

 

Your question makes good sense. What we all would like is to know what is the most discriminating method of detecting genuine differences in audio. Is listening by humans more discriminating than testing with difference testing or other conventional testing? Our first problem is which method of human listening is best. One camps says long term sighted listening trumps blind tests. Logically the blind testing seems the better.

 

My opinion is blind testing is more reliable once you get near the thresholds of audibility. Difference testing is clearly able to show differences much smaller than what audibility limits are shown to be by psycho-acoustics. Not that all possible limits are fully explored, but we can draw a reasonable envelope around what people can be shown to hear. You have a problem of relating levels of audibility in difference testing to accepted psycho-acoustic limits. On the other hand if you get difference signals down in or near the noise floor I don't think you have to wonder. If you have that level of performance you can assume it is subjectively transparent. Clearly many folks don't agree.

 

Now I have years back done the old style of difference testing with amplifiers. Take some headphones connect one lead to the positive of one amp and negative to the positive lead of the other. Load them with power resistors and hear the difference. Getting gain between them exactly right was tough. With tubed vs SS there was lots of difference audible. With solid state some of those could get pretty darn close. Now add speakers in place of power resistors and even SS amps had some considerable differences. But other than close, not close or a big difference I didn't quantify them. Always had in mind feeding that to a tape recorder, but never did that. People who have measured such report best nulls of -70db with speakers. Though not that common from the reports. My opinion on that is a -60 db null you can probably be comfortable is enough you won't hear them as different. Just for good measure to be safe I think if you get -80 db nulls to the speakers you could consider them subjectively indistinguishable. Maybe I will get around to doing some such measured testing later.

[esldude]

Looks like I didn't turn on comments for parts two and three. I have fixed that now. Sorry about that oversight.

[Speedskater]

Have you tried testing two different cables at the same time? One into the Left Channel ant the other into the Right.

[esldude]

Have you tried testing two different cables at the same time? One into the Left Channel ant the other into the Right.

 

Yes, I have. Level variation from channel to channel is worse. You can compensate for it, and then you are left with those phase shifts from timing and roughly the same level of nulls.

 

That does bring something to mind. I could use a mono adapter at first one end and then the other to see which end contributes most to those level variations, and I suppose timing variations as well.

[Julf]

Dennis,

 

thank you for the open and unbalanced reporting of your excellent experimental work.

[Jud]

Umm, Julf - I think you meant to say "balanced"? ("Unbalanced" refers in English, in the context in which you used it, to mental state, so you are thanking Dennis for his insane reporting. :-D)

[Julf]

Umm, Julf - I think you meant to say "balanced"? ("Unbalanced" refers in English, in the context in which you used it, to mental state, so you are thanking Dennis for his insane reporting. :-D)

 

Woah, yes! Meant unbiased. But balanced is good too. Thanks for noticing the linguistic mixup!

[esldude]

I don't know, I think there are some folks around who get paid for their unbalanced reporting.

 

But I thought that Julf's comment was just a slip up. Anyway, thanks guys.