Golden ears vs super scopes

[pkane2001]

41 minutes ago, SoundAndMotion said:

You're playing a semantic game with the meaning of "can". Does it mean "possible" or "able" (now). With your definition, I'd be able to say:

I can speak Russian. I can practice law. I can go to the moon. 

...but I should hasten to add, that I never have, and cannot today. I can if: someone teaches me Russian; I go to law school and pass the bar; NASA is willing to prepare me and give me a ride.

With my definition, I'd have to wait until I've learned what I need to know.

The question is what "thing" do I want to measure. Without a definition for "thing", I wouldn't know how to use an AP to measure it, and I wouldn't know if the popular measures help me with that. 

I would start with careful listening tests to confirm a predictable, verifiable, repeatable "thing". Then I'd try for an unambiguous, non-controversial name. If current hardware/software is up for it, then a measurement method can be created (possibly). Perhaps new hardware/software would need to be developed. Knowing what and how to measure is not a given, so I'd argue we can't measure everything.

 

-With your definition, yes, if a measurement method exists or can be developed. With my definition, no.

-Perceptual limits is my field. Do we know some (specifically defined) audibility limits? Yes. Do we know all that are necessary to understand perception of reproduced music and the ability of current equipment to do "well"? I don't know, but my experience tells me to truly doubt it.

-For the last 2 questions, I'm reminded of a thread (since deleted?) on WBF. Someone asked "what would it take for you to change your mind about XYZ". I forgot what XYZ was... cables, DACs or something... A small number of people (incl. me) wrote specific answers. Most answers were of the form: "I know I'm not wrong, so the question doesn't make sense". For both groups, the answer of the hardcore will be: nothing!

 

While we may not know every individual component of the distortion and how to measure it, we know quite well how to measure a waveform. Mathematically, we can compute just how close this waveform matches the original one. Empirically, we can subtract the two waveforms and listen to the differences. I don't need to know what TIM or jitter or THD or IMD is to be able to tell if two waveforms measure different, by how much, and whether this difference is audible.

 

One doesn't need to know how to measure individual electrons and forces acting on them to be able to measure current. Just like one doesn't need to know all the forces exerted by individual molecules of gas to determine pressure. In both cases, we know very well how to measure the aggregate behavior of all the underlying elements, even if we don't know what those elements are.

 

The flaw in these "we don't know how to measure something" discussions is that these are against some unknown measure that we have not discovered, say jitter, TIM, THD, IMD, ringing, phase, etc. What is missing is the understanding that we don't need to explain the individual distortions or to have specific tools to separate them from others to measure their effect: we can measure them all at once, in aggregate. Such tools are available today and can be used to measure minute differences. Even if we later discover another distortion source and find a new way to measure it, this will not change the aggregate measurements. It will just explain one more element that goes into the aggregate.

 

[pkane2001]

4 minutes ago, SoundAndMotion said:

 

Hi Paul,

I agree with everything you say, except what I have quoted and bolded. From your work with deltaw, I assume you'll correct this assumption of mine: Always (or nearly so), when you subtract the two (digitized analog) waveforms, the result is never all zeroes. This leads to two problems:

You assume that if you listen to that difference and can't hear anything identifiable, that means you can't hear the difference between the two waveforms. Similarly, you assume that if you generate that delta-wave, and you "know the limits of audibility", you can just declare the difference is inaudible. For any pair of waveforms you've encountered, the assumptions may have been true. But I reject the notion that your assumptions describe the way differential and absolute thresholds actually work. So I reject the idea that you can tell whether this difference is audible with deltaw. I wouldn't be convinced without a listening test. I mentioned that is the starting place for me in my post.

 

So if 2 analog waveforms, distinguishable with a listening test, never null completely, and you can't declare the difference is inaudible, you have not measured the effect.

 

What effect? I'm dangerously close to the argument "if you haven't measured everything and you don't know everything, then you just can't be sure". But I'm quite comfortable walking this tightrope in this way: I do know about the nonlinearities in perception and that superposition does not always apply (not speculation). I do know how nearly all the threshold experiments you use to discuss "audibility" were done (do you?), or perhaps you also do and have a relevant reference. I do know sensory perception is fine-tuned for pattern recognition, not detection or discrimination of simple stimuli or noise(s). The ear is not a good microphone and the eye is not a good camera, but both do their tasks better than their mechanical counterparts. In your favor: most complex stimuli have higher absolute and differential thresholds than simple stimuli. In my favor, not all.

Cheers, SAM

 

I agree the hardcore on both extremes have unshakable beliefs that can be called religion. 

 

Your list (derived from mine) of do-ables don't apply to me or anyone with the same level of training as me. For my list, there are people who are better trained. If an effect that can be verified by listening test has never been measured, then no one knows how or has had the training. It may be easy to measure, or not. But you have to know how/what to measure. The semantic game: I say they can't (without more info); you say they can. Some people will agree with me, some with you. There is no need to play with the words anymore.

 

Hi SAM,

 

Sure: analog captures will never null completely. And audibility is a complex question. What isn't complex is the actual mathematical differences that can be computed and compared to the limits of our audio equipment. If the differences are well below the noise level or the audio equipment ability to record or reproduce, they will not be audible.

 

I agree with you that listening to the null difference between two waveforms may not be a perfect way to determine audibility of distortions. The reasons are varied, from masking effects to different frequency sensitivities to the thresholds of timing/phase differences, to other non-linear effects.

 

While I spent some time researching audibility testing methods, thresholds, and sensitivities out of personal curiosity, I can't claim any professional experience in this area. I've built a few blind testing tools into DeltaWave, so that the two captures can be compared directly. Mostly, this was for me to experiment with. I've also built a number of distortion generation tools into DISTORT, so that I can test the audibility of certain distortions directly, on music material of my choice, using my own audio system and preferred testing methodologies, blind or otherwise. 

 

I've not published these, but I've also built tools to measure hearing frequency thresholds, sensitivity to phase differences between left and right ear, and a few others.  All of this was done so I can explore the subject of audibility in a bit more detail, but also to let others do their own experiments.

 

I accept that the audibility question is complex. But I don't accept that it is impossible to study. Since you have some professional experience in this area, maybe you can help to better formulate the issues with audibility testing and why this area is more complex than many of us think. A better understanding of the problems and issues gives me better ideas about how to go about solving them

 

[pkane2001]

12 hours ago, SoundAndMotion said:

Hi Paul,

I'm reminded of our 2 week long PM exchange in Sept. 2018, which morphed from audibility testing to a discussion of HRTFs. 

Of course I agree with you, otherwise accepting money for my work would be fraud. I'll add that the complexity does not come from difficult to understand concepts and principles (actually not difficult), but from the sheer number of variables that influence the result, thereby producing a set of results. Obtaining a "complete set" is both prohibitively tedious, but thankfully mostly unnecessary. It is important though to use an appropriate value for your specific goals.
 

I didn't reread our entire exchange, but I found a couple of early quotes that may relate here. I said: "My first question is: what is your goal?" Tailoring an answer to your specific goals and background is much more efficient. On the other hand, with about 15 min. prep, I could prepare 1 or 2 hour-long lectures for a group with comparable backgrounds (colleages, grad, undergrad, 4th grade, kindergarten - I've done all of these), but I won't transcribe such a thing now. We could have a very efficient back-and-forth by phone or FaceTime chat, but...
A couple quick tips:
Finding *the* value for *the* threshold of audibility is meaningless. There is no single value; it depends on the type of stimulus (e.g. sine, noise, music, etc.), the method used (e.g. rising, falling, adjustment, 2AFC, etc.), absolute vs. differential, and must account for significant intra- and inter-subject variability. You wrote: "I believe that 'average' results often reported for others don't apply to me, so I tend to want to try to measure things for myself, with me as a test subject." Trying to test all variables, results in a combinatorial explosion.

 

One important misconception is that a sensory-perceptual threshold is binary: you hear it or you don't. It is statistical, not single-valued, and follows a sigmoidal probability density function, called a psychometric function:

 

The cyan line is not correct. You can make it "look" correct if you choose a very large range for the stimulus level scale, but that would be deceptive. BTW, interesting things happen during the rising part of the curve. 

 

Let me know if I can help more...

 

EDIT: I misstated an opinion in a post yesterday that you address in your first paragraph above. There are things that can be considered inaudible, if you are careful with the assumptions made. It is safe to say that both those with normal hearing and those with freakishly, aberrantly good hearing cannot hear a tone at -16dBSPL (3µP).

 

Hi SAM,

 

Appreciate the continued conversation!  While some of the details have changed since our conversation a few years ago, generally my goal remains the same.

 

Simply stated, I'm after the most realistic audio reproduction possible. The number of variables is large, which is why I've  been taking the long, scenic route to get there, exploring, but also trying to narrow down the things I need to look into. I learn better through this sort of random walk, where I discover things that might or might not be relevant, often going off on tangents, but I'm a firm believer these will give me a better understanding in the long run, or at least provide a different perspective. 

 

Testing for audibility is, of course, a large part of this journey. And perhaps that's where you can help. To simplify the discussion, let me just propose a list of questions that I've formed over the last few years. I've seen some studies with attempted answers, but nothing I can accept as definitive. I've done a large number of ABX tests, but I'm always looking for better ways to test. Sorry to dump all of these on you, and if a voice or video chat works better, we can do that, as well 

 

1. Are there any test protocols that you've feel work best for minor impairment detection/discrimination testing? ABX, Triangular, paired, MUSHRA, etc.? Any new ways that have been proposed recently? As an example, someone suggested playing the two DUTs in stereo (one left the other right ear). I didn't find this test extremely good at discrimination, but perhaps it might be good for detecting phase anomalies.

 

2. Are discrimination tests best conducted with very short snippets and fast switching, or long-term evaluation? (long term here meaning anything over 10 seconds. Sometimes weeks are recommended by our subjective brethren)

 

3. Is there an issue with fatigue in fast-switching tests that must be dealt with, and what is the 'threshold' when that sets in (# of repetitions, length of time)? 

 

4. Are qualitative tests better than the binary discrimination ones in detecting differences? 

 

5. Any other variables in such testing that we, audiophiles, either don't know about or ignore that can have a large influence on the outcome of the test? I know, there are a huge number of variables. Can you list say the top 5 that have the most influence on the outcome?

 

Looking forward to your answers!

[pkane2001]

6 hours ago, SoundAndMotion said:

Hi Paul,

For a handful of reasons, I'm not in the "right space" to thoroughly and thoughtfully answer your post in written form. I've sent a PM to see if a voice chat might work. Perhaps you or I can/will record it and create a nice post. I'm totally with you on taking the scenic path. That is the path that I and every colleague/friend of mine has taken. The shortcut is overrated: very often muddy and full of bugs, thorns, poop and poison ivy.

But just quickly: I asked "what is your goal?" and you say testing for audibility with the ultimate goal of the most realistic audio reproduction possible. I was unclear. What exactly, specifically is your goal? For example: I want to test my own absolute threshold for SINAD, that is created and varied using this specific method-..., is an entirely different goal from: I want to find the range of estimates of "normal" humans for the angular width of the soundstage using a specific musical sample. And my answers below would need to take that goal into account.

 

What exactly is my goal? Hard to define it beyond "the most realistic sound reproduction possible"

 

My path has been to eliminate the obvious errors in reproduction, first. As we already discussed, there's never going to be 100% perfect analog reproduction, so by necessity, we must deal with some lower threshold to know when an error can be safely ignored. Audible thresholds for THD, IMD, FR, jitter, TIM,  etc. I'm not so much interested in the absolutely lowest possible threshold. If there are a couple of people on this earth that can hear THD at 0.0000001% level, I'm not all that concerned as long as I can only hear it to 0.1% 😄

 

Quote

1 - It really, really depends... the idea @Jud used of separating samples to each ear is a trade-off (see @sandyk response above). I participated in his experiment, but found it to be 1 creative step forward, but two back.

 

Yes, Jud convinced me to add this as a testing method into DeltaWave. To tell you the truth, I didn't find that it was an improvement over standard A/B/X, but seemed like an interesting idea worth investigating. Maybe this test would be more sensitive to large phase differences.

 

Quote

2 - Depends on the variable you choose to measure. snippet time and switch times are not the same. Simple detection, like audibility of measured specs (THD, FR, etc.), probably requires echoic memory, so a short switch time. But it is easy to come up with variables/tests that would require longer listening times and greater immunity to long switch times.

 

Common wisdom here is that the echoic memory is short, so it is impossible to compare A to B if one no longer has the details of the sound in their short-term memory. So, does the long-term listening test actually produce usable results? Results that identify differences consistently? Anecdotally, through my own testing, I find shorter snippets and short switch time to be much more sensitive than long term comparison. But I also don't always find the differences consciously. I've passed a number of blind tests by forming an overall impression of a short, 10-second comparison between snippets rather than trying to consciously decide if they are different. I failed the same test trying to consciously make that same distinction. So, can a long-term test be more sensitive, while remaining accurate?

 

Quote

3 - There is a fatigue issue for all tests: avoid it! There is no threshold that I know of; it is equivalent to the question of how long can people pay attention. If a test is boring, people must fight to not "drift away". But similarly, if a task is pleasurable, people must fight to focus on the task and not just sit happily listening. I always require the subject announce they are ready for the next trial (allowing zero to several seconds or a minute of self-controlled break, to gather themselves). I force a few minute (5+) break every 15 minutes. I stop for at least a half day, after about 1-1.5 hours testing. My experiments are "lots of fun" for the first 1 or 2 dozen trials, but I usually need 100-300 trials, so it gets boring and I must query and push for alertness and attention.

 

That's great info! The other part I meant to ask about: does our memory and learning ability stop us from being able to distinguish A from B if repeated more than a number of times? In other words, does our mind fill in gaps and the details and smoothes out the differences by learning what to expect? If so, perhaps switching frequently between different A/B snippets would be a way to avoid that.

 

Quote

4 - "in detecting differences" No, because of your stated goal. If I can easily detect the difference, but my preference depends on the snippet of source material (stimulus), I may answer in an apparently random fashion. It is a confound. 

 

That makes sense, also.

 

Quote

5 - My top 4, for those forgotten, misunderstood or taken for granted...
    Instructions (depending on goal; super-important for me).
    Subject selection. 
    Method, including what statistics will be used: know before start.
    Stimulus selection.   
    Others are also important, but are often thought of: bias control, calibration of all measures, subject-comfort...

 

Excellent! These all make sense. While I'm doing a lot of testing on myself I also have been involved in a few internet blind-tests. It's always a challenge to set these up correctly and to make sure the results are meaningful and interpreted the right way. The worst part is the lack of any control over how the test itself is conducted.

 

 

[pkane2001]

SAM (@SoundAndMotion) and I spent a good hour chatting on Skype yesterday about testing, audio thresholds, and experiments. A number of my questions were answered, and I really enjoyed the conversation. In-person (even virtually) works so much better than exchanging forum posts!  Looking forward to future discussions. Thank you, SAM!

[pkane2001]

3 hours ago, The Computer Audiophile said:

So cool. Thanks for sharing.

 

I wish we could record stuff like that for others to see and learn. 

 

I'll see if I can put together a short summary and run it by SAM before posting. We could've easily gone for another hour if the younger generation didn't start complaining about us hogging up the internet 🤨