Misleading Measurements

[SoundAndMotion]

If minimal- or zero-urine manufacturing is desired, perhaps “piss-poor manufacturing” is a good thing... No?

[SoundAndMotion]

Validation is an important issue. But I fear the calls for it are not done in good faith. I hope I'm wrong.

 

The punchline: after reading the last several pages, I downloaded DISTORT to play with it. At this stage, from posts in this thread, I'm interested in the harmonic distortion part (not jitter or noise... yet!). I validated it for my own needs, and I'm confident I could validate it for yours (if you are reading this, that means you!). In a moment, I'll tell you why I probably won't do that.

 

First, let's go over 4 possible validation methods:
1- DISTORT is essentially self-validating. You specify the nature of the distortion from presets (or your own creation/measurement). You choose a test signal from presets (or use your own). You look at the built-in spectrum (or do your own). Voilà! If the spectrum includes everything you ask for and nothing else, you're done.
2- Paul mentioned others had independently validated it. I really doubt a post exists that says: "My name is ... My credentials are ... I hereby certify that DISTORT is validated." I assume Paul inferred from exchanges like this:
User: Paul, how can I XYZ?
@pkane2001: In DISTORT, you ABC.
User: C?
@pkane2001: You can use a preset, or FGH.
User: That worked! Thanks.
3- Get rid of all the parenthetical "or"s in 1, and use "and"s. Do everything on you own AND with built-in functionality, and compare. You then validate all the built-ins AND the design goal.

 

The problem with these 3 is I can already anticipate the complaints/perceived problems. That leaves only:
4- You specify exactly what validation looks like to you... exactly. If you need UL or ISO certification, maybe apply ISO 900003 to get an ISO:9001:2015, forget it, you are beyond ridiculous. If you ask for a DBT, I'll pull out the hypocrisy card, since you don't make the same demand for the ER.
Also DISTORT is intended for others to do their own blind testing, if desired.

 

Anyone can do 1. Everyone can trust 2 or not. I can do 3, but doing the work, writing it up with reasonable and clear graphic support, and presenting it here is a lot of work. I'm not willing to put in an hour or or more if there is a likely 10 sec response "I don't buy it." So that leaves 4. I'm willing to do 4 ("exactly"), if it's reasonable and those who request it will accept it. 
 

[SoundAndMotion]

@bluesman I like the way you approach finding answers. You pick up your guitar, record it, and make your points. Using different tools, I often take a similar approach.

 

I have said for years, I'm an open-minded skeptic (similar to the Sagan quote above). So, a bit of the friendly skeptic part: I don't (yet!) see evidence of sum and difference frequencies that you mention, other than the linear effect of beats. Your recording doesn't help me (yet). Can you help me? I'd like to work through this with you to understand the sum and difference vs. IMD issues.

 

Some comments on your recording:
- I don't like the Spectrum function in Audacity (which I otherwise love), so I use my own.
- I don't find any "notes" (frequencies) near 246Hz
- I find 2 peaks at about ~221+ and ~226+Hz, giving the noticeable ~5Hz beats. I looked at the 2nd, 3rd and 4th harmonics: they are there, but I don't see anything at (F1 + F2) frequencies.
- There is not any signal near 5Hz (F2-F1)
- There's a lot of "stuff" between 20-34Hz, with a peak at 26+Hz. I have no idea about the origin.
- "Something" changes at about 1.0s. It is audible and visible as a sudden amplitude drop. Any ideas? Could you have adjusted your hand, or...?
- All the "stuff" around 26Hz is gone after 1.5s
 

[SoundAndMotion]

3 minutes ago, pkane2001 said:

But I already mentioned the ASR thread on testing and validating of DISTORT while it was being developed (still is, really) easy to look up and find what testing was done and what results were produced by me and by others. 

Hi Paul,

I understand, have seen the other thread, and will have some questions for you there.

But here, keep in mind you and @Audiophile Neuroscience are stuck in the same place I am with my cousin. I'm waiting for her to reach out to me, and she is doing the same in reverse. You want AN to put in some effort (define, read/search, try out), while he is waiting for you to put in effort to provide evidence of validation. It won't work. Either one of you has to flinch first (I won't hold my breath) or others will need to jump in. I attempted above.

Cheers, SAM

[SoundAndMotion]

2 hours ago, pkane2001 said:

Welcome to the loop (get out while you still can )!

Hi Paul,

 

I understand your perspective, and given the complexities of online relationships, I don't wish to convince you of anything

 I've interacted a few times with AN also, even before "the big break", sometimes very satisfyingly. He does have a few issues upon which he chomps down like a pit bull and won't let go - even if you turn the hose on him. Like validation. If it is done in good faith, he reminds me of a couple of reviewers I've dealt with a few times. They have issues they chomp down on, but even after a bit of swearing and name-calling during the rewrite/resubmit phase, I have to admit that the final product (the paper) was better after dealing with the "issues".

In this thread, the question of good faith arrises, and I hope giving the benefit of the doubt is warranted.

SAM

[SoundAndMotion]

5 minutes ago, bluesman said:

 What we still don't know is what else it may be doing to the signal passing through it.  That is not an unreasonable thing to want to know.

 

 

[snip]

 

 

So, specifically, what I'd like to see is a list of the other effects on the source signal that were considered possible and shown not to occur with use of the app in question.  This is a very reasonable thing to ask - it's neither pejorative nor argumentative.  It's just a factual inquiry that deserves an answer.  Since I didn't get one and I want to try the app myself, I'll try to determine the answer myself.  And I promise to try hard to avoid misleading measurements.

I guess you didn’t see post#776. If you answer mine, I’ll answer yours.

[SoundAndMotion]

15 minutes ago, bluesman said:

I read it - I just don’t see anything in it that relates to my question in any way at all.  Confirming the presence of desired effects is part of validation.  So is confirming the absence of undesired effects.  Surely a skilled, experienced, knowledgeable, creative engineer could think of at least one effect of adding a nonlinear twist to that straight wire with gain.

I'm not trying to debate you. If it seems so, I apologize. I'm trying to understand you and understand why you don't understand me. I tried to address both confirming the presence of desired and absence of undesired effects. FYI, I don't know if you meant me, but I'm not an engineer. And yes, adding a nonlinearity will have more than one effect, but it is, IMO, beyond the scope of determining the correct/incorrect functioning of software to explore every ramification of a nonlinearity.

 

15 minutes ago, bluesman said:

There’s a lot of excellent audiology literature on the effects of nonlinear processing on sound quality, most of it from research into hearing aid design and function.  Compression is very well studied as a nonlinear distortion of auditory input - and it’s well known and validated with good solid metrics that compression affects perceived sound quality.  So one might reasonably ask if an app that adds nonlinear processing to a vocal or musical data stream changes its dynamic range at all.  If it does, how?  Is it an across the board attenuation? Is it frequency related? Amplitude related? Random?

DISTORT seems to be a companion tool for those who wish to look into this stuff, perhaps even research it. I can't imagine you expected Paul to do all the research to confirm that the research results can be obtained with his app. Compression will add harmonic distortion, and adding HD of a certain type will cause compression. Which "certain types"? Those that create a symmetric, sigmoid (s-shaped) time-domain transfer function. Those will compress.  I believe that means no even harmonics, only odd, but I could be mistaken. My urge is to check it with a tool... my own or something like DISTORT.

So, yes, use DISTORT to create a "compressor"-like transfer function and study it. Or use Audacity's compression tools and see what harmonic distortion is created. Same issue from different sides. A fun project (for me; I'm not giving orders!) would be to use the Audacity compressor, measure the harmonic distortion components, enter those components in DISTORT and see if the resulting file is identical to that from Audacity!

Anyway, to repeat: I think your questions can be answered by DISTORT. They are not part of a test of DISTORT.

Peace.☮️

[SoundAndMotion]

@bluesman I realize this is frustrating for you, but I think we're about to punch through a communication barrier.

 

What is missing in your example is that anomaly X and the compression are both manifestations of the same thing: a particular nonlinear time-domain transfer function. If you want to add anomaly X, you have to be aware that the compression comes with it.

If you get a dog, fully aware that you'll have to feed it, don't be surprised that you also have to clean up the poo. It's part of the same process. If you didn't know, don't blame the kennel or the dog food company.

I'm sure Paul knows that creating a compressor and getting harmonic distortion go together.

I disagree that it is a reasonable conclusion that hearing a difference means it is due to anomaly X. You can't expect to create anomaly X without knowing that your method also added compression. They are, in this case, inextricably linked.

Paul has created a nonlinear time-domain transfer function generator. He lets you set it up by specifying the harmonic content. No one can offer you an anomaly X generator that doesn't also add compression, if anomaly X is addition of certain harmonic distortions.

 

I think I understand what you are saying. I hope you're not so frustrated that you won't try to understand what I'm saying.

[SoundAndMotion]

On 7/21/2020 at 3:28 AM, Audiophile Neuroscience said:

Well, I will have to just survive the veiled slurs on my motivations and character #776 #780 #781 but that aside nothing here or in the other dedicated thread changes one inconvenient truth.

 

In the spirit of an audiophile forum, let's "lift the veils".

Speaking as the writer of #776 and #781, I can tell you that there was no desire to slur your motivation or character, but rather to question your motivation, albeit giving you the benefit of the doubt. I admit "chomps down like a pit bull" seems a slur on your methods - I'm sorry - I should have used "relentlessly persistent", and I regret the offensive tone. 

When it comes to controls and validation, you strongly and persistently argue the case for perfection. What is "perfect" should never be forgotten in discussions of experimental design (or software development). But realizing that in some cases "perfection is the enemy of the good" means often choosing not to halt progress, even if that means good methods over perfect methods.

I have always assumed that the case you argue, even if extreme, is motivated by principle. I would call that "good faith". With that assumption, I expect to see the same argument with those you like and those you dislike. It is clear you dislike Paul.
I assumed a good faith motivation, and that is supported by, IIRC, your never being reticent in describing exactly what you mean and expect in the case of controls. This thread has challenged my assumption, because you don't describe what validation looks like to you, even though that is essential to providing validation.

 

I understand the need for validation in some cases: it was a big part of my creating a multi-sensory perception model a few years ago. I validated first by defining within which range of stimuli I would test, and then simulating standard published stimuli groups. I compared my model's output to the published perceptual responses; it was not identical, but demonstrated all the necessary features and was considered by others a success. It did not check for out-of-range inputs, so would easily provide GIGO (garbage in, garbage out).
I used Simulink. The MathWorks validated the mathematical performance of Simulink, but would never consider validating it for perceptual models. I had to know what I was asking of it and set the simulation parameters (time step, ODE solver, etc.) correctly. I had to validate my choices myself, and did so successfully.

DISTORT is analogous to Simulink. Paul (and others) have validated the basic mathematical performance, but not whether every person's uses and choices are correct. If you try to calculate your taxes with it, watch out! If you read all of #776, and got past the "good faith" phrase at the beginning, you'll see I offered to validate it for you and document it appropriately, if you spell out what you mean. (I also put the word "reasonable" in there. I won't spend a month on it... all subject to discussion).
Peace 🕊️☮️

[SoundAndMotion]

2 hours ago, 992Sam said:

 

look man, I know what I saw.... just go find the PS Audio DirectStream DAC review thread and you'll find at least 3 banned users.. all of them dared try to defend the DAC they felt sounded good against the measurement nazis in that thread including Amir.   I saw the same all over the board while lurking... 

hey man, @danadam wasn't calling you a liar... he just asked for examples. Some people, like me, don't automatically believe everything they read on the internet, especially interpretations of "what happened". It's nice to interpret for oneself.

 

Disclaimer:I don't know how to be a good moderator for internet forums. I notice on several forums, including this one, that some people are banned quickly after what I would consider minor infractions (although there could be a backstory I don't know), while other repeat-rule-violators are tolerated.

 

That said, I went to look at the PS Audio DAC thread, as you suggested. I found:
- With over 1500 posts, I wasn't willing to look at all of them. I perused here and there (beginning, middle, end of thread) and didn't see anyone labelled "Banned", but I found 2 users who were banned.
- One was a "Major Contributor" with over 4000 posts who seemed to be banned around February. I could not figure out why he was banned, but it was clear he was a fan of measurements, but also sometimes a bit abrasive.
- The other was you. You know why, and the reason you were banned was *not* an unwillingness to toe the pro-measurement line. People here have been banned for the same reason you were banned there.

[SoundAndMotion]

2 minutes ago, The Computer Audiophile said:

Examples please. 

I'm not going to start an open fight with you. I'll PM you a couple examples, and you are free to post them. As you know, or can check, I've praised your moderation (but included some bafflement). Also, read my disclaimer.

[SoundAndMotion]

44 minutes ago, stereo coffee said:

But normal CD players do not have such a feature to increase level.   

 

Out of curiosity, how many CD players have you tested/measured? Do you think Stereophile has perhaps tested more than you?

See what they say here (link):

Quote

The standard CD-player output voltage is 2V RMS, with units varying between 1.74V on the low side (the Audio Research DAC1) and a whopping 7.2V on the high side (the Theta DS Pro Basic). Most CD players and processors put out between 2.2V and 3.5V. Note that this value is the highest RMS output voltage possible from the player—there's no digital signal greater in amplitude than 0dBFS (see the "Decibels" sidebar).

 

[SoundAndMotion]

It might help to not spend too much effort understandIng AES17-1998 given the Forward of AES17-2015 states:

Quote

This document substantially revises and updates AES17-1998

and the Forward of AES17-2020 states:

Quote

This document clarifies the definition of levels, the units FS and dBFS.

I don't have the new document (nearly twice as long as the 1998 standard), but perhaps all is made clear!?