Misleading Measurements

[SoundAndMotion]

Validation is an important issue. But I fear the calls for it are not done in good faith. I hope I'm wrong.

 

The punchline: after reading the last several pages, I downloaded DISTORT to play with it. At this stage, from posts in this thread, I'm interested in the harmonic distortion part (not jitter or noise... yet!). I validated it for my own needs, and I'm confident I could validate it for yours (if you are reading this, that means you!). In a moment, I'll tell you why I probably won't do that.

 

First, let's go over 4 possible validation methods:
1- DISTORT is essentially self-validating. You specify the nature of the distortion from presets (or your own creation/measurement). You choose a test signal from presets (or use your own). You look at the built-in spectrum (or do your own). Voilà! If the spectrum includes everything you ask for and nothing else, you're done.
2- Paul mentioned others had independently validated it. I really doubt a post exists that says: "My name is ... My credentials are ... I hereby certify that DISTORT is validated." I assume Paul inferred from exchanges like this:
User: Paul, how can I XYZ?
@pkane2001: In DISTORT, you ABC.
User: C?
@pkane2001: You can use a preset, or FGH.
User: That worked! Thanks.
3- Get rid of all the parenthetical "or"s in 1, and use "and"s. Do everything on you own AND with built-in functionality, and compare. You then validate all the built-ins AND the design goal.

 

The problem with these 3 is I can already anticipate the complaints/perceived problems. That leaves only:
4- You specify exactly what validation looks like to you... exactly. If you need UL or ISO certification, maybe apply ISO 900003 to get an ISO:9001:2015, forget it, you are beyond ridiculous. If you ask for a DBT, I'll pull out the hypocrisy card, since you don't make the same demand for the ER.
Also DISTORT is intended for others to do their own blind testing, if desired.

 

Anyone can do 1. Everyone can trust 2 or not. I can do 3, but doing the work, writing it up with reasonable and clear graphic support, and presenting it here is a lot of work. I'm not willing to put in an hour or or more if there is a likely 10 sec response "I don't buy it." So that leaves 4. I'm willing to do 4 ("exactly"), if it's reasonable and those who request it will accept it. 
 

[SoundAndMotion]

@bluesman I like the way you approach finding answers. You pick up your guitar, record it, and make your points. Using different tools, I often take a similar approach.

 

I have said for years, I'm an open-minded skeptic (similar to the Sagan quote above). So, a bit of the friendly skeptic part: I don't (yet!) see evidence of sum and difference frequencies that you mention, other than the linear effect of beats. Your recording doesn't help me (yet). Can you help me? I'd like to work through this with you to understand the sum and difference vs. IMD issues.

 

Some comments on your recording:
- I don't like the Spectrum function in Audacity (which I otherwise love), so I use my own.
- I don't find any "notes" (frequencies) near 246Hz
- I find 2 peaks at about ~221+ and ~226+Hz, giving the noticeable ~5Hz beats. I looked at the 2nd, 3rd and 4th harmonics: they are there, but I don't see anything at (F1 + F2) frequencies.
- There is not any signal near 5Hz (F2-F1)
- There's a lot of "stuff" between 20-34Hz, with a peak at 26+Hz. I have no idea about the origin.
- "Something" changes at about 1.0s. It is audible and visible as a sudden amplitude drop. Any ideas? Could you have adjusted your hand, or...?
- All the "stuff" around 26Hz is gone after 1.5s
 

[pkane2001]

45 minutes ago, SoundAndMotion said:

Validation is an important issue. But I fear the calls for it are not done in good faith. I hope I'm wrong.

 

The punchline: after reading the last several pages, I downloaded DISTORT to play with it. At this stage, from posts in this thread, I'm interested in the harmonic distortion part (not jitter or noise... yet!). I validated it for my own needs, and I'm confident I could validate it for yours (if you are reading this, that means you!). In a moment, I'll tell you why I probably won't do that.

 

First, let's go over 4 possible validation methods:
1- DISTORT is essentially self-validating. You specify the nature of the distortion from presets (or your own creation/measurement). You choose a test signal from presets (or use your own). You look at the built-in spectrum (or do your own). Voilà! If the spectrum includes everything you ask for and nothing else, you're done.
2- Paul mentioned others had independently validated it. I really doubt a post exists that says: "My name is ... My credentials are ... I hereby certify that DISTORT is validated." I assume Paul inferred from exchanges like this:
User: Paul, how can I XYZ?
@pkane2001: In DISTORT, you ABC.
User: C?
@pkane2001: You can use a preset, or FGH.
User: That worked! Thanks.
3- Get rid of all the parenthetical "or"s in 1, and use "and"s. Do everything on you own AND with built-in functionality, and compare. You then validate all the built-ins AND the design goal.

 

The problem with these 3 is I can already anticipate the complaints/perceived problems. That leaves only:
4- You specify exactly what validation looks like to you... exactly. If you need UL or ISO certification, maybe apply ISO 900003 to get an ISO:9001:2015, forget it, you are beyond ridiculous. If you ask for a DBT, I'll pull out the hypocrisy card, since you don't make the same demand for the ER.
Also DISTORT is intended for others to do their own blind testing, if desired.

 

Anyone can do 1. Everyone can trust 2 or not. I can do 3, but doing the work, writing it up with reasonable and clear graphic support, and presenting it here is a lot of work. I'm not willing to put in an hour or or more if there is a likely 10 sec response "I don't buy it." So that leaves 4. I'm willing to do 4 ("exactly"), if it's reasonable and those who request it will accept it. 
 

 

Thank you SAM! You have it exactly right. But I already mentioned the ASR thread on testing and validating of DISTORT while it was being developed (still is, really) easy to look up and find what testing was done and what results were produced by me and by others.  A recent blind test was conducted by Archimago using DISTORT with some very interesting results with nearly 70 test subjects reports and analysis.

 

But I'm really not clear as to what other kind of validation @Audiophile Neuroscience is seeking. I've started another thread dedicated to DISTORT discussion, since it really doesn't belong here: DISTORT is not measuring anything.

[SoundAndMotion]

3 minutes ago, pkane2001 said:

But I already mentioned the ASR thread on testing and validating of DISTORT while it was being developed (still is, really) easy to look up and find what testing was done and what results were produced by me and by others. 

Hi Paul,

I understand, have seen the other thread, and will have some questions for you there.

But here, keep in mind you and @Audiophile Neuroscience are stuck in the same place I am with my cousin. I'm waiting for her to reach out to me, and she is doing the same in reverse. You want AN to put in some effort (define, read/search, try out), while he is waiting for you to put in effort to provide evidence of validation. It won't work. Either one of you has to flinch first (I won't hold my breath) or others will need to jump in. I attempted above.

Cheers, SAM

[pkane2001]

1 hour ago, SoundAndMotion said:

Hi Paul,

I understand, have seen the other thread, and will have some questions for you there.

But here, keep in mind you and @Audiophile Neuroscience are stuck in the same place I am with my cousin. I'm waiting for her to reach out to me, and she is doing the same in reverse. You want AN to put in some effort (define, read/search, try out), while he is waiting for you to put in effort to provide evidence of validation. It won't work. Either one of you has to flinch first (I won't hold my breath) or others will need to jump in. I attempted above.

Cheers, SAM

 

SAM,

 

Welcome to the loop (get out while you still can )!

 

This one, between AN and I, was started many years ago, long before DISTORT. By nature I'm a skeptic and so I ask questions, just like AN does. But that's where we differ: I don't ask questions just to prove the other side wrong. I ask questions because I'm really interested in finding and understanding the answer, and if I don't believe or understand the answer, I look for ways to learn and verify.  That's why I created DISTORT, that's why I created DeltaWave.

 

I have a couple more apps I'm working on now, and have a few more planned for when I'm done. These tools I create primarily to help me understand the issues and not to have to rely on others to tell me.  As I said, I'm a skeptic, and I learn by doing, not by arguing.

 

That said, I like when interesting questions are being asked. The one by @bluesman, for example, is interesting to me since it intuitively makes some sense, but I have not seen it being studied or tested. And since it appears to be something that could be tested, I am curious. 

 

[SoundAndMotion]

2 hours ago, pkane2001 said:

Welcome to the loop (get out while you still can )!

Hi Paul,

 

I understand your perspective, and given the complexities of online relationships, I don't wish to convince you of anything

 I've interacted a few times with AN also, even before "the big break", sometimes very satisfyingly. He does have a few issues upon which he chomps down like a pit bull and won't let go - even if you turn the hose on him. Like validation. If it is done in good faith, he reminds me of a couple of reviewers I've dealt with a few times. They have issues they chomp down on, but even after a bit of swearing and name-calling during the rewrite/resubmit phase, I have to admit that the final product (the paper) was better after dealing with the "issues".

In this thread, the question of good faith arrises, and I hope giving the benefit of the doubt is warranted.

SAM

[Patate91]

7 hours ago, Audiophile Neuroscience said:

 

You seem to be saying that this App equates to using a hammer or it's truth is self-evident, no objective data required, correct?

 

.....I see our posts have crossed and now you are saying something different about hammers! Objective data is useful.You have changed tacks to pre-requiste knowledge.....

 

Yet So, Paul who invented the App says his Apps "do exactly what they were designed to do and have been validated through independent testing".

 

So, in an objective forum I am asking any objectivist (or non objectivist) for the objective data that tells me it does indeed  "do exactly what they were designed to do and have been validated through independent testing".

 

This is separate and distinct from understanding how the App does it, or any other specific knowledge, or how to use it. "Independent testing" is by its very nature , well, independent of my understanding or knowledge and does not rely on my expertise or lack thereof, to validate the App.

 

Is that so hard to understand?🤷‍♂️

 

I suggest we have an intermission to let others have a say (if they're interested)

 

 

 

You'll have to learn how to code an application to understand how it works, this is out of scope of the thread and "irrevelant" for now. 

 

Like it has been said earlier the app has already been "validated" and verified . Unless you have the expertise too objective data about how it bas been code will be "useless" for this goal of this thread.

 

And sure I'll change my position whenever a rationnals arguments will be provided. It includes things that I can think of in the future.

 

So for OP, objective data are always welcome.

 

 

[pkane2001]

4 minutes ago, SoundAndMotion said:

Hi Paul,

 

I understand your perspective, and given the complexities of online relationships, I don't wish to convince you of anything

 I've interacted a few times with AN also, even before "the big break", sometimes very satisfyingly. He does have a few issues upon which he chomps down like a pit bull and won't let go - even if you turn the hose on him. Like validation. If it is done in good faith, he reminds me of a couple of reviewers I've dealt with a few times. They have issues they chomp down on, but even after a bit of swearing and name-calling during the rewrite/resubmit phase, I have to admit that the final product (the paper) was better after dealing with the "issues".

In this thread, the question of good faith arrises, and I hope giving the benefit of the doubt is warranted.

SAM

 

Agreed, SAM. I've had more problems getting anything specific or helpful out of my conversations with AN, but I'm willing to consider that I am 50% responsible for such an outcome

 

[Summit]

1 hour ago, Patate91 said:

 

You'll have to learn how to code an application to understand how it works, this is out of scope of the thread and "irrevelant" for now. 

 

Like it has been said earlier the app has already been "validated" and verified . Unless you have the expertise too objective data about how it bas been code will be "useless" for this goal of this thread.

 

And sure I'll change my position whenever a rationnals arguments will be provided. It includes things that I can think of in the future.

 

So for OP, objective data are always welcome.

 

 

 

Welcome to audiophile style! I think it’s a great capability to be able to consider other perspectives and to change position if sensible evidences are provided. Opened minded is the word for it.

 

To be blunt, I don’t think that your examples with planes and hammers are related to the issue/phenomenon that is discussed here, and more important doesn’t make your perspective stronger or easier to grasp. I would even say that it has the opposite effect.  

[Patate91]

1 minute ago, Summit said:

 

Welcome to audiophile style! I think it’s a great capability to be able to consider other perspectives and to change position if sensible evidences are provided. Opened minded is the word for it.

 

To be blunt, I don’t think that your examples with planes and hammers are related to the issue/phenomenon that is discussed here, and more important doesn’t make your perspective stronger or easier to grasp. I would even say that it has the opposite effect.  

 

Distort app is a tool, hammer is a tool. All objective data for a tool cannot be published.

 

Same thing goes for plane, we don't have access to all the objective data before we buy a ticket. There's no doubt or skeptism to have about the plane's abilities to fly and bring you from point A to B.

 

But people can still ask for it if needed. Manufacturers or creators will provide it or not. For our situation feel free to ask everything toi want, we'll  see what Pkane anwser.

 

Again objective data can only be misleading if you don't have the knowledge to interpret it.

[The Computer Audiophile]

5 minutes ago, Patate91 said:

Again objective data can only be misleading if you don't have the knowledge to interpret it.

That's the rub. Nobody has the knowledge of everything.

[Patate91]

Just now, The Computer Audiophile said:

That's the rub. Nobody has the knowledge of everything.

 

Sure, but that's not a "problem".

 

Irrational buying habits is a problem.

[The Computer Audiophile]

2 minutes ago, Patate91 said:

 

Sure, but that's not a "problem".

 

Irrational buying habits is a problem.

I believe it's a problem for consumers trying to weed through buying options.

 

With respect to buying habits, I will never judge another person for what s/he purchases. 

[Summit]

16 minutes ago, Patate91 said:

 

Distort app is a tool, hammer is a tool. All objective data for a tool cannot be published.

 

Same thing goes for plane, we don't have access to all the objective data before we buy a ticket. There's no doubt or skeptism to have about the plane's abilities to fly and bring you from point A to B.

 

But people can still ask for it if needed. Manufacturers or creators will provide it or not. For our situation feel free to ask everything toi want, we'll  see what Pkane anwser.

 

Again objective data can only be misleading if you don't have the knowledge to interpret it.

 

Which type of objective data that’s relevant varies greatly between a plane, a hammer and an app for measuring. Which data that are of importance depends, and I think it’s safe to say that the data/measurements that we needed or want to know for a simple device like a hammer and a complex thing like an airplane is not comparable.  

[Patate91]

1 minute ago, Summit said:

 

Which type of objective data that’s relevant varies greatly between a plane, a hammer and an app for measuring. Which data that are of importance depends, and I think it’s safe to say that the data/measurements that we needed or want to know for a simple device like a hammer and a complex thing like an airplane is not

 

In fact I don't even need the hammer's data to use it and find that it does what's supposed to do.

 

Now what objective data do you expect from a tool like Distort? Seems that no one want to anwser it.

[Patate91]

9 minutes ago, The Computer Audiophile said:

I believe it's a problem for consumers trying to weed through buying options.

 

With respect to buying habits, I will never judge another person for what s/he purchases. 

 

Being irrationnal is not a judgment, at different degree we are all irrationnal and full of biais.

 

If a buyer has to weed out option there won't be free ticket, he'll have to inform himself, simple has that. If a buyer doesn't want or doesn't have time to learn and fer informed he'll have to trust sellers and or manufacturer. At this point objevtive data, even a "pass" /no pass is irrevelant

[Summit]

14 minutes ago, Patate91 said:

 

In fact I don't even need the hammer's data to use it and find that it does what's supposed to do.

 

Now what objective data do you expect from a tool like Distort? Seems that no one want to anwser it.

 

The reason you and I don’t need to know everything is because much is regulated and tested one way or the other, some directly by governments others by the branches.

 

I haven’t asked for objective data from a tool like Distort. I don’t think anyone have actually asked for anything besides the validated and verified independent tests that has been conducted.

[Summit]

32 minutes ago, The Computer Audiophile said:

That's the rub. Nobody has the knowledge of everything.

 

Knowledge is a familiarity, awareness, or understanding of someone or something, such as facts (propositional knowledge), skills (procedural knowledge), or objects (acquaintance knowledge). Knowledge can be acquired in many different ways and from many difference sources, including but not limited to experience, reason, memory, testimony, scientific inquiry, education, and practice. The philosophical study of knowledge is called epistemology.

 

https://en.wikipedia.org/wiki/Knowledge

 

[bluesman]

50 minutes ago, Patate91 said:

In fact I don't even need the hammer's data to use it and find that it does what's supposed to do.

 

Now what objective data do you expect from a tool like Distort? Seems that no one want to anwser it.

I fear you're all missing the issue in questioning the concept of validation, especially given how it's being used in this thread.  I was the one who first raised the question, and what I asked repeatedly (still without an answer) is whether and how we know that the app in question does ONLY what it's said to do.  Some participants are turning this into a bizarre debate over a non-issue. I did not and am not questioning whether it does what its creator says it does - as has been pointed out, that's easy to validate.  What we still don't know is what else it may be doing to the signal passing through it.  That is not an unreasonable thing to want to know.

 

It's simply hard to believe that any transfer function applied to a source file would do absolutely nothing but add one specified anomaly to that file.  The analogy to medication is a sound example of the fact that few if any interventions into anything have only one effect.  In most cases, additional effects are unintended and often undetected prior to widespread use.  Many common medications like Vioxx were "validated" through FDA mandated clinical trials, only to be found after marketing to cause serious cardiac complications that resulted in withdrawal from the market.  The Boeing 737 Max was "validated" for use by the FAA, and we all know how that worked out.

 

Even a hammer can have undisclosed and/or unknown issues that potentially affect its function and its users.  I once bought an inexpensive 16 ounce framing hammer at a local hardware store while visiting friends far from home, to help them build a bench.  Halfway through the project, I struck a 10 penny nail (unhardened) and the hammer's handle broke just below the point at which the shaft entered the metal head.  When I looked closely at it, I saw that the wood was "secured" in the head by driving an old fashioned cut nail into the end of it to widen it and wedge it in.  That cut nail was too large, driven in well off center and penetrated below the hammer's head, weakening the shaft and causing the break.  I always wear safety glasses when working, so the pieces of splintered wood that flew off didn't cause me any harm - but they could have.  I might also have questioned what kind of wood the handle was made of, since it was obviously too soft and easily splintered for the use to which it was put.

 

So, specifically, what I'd like to see is a list of the other effects on the source signal that were considered possible and shown not to occur with use of the app in question.  This is a very reasonable thing to ask - it's neither pejorative nor argumentative.  It's just a factual inquiry that deserves an answer.  Since I didn't get one and I want to try the app myself, I'll try to determine the answer myself.  And I promise to try hard to avoid misleading measurements.

[SoundAndMotion]

5 minutes ago, bluesman said:

 What we still don't know is what else it may be doing to the signal passing through it.  That is not an unreasonable thing to want to know.

 

 

[snip]

 

 

So, specifically, what I'd like to see is a list of the other effects on the source signal that were considered possible and shown not to occur with use of the app in question.  This is a very reasonable thing to ask - it's neither pejorative nor argumentative.  It's just a factual inquiry that deserves an answer.  Since I didn't get one and I want to try the app myself, I'll try to determine the answer myself.  And I promise to try hard to avoid misleading measurements.

I guess you didn’t see post#776. If you answer mine, I’ll answer yours.

[semeniub]

This link is for a research paper that contains the parameters used to estimate IMD analytically for a Class D amp case, and is compared to a Matlab DSP simulation, and also measurements on the actual amplifier:

 

https://core.ac.uk/download/pdf/162670379.pdf

 

Can this case be used as a test case for DISTORT? This would help to stop the infinite-loop discussions about what DISTORT actually is doing...

 

Also, as far back as 2010, there were publications about a Distortion Analysis Toolkit which is Matlab based. Is DISTORT doing anything different than this available software, or Matlab DSP functions for instance?

[bluesman]

7 minutes ago, SoundAndMotion said:

I guess you didn’t see post#776. If you answer mine, I’ll answer yours.

I read it - I just don’t see anything in it that relates to my question in any way at all.  Confirming the presence of desired effects is part of validation.  So is confirming the absence of undesired effects.  Surely a skilled, experienced, knowledgeable, creative engineer could think of at least one effect of adding a nonlinear twist to that straight wire with gain.

 

There’s a lot of excellent audiology literature on the effects of nonlinear processing on sound quality, most of it from research into hearing aid design and function.  Compression is very well studied as a nonlinear distortion of auditory input - and it’s well known and validated with good solid metrics that compression affects perceived sound quality.  So one might reasonably ask if an app that adds nonlinear processing to a vocal or musical data stream changes its dynamic range at all.  If it does, how?  Is it an across the board attenuation? Is it frequency related? Amplitude related? Random?

 

If I haven’t made my point by now, I don’t think I ever will. Enjoy your debate.

[Patate91]

1 hour ago, bluesman said:

I fear you're all missing the issue in questioning the concept of validation, especially given how it's being used in this thread.  I was the one who first raised the question, and what I asked repeatedly (still without an answer) is whether and how we know that the app in question does ONLY what it's said to do.  Some participants are turning this into a bizarre debate over a non-issue. I did not and am not questioning whether it does what its creator says it does - as has been pointed out, that's easy to validate.  What we still don't know is what else it may be doing to the signal passing through it.  That is not an unreasonable thing to want to know.

 

It's simply hard to believe that any transfer function applied to a source file would do absolutely nothing but add one specified anomaly to that file.  The analogy to medication is a sound example of the fact that few if any interventions into anything have only one effect.  In most cases, additional effects are unintended and often undetected prior to widespread use.  Many common medications like Vioxx were "validated" through FDA mandated clinical trials, only to be found after marketing to cause serious cardiac complications that resulted in withdrawal from the market.  The Boeing 737 Max was "validated" for use by the FAA, and we all know how that worked out.

 

Even a hammer can have undisclosed and/or unknown issues that potentially affect its function and its users.  I once bought an inexpensive 16 ounce framing hammer at a local hardware store while visiting friends far from home, to help them build a bench.  Halfway through the project, I struck a 10 penny nail (unhardened) and the hammer's handle broke just below the point at which the shaft entered the metal head.  When I looked closely at it, I saw that the wood was "secured" in the head by driving an old fashioned cut nail into the end of it to widen it and wedge it in.  That cut nail was too large, driven in well off center and penetrated below the hammer's head, weakening the shaft and causing the break.  I always wear safety glasses when working, so the pieces of splintered wood that flew off didn't cause me any harm - but they could have.  I might also have questioned what kind of wood the handle was made of, since it was obviously too soft and easily splintered for the use to which it was put.

 

So, specifically, what I'd like to see is a list of the other effects on the source signal that were considered possible and shown not to occur with use of the app in question.  This is a very reasonable thing to ask - it's neither pejorative nor argumentative.  It's just a factual inquiry that deserves an answer.  Since I didn't get one and I want to try the app myself, I'll try to determine the answer myself.  And I promise to try hard to avoid misleading measurements.

 

To validate the application you have to use it, and as said earlier Pkane already asked other people to validate it.

 

If I create an algorithm that do addition. Then if you enter 2 and 2 and it gives you 4 I guess you'll know it works correctly. No need for external validation.

[SoundAndMotion]

15 minutes ago, bluesman said:

I read it - I just don’t see anything in it that relates to my question in any way at all.  Confirming the presence of desired effects is part of validation.  So is confirming the absence of undesired effects.  Surely a skilled, experienced, knowledgeable, creative engineer could think of at least one effect of adding a nonlinear twist to that straight wire with gain.

I'm not trying to debate you. If it seems so, I apologize. I'm trying to understand you and understand why you don't understand me. I tried to address both confirming the presence of desired and absence of undesired effects. FYI, I don't know if you meant me, but I'm not an engineer. And yes, adding a nonlinearity will have more than one effect, but it is, IMO, beyond the scope of determining the correct/incorrect functioning of software to explore every ramification of a nonlinearity.

 

15 minutes ago, bluesman said:

There’s a lot of excellent audiology literature on the effects of nonlinear processing on sound quality, most of it from research into hearing aid design and function.  Compression is very well studied as a nonlinear distortion of auditory input - and it’s well known and validated with good solid metrics that compression affects perceived sound quality.  So one might reasonably ask if an app that adds nonlinear processing to a vocal or musical data stream changes its dynamic range at all.  If it does, how?  Is it an across the board attenuation? Is it frequency related? Amplitude related? Random?

DISTORT seems to be a companion tool for those who wish to look into this stuff, perhaps even research it. I can't imagine you expected Paul to do all the research to confirm that the research results can be obtained with his app. Compression will add harmonic distortion, and adding HD of a certain type will cause compression. Which "certain types"? Those that create a symmetric, sigmoid (s-shaped) time-domain transfer function. Those will compress.  I believe that means no even harmonics, only odd, but I could be mistaken. My urge is to check it with a tool... my own or something like DISTORT.

So, yes, use DISTORT to create a "compressor"-like transfer function and study it. Or use Audacity's compression tools and see what harmonic distortion is created. Same issue from different sides. A fun project (for me; I'm not giving orders!) would be to use the Audacity compressor, measure the harmonic distortion components, enter those components in DISTORT and see if the resulting file is identical to that from Audacity!

Anyway, to repeat: I think your questions can be answered by DISTORT. They are not part of a test of DISTORT.

Peace.☮️

[bluesman]

2 hours ago, SoundAndMotion said:

yes, adding a nonlinearity will have more than one effect, but it is, IMO, beyond the scope of determining the correct/incorrect functioning of software to explore every ramification of a nonlinearity.

I never said anything about how well or correctly it functions. Do you not understand or believe that an undetected secondary effect of the app in question might affect its use and cause misleading results?  
 

For example, imagine using the app to add an anomaly we’ll call anomaly X to a music file to see if you can hear it.  But unknown to you and the creator of the app, it also compresses the output at one end of the spectrum enough to make an audible difference to you.  
 

You hear a difference in SQ between app off and app on, so you reasonably conclude that you can hear the effect of anomaly X on the source file.  What you’re actually hearing is the effect of slight compression, but you didn’t know it was added and neither did the app designer because it was never considered and not looked for during the validation process.

 

If you don’t understand this valid real world example and see that it’s a practical use cases for us as audiophiles, I don’t know what else I can add to help you.