Objective-Fi Optical Networking & SFPs

[Audiophile Neuroscience]

5 hours ago, plissken said:

 

If I came out to you on you're setup with my playback computer with my dual SFP+ Solar Flare card and did this with you blind, would you evaluate that way?

 

4 hours ago, kennyb123 said:

 

 You don't seem to be able to understand how to properly conduct experiments.  Nor do you seem to understand the need for experiments to be done properly.

 

 

What could possibly go wrong? 🙄

 

I didn't know Mark still offered his mobile blind testing debunking service. IIRC it used to come with some kind of monetary incentive.

[Audiophile Neuroscience]

4 hours ago, ray-dude said:

FWIW, at the engineering schools I went to the reaction would have been "No way!  Let me hear that...  Huh...  Wow...  What the hell is going on?  Cool, let's try to figure this out!"  This is the exciting stuff!

The most exciting phrase to hear in science, the one that heralds the most discoveries, is not Eureka! (I found it!) but 'That's funny.....What the Fuck !

(to pararphrase Asimov 🙂)

[Audiophile Neuroscience]

14 minutes ago, plissken said:

 

At this point your only option is to win the lottery and hire a permanent live band. 

 

I think before squandering money on a permanent live band, to be objective about it, you would need to run them through a very good ADC then DAC output into a very good ADC. Tested blind of course!🤷‍♂️

[Audiophile Neuroscience]

2 hours ago, cat6man said:

of course DBT is a valid and immensely useful scientific method, and

i agree that most amateur DBT are relatively useless.

 

however in most(?) cases at the medical center, i'd guess that you have some (what i'll call) objective output measure such as blood pressure, survival rate, visual acuity, heart ejection fraction........i.e. things you can measure. 

 

Yes, I agree and it is a point that seems often overlooked IMO.

 

The best outcome measures pass what I have referred to previously as objective 'test of test' parameters.

 

Also, those tests are ideally out of the subjects direct control or influence or even knowledge. Ideally subjects don't have a clue what is being tested like a blood measure they have never even heard of, let alone something that tests their skill or potentially their reputation. Things however cannot be always ideal.

 

Without test of test parameters however, I won't go into detail here but, one is basically using an uncalibrated tool. Audio perceptual testing is interesting in this context, adding a layer of increased difficulty as you have to develop methodology that in essence uses a perceptual test to test perception. There is no gold standard to which we can 'calibrate' against such as using a blood test serving as a surrogate marker of a particular cancer but comparing results against gold standard biopsy results.

 

Even with well done medical research confounders often lead to less than conclusive results and this often has  nothing to do with blinding. Just look at the number of peer reviewed publications on the role of statins in heart disease, or the place of coronary CT calcium scores, what you might expect should be fairly straight forward is anything but.

 

Eliminating bias is not the issue , it is the "how" you do it. Methodology is key. Blinding does not automatically make a methodology valid and indeed you cannot guarantee it does not actually increase false negatives through other means such as interdependent variables . In order to know this you would need to know the false negative rates and this poses difficulty without a gold standard reference for calibration.

 

 

 

[Audiophile Neuroscience]

11 hours ago, kennyb123 said:

 

DBT proponents seem to see this as feature and not a bug.  Their desired outcome is guaranteed and made bulletproof (so they think) because properly-conducted DBT has been demonstrated to be a valid research tool.  Valid questions about how the test was conducted are ignored.  That's why I've said that those who press for amateur DBTs are actually the ones behaving like snake oil salesmen.  

 

I agree it seems to be assumed by some as fairly simple and infallible procedure, or at least highly validated, and can even be used by amateurs as a weapon to push a particular view. It is just a test tool and as alluded to above, like all tools, one needs to know how good the tool is at telling you what you think it is telling you - in the context and setting of the whole test methodology .........I think we are heading OT.....

[Audiophile Neuroscience]

12 hours ago, bluesman said:

The critical factor in choosing a test is whether it can answer the question being asked.  As you point out, the question asked by audiophiles is often unanswerable and therefore untestable as asked.  It’s not possible to determine if a given cable improves SQ with blinded testing - for a start, one audiophile’s smooth is another’s muddy. But it’s possible to determine if there’s a consistent difference between two that subjects can identify with 95+% consistency in enough well done blinded listening trials to be statistically significant.  Correctly identifying one alternative from another with 95% certainty in well designed and well conducted DBT is objective.  And this can be useful info.
 

The “best” questions for any study are objectively measurable, as you point out.  But many healthcare decisions are made on the basis of parameters that can be as vague and nebulous as “how real it sounds”, e.g.  sense of well-being, intensity of pain, Quality of Life Years, patient satisfaction, and likelihood of recommending. Picking and using good tests to get valid, repeatable results is dependent on what question is asked, how it’s asked, and in what form an answer is sought.  And measurement systems are less reliable than we believe.  Even “simple” blood pressure measurement is not simple, e.g. 3 consecutive readings 5 minutes apart can vary widely.

 

Other factors ignored by those who oversimplify DBT include consistency among multiple raters and consistency of the same rater over time.  If a subject correctly identifies A or B in 95% of presentations in one trial session but averages 53% over 5 identical sessions, that 95% session is not representative - it’s random chance.  Controls are also needed.  Present the same alternative to raters multiple times and see if they think they perceive differences that aren’t there.

 

I mostly agree but a couple of comments.

 

In the absence of objective direct outcome measures, correctly identifying a perceptual difference to a statistically significant degree is indeed useful and objective information. As you point out p=.05 is the 'usual' figure nominated but some would like a bit better than this depending on the situation. @manisandher in the Red Pill/ Blue Pill scored p=0.1, a 99% consistency ie probability that his perception was not a product of random chance or guesswork. But, without direct objective measures the problems of inductive reasoning are amplified - there is no inductive law to prove a hypothesis. Hence, that thread was very long and hotly debated - basically the result was challenged, by some, based on supposed methodological issues. So while "well designed and well conducted DBT" is prerequisite, achieving same is not always, and I would suggest often not, without challenge. A bit like audio, its 'all in the implementation'.

 

You touched on intra-observer false positives. These are important but so are intra-observer false negatives. IMO false negatives in general are really more the issue in the context of blind listening tests. It has to do with the whole "blind test it or it didn't happen" claim. If a perception disappears on a blind listening test, is it due to the perception never existing (beyond biased imagination) or is it that the test has failed in some way ie it is a false negative. It's a rhetorical question as seemingly most have preconceived unshakable positions.

 

Whatever the case, if you don't know the false positives or false negative rates in the first place (together with other test of test parameters) of the test procedure you are working in the dark. The blind leading the blinded. It is analogous to using an uncalibrated tool. At the very least one needs to accept certain assumptions of validity about the test procedure. I think there is more light shed on the situation if the subject passes the test with flying colors, but in either case it comes down to acceptance of the methodology.

 

Not directed at you but asking the question how do we know what we know is not just philosophical musing but strikes at scientific method. Whether we are assessing SFP modules or considering the credibility of reviewers on this very forum site, there will always be claims that "if it wasn't blind tested it didn't happen". In the absence of proof one way or another I remain open but skeptical about reported observations.

[Audiophile Neuroscience]

51 minutes ago, bluesman said:

A p value of 0.1 means a 90% probability, not 99%.

My typo. should have been p value =0.01, 99%

 

Of interest, the first blind test methodology based on AB XXXXXXXX failed ( false negative test result) whereas the second blind test procedure, ABX ABX etc succeeded to a p-value of 0.01 being highly significant.

[Audiophile Neuroscience]

On 6/27/2020 at 5:14 AM, cat6man said:

 

I'm a big fan of WU! (and still have my coffee mug)

 

This discussion reminds me of situations that happened repeatedly at work over the years.

We'd measure something in the lab and some arrogant PhD would say it was impossible because he had determined that such and such was optimum, blah blah blah.

 

I always insisted that the theoreticians in my department/lab get their hands dirty and verified that their algorithms behaved as expected when implemented.

In my personal experience, I obtained some of my most interesting insights (and patents) from the lab (i.e. real world) experiments not lining up with the oversimplified theories.........like "Why the hell is that happening?".............answer:  the experiment is trying to tell you something dummy!

 

Back to audio:

It really bugs the heck out of me that putting a better power supply on a digital music server can have such a profound impact on my musical enjoyment................I love the enjoyment part but am frustrated that I don't understand the mechanism (actually worse, I don't have a clue!) by which this happens. 

 

I have no specific theory or hypothesis to fit my experience but I think a framework is in sight.  My engineering intuition (or hubris?) suggests that it is related to behavior on the tail of distributions (a phenomenon I've seen repeatedly in my professional work).

A possible framework, though unproven, might look as follows:  (YMMV, ignore the rest if you detest speculation)

 

Whatever mechanism is at work is likely not measurable in an average sense but is likely, IMHO, to be a low probability event.  Let's hypothesize 1% of the time this "something happens".  For red book, this would be ~440 times/sec.  I can certainly imagine that this might be discernible if it affected 440 samples/sec but would yet be unmeasurable on average due to being masked by the other 43,000+ samples.  Occasional long latencies between samples?  Clock jitter spikes?  Short infrequent power supply glitches?  I've no idea but something is going on.

 

[start lecture]

Case in point/analogy:  In wireless communications, using 2 spatially separated antennas give improved immunity to fading and is called space diversity reception, the idea being that if the fading is independent on each antenna the probability that they both are bad (i.e. faded) is greatly reduced.  That's the theory.....and we saw exactly that with a fading simulator in the lab, but not so when testing around a real cell site with space diversity antennas.  WTF?  It turned out that the two antennas have almost independent fading but there was a residual 1-2% correlation in the fading statistics.  On average that would make no difference to the measurements, however the wireless receiver  behaved differently.  Now a 1% correlated signal could be simply modeled as 99% of the time uncorrelated and 1% of the time completely correlated (i.e. both fading at the same time).  The radio receiver was happy as a clam 99% of the time but very unhappy the other 1% of the time, and whenever that 1% of the time occurred the receiver would have to change its operating point and required SNR and the entire receiver loop would have to re-converge.  When we programed the fading simulator to have 1% correlation between the two receiver inputs, we obtained the exact same results as in the outside world.  The 1% correlation did not make any change to the average fading statistics but it had a huge impact on the receiver due to dynamic effects.

[end lecture]

 

So let me put down a friendly challenge.  In the interest of being 'objective' and not 'subjective' in this sub-forum, how about a sub-sub-group interested in getting to the bottom of this technically and not just acting like my old 'arrogant PhD' colleagues who already knew all the answers (but had oversimplified the situation and therefore hadn't formulated the problem accurately)?

 

This will be my last shot at trying to see if anyone else is interested in approaching this in a similar manner to my current way of thinking.  I'm not interested in debating  philosophy of testing and will not reply to such.  Anyone want to try to figure out what is going on here?

 

 

I expect to be excluded from such a group on "philosophical grounds". No problem but despite my philosophical bent, and probably because of it, I do applaud the open minded experimental approach to knowledge as you suggest. I can only offer then, some philosophy stolen from some of my science heroes.

 

If it disagrees with experiment it is wrong....... It's difficult to be absolutely sure about anything (scientifically speaking)..... among the easiest to fool are ourselves (for or against a proposition), and a direct quote from The meaning of it all: Thoughts of a citizen scientist - "Looking back at the worst times, it always seems that there were times in which there were people who believed with absolute faith and absolute dogmatism in something. And they were so serious in this matter that they insisted that the rest of the world agree with them. And then they would do things that were directly inconsistent with their own beliefs in order to maintain that what they said was true.”

 

Now, if I could put it to music...ah somebody beat me to it 😁

"So I hope, through life you never will decline

In spite of philistine
Defiance
To do what all good scientists do

Experiment"

 

Good Luck