Why you can't trust measurements

[Confused]

9 hours ago, Rexp said:

'To get an obvious question out of the way: Yes, I did listen to the Ayazi without the Master Time. By its lonesome, the Ayazi reproduced music with less resolution and timbral accuracy and created a spatially smaller, less lifelike sound. Music sounded duller and less compelling.' Alex Halberstadt

 

Overall, Ideon's Ayazi did well on the test bench. I didn't find any difference in its performance when fed USB data directly or via the 3R Master Time Black Star.—John Atkinson

 

https://www.stereophile.com/content/ideon-audio-ayazi-mk2-da-processor-3r-master-time-black-star-usb-clock

 

There are at least two possibilities here:

 

1. Mr Halberstadt was imagining the improvement from the Master Time, hence the measurements correctly show no improvement in performance.

 

2. The improvement in sound quality from the Master Time is real, but the correct measurements to show why this might be the case were not performed.

 

If I was the editor of a magazine that planned to publish this, I would set up a blind test with Mr Halberstadt (and any other golden eared hacks that I had available) to see if he can reliably pick up the resolution and timbral accuracy and the spatially smaller, less lifelike sound, without the Master Time.

 

If he could not, I'd get him to rewrite the article. If he could, I'd get Mr Atkinson to take another look at how to perform his measurements.

 

As things stand, the article does not tell us much, apart from demonstrating that the audio press can be a bit useless at times.

[Confused]

2 hours ago, Rexp said:

The fact is the reviewer heard a significant difference which was not measurable. What fact(s) lead you to believe either the reviewer was wrong or the measurements faulty?

I have no reason to believe the measurements are “faulty”. It is of course possible that different or additional measurements could be performed.

 

What I see as a “fact” is that the ultimate conclusion from the measurements and the listening tests are contradictory.

 

Do you think that it is 100% impossible that the reviewer was not influenced by expectation bias?

 

It is of course possible that what the reviewer heard was real but not picked up by the specific measurements that were performed.

[Confused]

26 minutes ago, Superdad said:

This. 

Do you think that it is 100% impossible that the reviewer was not influenced by expectation bias?

[Confused]

22 minutes ago, Iving said:

convincing data near impossible with single reviewer - 

and multi participant studies all sorts selection and expense problems.

Some very fine wise words in your earlier post Mr Iving. You kind of stated what I am thinking, although you have phrased it far better than I could.

 

Although re the quoted words above, I do not think we need expensive multi participant studies in this particular case. What we have is a reviewer who has clearly stated that he can hear very specific differences with the $4K Master Time in the chain. This would take only two people and a couple of hours to establish if he could do this "blind". If he could, this would be convincing data from one reviewer, and thus provide good reason to investigate further in the measurements.

[Confused]

58 minutes ago, Superdad said:

I was not in the room where it happened so have no idea what was actually heard or imagined.

Then again, I don't see why IDEON charges $4,000 for what is essentially a copy of our original 2015 USB REGEN ($175) with exact same SMSC hub chip, some enhanced power networks, and some unknown clocking. (And that whole layout with the daughter board is terrible for USB signal integrity.) 

 

Personally, I get a far greater expectation bias invoked SQ hike from a $4000 item than I do from a $175 product. 🙂

 

Joking apart, I know the Master Time has excellent clocking properties because it has "advanced femtoclock anti-jitter technology" written on it in big letters on the circuit board.

[Confused]

6 hours ago, fas42 said:

 

Groupthink is everywhere, in audio. Anyone who strays from the narrow thinking of the two fraternities, objectivists and subjectivists, is usually hounded out from the area, because the variation from the mindset of the majority there makes too many people uncomfortable - it can't be allowed to remain!

 

And here we see it again ... @Rexp points to a single article where there weren't measurements to support an audible difference - and what do we get? Not an interesting discussion of why this may be the case, because of the technical aspects of the particular situation; but rather the usual argy-bargy between the two camps of thought. And not a single, tiny bit of movement is made forward, in terms of better understanding of what matters ... 

My posts were specifically trying to move away from the narrow thinking of the two fraternities, objectivists and subjectivists. 

 

My frustration is with Stereophile. They have such an advantage over us mere mortals posting on a forum. They have access to the actual equipment being discussed, they have a reviewer who states he can hear clear differences when a specific piece of equipment is used in the chain, they have a measurement specialist armed with an Audio Precision SYS2722. Yet all they do is publish a subjective review which states that the DAC being reviewed has "less resolution and timbral accuracy and created a spatially smaller, less lifelike sound" when the USB clocking device is not used in the chain, and then publish measurements that do not reveal why this might be the case.

 

Also note that when the DAC was used without the Master Time Black Star. it was being fed by a Sonore opticalRendu.

 

So lets take the subjective review at face value. Why did the Master Time Black Star make such a profound difference? A reduction in noise? Maybe, but this should be measurable. Also the opticalRendu is a relatively low noise device, the entire point of the device is to produce a low noise USB feed, with optical isolation of the Ethernet feed eliminating any upstream noise.

 

Maybe there is a benefit from the USB clocking? Although as @Superdad pointed out there are no details of what the clock actually is, so this would just be speculation.

 

We could of course speculate that very many other things created the audible differences, but it would be just that, more speculation.

 

We could also speculate that the reviewer imagined the audible differences or was otherwise mistaken. That too is just speculation.

 

My frustration is that Stereophile are happy to publish such stuff without making any apparent effort to did a little deeper when basically they are the only entity with actual access to the kit, a reviewer who can hear the difference, and measurement kit. Yet all they do is publish something almost deliberately aimed at fuelling the objective subjective debate. Frankly I think it is lazy and unscientific , from a publication that takes great pride in the fact that they stated publishing measurements decades ago.

[Confused]

I can think of more....

 

The review chain was a Sonic Transporter i5 running Roon Core, feeding an optical Rendu. In practical terms, most people would consider this to be a fairly decent digital "front end".

 

So what would the DAC sound like being fed direct from the Sonic Transporter? What if the Sonic Transporter was used to to feed the Master Time Black Star directly, taking the opticalRendu out of the chain? A few experiments like this this might provide some data as to exactly what was producing the profound subjective improvements that the reviewer could here, and give some insight into why. What if the DAC is fed from another different source, maybe something other than Roon? Could they not have tried something else?

 

But all we have is a review which basically states "with the $4k extra box in the chain I heard much better sound".

 

Then take a look at Ideon's own webpage. Even they can't be bothered to provide any technical data or reasons as to why this device improves sound quality:

 

https://ideonaudio.com/3r-master-time-black-star/

 

Although they do state this: 

Based on Ideon Audio’s proprietary technologies, dramatically improves the sound from digital audio computer files, and music streaming services.

It achieves amazing sound performance by uncovering lost detail, enhancing dynamics, re-clocking the signal using ultra-low jitter oscillators, and minimizing transmission losses.

[Confused]

38 minutes ago, fas42 said:

The problem is that to really, really do it properly the whole exercise would gravitate to a full blown scientific analysis; with every possible confounder evaluated. Which just won't happen. If lucky, some clues will emerge, to give buyers a bit of guidance - but I wouldn't bet on it!

The first step would be to establish if the reviewer really did hear the differences that he reported. As mentioned earlier in this thread, this would be very easy to do.

 

The above could be repeated with and without the opticalRendu in the chain.

 

Assuming this provided a positive result, this would provide some reason to explore further in the measurements.

 

None of this would be difficult.

 

I agree that we are never going to get a a full blown scientific analysis; with every possible confounder evaluated, but the Stereophile review and measurements offer nothing, when they could very easily have at least offered something.

 

Or put it this way, don't let perfection be the enemy of good. What Stereophile offered was not good.

[Confused]

7 hours ago, fas42 said:

 

The industry has had decades to pull its finger out, and try and do things better. Do you think there is any chance that anything is going to change any time soon, based on how far things have progressed, to this point?

 

I do not really have a view on this. 

 

Generally I tend to side with the words of Physicist  Niels Bohr "Prediction is very difficult, especially if it's about the future". I am sure we could have a thread on this forum where we all give our very many views on what the future will bring, a lot of it will be contradictory and most will be wrong. Maybe even this view will prove to be wrong?

 

Meanwhile I intend to travel through life, trying to make a positive impact with respect to anything that matters to me.

 

7 hours ago, fas42 said:

Enthusiastic audiophiles are seen by most to be in the same category as model train buffs - a peculiar bunch of people who spend lots of money on something that is of little relevance to anything else in life, apart from the pleasure of fiddling with things. No-one spends any time, energy, or money analysing the bigger picture in model trains - and the same applies in audio. Only companies who create products that are interesting to enthusiasts are making efforts; and of course this will be devoted to ensuring that their output is seen to be superior ...

 

An interesting view.

 

In my experience, when I tell most people that I am an Audiophile, they mostly tend to respond with something along the lines of "oh, I could never tell the difference between speakers or amplifiers or anything", or maybe they might say "do you like to listen to jazz then?" In general, I find that most people are mostly not remotely interested.

 

As a final point from myself. Can we try to keep things at least vaguely on topic?

[Confused]

Can we measure Frank?

[Confused]

They are all at it!

https://railwaytraveller.com/inspiration/celebrity-tips/the-unlikely-celebrities-who-are-huge-model-train-hobbyists

 

 

https://www.theguardian.com/lifeandstyle/2022/mar/23/im-proud-to-be-called-a-nerd-the-pop-and-rock-greats-who-love-model-railways

[Confused]

5 hours ago, botrytis said:

 

Me thinks imaginary numbers would be involved.....

fas = square root 2?

[Confused]

9 hours ago, fas42 said:

Do you want to measure the interesting bits - or the boring stuff?

Both, we need an interesting bits to boring stuff ratio. (IB/BSr)