Why you can't trust measurements

[fas42]

Nicely done. People who like what they deal with neatly tied up in a well presented package will have a strong liking for measurements; even though the latter only tell part of the story, or are 'fudged' to suit the situation. My own experiences with my first high performance amplifier, 30 years ago, a Perreaux, pretty well sum up the realities in the audio world: I chose this amplifier by ear; it easily beat the big Krell that was also in the demo room at the time; and in further listening to other ambitious rigs using high end amplifiers I had my opinion that I had chosen a good'un confirmed. What was also impressive is that the user manual had an excellent rundown of measurement data, which far exceeded what one gets in any current day amplifier guide. And the suite of numbers in the manual "said" that I shouldn't hear any distortion ...

 

Yet the amplifier had a problem with treble distortion. Quite easy to hear - when the amplifier was driving above a certain level, quite moderate in comparison to how loud it could go, the treble content started to show an issue. Over a period of time I determined that the unsophisticated power supply it used was the problem - I made major modifications to this area; and the treble distortion was eliminated.

 

The point is that the fulsome measurements, on paper, gave zero hint that the amplifier had a weakness - only longer term living with the part revealed the lacking. So, if you rely on measurements to "tell the full story", well then, I've got a nice big bridge to sell you ... .

[fas42]

Sorry 'bout that ...

[fas42]

It's curious how some people don't trust their ears, believe they are of such poor quality as a means of registering what's going on that they constantly run to a piece of equipment, to justify some stance on the audio equipment game.

 

Personally, what I have found intensely satisfying over the decades, is seeing how a particular setup can steadily be pushed, evolved, to slowly eliminate its unrefined personality to the point where that of the recording starts to take over. The music event that was captured, which is now effectively eternal, just bides its time, patiently, until its "true colours" can properly emerge - each configuration is better able to do such in some areas than others; the 'truth' of the recording is the combination of the very best that one has ever heard from a particular track, over the years of experiencing it on many setups and situations. This builds into a memory of the musical performance - and when one gets very close to the essence of what's in the source, the jolt of recognising all the little giveaways that you picked up over the years is similar to meeting someone you knew extremely well some years ago, but hadn't come across for a long time in between. The familiarity with what's on the recording is one of the pleasures of getting the SQ right - and this where conventional measurements are nowhere near good enough to do the job of determining the, 'accuracy' ...

 

 

[fas42]

20 minutes ago, botrytis said:

 

Human ears are poor quality, compared to dogs and other species. Humans are a vision and tactile centric species.

 

Yet a conductor in the storm of an orchestral crescendo drowning him in sound, from a flotilla of instruments feet away, all giving their all, can then point to a particular player in a section - and say, "You did that wrong!" ...

[fas42]

1 hour ago, botrytis said:

I have no idea what 'crazy uncle Frank' is talking about.

 

What I'm talking about, is that human hearing can be as precise as needed for the situation at hand - if the owner of the ears cares about he's listening to, .

 

How precise ... oooh, about 120dB worth - what's that mean? How about, you get in a car, drive like crazy for just under a km; and as you get to the km point, slow down; and finally stop - to an accuracy of 1 mm, over the whole distance of 1km .. translating that to human hearing, not too bad for a "poor quality" sense  ...

[fas42]

The interesting thing for me is that I started my digital journey with one of the best DACs you could buy at the time, which had very good specs, and capabilities - which delivered 'transparent' SQ, when the whole system was sufficiently optimised. And this made the lack of transparency of nominally extremely good measuring, expensive gear I came across elsewhere painfully obvious. This was one side of the equation. The other side was then trying modding of cheap 'n' cheerful stuff, which used parts that had only mediocre measured performance - and extracting subjectively transparent SQ from them.

 

This demonstrated that the measurements of equipment had close to zero correlation with their ability to deliver satisfying listening. As of now, things have barely improved - not many would accept having to live indefinitely with some combo of gear, bought purely on the basis of numbers - because, "they were right!" ...

[fas42]

I've had run-ins with Amir ... he has a powerful need to have the last word, and will end up throwing everything he can possibly think of, so that he can come out on top ... when you realise the exchange has got to this point, the best option is to withdraw quietly; to continue will only cause the mud to get thicker, and more smelly ...

[fas42]

And done in such a gentlemanly way, too ... pity, it was indeed a forum for "sound, reasoned argument, and meaningful discussion" ... as we all know, .

[fas42]

7 hours ago, Iving said:

https://forums.stevehoffman.tv/threads/just-got-banned-by-asr-for-pointing-out-the-inconvenience-truth.1144772/

 

The OP is not me btw!

 

What's going on ... that thread is no longer there?

[fas42]

1 hour ago, pkane2001 said:

 

It's important to know what measurements are and what they are good for. Certainly, poor measurements can point to poor design and/or poor engineering. Or, they may demonstrate intentional deviation from transparency.

 

 

The severe tribalism in the audio world does no favours, for anyone. Measurements have almost no value for someone who wants to assemble a system for maximum listening satisfaction; it's become a fetish to either put numbers on a sacred altar, or treat them with contempt - the truth, as is often the case, is elsewhere .

 

It has been repeated an infinite number of times, that conventional measurements are a poor guide to a system's subjective qualities - personal experience, 35 years ago, made this obvious. Ongoing beating on the worn out drum that it's possible to assess a setup by checking each component on a testbench is getting the audio world nowhere - it's time to move to better methods ...

[fas42]

Gosh, you just reminded me ... 30 years ago, I said to someone, 5 ...

[fas42]

Yes, I'm aiming for a googolplex - at least ...

[fas42]

2 hours ago, Allan F said:

 

I think some people think that a microphone hears the same way as our ears. So we have these room correction devices that fool you into believing that they create the perfect listening environment because the microphone tells you so. The same thing happens when you look at your distortion analyzer or whatever makes you think you are okay. You're far from okay.

 

 

 

Exactly.

 

2 hours ago, Allan F said:

 

There are sonic differences that cannot be measured.  I won't say that they never will be able to be measured, but as far as I know, there's no current measurement that will guarantee sound. I can give you an example. When solid state amplifiers came on the scene, people hated the sound because they used maximum power distortion measurements [to gauge success]. But all the problems were at zero-crossing distortion, which was a measurement that wasn't really included. As the level was going down, the distortion was going up relative to it. So, it was only after listening that the measurement was finally included. (bold emphasis added)

 

There will always be a new measurement that will be discovered. For example, the effect of jitter was initially ignored”.

 

 

Completely agree. That is, "there's no current measurement that will guarantee (competent SQ)". It's trivially easy to sabotage the subjective SQ by making a minor adjustment, which undoes the effectiveness of an optimisation - and I'm certain no normal metric would pick what just happened.

[fas42]

21 hours ago, Archimago said:

Again, what does this mean "sonic differences that cannot be measured"? Since we're talking about a DAC, other than reproducing the intended digital data as analogue electrical waveforms, what does he think is still out there to be "discovered"?

 

 

What needs to be "discovered" is that digital replay requires exceptional "hygiene", compared to the typical analogue based setup, for the subjective SQ to switch into, 'magic' mode - every system I've played with over the decades has needed love and attention to detail to get this; the need for this doesn't go away just by throwing enough expensive, street cred gear at the situation. The industry still hasn't properly woken up to the requirement that the "minor things" like noise isolation, and reduction of spurious causes of interference have to be taken really seriously - until this happens it's always going to be a lottery as to whether a particular rig just goes through the motions of playing some recording, as compared with conjuring up an enthralling, immersive listening experience ...

[fas42]

6 hours ago, Iving said:

all the uncertainty makes a lot of room for individual hobby fun, forum conflicts - and industry confidence trickery.

 

antidote to all above is education of all types - and learning to think for ourselves

 

There's lots of uncertainty ... but very little trickery, in the normal sense. The whole thing falls in the same arena as the massive, personal cosmetics industry - do you want to send all the executives of these companies to prison, for defrauding the public?

[fas42]

It's interesting how so many people can't stand back from what results after making some changes to a system; there is almost a gritted determination to find some improvement - there must be, because I have put so much energy, or money into doing this; it better be better !!!

 

I have had far too many disappointments over the years to see any use for this attitude - a rig is what it is: it may be incredibly expensive, with layers of bling to bedazzle, but if it sounds like a mess, well, it is a mess ...

[fas42]

1 hour ago, Iving said:

 

confidence trickery - believe what i say because of the way i say it - don't examine merits 

 

it's clear what i mean

 

It is? From https://en.wikipedia.org/wiki/Confidence_trick,

 

Quote

A confidence trick is an attempt to defraud a person or group after first gaining their trust. Confidence tricks exploit victims using their credulity, naïveté, compassion, vanity, confidence, irresponsibility, and greed. Researchers have defined confidence tricks as "a distinctive species of fraudulent conduct [...] intending to further voluntary exchanges that are not mutually beneficial", as they "benefit con operators ('con men') at the expense of their victims (the 'marks')

 

So, say, every person who was purchased an expensive audio cable, which costs many, many times more then any person could conceive that it would cost to manufacture, has been defrauded ... correct?

[fas42]

Yes, silly of me to be annoyed by the seemingly obligatory put down of people who produce tweaks - who are supposedly always doing for the money, as compared to having a belief in the value of their product.

[fas42]

https://www.webmd.com/mental-health/what-is-a-mob-mentality ...

 

Quote

Mob mentality, herd mentality, pack mentality, groupthink, or crowd psychology — the concept has many names. These all boil down to the same idea: Individuals are influenced by a larger group. Regardless of whether that group includes people in your class, your neighborhood, or an entire nation, you may experience mob mentality.

In the 1950s, researchers conducted a famous conformity experiment that showed how readily people conform or change their behavior to match social norms. It involved:

• A single participant was put in a room with seven undercover accomplices.
• Asch presented the group with four lines, and the goal was to determine which two lines were the same length.
• Even though the answer was obvious, the undercover accomplices purposefully gave an incorrect answer.
• The real participant answered last.
• The intent of the experiment was to see if the real participant would give a false answer — conforming with the accomplices — even if the correct answer was clear.

‌The results were surprising. About one-third of the real participants answered incorrectly. They conformed to the wrong answer given by the rest of the group

 

Why did they do it? When the participants heard the incorrect answer, some started to believe the incorrect answer was actually correct. The participants conformed mainly because they wanted to fit in with the rest of the group and thought the rest of the group was more informed than they were.

 

 

.

[fas42]

10 hours ago, kumakuma said:

 

I'm having a hard time seeing how this relates to the subject being discussed in this thread. Can you elaborate? 

 

Groupthink is everywhere, in audio. Anyone who strays from the narrow thinking of the two fraternities, objectivists and subjectivists, is usually hounded out from the area, because the variation from the mindset of the majority there makes too many people uncomfortable - it can't be allowed to remain!

 

And here we see it again ... @Rexp points to a single article where there weren't measurements to support an audible difference - and what do we get? Not an interesting discussion of why this may be the case, because of the technical aspects of the particular situation; but rather the usual argy-bargy between the two camps of thought. And not a single, tiny bit of movement is made forward, in terms of better understanding of what matters ... 

[fas42]

1 hour ago, kumakuma said:

 

This is a convenient way to you to blame others for a situation of your own making. 

 

A situation of my own making? ... Clarify, please.

[fas42]

 

1. The principle that the mere occurrence of some types of accident is sufficient to imply negligence.

    

    

   Yep, clear as mud ...

[fas42]

35 minutes ago, Confused said:

So lets take the subjective review at face value. Why did the Master Time Black Star make such a profound difference? A reduction in noise? Maybe, but this should be measurable. Also the opticalRendu is a relatively low noise device, the entire point of the device is to produce a low noise USB feed, with optical isolation of the Ethernet feed eliminating any upstream noise.

 

Should be measurable - but what specifically is to be measured? And if a variation is seen, by some "new" method of assessing the performance, how can we be certain that there is strong correlation between these numbers, and what was heard? There may be some other difference, which is the crucial one as far as the subjective assessment is concerned, but it's completely missed. ...

 

35 minutes ago, Confused said:

 

We could of course speculate that very many other things created the audible differences, but it would be just that, more speculation.

 

Like the cables used ... I'm trying optimising a digital link between components for the first time ever, right now; and it's a can of worms! Okay, standard quality optical rather than USB, but the situation from reports I read is equivalent: how much does the precise way the cable is organised between any digital out and any digital in matter ... so far, the answer, for me, is a lot!

 

35 minutes ago, Confused said:

We could also speculate that the reviewer imagined the audible differences or was otherwise mistaken. That too is just speculation.

 

My frustration is that Stereophile are happy to publish such stuff without making any apparent effort to did a little deeper when basically they are the only entity with actual access to the kit, a reviewer who can hear the difference, and measurement kit. Yet all they do is publish something almost deliberately aimed at fuelling the objective subjective debate. Frankly I think it is lazy and unscientific , from a publication that takes great pride in the fact that they stated publishing measurements decades ago.

 

The problem is that to really, really do it properly the whole exercise would gravitate to a full blown scientific analysis; with every possible confounder evaluated. Which just won't happen. If lucky, some clues will emerge, to give buyers a bit of guidance - but I wouldn't bet on it!

[fas42]

14 hours ago, Confused said:

The first step would be to establish if the reviewer really did hear the differences that he reported. As mentioned earlier in this thread, this would be very easy to do.

 

The above could be repeated with and without the opticalRendu in the chain.

 

Assuming this provided a positive result, this would provide some reason to explore further in the measurements.

 

None of this would be difficult.

 

The industry has had decades to pull its finger out, and try and do things better. Do you think there is any chance that anything is going to change any time soon, based on how far things have progressed, to this point?

 

Enthusiastic audiophiles are seen by most to be in the same category as model train buffs - a peculiar bunch of people who spend lots of money on something that is of little relevance to anything else in life, apart from the pleasure of fiddling with things. No-one spends any time, energy, or money analysing the bigger picture in model trains - and the same applies in audio. Only companies who create products that are interesting to enthusiasts are making efforts; and of course this will be devoted to ensuring that their output is seen to be superior ...

[fas42]

5 hours ago, Confused said:

 

In my experience, when I tell most people that I am an Audiophile, they mostly tend to respond with something along the lines of "oh, I could never tell the difference between speakers or amplifiers or anything", or maybe they might say "do you like to listen to jazz then?" In general, I find that most people are mostly not remotely interested.

 

As a final point from myself. Can we try to keep things at least vaguely on topic?

 

To keep it on topic, why you can't trust measurements is that they don't tell you when a combination of gear gets into the zone when people who haven't the slightest interest in the game of audiophilia are turned on by what they're hearing. Things like, that the women come in and really, really listen - or start bouncing around the room, from the energy of the music. That means, that all the technical nonsense is no longer relevant - because the message of the recording is finally properly communicating ..