Jump to content
IGNORED

Why are objective assessments important...


Recommended Posts

6 hours ago, Archimago said:

Human perception has its limits and our attention to things also can be limited, missing out on what we actually CAN hear but didn't notice. For example, look at all the positive comments about the recent AudioQuest Dragonfly Cobalt. From my perspective, it totally sucks as a USB DAC at this price point. Good that Mans found similar issues with distortion that I saw. Once one is tipped off to these anomalies, one can start picking out examples and select music that can bring out the anomaly that one might have missed before. This is what "perfectionist audio" IMO is about.  If I am going to pay big(er) bucks, it certainly would be nice to be clear about what performance I'm buying.

 

The opinion of any specific listener is nice, but IMO, not as strong as what objective means might reveal.

 

And that is how one can be "objective" in areas where the people usually will use the term, subjective ...what one does is use tracks of music whose content very strongly provokes the system playback to distort - the anomalies are obviously audible; so in that area it's a fail, for the setup.

 

The process of "sorting out" is eliminating each of the failure 'modes', one by one.

Frank

 

http://artofaudioconjuring.blogspot.com/

 

 

Over and out.

.

 

Link to post
Share on other sites
On 2/21/2020 at 12:05 PM, tapatrick said:

As you seem to be a helpful person I have some basic questions as I have very little understanding of measurements or their meaning. I know what a frequency curve is and the importance of low jitter and noise levels etc. but I'd much rather read a review by someone who knows how to interpret the measurements presented.

 

Apart from the obvious (noise spikes or reduced frequency range), opinion seems to be that these are open to interpretation or only show a partial picture of what constitutes the sound coming from system. I spend time on DIY forums and I put together basic components and modify them following the advice or example of those more qualified - changing wiring, building cables, replacing capacitors and replacing clocks etc as I like to tinker then try them by listening carefully over several weeks of trial in order to hopefully enhance my system. For instance my reference source is a $50 chinese SD card player heavily modified and powered by LifePO4/Ultracaps. So my interest in measurements is in relation to things like this.

 

For myself and other non techies could you please list the types of measurements everyone is referring to and any good resources to understand the relationship between these and sound quality? Thanks in advance

 

Hi @tapatrick,

Yeah, very good question but a broad one also...

 

I think it starts at first principles which is that we need to appreciate the limits of human hearing; from there we can then talk about specifics like what makes a frequency response "sound different", what noise levels we need be concerned about, then the time domain parameters like phase shifts and related words like "group delay", etc...

 

I suggest making sure you read this first:

http://archimago.blogspot.com/2015/10/musings-meditations-on-limitations-of.html

 

In it, I speak about some of these basics. Then perhaps we can touch on each of these measurement parameters and see if together we can discuss the implications and where we can find evidence of audibility and significance then look at the hi-fi gear we own and see if these characteristics may be inadequate. Some of this I've discussed in my blog and will point there as appropriate...

 

Archimago's Musings... A "more objective" audiophile blog.

Free The Music - No MQA!  :nomqa:

Link to post
Share on other sites
22 hours ago, fas42 said:

 

And that is how one can be "objective" in areas where the people usually will use the term, subjective ...what one does is use tracks of music whose content very strongly provokes the system playback to distort - the anomalies are obviously audible; so in that area it's a fail, for the setup.

 

The process of "sorting out" is eliminating each of the failure 'modes', one by one.

 

Sure, it's good to one-by-one look at issues and sort them out in our systems. Over time, this should lead to optimization as the system "evolves" to resolve audible issues...

 

Archimago's Musings... A "more objective" audiophile blog.

Free The Music - No MQA!  :nomqa:

Link to post
Share on other sites
8 minutes ago, Archimago said:

 

I think it starts at first principles which is that we need to appreciate the limits of human hearing; from there we can then talk about specifics like what makes a frequency response "sound different", what noise levels we need be concerned about, then the time domain parameters like phase shifts and related words like "group delay", etc...

 

I suggest making sure you read this first:

http://archimago.blogspot.com/2015/10/musings-meditations-on-limitations-of.html

 

 

Howdy, just curious whether you're familiar with Bregman's "Auditory Scene Analysis", and subsequent research?

Frank

 

http://artofaudioconjuring.blogspot.com/

 

 

Over and out.

.

 

Link to post
Share on other sites
37 minutes ago, fas42 said:

 

Howdy, just curious whether you're familiar with Bregman's "Auditory Scene Analysis", and subsequent research?

Not familiar with this Frank.

 

I see there's a review here that seems like a good read perhaps:

https://www.frontiersin.org/articles/10.3389/fnhum.2011.00158/full

 

What are your thoughts on this?

 

Archimago's Musings... A "more objective" audiophile blog.

Free The Music - No MQA!  :nomqa:

Link to post
Share on other sites
2 hours ago, Archimago said:

 

Hi Bill,

I certainly agree that many things we measure today are either minor or do not lead to audible differences. A great example I think is JITTER. Yes, we can measure this with all those sidebands down below -100dBFS in modern asynchronous DACs, but seriously, who can hear them unless VERY severe - basically "broken", "engineering flaw" gear? (See my DEMO post awhile back.)

 

Thank you for the long reply, Archimago.  It is interesting.  I agree with a lot of what you wrote, but there are some subtle differences, even just in context, or history, or something.  I'll try to explain.

 

I do agree with the above in the sense that Jitter is a "solved" problem and should not cause audible problems or be an issue in a competent design.  I can't think that I would ever consider buying a DAC that tested poorly in this regard.  I have to think back, though to the (?) 80s when this problem wasn't being addressed by designers of audio components (as opposed, possibly, to the users of digital circuits in other fields).  Objectivists at the time thought digital as it existed was "solved."  DACS, I think it is clear to see in retrospect were "tweaked," or "kludged" by "high-end" designers in the search for "better sound" (?euphony).

 

As things evolved I learned about jitter via Stereophile, saw early measurements being conducted and how poorly some DACs measured (as well as some of the tweaky boxes that were designed to fix this and didn't):

 

https://www.stereophile.com/content/2020-jitter-measurements

 

A subjectivist provocateur would say that before jitter was fixed in DACs that they were being told by objectivists that they were "crazy," that digital was "solved," and that no additional measuring techniques were needed as all important parameters could be characterized with the current state of the art......

 

But yes, I agree jitter is solved and am not holding my breath for another clear, measurable area for improvement in digital-reproduction to be found.

2 hours ago, Archimago said:

Despite this, notice how "jitter" continues to be claimed by various companies as being important... Go talk to Paul McGowan and Ted Smith and their PS Audio DAC. Isn't it more than a little suspicious that they never provide measurements/evidence to show the audible issue despite their verbal claims!?

 

Yeah, not sure I get what is going on there, to say it politely.  I have seen the measurements on Sphile and ASR and am not terribly impressed.  Some people that I respect like them, so....heck, I don't know.

 

As a consumer I don't really care if they don't provide measurements, it's up to them.  Would I buy one? No.  Would I try to "save" someone from being "fooled" into buying one?  No.  Would I take the opportunity to discuss the facets if they wanted?  Sure.  If they are happy and enjoying their music, more power to them.  I am not a crusader.  Don't think it would work anyway, people are so, so variable.

 

2 hours ago, Archimago said:

Your comment "develop a body of measurements that correlate with perception (positivenegative, and euphonic)" is insightful. Nice. I believe that it is only with objective testing and controlled listening (measurements of the subjective experience while minimizing non-sound-related biases) that we can differentiate "positive, negative, and euphonic". I think in fact we are getting to the point where of the three, "euphonic" is becoming most important.

 

Again, I agree to some degree :).  Objective testing? Yes!  I'll take all the sources of information I can get!  Controlled testing?  Sure, but.....  I like the blinded testing at Harman as an example.  I am simply not yet convinced that it is universally applicable to the testing of all audio gear, though hopefully, as a physician you won't find me too hypocritical for believing in blinded, placebo-controlled trials for medicines, etc...... :)

 

Not sure if you recall, but decades ago JA organized a carefully controlled DBT of amplifiers.  One a tube amp, the other SS.  The tube amp had an output impedance that would have to produce frequency-response changes into the varying impedance v frequency of the speakers that were audible by anyone's criteria.  Null result.  Why?  I don't know.  I have some guesses but I don't know.

 

Yes, euphony the way you describe it is huge.  Are subjectivist reviewers now simply describing differences in euphony?  Certainly possible (thought not exclusively, my gut says).

 

2 hours ago, Archimago said:

Would be rather unexpected if audiophiles ever advocated for such a thing because for some it's "euphonic"!


Some, probably.  I hope others, though, are more self-aware and consider it a possibility.  Or maybe not :)

 

I suspect you are familiar with Nelson Pass' work in this regard.  He is fairly matter-of-fact that he knows 2H HD is euphonic, this informs his designs, and has even produced a gizmo that allows manipulation of its phase and level and told folks to have fun.

 

Re TTs.......In the 90s I built one using Well Tempered Reference components (arm, motor, platter, bearing).  I used multiple levels of Aluminum alternating with Isodamp for CLD, all mounted on a Newport laser table (just the base), pneumatically isolated and self-leveling.  Enjoyed lots of good sound.

 

Now, though just listen to music from an HD, having sold off most of our stuff (not just audio) in a wonderful, simplifying way.

 

I still have lots of files I recorded digitally, though :)

 

Thanks again,

 

Bill

 

 

Labels assigned by CA members: "Cogley's ML sock-puppet," "weaponizer of psychology," "ethically-challenged," "professionally dubious," "machismo," "lover of old westerns," "shill," "expert on ducks and imposters," "Janitor in Chief," "expert in Karate," "ML fanboi or employee," "Alabama Trump supporter with an NRA decal on the windshield of his car," sycophant

Link to post
Share on other sites
15 hours ago, fas42 said:

Yes, would be worth diving in ... note a thread over on ASR, where John Kenny and I posted a lot of thoughts, and links on this area of research, https://www.audiosciencereview.com/forum/index.php?threads/auditory-scene-analysis.236/

 

Thanks Frank,

I'll take a look at the review first and consider. The thread looks pretty unwieldy and will need to have a look at the basics first to see if the topic seems to correlate significantly towards audiophilia...

Archimago's Musings... A "more objective" audiophile blog.

Free The Music - No MQA!  :nomqa:

Link to post
Share on other sites
2 hours ago, Archimago said:

Good points there Bill. I was but a lad back in the 80's (especially early to mid 80's) before having the resources to own or explore hi-fi so can't really speak to what the debate was like between objective and subjective folks...

 

:)  

 

I was 14 in 1982 when the first CD players came out(?).  I began reading everything about this hobby that became my lifelong love then, went through all the normal stages (gear-lust, reading the Audio Magazine annual equipment catalog like a kid used to read the Sears catalog before Christmas, etc.).  I would watch the mailbox for the latest issues of Audio, Stereophile, and, I must confess, Stereo Review and inhale all of them in less than a day, then resuming the month-long wait.  Can't believe this all makes me feel old at 51!  This was basically the ONLY way to get info on the hobby except for the one friend I found in college that shared my passion!  An AMAZING contrast with today.  Wow!

 

History is important to me.  Subjective reviewing grew from JGH for Stereophile, and a bit later HP from TAS as a reaction to the measurement only era preceding it, seen in magazines like High Fidelity(?).  JGH swung the pendulum towards subjectivity and I think it was needed.  Remember, though, that he used measurements as able at the time- find one of his old reviews and check out a primitive FR measurement of a phone cartridge!  He was in pursuit of "high fidelity," in, I think, the truest definition of the term, and actually decried the further swing of the pendulum towards the subjectivist-only, "how the sound makes me feel," "if it sounds good to me" type of pursuit.

2 hours ago, Archimago said:

I say things but I too am not all that interested in the "crusade". No need to "fight" or force anyone's personal beliefs.

 

Yes, I sense a healthy, pragmatic, non-judgemental approach to your writing and the engineering chops you bring to the table- whoever the heck you are! :)

 

I was thinking of the things this morning that would lead me to a "righteous crusade."  There are some, perhaps, but certainly none in this realm.

2 hours ago, Archimago said:

I will say one thing about the importance of blind tests though. When we come across a claim by a "pure subjectivist" that he/she "can easily hear the difference between the two USB cables" (or whatever contentious issue), I would consider that person as a prime candidate for blind testing.

 

Yes, why someone would set themselves up for that is a mystery to me.  I am confident about some stuff, but........ I try to minimize the possibility of getting even close to Hubris.

 

3 hours ago, Archimago said:

what is the effect of the tube amp's lower damping factor on the frequency response?

 

I can't remember the exact numbers, but certainly differences that should be perceived as outlined in the relevant literature, my understanding of which suggests that narrow cuts and peaks are difficult to perceive, but that (even fairly subtle) frequency response changes that affect wider frequency bands are perceptible.  Note that in this case, as suggested by JA's measurements of amps into a simulated load, that they should have been.  Again....euphony?

 

I don't want to overstate, though, the test as a "gold standard."  I found it intriguing at the time and it remains in the back of my mind, especially with my personal experiments with blind testing.

 

Bill

Labels assigned by CA members: "Cogley's ML sock-puppet," "weaponizer of psychology," "ethically-challenged," "professionally dubious," "machismo," "lover of old westerns," "shill," "expert on ducks and imposters," "Janitor in Chief," "expert in Karate," "ML fanboi or employee," "Alabama Trump supporter with an NRA decal on the windshield of his car," sycophant

Link to post
Share on other sites
6 hours ago, John Dyson said:

 

Some of the problems with objective review might include: the impairments aren't adequately defined/described, sometimes the impairments are difficult to measure, and sometimes even the impairments come from multiple sources that meld together into something wrong with the sound.

 

 

Some of the impairments are easy to detect, and describe - my first good amplifier of 35 years ago, a supposed powerhouse, would start distorting in the treble, when its circuit reached a certain load into the speakers - I used the splashing of cymbals in a driving rock track to pick this; below the particular SPLs a very natural shimmer to the sound, above that sound level, they became saucepan lids - majorly distorted. To put this into context all other amplifiers I used this test on, that I came across, fared far worse - a Krell was particularly poor, 😜.

 

This was straightforward to resolve, though it took some time - for a couple of other reasons I suspected the amplifier's power supply, and after some major re-engineering of this area, this problem then disappeared.

 

The point is that it's not always hard - if one has the right test material, and a good understanding of electronics then the logical steps needed to resolve less than stellar SQ can be followed without too much difficulty.

Frank

 

http://artofaudioconjuring.blogspot.com/

 

 

Over and out.

.

 

Link to post
Share on other sites
10 hours ago, STC said:

Our hearing is not flat nor consistent. It varies through out the day.

Certainly would explain a lot. I always wonder when listening faculties are treated as some kind of scientific instrument.  

ATL DC Blocker > Topaz 2.5Kva Isolation Transformer > Sine SA5 Cryo 5 power strip > EtherRegen switch powered with Ciunas Supercaps 7.5v LPS > Laufer Mini Memory Player> EC Designs U192 USB Transport plus Fractal DAC > Decware SE84UFO3 Mono Amps > TNT UBYTE-2 Speaker cables > Omega Super Alnico Monitors. PH SR4 & Ian Canada UConditioner Ultracap 5V power supplies. 

Link to post
Share on other sites
On 2/23/2020 at 9:10 AM, John Dyson said:

For people who aren't tied up in with strong opinons, I see the disagreement between those who tend to be 100% subjective vs. those of us who tend (but not 100%) objective partially described in the following statement:  it is mostly related to the fact that SOMETIMES it is difficult to measure certain impairments.   I think that people move towards the subjective out of frustration caused by inadequate objective information.  (There can probably be a lot of  reasons for the incomplete/inadequate objective information.)

 

Some of the problems with objective review might include: the impairments aren't adequately defined/described, sometimes the impairments are difficult to measure, and sometimes even the impairments come from multiple sources that meld together into something wrong with the sound.

 

Example: on the case of TIM, which can really exist (and used to REALLY exist in older designs), we didn't initially 1) understand what caused the impairment, and 2) it can be tricky to measure, esp with techniques used in the '60s/'70s.   TIM is a sibling of modulation distortion in gain control devices also, it happens when signals are changing character...  It can take a while to understand,define, describe an impariment well enough to quantify it.  TIM and siblings can be tricky to measure, and there more than 'one' kind of TIM in the sense it is dependent on lots of variables.   This is ONE example that might have discredited 'measurements' in some peoples minds.

 

This doesn't mean that objective measurement can be discredited, in fact when it is applicable, it must be a PRIMARY way of evaluating a design, and eventually the subjective becomes a double check.  On the other hand, complex designs can demand certain kinds of testing where there is no appropriate measurement device.  Subjective evaluation is needed until (if ever) an objective method is developed.

 

Hi John, yes, well put!

 

Many of the fights we get into originates from the "all or none", "black or white", "100%" mindset. As mature adults, we know that the only way to handle the complexities of life (of which the squabbles of audiophilia is but a tiny microcosm) is to find the middle ground... The "shade of grey" between the subjective and objective. Some things do need to be "more subjective" just as my preference is to be "more objective" when it comes to audiophile gear.

 

It's sad to see how there was a time when at least we appreciated some of the "basics" of objective analysis (like Bill mentioned about JGH and phono frequency response). These days, the majority of online sites and at least a big proportion if not the majority of magazines have taken out even a simple frequency response; much less consideration for noise level or time-domain characteristics.

 

In part, I think that as technology has progressed, we can often "assume" that most devices are quite "good" already. Or at least good enough that most listeners will not hear a problem. While technically not great, it's fine when the subjectivist reviewer doesn't hear a problem with a DAC but yet John Atkinson measures relatively high jitter... Since jitter (IMO) was never a terrible boogeyman, the device still "sounds good" assuming the rest of the performance is reasonable. The issue then becomes one of "value" for the price (given suboptimal engineering), and whether it would be appropriate for the company to hype up a product as "one of the best DACs in the world" for example.

 

I can certainly agree with the idea that objective parameters should take primacy in design and then of course confirmed with listening. Sure, we have some products "designed by ear" and who knows, some will sound good as well. Probably best to at least double check with objective measures if something is designed by the ears of a man with many decades of life behind him 😲.

 

When it comes to audible distortions, using multiple blinded listening tests to identify and verify audible anomalies then use objective testing to measure the problem probably is the best way to effectively weed out issues, and keep findings relevant (eg. audible TIM back in the day might be an example). These days, the "golden ears" who claim to hear differences but never bother to run a blind test to prove to themselves and others that the impression is anything more than a mirage would not be helpful in this endeavor.

 

On 2/23/2020 at 9:10 AM, John Dyson said:

I can give parallels to the situation on my current project -- but I do not want to divert attention from the matter at hand.  The bottom line is that objective measurement and evaluation is critical for a design to be created and completed.  A pure 'design by sounds good' is only going to work for the most simple design with simple interactions.   A 'design by spec' and requirements to meet objective criteria is important for a non-trivial design to be workable.

 

NOTE:  design by finding 'sweet spots' and avoiding 'rabbit holes' is NOT the best engineering design method and can sometimes make the best of us into a sucker, wasting lots of time doing tweaking.   If the tweaking can be avoided, then tweaking MUST be avoided and it is very worthwhile to sit down and do a real design -- the worst of time wasting chasing rabbits into rabbit holes can be avoided.

 

On conventional circuitry and software, where the specifications, requirements and behavior can be accurately measured, almost pure objective design is best.  Secondary subjective review is also important, sometimes specs and measurements miss details that are unforseen.

 

On complex circuitry and software, that is, 'stuff that hasn't been done very often', then subjective review is so important, but objective focus is necessary -- the degrees of freedom and interactions could cause a 'design by sounds good' to become a random walk, falling into rabbit holes all of the time.

 

I think your current project with the "feral" Dolby A material is a beautiful example and some of the results I've heard certainly speaks to this. A nice melding of the objective science of how the old analogue noise-reduction system worked with its various filter bands yet there is the need for subjective listening to get the variables sounding "right" - the "art" side of things... A reflection of how folks in the studio also used their ears to tweak the music for the best sound and would have applied customized settings here and there to achieve the ultimate goal.

 

Needless to say, the moment we have real music involved (not just reproduction of "data" with high fidelity), there is no option but to integrate "art" and subjectivity into the equation.

 

On 2/23/2020 at 9:10 AM, John Dyson said:

WIth my mostly objective view, sometimes I must unfortunately depend on my hearing, or accept input from other peoples perception.  Very often, the subjective feedback has been helpful in resolving actual bugs that I couldn't measure.   Subjective feedback from uncontrolled experiments is notoriously unreliable though -- and must be considered on a statistical basis and not as a measurement with negligible error.   Subjective review can become so distorted and become totally emotional.   When feedback is too distorted by emotion and/or all of the human foibles that can affect subjective review, then that data source should be ignored.  I run into that problem all of time, and must quit depending even on my own senses  -- human perception is definitely unreliable, but good information can often be derived.


Both subjective and objective review are necessary -- but each has it's limitations.

In a way, after a person really considers and truly understands the complexity of their equipment, I cannot imagine why someone would be strongly biased away from objective review.   It just doesn't make sense to disparage good objective review of any technical device.   Subjective review is important also -- it is the attitude that disparages one or the other -- doesn't make sense.

 

John

 

 

"Subjective feedback from uncontrolled experiments is notoriously unreliable though". Yes, for sure. I suspect most audiophiles appreciate this fact even though strangely enough, it seems like some have unwavering faith in certain high profile "golden ears"... "In Mr. Purely Subjectivist Reviewer's ears we trust!" Without a fair amount of critical thinking, seems a little dangerous 😟.

 

Thanks again for the work and thoughtful insights John.

Archimago's Musings... A "more objective" audiophile blog.

Free The Music - No MQA!  :nomqa:

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×
×
  • Create New...