Invite: Blind Listening Survey - "High-End" DACs...

[jabbr]

7 hours ago, pkane2001 said:

Laugh all you want, but Archi is under no obligation to make every one of his blog posts into a scientific study. Your perception of objectivity is a bit skewed.

Good point. There is a common misconception that objective results mean valid results

[PeterG]

On 5/14/2024 at 7:54 AM, pkane2001 said:

Laugh all you want, but Archi is under no obligation to make every one of his blog posts into a scientific study. Your perception of objectivity is a bit skewed.

 

19 hours ago, firedog said:

1. It's a lot more scientific and objective than single user sighted listening impressions;

2. The information gleaned may or may not be useful. I don't think anyone is going to claim it's scientific proof of anything. Each person can use or discard the results as they see fit. 

 

I have no objection at all to non-scientific posts.  Very few of my own posts are scientific.  My objection is to non-scientific posts/studies dressing up as scientific studies--it bastardizes real science, and enables the spread of misinformation.  You can be sure that months from now when there's a review of a $30,000 DAC on AS, somebody will be chiming in with "Archi showed a while back that people can't tell the difference between DACs".

[pkane2001]

10 minutes ago, PeterG said:

I have no objection at all to non-scientific posts.  Very few of my own posts are scientific.  My objection is to non-scientific posts/studies dressing up as scientific studies--it bastardizes real science, and enables the spread of misinformation.  You can be sure that months from now when there's a review of a $30,000 DAC on AS, somebody will be chiming in with "Archi showed a while back that people can't tell the difference between DACs".

 

Your objection seems to be to what others might claim about the survey. Probably best to address those directly, as they come up. Again, it's a survey. It can't "bastardize real science" as it is not a scientific study or experiment. It's a survey.

 

I presume Archi's results will be detailed enough for anyone who wants to do so to reach their own conclusions. Some may conclude that the design was flawed, some may conclude there is no conclusion possible, and some may reach their own conclusion, one way or the other. The benefit of such a survey is that it can be discussed intelligently, with conclusions and merits judged from the data and not from the anticipation of what someone, somewhere might say or claim about it.

 

[Iving]

Scenario #1

 

If Archimago shows that participants preferred a particular DAC, or ranked DACs in a consistent order, that will be most interesting in itself. It will demonstrate that either:

1. His approach was legitimate - that is - the one in which he argued that DACs at hugely variable price points might (or ought) to be discriminable in our domestic listening environments - even through another ADC-DAC loop;

or,

2. That some rogue attribute of the stimulus germane accounted for that preference. In my submission, I advised Archimago that I wasn't convinced by reassurances of volume levelling (notwithstanding the method/evidence presented). My preferred DAC seemed (marginally) louder to me. It might have been my brain telling me so. If a substantial # participants preferred different DACs to me, then I would be inclined to accept the "my brain" explanation. If a significant number of participants preferred the same DAC as me, I would say a discussion about apparent loudness is warranted.

 

Archimago could matrix-out the preferences (A, B, C) against their price points. Of itself the Table will be interesting. Default is that participants will prefer the more expensive DAC. If participants prefer the least expensive one, Objectivists of the ASR (Topping) hue will squirm with delight (and we shall need to rule out "loudness" explanations).

 

DAC	Retails at $	# participants preferred as 1st Choice
A		n
B		n
C		n

 

 

Scenario #2

 

It's implicit in the expression of preferences that participants (think they) can hear a difference. Scenario #2 is that participants say they cannot. I think it's pretty unlikely that this will happen in large numbers. The power of suggestion is too strong. I am assuming the whole exercise is not a ruse / placebo trick in which Archimago is presenting three identical digital stimuli!

 

 

Scenario #3

 

Participants express preferences, but the data are not consistent. People vary greatly in which DACs they prefer. This is going to produce a lot of statistical noise. The conclusions are either:

1. The DACs are not really discriminable at all, and that the power of suggestion/expectation is just causing people to pick DACs pretty much at random. Those who argue that the additional ADC-DAC loop is an obfuscation may feel vindicated. People may have other arguments about sensitivity of Method/Procedure.

2. The DACs are discriminable, and people just vary in which they prefer. Their comments/remarks might help Archimago understand which attributes of which DACs were important to which participants.

In his blog for this exercise, Archimago introduced notions of "audibility" and types of "subjectivism".

In the event of Scenarios #s 2 and 3.1 it's going to be very difficult to assert that actual audibility has not been adulterated by ADC-DAC loop lens or some other aspect of the exercise that introduced error.

Scenario 3.2 would constitute support for a particular type of subjectivism (akin to liking tube amps etc) – but I'm not sure whether Archimago will be able to effectively differentiate between 3.1 and 3.2 in the event of Scenario #3.

 

 

Scenario #4

 

Only those with expensive systems were consistent in their preference for one or more DACs. This is a moderating effect; i.e., the relationship between preference (Dependent Variable) and DAC/price (Independent Variable) is moderated by cost of participant listening system. I'd be surprised to see a strong moderating effect. But it would suggest that audiophiles are indeed hearing good stuff that common-or-garden music lovers (with cheap systems) aren't.

Archimago solicited other data which could affect/moderate preferences: age, sex, whether musician etc.

 

 

Scenario #5

 

There is a lot of random reporting – but one or two participants are curiously on beam with their choices and their remarks. Archimago has identified a "Golden Ear" in one of his past exercises. I imagine this suggests that there is a minority of folks around with "attuned brains" whatever the explanation for that tuning might be. [Jud enters Stage Left with "pattern" assertions and anecdotes.]

 

 

The point of my post is not to argue that Archimago's exercise is - or is not – "scientific". It is to argue that interpretation is everything. This exercise can produce a "strong" result – Scenario #1 – and that will be upheld as "scientific" by those who identify with the effect demonstrated. [DACs do sound different. Expensive ones are justified/not justified.] It can produce a "weak" result – Scenarios #s 2 and 3. It is here that the likes of Peter G have a point. Ardent but unthinking Objectivists may say Archimago (an authority figure!) demonstrated that participants were just as likely to prefer a cheap DAC as an expensive one. But that may not be the case at all. There are several possible explanations for dispersion of expressed preferences.

 

 

If I haven't been clear enough – it's not so much about whether fun studies like this – a great service to the hobby – are "scientific" (sufficiently rigorous) or not. It's about whether we have our thinking caps on straight when we read the Results.

[krass]

I don’t think this gives anything away - I enjoyed the music selection so for future reference on this website, the tracklist is:

1. Eiji Oue & Minnesota Orchestra - “Infernal Dance of the King Kashchei” from Stravinsky: The Song of the Nightingale / The Firebird Suite / The Rite of Spring

2. TakéDaké & John Kaizan Neptune – “Japanese Roots” from Asian Roots

3. The Ray Brown Trio – “Summer Wind” from Live at the Loa: Summer Wind

4. Eva Cassidy – “Fields Of Gold” from Songbird

5. Benjamin Clementine – “Winston Churchill’s Boy” from At Least For Now

6. Dua Lipa – “Love Again” from Future Nostalgia

[Superdad]

While there is not much I can say that has not already been said here, I am very surprised that nobody brought up this:

 

The survey sample files should have included one file (still secret which one) that is the original—that is, the source material—not put through any DAC or ADC. 
Such would be a great “control” for the test—a sort of “placebo.” With that file people would be hearing only the effect of their own playback DAC. 

And it might be very telling if either a large number of people preferred that original source file or some of the others. 
 

Thoughts?

[Iving]

32 minutes ago, Superdad said:

While there is not much I can say that has not already been said here, I am very surprised that nobody brought up this:

 

The survey sample files should have included one file (still secret which one) that is the original—that is, the source material—not put through any DAC or ADC. 
Such would be a great “control” for the test—a sort of “placebo.” With that file people would be hearing only the effect of their own playback DAC. 

And it might be very telling if either a large number of people preferred that original source file or some of the others. 
 

Thoughts?

 

immediate thoughts:

a) All stimuli must be in same bit/freq etc format - thus "experimental" files would have to match your original/control - either that or original would need conversion - more futzing about for Archimago who has a convoluted-enough data set. [Edit: NB: Archimago's stimulus files are a Medley.]

b) Has Design issue for current proposal "telling the difference between DACs" - since there is no DAC with your suggestion ... A control level of the IV is best embedded in a Design where effect on DV of an experimentally deliberate change in IV is interpreted ... In your suggestion how would you interpret DV variation across levels of the IV including "no DAC"? Your hypothesis would be that no DAC is purest possible stimulus and should result in higher preference ratings? That would answer a research question that might not interest Archimago - who is primarily concerned with whether DACs varying hugely in price point are discriminable (from each other - not a baseline).

[Superdad]

4 minutes ago, Iving said:

Your hypothesis would be that no DAC is purest possible stimulus and should result in higher preference ratings? That would answer a research question that might not interest Archimago - who is primarily concerned with whether DACs varying hugely in price point are discriminable (from each other - not a baseline).

 

Thanks for your reply. But it all seems like excuses for a dumb test anyway.  

The whole test is like doing a comparison of high-res TV displays by watching them on your own home TV as captured by a video camera--and expecting to make judgements about fine details. 

[Iving]

Just now, Superdad said:

Thanks for your reply. But it all seems like excuses for a dumb test anyway.  

The whole test is like doing a comparison of high-res TV displays by watching them on your own home TV as captured by a video camera--and expecting to make judgements about fine details. 

 

You shall be disproven is there is an overwhelming or statistically significant consistent preference or ranking from participants!

 

otoh many feel like you do - myself included somewhat - equates to Scenario #3 in my earlier post - yes - thorny issues in interpretation.

 

Secretly I hope I am the only person who ranked the DACs in their "correct" order. I shall set myself up as Consultant to High-End.

[danadam]

4 hours ago, krass said:

I don’t think this gives anything away

Given that he himself links to the detailed description of the test track, no, it doesn't give anything away 🙂

Archimago's Musical Performance Track (AMPT)

[danadam]

3 hours ago, Superdad said:

Thoughts?

Anyone who is so inclined (admittedly, probably not many) can try to ABX the tracks between each other or with the original.

[Superdad]

15 minutes ago, danadam said:

Given that he himself links to the detailed description of the test track, no, it doesn't give anything away 🙂

Archimago's Musical Performance Track (AMPT)

 

7 minutes ago, danadam said:

Anyone who is so inclined (admittedly, probably not many) can try to ABX the tracks between each other or with the original.

 

Thanks for posting the 2021 link to his original files.  I downloaded it. In 16/44.1KHz but I can SRC it to 96KHz to match the DAC>ADC processed files.

[Mops911]

results?

[audiobomber]

3 hours ago, Mops911 said:

results?

Yes, please! I didn't do the test until the last day because I hate waiting for results. C'mon Arch, we don't need the full analysis. just tell us which DAC was which. 🙂

[Archimago]

Well folks, time to unblind:

 

PART I: "High-End" DAC Blind Listening: Devices Unveiled!

 

Sample A = Apple USB-C dongle - ~$10

Sample B = Linn Majik DS + Dynamik Power Supple - ~$3,000 when new

Sample C = Linn Klimax DSM/2 - ~$20,000 when new

 

Both Linns used much better cable, streamed over ethernet from Roon than the Radio Shack wire for the Apple dongle played back on a laptop over Windows 11 in Foobar.

 

105 responses. Thanks everyone. Analysis to follow...

 

How did you do?

[Iving]

17 minutes ago, Archimago said:

How did you do?

 

:-)

 

iirc

I dismissed A.
I said B or C (to myself).
At first thought B as mellower.
But chose C with confidence.

 

This was on a cheap Desktop system I use to tag up my flac Library - not my main system.

And I spent no more than 2 mins. total.

 

Your essential thesis - that DACs should sound different by price is supported in my case. Of course the overall Results will speak much louder.

 

Talking of loudness, I did remark in my comments about C seeming marginally louder to me. Again - what other people say matters.

 

Thanks again and again for this.

 

Edit: P.S. Brand-wise this is a bit of an ad for Linn! Generalisable across manufacturers?

[copy_of_a]

My ranking was C, B, A (so if you want so it was "correct").

I've decided to opt for the differences are not significant.

I've hestitated with that choice because A was horrible (lifeless and noisy).

But regarding B and C I would very well enjoy music with both of them.

C has a more refined high end and B tends to be a bit more beefy in the low mids (depending on the track).

However, without doing a comparision and only listening to B in isolation I don't think I'd really miss anything.

[krass]

I slightly preferred B, followed by A&C / C&A

 

All of them sounded a bit like my own system ;)

 

 

[Pour Decisions]

On my system, I preferred B, and didn't care for A.  Ranked them B, C, A in the poll.

[robocop]

I voted the same B,C and A last. Interesting when I saw that he used Linn players I'm not surprised I've always had trouble with Linn audio. Going back to the original Linn Sondek turntable which I did own once, over rated and expensive.

 

Would have been interesting to see a player like the T&A 200 or Emmlabs DV2 in the mix rather than the Klimax.  

 

The point is the difference is marginal so why spend more than $10!!!! 

[STC]

Dissapointed. I chose C,B and A. But unlikely to guess which is better with A/B’ing. 

[Iving]

7 hours ago, robocop said:

Interesting when I saw that he used Linn players I'm not surprised I've always had trouble with Linn audio. Going back to the original Linn Sondek turntable which I did own once, over rated and expensive.

 

ime Linn make great electronics but unconvincing speakers. Perhaps you are in a minority having a poor view of the LP12. My own pov is that Linn are overpricing the LP12 in the 21st Century - you can spend many k on a new LP12 and not get so much as an Ittok.

 

What strikes me about Archimago's selection of 2 x Linn devices as the serious contenders in this survey - aside from the brand restriction per se - is that the devices are not DACs as such. They are internet streamers and more besides, and have sophisticated circuitry going way beyond the DAC brief. They don't just DAC. What's more - they are separated by many years regards their production date - during which time Linn tends to move apace with upgrades to e.g. PSs. Also – Linn has evolved an approach to music reproduction – specifically hinged on digital routes through a system – that is market-atypical (even unto digital RIAA in the Klimax LP12 record player). iow any discernible differences in SQ between these two Linn DUTs could have as much to do with attributes of theirs barely related to Digital-Analogue conversion specifically.

 

I really appreciate Archimago presenting us with opportunities such as this. And we have not yet seen even the fundamental Descriptive Statistics (how many people preferred which DUTs or how they were ranked). But I am coming round to thinking that - even when we achieve a sensible interpretation of the data - "DAC Blind Listening Survey" shall seem a misnomer. "Can you hear the difference between two Linn streamers?" will seem a fairer depiction.

 

Going on just the very few self-reports I have seen here, on SH – and on Archimago's blog – it might be a close run thing between B and C. imo we are going to struggle with interpretation given anything but a clear favourite or ranking.

[lmitche]

C,B,A here.

 

Bass was thin on A.

Bass was full, but smeared, on B.

Bass was full, and more detailed, on C.

[Superdad]

"A" was indeed awful, but honestly all three versions were poor in comparison to the original tracks (and yes, I compensated for the level difference).  I used just the Ray Brown selection as piano, drums, and audience claps made the differences most obvious.

 

I still have no idea about the point of this comparison.

Odd choice of DACs, tracks, no original rip thrown in as control, somewhat small survey size (105 people worldwide), so I hope this does not get touted as some big objective test to prove that people can't hear the difference between DACs.

[danadam]

Part 2

https://archimago.blogspot.com/2024/05/high-end-dac-blind-listening-results.html