Is Someone Playing Fast & Loose With Measurements?

[The Computer Audiophile]

Hi Guys - I haven't paid much attention to the measurements published by LH Labs, a division of Light Harmonic, but they were recently called to my attention by AudioQuest. Much of this is beyond my technical expertise, but the information is certainly interesting. Hopefully we can get comments from both sides to help us figure out what's fact and what's fiction. Below is information claimed by LH Labs and refuted by AudioQuest.

 

 

Please let me know if any of the information I posted is incorrect or misleading, I can update the data anytime. I just want to get to the facts. Thanks guys.

 

 

 

 

[TABLE]

[TR]

[TD]Date: 8/16/2013 Testing: GEEK vs. the Others[/TD]

[TD][/TD]

[TD][/TD]

[/TR]

[TR]

[TD]Item[/TD]

[TD]LH Claims[/TD]

[TD]AudioQuest Claims[/TD]

[/TR]

[TR]

[TD]Volume Control[/TD]

[TD]Software[/TD]

[TD]Software Controlled Analog[/TD]

[/TR]

[TR]

[TD]Output impedance[/TD]

[TD]12 (note 1)[/TD]

[TD]0.65 ohms (note 2)[/TD]

[/TR]

[TR]

[TD]Amplifier Type[/TD]

[TD]Class A/B[/TD]

[TD]Class A[/TD]

[/TR]

[TR]

[TD]THD+N[/TD]

[TD]0.041% (note 2)[/TD]

[TD]0.041% (note 2)[/TD]

[/TR]

[TR]

[TD]Max Native Bitstream[/TD]

[TD]None[/TD]

[TD]Sample Rate Equal (note 3)[/TD]

[/TR]

[/TABLE]

1. 12 ohms is specified in the DragonFly manual as the minimum load to meet the specifications. Not the output impedance.

2. Stereophile a Source Interlink Publication Vol 35, Number 10 October 2012, tests by John Atkinson pages 143-148, review by Art Dudley. Link to measurements

3. Sample Rate Equal, implies that the correct fixed oscillator that is a base 2 multiple of the selected sample rate.

 

 

 

 

[TABLE]

[TR]

[TD]Date: 9/5/2013 Testing: GEEK vs. the Others, v2[/TD]

[TD]LH Claims[/TD]

[TD]AudioQuest Claims[/TD]

[/TR]

[TR]

[TD]Output Impedance[/TD]

[TD]5.9 ohms[/TD]

[TD]0.65 ohms (note 2)[/TD]

[/TR]

[TR]

[TD]Maximum Output V[/TD]

[TD]2.106 and 2.105 (note 4)[/TD]

[TD]1.86V (note 2)[/TD]

[/TR]

[TR]

[TD]THD+N[/TD]

[TD]5.038965 and 4.860163%[/TD]

[TD]0.041% (note 2)[/TD]

[/TR]

[/TABLE]

 

4. The DragonFly is only capable of these output voltages when the testing level is greater than a gain of 0dBFs. Meaning the tester is deliberately clipping the signal before it reaches the DragonFly.

 

 

 

 

 

 

[TABLE]

[TR]

[TD]Date: 3/15/2014 Testing: Geek Out vs. the Others, 3rd Installment[/TD]

[TD]LH Claims[/TD]

[TD]AudioQuest Claims[/TD]

[/TR]

[TR]

[TD]Noise Floor[/TD]

[TD]~ -120dB[/TD]

[TD]See Sine1KHz-44.1Khz-AC-and-Battery[/TD]

[/TR]

[TR]

[TD][/TD]

[TD][/TD]

[TD]See Sine1Khz-96Khz-ACPower[/TD]

[/TR]

[TR]

[TD]Volume Control[/TD]

[TD]It doesn’t appear to[/TD]

[TD]Computer Controlled Analog (Note 5)[/TD]

[/TR]

[TR]

[TD][/TD]

[TD]have computer-[/TD]

[TD]not Digital volume control (note 5)[/TD]

[/TR]

[TR]

[TD][/TD]

[TD]controlled digital[/TD]

[TD]See file DragonFly-Enumeration[/TD]

[/TR]

[TR]

[TD][/TD]

[TD]volume attenuation,[/TD]

[TD][/TD]

[/TR]

[TR]

[TD][/TD]

[TD]which may cause[/TD]

[TD][/TD]

[/TR]

[TR]

[TD][/TD]

[TD]issues with software[/TD]

[TD][/TD]

[/TR]

[TR]

[TD][/TD]

[TD]that doesn’t have it’s[/TD]

[TD][/TD]

[/TR]

[TR]

[TD][/TD]

[TD]own implementation.[/TD]

[TD][/TD]

[/TR]

[TR]

[TD][/TD]

[TD][/TD]

[TD][/TD]

[/TR]

[TR]

[TD]Output voltage[/TD]

[TD]2.077 and 2.076Vrms[/TD]

[TD]1.8609V, 1.85965Vrms see Sine1KHz-96KHz-100percent -Notes 7&8[/TD]

[/TR]

[TR]

[TD]THD+N[/TD]

[TD]5.086305 and 5.26774%[/TD]

[TD]See file #4 0.76905 and 0.7072%[/TD]

[/TR]

[TR]

[TD]Channel Separation[/TD]

[TD]-62.003dB[/TD]

[TD]-65dB (note 2)[/TD]

[/TR]

[TR]

[TD]5KHz Square Wave[/TD]

[TD]Why wasn’t a scope used?[/TD]

[TD]See file TEK-5Khz (Note 6)[/TD]

[/TR]

[TR]

[TD]0dBFS 1KHz Sine[/TD]

[TD][/TD]

[TD]See Sine1KHz-96KHz-100percent[/TD]

[/TR]

[TR]

[TD]Jitter?[/TD]

[TD]Why No Jitter Testing?[/TD]

[TD]See file JTEST-DragonFly[/TD]

[/TR]

[/TABLE]

 

5. Digital volume controls reduce the sample size. Analog volume controls retain the full sample and control volume in the analog domain.

6. The Ap525 should not be used as an oscilloscope. These have very limited processing range. The only true test is using a good calibrated oscilloscope for this kind of testing.

7. The only way to get 2.077Vrms out of the DragonFly was to send a 100% signal then add another 10% (meaning the sine wave is distorted before it reaches the DragonFly).

8. As you can see from file Sine1KHz-96K-110percent.gif The testing done by LH/Geek was causing clipping distortion before it left the computer. This is the only way to get 2.077vrms output

 

 

 

Sine1Khz-96Khz-ACPower

 

 

 

 

 

 

 

Sine1KHz-96KHz-100percent

 

 

 

 

 

 

 

Sine1KHz-96K-110percent

 

 

 

 

 

 

 

Sine1KHz-44.1Khz-AC-and-Battery

 

 

 

 

 

 

 

 

JTEST-DragonFly

 

 

 

 

 

 

 

 

TEK-5KHz

 

 

 

[barrows]

Chris: Respectfully, I trust that you ran a copy of this posting by both the folks at AudioQuest and Light Harmonic for comment before publishing? If not, please send both a link to this post and ask them to comment for increased clarity.

I doubt that either party is trying to willfully deceive the consumer, but clearly something is up here, and it would be very helpful for potential customers of either product to understand the nuances of what is going on with these claims and measurements.

[Andrew Allen Ballew]

The Dragonfly volume control is analog. As a matter of fact, the analog nature is a key design point and selling feature of the Dragonfly. For a competitor to suggest otherwise is at best irresponsible. Of course, the used car salesman bit isn't surprising giving in such a competitive field. Buyer beware.

[junker]

The DragonFly did clip when ran as designed if it was run in the top 3 steps of it's volume control. So ran at FS this could indeed be the case... I believe Gordon even said as much here at CA and then later there was a firmware update if I am not mistaken?

 

You can search for my reviews early on here at CA. The DragonFly sounded no better than the built-in audio in my Mac and went back to Crutchfield. I even tried Kimber interconnects. In the end it cost me a couple hundred bucks for a product that could not exceed the digital format of my Mac (24/96), or support USB 2.0, or DSD.

 

http://www.computeraudiophile.com/f6-dac-digital-analog-conversion/audioquest-dragonfly-24-96-asynchronous-universal-serial-bus-industry-standard-cables-connectors-and-communications-protocols-between-computers-and-electronic-devices-digital-analogue-converter-headphone-amp-12353/index12.html#post184175

 

They were first-to-market but my experience with that product is that the SQ was extremely disappointing.

[junker]

Also in Fair Disclosure it should be noted that AudioQuest is a paid advertiser at Computer Audiophile.

[Steve Silberman]

Hello Junker –

 

As you point out – the original (first production run) DragonFly did in fact clip when a 0db 24/96 signal was presented. Upon discovering this we took immediate steps to resolve the issue. It was only in the first production run (1.0) that the problem existed. the following production runs had a version 1.0c, which we used to identify amended DragonFly's (also measured by John Atkinson in Stereophile's review). Then in late November of last year we released the improved 1.2 version.

 

Regards,

 

Steve Silberman

AudioQuest

[email protected]

[junker]

Thanks for the clarification Steve. This makes sense.

 

Hello Junker –

 

As you point out – the original (first production run) DragonFly did in fact clip when a 0db 24/96 signal was presented. Upon discovering this we took immediate steps to resolve the issue. It was only in the first production run (1.0) that the problem existed. the following production runs had a version 1.0c, which we used to identify amended DragonFly's (also measured by John Atkinson in Stereophile's review). Then in late November of last year we released the improved 1.2 version.

 

Regards,

 

Steve Silberman

AudioQuest

[email protected]

[Larry Ho]

Few simple facts here...

 

1. Dragonfly do have clipping in -0dBFS. John Atkinson also mentioned in his review. He needs to dial down a few clicks to get normal result.

 

IF there is a need to do a special favorable treatment to Dragonfly. I could do that, actually I did it in our first test. The THD+N is still around -70dB or so.

 

Or if I do that, Meridian and iFi will jump out say: Not Fair? ;-)

 

2. JTEST is only a simulation of the simplest way jitter by toggle LSB... The JTEST doesn't have the 32 bit version yet, at least to my knowledge so I decide to skip it. Also, there are many professions claim that test is not accurate.

 

3. As an US manufacturer, I won't have any special favor for an UK vendor. We did Explore, iDSD and Dragonfly at the same time.

Meridian Explorer is normal there... Machine won't discriminate.

 

4. We have the most advanced 1G bandwidth scope on hand. With the distortion I saw on Dragonfly's output wave, we really don't need the big gun scope there.... Again, even if Audio Precision is NOT good enough, and don't need to see Geek out's test result. Why iDSD is performing much better than Dragonfly? Do I need to favor iDSD? :-)

 

One friend from Google is sending me 3 more good DACs, including Herus, micro streamer and D3.

 

I'm gonna publish that with the original test results.

 

Don't worry. Truth is the daughter of the time.

 

Enjoy!

 

The DragonFly did clip when ran as designed if it was run in the top 3 steps of it's volume control. So ran at FS this could indeed be the case... I believe Gordon even said as much here at CA and then later there was a firmware update if I am not mistaken?

 

You can search for my reviews early on here at CA. The DragonFly sounded no better than the built-in audio in my Mac and went back to Crutchfield. I even tried Kimber interconnects. In the end it cost me a couple hundred bucks for a product that could not exceed the digital format of my Mac (24/96), or support USB 2.0, or DSD.

 

http://www.computeraudiophile.com/f6-dac-digital-analog-conversion/audioquest-dragonfly-24-96-asynchronous-universal-serial-bus-industry-standard-cables-connectors-and-communications-protocols-between-computers-and-electronic-devices-digital-analogue-converter-headphone-amp-12353/index12.html#post184175

 

They were first-to-market but my experience with that product is that the SQ was extremely disappointing.

[Larry Ho]

Hi, Steve

 

Thanks for clear this out. Now we know why.

If possible, I will try to get the Dragonfly V1.2 and test again.

Is that public available now?

 

Larry

 

Hello Junker –

 

As you point out – the original (first production run) DragonFly did in fact clip when a 0db 24/96 signal was presented. Upon discovering this we took immediate steps to resolve the issue. It was only in the first production run (1.0) that the problem existed. the following production runs had a version 1.0c, which we used to identify amended DragonFly's (also measured by John Atkinson in Stereophile's review). Then in late November of last year we released the improved 1.2 version.

 

Regards,

 

Steve Silberman

AudioQuest

[email protected]

[MikeJazz]

Hi, Steve

 

Thanks for clear this out. Now we know why.

If possible, I will try to get the Dragonfly V1.2 and test again.

Is that public available now?

 

Larry

 

Larry, I know these competitive/comparative marketing is a very american way of doing thinks but I am getting the feeling that you are too much focused on that.

 

My suggestion:

1) send all equipment to a third-party to validate your conclusions, with a detailed test script (conditions of test)

2) only publish competitive results of your results is in line with the third party conclusions

 

This way you eliminate any potential criticism.

We, consumers, are well aware that a vendor never publishes unfavorable comparisons.

But at the end of the day, we listen to audio, not raw data from tests...

And I admit I understand your excitement - I am also excited...

 

Both Dragonfly (first version) and Meridian (first version) failed my real world test.

Let's hope Geek Out will not!

 

Thanks!

[mav52]

Larry, I know these competitive/comparative marketing is a very american way of doing thinks but I am getting the feeling that you are too much focused on that.

 

My suggestion:

1) send all equipment to a third-party to validate your conclusions, with a detailed test script (conditions of test)

2) only publish competitive results of your results is in line with the third party conclusions

 

This way you eliminate any potential criticism.

We, consumers, are well aware that a vendor never publishes unfavorable comparisons.

But at the end of the day, we listen to audio, not raw data from tests...

And I admit I understand your excitement - I am also excited...

 

Both Dragonfly (first version) and Meridian (first version) failed my real world test.

Let's hope Geek Out will not!

 

Thanks!

 

Dead on Mike. I view any vendors own review as "car" advertising with all the little checks in the boxes, colors etc.., nice to see, but a third party review just tells it like it is.

[magnum innominandum]

Dear Mr. Ho,

 

Few simple facts here...

 

Truth is the daughter of the time.

 

Yes please. Let's have some facts and truth.

 

Stereophile published a set of measurements of the Dragonfly here:

 

AudioQuest DragonFly USB D/A converter Measurements | Stereophile.com

 

It states clearly that the Dragonfly measures 0.041% THD & N at 0dBFS, if the volume control is set correctly to avoid to avoid clipping.

 

Your PR man published measurements here:

 

Geek Temple :: Topic: Geek Out vs. the Others, 3rd Installment (1/2)

 

It clearly states that the Dragonfly has > 5% THD+N.

 

The difference in measured results is over 100 times. One of the measurements is in error, severely so, this is not a small mistake by one party.

 

John Atkinson at Stereophile has measured hundreds if not thousands of DAC's over the decades. He is probably THE person in the world with the greatest body of practical experience in measuring audio gear in term of numbers of units measured.

 

If you claim that your measurements are correct, then JA's must be in error, or worse falsified. If you do not claim your measurements are correct, then why publish them, if anyone can illustrate them as grossly inaccurate by showing those of a third party with a decades long record of truthfulness in measuring equipment?

 

Your measurements incidentally also contradict (if not as grossly as with the dragongfly) the tests results of Stereophile for the Explorer:

 

Meridian Explorer USB D/A processor/headphone amplifier Measurements | Stereophile.com

 

The measurements you published also contradict the specifications shown on the iFi website:

 

iFi-audio Nano iDSD

 

There are no independent tests for the iDSD, so for now we do not know if their website is wrong, but it is very specific about the details and conditions, so it should be easy to replicate them.

 

Further in your measurements you claim that a digtal volume control is mandatory for USB Audio devices. This does not agree with my reading of the relevant USB standard documents:

 

USB.org - USB Device Class Specifications

 

"the Feature Unit optionally provides Audio Controls for the following features: Mute • Volume • Tone Control (Bass, Mid, Treble) • Graphic Equalizer • Automatic Gain Control • Delay • Bass Boost • Loudness • Input Gain • Input Gain Pad • Phase Inverter"

(from section 3.13.6 USB Audio Class 2 standard - USB Audio Class 1 standard contains similar wording)

 

Note the word OPTIONALLY?

 

Your product appears to measure very well, even if we compare to the Stereophile measurements of the Dragonfly and Explorer and the paper specifications of the iDSD. I do not think you need to use any questionable approaches in order to illustrate the superiority of your product, such as you use right now by posting highly questionable measurements and take cheap shots.

 

Why don't you just measure the Geek in the exact format, scaling and other details as the measurements Stereophile commonly publishes? Then your customers can be directed to Stereophile's extensive on-line archive where they could directly compare your product against those of the competition from very cheap all the way to extremely expensive items.

 

Or even better, send a unit to John Atkinson for review and let him do the measurements.

[The Computer Audiophile]

Or, send me a unit and I'll have it measured by a third party.

[Miska]

One friend from Google is sending me 3 more good DACs, including Herus, micro streamer and D3.

 

I did measurements on my Herus too, we can then compare...

 

Btw, I would always suggest to also make standard IMD measurement and spectrum analysis plot of at least 3.1 MHz band to detect leaky digital filters.

 

(I also did some measurements on hiFace DAC which uses same DAC chip as the Explorer)

 

 

P.S. Generating 32-bit J-test signal is easy, but you can of course as well use padded 16- or 24-bit ones.

[christopher3393]

Michael Lavorgna at Audiostream has posted on this. Some may find it helpful:

 

Are Measurements Subjective? | AudioStream

[junker]

AudioQuest DragonFly USB D/A converter Measurements | Stereophile.com

 

With the first sample of the DragonFly, a full-scale 24-bit signal actually clipped the bottom halves of the waveform with the computer's volume control set to its maximum, giving a THD+noise level of 3.8%. Backing off the control by one click (–0.17dB) reduced the THD to 2.14%, by a second click (–0.34dB) to 0.627%, and by a third click (–0.51dB) to 0.054%, below which the THD+N percentage plateaued. The second sample didn't clip with a 0dBFS signal at maximum volume, and the THD+N was 0.041% rather than 3.8%. According to Gordon Rankin, the volume control offers 64 steps of less than 1dB to –60dB and then mute (–100dB), but he used only 60 of those steps in the DragonFly, as the top four steps suffered from significant clipping into high impedances. "In retrospect," he wrote of the first sample, "I could have changed the maximum volume down a few more steps and then this would not have been an issue."

 

JA DID see similar test result - which I would consider this a 3rd party - until the volume control on the DragonFly was dialed back. Is this mentioned in the anywhere in their literature or instructions? To give a fairer comparison this should be done.

[Steve Silberman]

AudioQuest DragonFly USB D/A converter Measurements | Stereophile.com

 

With the first sample of the DragonFly, a full-scale 24-bit signal actually clipped the bottom halves of the waveform with the computer's volume control set to its maximum, giving a THD+noise level of 3.8%. Backing off the control by one click (–0.17dB) reduced the THD to 2.14%, by a second click (–0.34dB) to 0.627%, and by a third click (–0.51dB) to 0.054%, below which the THD+N percentage plateaued. The second sample didn't clip with a 0dBFS signal at maximum volume, and the THD+N was 0.041% rather than 3.8%. According to Gordon Rankin, the volume control offers 64 steps of less than 1dB to –60dB and then mute (–100dB), but he used only 60 of those steps in the DragonFly, as the top four steps suffered from significant clipping into high impedances. "In retrospect," he wrote of the first sample, "I could have changed the maximum volume down a few more steps and then this would not have been an issue."

 

It's clear to get better measurements from the DragonFly the volume control needs to be dialed back.

 

Hello Junker –

 

Only on the first production run (1.0) was dialing back the volume control by .5db required to eliminate the clipping; and this clipping was present only when a 0db 24/96 test signal was sent. 1.0c and 1.2 do not have this issue.

 

Steve Silberman

AudioQuest

[Selarom]

Or, send me a unit and I'll have it measured by a third party.

 

+1

 

Although I have no reason to believe there was some fudging of the measurements done by LH

[Larry Ho]

Michael Lavorgna at Audiostream has posted on this. Some may find it helpful:

 

Are Measurements Subjective? | AudioStream

 

I already list our testing settings there... Including 90K bandwidth, 300R load and none-weighted there... You could check it out.

 

Of course a lot of media people will get Geek Out and test it. Actually a lot of them already bought it!

 

I'm waiting for their test results show up. Time will tell.

 

Under different testing settings will have different test results. And all the DACs are tested in the same condition, same machine, same cable by the same people during a short period of time. So instead of picking one absolution value, using JA's measuring value compared with mine, the relative values is more important.

 

For example, JA got 3.8% THD from Dragonfly. I got 5%... I would say both are right, because measuring difference between 90K Hz and 20KHz, loading from 100K (200K) to 300Ohm... all possible. And when analog IC got clipping, one could be 4%, one could be more, since that is not a ideal working condition. Using that 3.8% vs 5% as a prove of I'm intentionally against our competitor is a very weak argument.

 

And one thing I'm quite amazing: We publish that testing result inside our Geek Force forum. And we just hope our Geek Out backers knows how good their perk will be. Why suddenly it looks like this becomes a media focus? We didn't quote it here or in Audiostream...(I know Audioquest is a bigger company. Especially when I wrote this message, its big read banner is hanging on top of this forum.)

 

Time will tell....

 

Larry

[valve5425]

 

And one thing I'm quite amazing: We publish that testing result inside our Geek Force forum. And we just hope our Geek Out backers knows how good their perk will be. Why suddenly it looks like this becomes a media focus? We didn't quote it here or in Audiostream...(I know Audioquest is a bigger company. Especially when I wrote this message, its big read banner is hanging on top of this forum.)

 

Time will tell....

 

Larry

 

Touché. Well put Larry.

[Audio_ELF]

And one thing I'm quite amazing: We publish that testing result inside our Geek Force forum. And we just hope our Geek Out backers knows how good their perk will be. Why suddenly it looks like this becomes a media focus? We didn't quote it here or in Audiostream...(I know Audioquest is a bigger company. Especially when I wrote this message, its big read banner is hanging on top of this forum.)

With respect Larry - you can publish measurements showing how good you feel the Geek Out is without publishing figures which disparage your competition.

 

As soon as you test competitors products you're inevitably going to open yourself to criticism of bias.

 

Eloise

[Paul R]

And one thing I'm quite amazing: We publish that testing result inside our Geek Force forum. And we just hope our Geek Out backers knows how good their perk will be. Why suddenly it looks like this becomes a media focus? We didn't quote it here or in Audiostream...(I know Audioquest is a bigger company. Especially when I wrote this message, its big read banner is hanging on top of this forum.)

 

Time will tell....

 

Larry

 

(grin) A lot of us never see the advertising anymore, and it actually took me a minute to realize what you were talking about. I suppose you could take a bit of that million+ and use it to advertise here a bit, if you wanted to even things out.

 

In the grand scope of things, AudioQuest is not that big a company, nothing at all like comparing LH to say, Apple.

 

-Paul

[magnum innominandum]

Dear Mr.Ho,

 

Time will tell...

 

So it will. I find it interesting that with the kilobytes of virtual ink you have so far spilled in all the various venues, you have failed to address directly any of the criticisms raised.

 

Measurements can differ, but a 100 fold difference in distortion measurements is not among the differences that can brushed off like that. If your bank overstated the debt on your mortgage 100 fold and then claimed you have to pay it like that as it is just another way of measuring your true debt, I suspect you would take exception. The difference is just too large to be brushed off with "different measurement settings".

 

Claiming that a given feature is demanded by USB audio class standards and disparaging competitors products for not implementing this feature, when the relevant standards (which are readily accessible in the public domain) in fact show this feature to be one of many OPTIONAL features that can be implemented would seem to go well past bending the truth.

 

As far as claiming these test results were "geek force only" and not meant to be made public, it pays to do a google search (for geek-out-vs-the-others-3rd-installment). It turns up links from:

 

http://www.facebook.com/geekbylhlabs/posts/10202418961769551 - the official Geek by LH Labs Facebook page

- seems a fanpage on Facebook

Light Harmonic GEEK - Page 28 -Headfi "semi-official" Geek Thread

Archimago's Musings: MEASUREMENTS: Audioengine D3 USB DAC / Headphone Amp. - Some blog on measurements

WiredState Audio Community • View topic - PureSound 2014 - What's New? - a popular forum

 

Next time if you don't want things to get out, don't post them on facebook and make sure they are behind a sign-in-wall. Anything else is public advertising. As your lawyers.

[wkhanna]

When ever I hear someone touting how 'their' measurements show their product, idea, research, or what-ever is better than someone else's...... I am often reminded of the conversation I had with my accountant.

 

Me; "What is the total sum of 1 +1?"

 

My accountant; "What do you want it to be?"

[Matias]

I think it is no big deal. One can see in the picture below that DragonFly clipping is at 0dBFS but drops back to normal in -10dBFS. LH also stated this in the text below the graphs... just like JA did in Stereophile.