Is Someone Playing Fast & Loose With Measurements?

[Skeptic]

Am I missing something here?

It looks like all that has happened is that someone was not following good practices when testing - whether intentionally or not.

 

 

OK, at 0dB the first revision of the DragonFly clips the signal. That should not be happening, but for whatever reason, it did.

Good test practices would suggest that you lower the input until the signal is no longer being clipped before taking measurements, and it should be noted that this is the case.

This is not cheating the test, as lowering the volume is likely to reduce your SNR numbers.

 

It's very possible that this is an honest mistake if proper test procedures were not followed.

I've seen this sort of thing happen quite a lot in the video world when people have $10,000+ invested in hardware to analyze the display but do not have proper training or experience to know the best practices to use when calibrating/measuring it, so the end results are not as good as they should be.

 

 

Of course if you want a competitor to look bad, you could compare the numbers from an 0dB signal on a Rev 1.0 DragonFly but that's hardly being fair, and it suggests to me that your product is not actually going to be any better (or possibly worse) when this clipping has been taken into account. Or at least it won't appear to be such a drastic improvement if it does still perform better.

 

Something else to consider is that while most DACs should not be clipping an 0dB test signal like that, few DAC designs have headroom to account for inter-sample clipping.

The only DAC design I know of which advertises this fact is Benchmark with their DAC2 series that have 3.5dB headroom in the digital domain, which should be enough to avoid inter-sample clipping with most lossless files.

 

JRiver measures inter-sample clipping (listed as Peak Level (R128)) in its audio analysis, and in my library there are currently only three lossless files which exceed that 3.5dB headroom.

However, about 40% of my library has inter-sample peaks above 0dB - so it could be argued that most DACs are going to be clipping the signal even with lossless files if you are leaving the volume at 0dB, and it will be even worse if you are using lossy files.

You have to be very careful with DSD playback as well - many devices or players will automatically add 6dB of gain, which can clip the signal if the track is not mastered correctly. You would be surprised at how many SACD discs/DSD tracks have less than 6dB of headroom.

 

 

So with just about any DAC, you should be avoiding playback at 0dB on the computer, regardless of whether or not they're clipping this test signal.

JRiver's Volume Leveling feature takes this inter-sample clipping into account, but I don't know of any other players which do. (and JRiver won't prevent inter-sample clipping without Volume Leveling or Peak Level Normalization enabled)

I would suggest reducing the volume by at least 4dB and probably 6dB to be safe, if you are using player which does not account for this.

 

Of course it would be better if the DragonFly did not require playback to be below 0dB, but it sounds like that has been addressed in newer revisions of the hardware.

 

None of this suggests that measurements are "subjective" but simply that care needs to be taken to follow good testing procedures.

[Miska]

The only DAC design I know of which advertises this fact is Benchmark with their DAC2 series that have 3.5dB headroom in the digital domain, which should be enough to avoid inter-sample clipping with most lossless files.

 

Top Wolfson DAC chips have option to leave 2 dB headroom to avoid inter-sample clipping.

 

And I recommend my users to use max -3 dB volume setting when upsampling to avoid limiter kicking in at inter-sample overs. But in any case this is made visible through indicator display.

 

JRiver measures inter-sample clipping (listed as Peak Level (R128)) in its audio analysis

 

Hmmh, I don't think EBU R128 has anything to do with inter-sample overs... Inter-sample clipping also depends on the oversampling filter and factor used, so it is not easily determined.

 

However, about 40% of my library has inter-sample peaks above 0dB - so it could be argued that most DACs are going to be clipping the signal even with lossless files if you are leaving the volume at 0dB

 

Yes, inter-sample overs are guaranteed whenever there is digital clipping, and that's these days happening on lot of content!

 

You have to be very careful with DSD playback as well - many devices or players will automatically add 6dB of gain, which can clip the signal if the track is not mastered correctly.

 

That is not problem when DAC is properly designed true DSD DAC. It is mostly problem only with PCM conversion and then there should be option to not apply 6 dB gain.

[Skeptic]

Top Wolfson DAC chips have option to leave 2 dB headroom to avoid inter-sample clipping.

Excellent - companies should make it clear whether they have this enabled or not. I'm not sure that 2dB is enough even for lossless files though, and it will definitely allow clipping with lossy files.

 

Hmmh, I don't think EBU R128 has anything to do with inter-sample overs... Inter-sample clipping also depends on the oversampling filter and factor used, so it is not easily determined.

It calculates the R128 "True Peak" value, which is upsampled (4x I think?) and then leaves an additional 1dB headroom to account for variance in the DAC's filters. (per spec)

 

That is not problem when DAC is properly designed true DSD DAC. It is mostly problem only with PCM conversion and then there should be option to not apply 6 dB gain.

Yes, it depends entirely on the DAC/player design. If you're bitstreaming DSD and it stays as native DSD in the DAC you should be fine.

It's something you need to look out for though, especially if you are converting to PCM or encoding PCM audio to DSD.

[magnum innominandum]

BTW some preliminary test results for the Geek Pulse were revealed in the indiegogo campaign today.

 

Thank you for pointing me to this. It made my day. It created hilarity here not known since that bit on TV where Rob Ford admitted he was on Crack!

 

In fact it will make my tomorrow.

 

Taking a 24 Bit SPDIF signal from the Geek Pulse and sending it to the AP Test set and posting the numbers. So VERY, VERY honest.

 

I am sure most people reading this on Indiegogo are falling over themselves and those who dropped their coin on this must surely be wetting themselves with pleasure - "I am going to get a DAC with -141dB THD + N and -185dB noisefloor".

 

Hey not even Weiss for almost 8 big ones can come close (Weiss DAC202 FireWire D/A converter Measurements | Stereophile.com) to the Geek Pulse and I only paid XXX Bucks. I am sure some will even print this out and slink off to the lavatory to jerk off over this.

 

How amusing.

 

Even more amusing is that that Fishy Guy should have got -144dB in the digital domain, so they are 3dB worse than very cheap PC sound cards or build in HD capable Audio Chips in any modern computer.

 

And someone at LH-Labs needs to take time to learn the difference between "Noisefloor" and "FFT Noisefloor", easy mistake to make for someone without formal training in electronics and thusly excused:

 

Noise floor - Wikipedia, the free encyclopedia

 

http://www.analog.com/static/imported-files/tutorials/MT-003.pdf

 

Why will it make my day tomorrow?

 

I will go to what the wife refers to as "that hole in the basement".

 

I will take my antique IBM Laptop (which normally plays music) and the ultra-cheap chinese musiland usb to spdif converter that feeds my Pass D1 DAC to there.

 

I will then use my second laptop and my 99 USD M-Audio USB Sound Card with SPDIF in and RMAA to measure the THD + N and SNR via SPDIF (I normally use it for my acoustic consultancy which pays the bills), which I know will actually be 146dB.

 

I will then post my results here to illustrate that sub 100 USD Sound Card and a Sub 100 USD SPDIF converter are better than that Geek Pulse AND the very expensive AP test-set LH Labs are using and I will use power rating and not the correct scaling, the way LH Labs does and claim that you only need 200 USD to be almost 10 times better than LH Labs and their AP...

 

I shall cackle my mad scientist cackle all the time, while listening to Heavy Metal Music VERY LOUD.

 

I know it will be totally pointless, but I will be having so much fun, who cares if a canadian beaver can pwn it?

 

Actually, if I really wanted to I'd just make it up using Photoshop, which would be just as good, but I believe in old-fashioned measurements done by hand and not faking it.

[germay0653]

Interesting read.

 

Internet troll personality study: Machiavellianism, narcissism, psychopathy, sadism.

[valve5425]

Interesting read.

 

Internet troll personality study: Machiavellianism, narcissism, psychopathy, sadism.

 

Many thanks for the link. Very enlightening!

[DM]

Thank you for pointing me to this. It made my day. It created hilarity here not known since that bit on TV where Rob Ford admitted he was on Crack!

 

Sorry, magnum innominandum, I don’t understand half of the technical stuff you were trying to say about their measurements…

 

 

Want to know what you're looking at right there? It's the final measurements of Geek Pulse's digital input circuit. What's so special about it? Well...

Looking at the THD+N, it's measuring better than -141 dB. This is a very geeky measurement that usually isn't published during the development of a product, so there's not much to compare it to. But it means that the digital input circuit adds 0.000008% distortion to the input signal. Not bad at all!

 

The second thing you can see in this pic is the FFT Spectrum analysis of the digital input circuit. It shows the noise floor at about -185 dB. The Audio Precision APx525's noise floor is -190 dB, so we're very close to the limits of what we can measure. We're feeling very good about this circuit!

 

http://www.indiegogo.com/projects/geek-pulse-a-digital-audio-awesomifier-for-your-desktop?c=activity

 

Are you saying they were measuring the digital to digital transmission lost inside the Geek Pulse?

 

If there is a case, it should have no error at all, so a 24bit signal should be -144dB.

 

This is just like measuring whether there is any error in the 10MB file that I have just copied to my USB thumb drive from my PC, and then say “Everyone look, the copying went perfect without error, not bad at all!"

 

What is the point of this completely meaningless test? A couple cheapest computer sound cards can achieve that …

 

Come on, LH Labs couldn’t be doing this, they must be measuring something else which I don’t quite understanding yet, they are surely more professional than this.

[junker]

Love it. Great link!

 

Interesting read.

 

Internet troll personality study: Machiavellianism, narcissism, psychopathy, sadism.

[magnum innominandum]

DM,

 

Are you saying they were measuring the digital to digital transmission lost inside the Geek Pulse?

 

That is what THEY are saying, but you must read very, very carefully. Please look again.

 

If there is a case, it should have no error at all, so a 24bit signal should be -144dB.

 

Precisely.

 

Come on, LH Labs couldn’t be doing this

 

Over at Audiostream Mr. Fish gave his side of the story, I think this says it all:

 

Not a great PR move by LH?

Submitted by gavn8r on March 27, 2014 - 10:17pm

We do have a lot of forums and publications on fire right now, all debating whether we're geniuses or loons. Either way, we win.

[Larry Ho]

Thanks for your question.

 

We already published the digital to analog performance of Geek Pulse long time ago, that is USB input vs balanced analog output.

 

The new test there is related to its digital input interface (SPDIF and AES), because this is another important part of the system so we hope our backers know the current progress.

 

Audio digital interface is 'not' like your hard drive transmission, because that is REAL time and NO re-try... If you said this measurement is useless, then basically that means every SPDIF or AES interface performance is the same... which I think a lot of people not only me won't agree.

 

SPDIF and AES signal is based on NRZ protocol and they embedded the clock signal within the audio data. On receiver side, it needs to make sure the input signal as jitter-free as possible, so when we use PLL to recover the master clock from the data, it will have the best performance for the next stage, which is digital-to-analog stage.

 

I have been asked many times for the similar question in USB interface too. People ask since they use USB Hard drive or USB printer that never got a bit of error. Why bother with Asynchronous USB interface or buffer... this kind of stuff. The answer is the same: the real time transmission nature of digital music, and the SPDIF or USB Audio protocol design itself could not guarantee the bit perfect, 100% error free transmission. That is why people are investing their time and money here. :-)

 

 

 

 

Sorry, magnum innominandum, I don’t understand half of the technical stuff you were trying to say about their measurements…

 

[ATTACH=CONFIG]11581[/ATTACH]

 

 

 

Geek Pulse: A Digital Audio Awesomifier for Your Desktop | Indiegogo

 

Are you saying they were measuring the digital to digital transmission lost inside the Geek Pulse?

 

If there is a case, it should have no error at all, so a 24bit signal should be -144dB.

 

This is just like measuring whether there is any error in the 10MB file that I have just copied to my USB thumb drive from my PC, and then say “Everyone look, the copying went perfect without error, not bad at all!"

 

What is the point of this completely meaningless test? A couple cheapest computer sound cards can achieve that …

 

Come on, LH Labs couldn’t be doing this, they must be measuring something else which I don’t quite understanding yet, they are surely more professional than this.

[Miska]

SPDIF and AES signal is based on NRZ protocol and they embedded the clock signal within the audio data. On receiver side, it needs to make sure the input signal as jitter-free as possible, so when we use PLL to recover the master clock from the data, it will have the best performance for the next stage, which is digital-to-analog stage.

 

But if you want to make any jitter measurements on it, you cannot measure it based on the recovered clock, because the result would always look perfect. Instead you need to measure it against external unrelated reference clock. So practically you would run spectrum analysis against frequency difference of the two clocks. Recovered master clock and reference - in optimal case reference being the original transmitter master clock.

 

Making analysis on digital output of S/PDIF receiver as-is, is completely useless. Unless something is really badly wrong, it always looks perfect.

 

I have been asked many times for the similar question in USB interface too. People ask since they use USB Hard drive or USB printer that never got a bit of error. Why bother with Asynchronous USB interface or buffer... this kind of stuff. The answer is the same: the real time transmission nature of digital music, and the SPDIF or USB Audio protocol design itself could not guarantee the bit perfect, 100% error free transmission. That is why people are investing their time and money here. :-)

 

I have moved over to use ethernet/WLAN as transfer media, guaranteed to be 100% error free.

 

Error rate is not really an issue, I don't remember having transfer errors even with S/PDIF... It is very likely that any transmission error would result in clearly audible error.

[andyv]

Audiostream have published some responses and measurements and comments from Audioquest and iFi which seem to be an objective rebuttal of LH Labs claims - see here.

[Miska]

Local hifi magazine "Hifimaailma" had group test of USB DACs with headphone amp and Dragonfly v1.2 was included.

 

Some of the Dragonfly v1.2 measurement results:

SNR: 95 dB

Noise: -120 dBV

THD 0 dBFS, 500 mV, 1 kHz 33/330 ohm: 0.18/0.17 %

Linearity: -92 dBFS

IMD: 19/20 kHz, 2nd, 33/330 ohm: 0.017/0.016%

 

Headphone output of MacBook Pro was included for comparison:

SNR: 97 dB

Noise: -120 dBV

THD: 0.0043/0.0013 %

Linearity: -93 dBFS

IMD: 0.0009/0.0008%

[magnum innominandum]

Hi,

 

Local hifi magazine "Hifimaailma" had group test of USB DACs with headphone amp and Dragonfly v1.2 was included.

 

Some of the Dragonfly v1.2 measurement results:

SNR: 95 dB

Noise: -120 dBV

THD 0 dBFS, 500 mV, 1 kHz 33/330 ohm: 0.18/0.17 %

Linearity: -92 dBFS

IMD: 19/20 kHz, 2nd, 33/330 ohm: 0.017/0.016%

 

Headphone output of MacBook Pro was included for comparison:

SNR: 97 dB

Noise: -120 dBV

THD: 0.0043/0.0013 %

Linearity: -93 dBFS

IMD: 0.0009/0.0008%

 

These measurements look very "interesting".

 

First time I see lower IMD than THD unless the whole thing was oscillating.

 

Also, I cannot square a noise of -120dBV (which seems excellent, that would be 126dB below 2V - something few if any DAC's manage) with an SNR of 95/97dB. Do you have more details how the measurements where conducted?

[magnum innominandum]

Dear Mr. Ho,

 

The new test there is related to its digital input interface (SPDIF and AES), because this is another important part of the system so we hope our backers know the current progress.

 

Audio digital interface is 'not' like your hard drive transmission, because that is REAL time and NO re-try... If you said this measurement is useless, then basically that means every SPDIF or AES interface performance is the same... which I think a lot of people not only me won't agree.

 

I am impressed AP now offer a I2S input option. This is nice.

 

Now given your were sending 24 Bit data I would not at all impressed if I were (say) seeing -144dB so I wonder what is wrong with this measurement.

 

Plus, what did you expect? Anyone can take any industry standard SPDIF receiver, apply it according to the datasheet/reference design and get better results than you got (e.g. the -144dB expected for 24 Bit PCM).

 

Transferring Digital Data via AES-EBU over several 100m (or even down phone lines) with no errors was a problem solved adequately in the 1980's. You would have to look at Jitter test signal (J-Test - cough, cough) to see if your device actually allowed low jitter.

[Miska]

First time I see lower IMD than THD unless the whole thing was oscillating.

 

I've seen it quite a lot. For example my Resonessence Labs HERUS when playing DSD128 has 2nd order IMD at around -100 dB (3rd @-87.5 dB) while 3rd THD component (highest) is at -85 dB. With PCM 2nd IMD is around -90 dB (3rd -85 dB) while 3rd harmonic is at -82 dB. Curious thing is also that with PCM THD spectrum is dominated by odd harmonics while with DSD it becomes more even harmonics.

 

2nd order IMD was lower than THD also in number of other devices. They listed 2nd and 3rd order IMD separately, so for total IMD you need to sum the two. For Dragonfly, 3rd order IMD was 0.017/0.014%.

 

Some devices had significantly higher 3rd order IMD than 2nd order: ADL X1, CA DacMagic XS, Meridian Explorer (0.30/0.23%!), Meridian Prime and Pro-Ject Headbox DS. MacBook Pro has challenge to lower 33 ohm impedance since 3rd IMD increases to 0.01% compared to 0.0008% to 330 ohm.

 

Do you have more details how the measurements where conducted?

 

Apart from what I already quoted, measurement is done using Miller Audio Research analyzer (IOW, same as used by HiFi-News magazine).

 

SNR is A-weighted while the background noise dBV figure must be unweighted.

 

Background noise is probably measured while playing silence. Hopefully dithered one, because otherwise most DACs would just mute output. (from past experience I know MacBook Pro mutes headphone output one second after playback has stopped)

[Miska]

You would have to look at Jitter test signal (J-Test - cough, cough) to see if your device actually allowed low jitter.

 

That is only useful for digital input if you discard the recovered clock and use some other reference clock. It would of course still look perfect if you check it against the recovered clock.

[magnum innominandum]

Miska,

 

I've seen it quite a lot. For example my Resonessence Labs HERUS when playing DSD128 has 2nd order IMD at around -100 dB (3rd @-87.5 dB) while 3rd THD component (highest) is at -85 dB. With PCM 2nd IMD is around -90 dB (3rd -85 dB) while 3rd harmonic is at -82 dB. Curious thing is also that with PCM THD spectrum is dominated by odd harmonics while with DSD it becomes more even harmonics.

 

Oranges and Lemons.

 

This is CRUCIAL.

 

THD or THD+N?

 

The first, IMD can never be greater than THD (mathematical rules), THD & N implies not just distortion but any noise, so THD & N can be greater than any IMD intercept, but only in the presence of noise greater than the actual distortion.

 

> SNR is A-weighted while the background noise dBV figure must be unweighed.

 

How can unweighed noise be lower than the A-weighted SNR? This truly seems an enigma to rival the Sphinx.

 

> Background noise is probably measured while playing silence. Hopefully dithered one,

 

You mean you are not sure how these results where arrived at, yet you still choose to share them "because they use Miller Beer Analyser"?

 

You cannot be cerious.

[magnum innominandum]

Miska,

 

That is only useful for digital input if you discard the recovered clock and use some other reference clock. It would of course still look perfect if you check it against the recovered clock.

 

Who knows, sounds like what LH Lab published, FWIW.

[Miska]

THD or THD+N?

 

The first, IMD can never be greater than THD (mathematical rules), THD & N implies not just distortion but any noise, so THD & N can be greater than any IMD intercept, but only in the presence of noise greater than the actual distortion.

 

Story doesn't tell...

 

Jumping to some measurements I did...

Here's how THD looks like on Resonessence Labs HERUS @DSD128, 0 dBFS:

 

Dropping volume to -10 dBFS lowers the distortion quite drastically, as is case for many DACs:

 

And here's how IMD looks like @DSD128, 0 dBFS:

 

 

How can unweighed noise be lower than the A-weighted SNR? This truly seems an enigma to rival the Sphinx.

 

At least SNR contains all the distortion components while my guess is that background noise is the idle noise level.

 

You mean you are not sure how these results where arrived at, yet you still choose to share them "because they use Miller Beer Analyser"?

 

You cannot be cerious.

 

Damn, I am trying to answer your questions, I didn't make the measurements, I don't have any affiliation to the magazine in question. I'm just a subscriber and quoted figures from the table of results and referenced the source.

 

I don't take sides on validity of measurements of anybody else's except my own. Just one more set of figures to already existing bunch.

 

I don't have Dragonfly, so I cannot measure it myself and I'm not interested to buy it because it doesn't fulfill my feature requirements (DXD & DSD128).

[Andrew Allen Ballew]

not that my layman opinion means anything. But, it does seem to me some fudging of the numbers is indeed happening. Negative marketing.

 

For instance, linked is a third party measurement of the iFi Nano. It confirms the specs as stated by the manufacturer, not as measured by LH.

 

I would also add that while I am sure LH makes an outstanding product, so do its competitors. I assure you audio designers like Thorsten at AMR and Gordon of Wavelength (who designed the Dragonfly for AQ) know what they are doing. And they are VERY invested in the quality of their designs. And proof of that is in the listening. The Dragonfly is an amazing sounding little device. The ifi iDSD is also an amazing sounding little device.

 

Google Oversæt

[kravi4ka]

It will be very interesting to have a device delivered by LH and have it measured. They have very interesting ideas about engineering, had some papers about LSB correction and stuff that later disappeared... Now interesting ideas about measuring...

 

For the sake of all who believed them and gave them their money I hope it will sound good, no matter how it measures... But Gavin Fish has really secured his place in my heart, that guy really made the whole thing disgusting.

[DM]

Looking at the 3rd party(AudioC) iDSD measurements and comparing to LH Lab's (Geek) own measurements:

 

Google Oversæt

 

iFi nano iDSD

 

Output voltage

Geek Measurement: 1.48V

AudioC Measurement: 1.65V

iFi Specification: 1.65V

 

THD + N @ 0dBFS

Geek Measurement: 0.15%

AudioC Measurement: 0.024%

iFi Specification: 0.02%

 

the difference in distortion is nearly 10 times! i.e. LH Labs measurements showed a 10 fold increase in distortion.

 

Also with the FFT's, the AudioC measurements show non of the problems remarked by LH Labs.

 

Hence I think the Chinese chief designer and president Larry Ho from LH Labs (Larry, did you name your company after your initials?) was very unlucky. He first got a bad (or a really really old version of） Dragonfly, then he also got a bad iDSD too.

 

So 2 out of 3 of the DACs (Dragonfly and iDSD) that LH Labs(Geek) had, were all bad units. Audioquest and iFi should really look into their QC control and make sure Larry doesn't get bad samples in the future.

[DM]

I am finalising the review on the iDSD and Sony PHA-2, the Japanese Sony is a better looking unit, the iDSD is like a tank built by the German (the designer Thorsten is German), but this tank is not the beautiful Tiger tank, but a couple version earlier ... i.e. not as nice looking ...

[Hiro]

the difference in distortion is nearly 10 times! i.e. LH Labs measurements showed a 10 fold increase in distortion.

 

Comparing measurements made by different people (and likely in different test conditions) strikes me as rather useless.