Popular Post pkane2001 Posted January 10, 2020 Popular Post Share Posted January 10, 2020 2 hours ago, manisandher said: So... If there were to be a retest, what would need to be done differently for those of you who remained sceptical to have more confidence in the results? Mani. Hi Mani, Here are some thoughts: 1. Test something that's more of a common concern. A setting in Phasure XXX software is too obscure and not of much interest to anyone who doesn't use this software (I'd say the majority here). 2. Run more tests. You had a few low-scoring attempts before getting 9/10 (I'm aware of the reasons you gave, but an explanation of why a test failed is usually not acceptable in lieu of a positive result). Two consecutive 9/10 tests or 8/10 would be much more convincing. If you do run multiple tests, the results of all of those tests should be included in the calculation, good or bad. 3. Use better-vetted equipment to record digital and analog feeds -- I was confused by the provided files when trying to analyze them, there were some issues with one ADC and different issues with another, etc. It wasn't clear to me which recordings were accurate representation of what was tested (if any). 4. Document and review the test procedure with us before the test, not after. 5. Some photos/video recording of the session might be helpful in reviewing. The blinding procedure should be carefully considered and any possible tells eliminated (not accusing you of cheating, but there are often tells that have nothing to do with the actual audio, like a small click, or a slightly longer delay before playing, etc.) I'll probably think of something else later manisandher, mansr and semente 2 1 -Paul DeltaWave, DISTORT, Earful, PKHarmonic, new: Multitone Analyzer Link to comment
pkane2001 Posted January 11, 2020 Share Posted January 11, 2020 3 hours ago, manisandher said: I reckon a 10-run ABX would take around 10-12 minutes. I'd go for 3 of these in total, with a small break between each. This would give a total sample size of 30. Would you agree that a score of >25/30 could be taken as a positive? Yes, 25/30 is an excellent result. 3 hours ago, manisandher said: There was no issue with the digital feed - it proved that the DAC received bit-identical signals during the ABX, where I scored 9/10. I used a Tascam DA-3000 ADC for the initial analysis of the analogue output of the DAC. Here are its specs: As I recall, the digital feed recordings had issues with not synchronizing right away. Some number of samples were different at the beginning. Since these were the main record of what had actually transpired, I would have to assume that the two captures were not bit perfect, at least at the start of each track. Again, an explanation of what went wrong isn't really a substitute for the real capture being bit-perfect Tascam THD+N of 0.003% can be a lot better. I have a simple pro Apogee interface (paid about $350 for it used) that does about 10x better (0.0003%). I think Benchmark or RME equipment was mentioned before in this thread, and these do 8-10dB better than my little Apogee. The point is to have as little distortion added to the recording as possible by the test equipment. Things like jitter and phase distortions need to be looked at, as well, since these can create audible differences when large enough. A few simple loop-back recordings of DAC/ADC playing music for 60 seconds would give us enough to analyze to see how the equipment behaves before the test. With DeltaWave software, I've compared now a large number of DAC/ADC loop-back recordings from various sources. The better equipment produces a null of -90dB or better. The recordings your sessions produced generated, at best, a -50dB null. That's a really poor conversion quality compared to some of the better equipment out there. esldude 1 -Paul DeltaWave, DISTORT, Earful, PKHarmonic, new: Multitone Analyzer Link to comment
pkane2001 Posted January 11, 2020 Share Posted January 11, 2020 3 hours ago, manisandher said: I find it useful to consider this in two parts: 1. the ABX listening test itself The specific DAC, amps and speakers used for the ABX, and where they're situated, etc. are all totally irrelevant. It's a red herring on Mans's part. The only thing that matters is showing that the DAC received bit-identical signals when replaying A and B. The digital feed to the DAC was captured throughout the ABX, and it was proven that the DAC had indeed received bit-identical signals throughout. And yet I heard consistent differences between A and B, as shown by my 9/10 score. 2. the analysis of the analogue output of the DAC This proved difficult. I've posted the specs of the ADC used in the original analysis, which show nothing untoward. But on analysing the analogue captures, the ADC proved unsatisfactory. Mani. Mani, there are some things that need to be considered in (1) as well. Bit-perfect transmission is not sufficient if the interface (SPDIF) also carries the clock signal used to drive the DAC output. Large amount of jitter on the interface can cause timing errors. For this reason, using a USB or another asynchronous connection to the DAC would be a much better choice, IMO. -Paul DeltaWave, DISTORT, Earful, PKHarmonic, new: Multitone Analyzer Link to comment
pkane2001 Posted January 11, 2020 Share Posted January 11, 2020 2 hours ago, manisandher said: So, use a USB DAC with an spdif output, and capture spdif output in real-time to test that A and B are bit-identical? Any way of checking A and B are bit-identical using a USB DAC without an spdif output? Mani. There are very inexpensive USB to SPDIF converters out there that could be used, but that wouldn't work simultaneously with the DAC unless there's some PC sound driver software used that can feed the same input to multiple outputs (easy to do on the Mac, not sure about Windows). -Paul DeltaWave, DISTORT, Earful, PKHarmonic, new: Multitone Analyzer Link to comment
pkane2001 Posted January 11, 2020 Share Posted January 11, 2020 2 hours ago, manisandher said: I've got the following ADCs to hand: - RME Fireface 800 (need to repair the SMPS first) - MOTU 896HD (mkI) - Prism AD124 (44.1/48 only) - Tascam DA-3000 (Shame I don't still have my Pacific Microsonics Model Two .) The Tascam should be fine for the digital captures, as there's no actual conversion going on. I'll do some DA/AD loops with my USB DAC and the other converters to figure out how they fare. Would a simple sine wave suffice, or would real music content better? Mani. For loop back, both, a simple 1kHz sine wave and a music files would both be useful. Sine wave is easier to interpret (get actual THD+N, etc.) but a music file will exercise a lot more of the converter chain. The Gearslutz interface loopback thread uses this WAV file (24/44.1kHz). -Paul DeltaWave, DISTORT, Earful, PKHarmonic, new: Multitone Analyzer Link to comment
Popular Post pkane2001 Posted January 11, 2020 Popular Post Share Posted January 11, 2020 10 minutes ago, fas42 said: The huge failure of the "scientific method" ... ends up with results which are largely 'tainted', or are shown to have poor relevance in the long run. Personally, I rely on ... what my immediate, 'gut' reaction is telling me - this has always worked out, in that it allows me to make decisions which I am satisfied with, long term. Ah, yes! Scientific studies obviously have completely failed us, which is why all of science is now done by gut reaction 🤣😂 Ralf11, Archimago, lucretius and 1 other 1 3 -Paul DeltaWave, DISTORT, Earful, PKHarmonic, new: Multitone Analyzer Link to comment
Popular Post pkane2001 Posted January 13, 2020 Popular Post Share Posted January 13, 2020 57 minutes ago, PeterSt said: Mani, we must assume that what we can incur for minute differences (and let's say that they emerge by accident, hence out of our control really), which is ever so small but audible, and that because we could not control it explicitly, we can not measure it. It is almost a physics law; Why wouldn't we be able to perceive noise-like modulations, especially if they oscillate (re-occur). And mind you, we already know that they oscillate because that is inherent to the processes (at least the ones I apply in XXHighEnd - all of them). Maybe take the analogy of a piano out of tune (or any string instrument for that matter). Would you be able to hear the two strings out of tune (but the ever so slightest !) when individually played ? maybe a very best musician can. But I wonder. Suppose this level of distortion is sow low (the deviation to the absolute A key etc.) that no analyser would be able to measure it. This could be so already because of present system noise. But for sure you'd hear the out of tune when two strings are played (one so slightly out of tune) ! Maybe the audible oscillation would be once per minute only, but you'd hear it still. I think this compares to IMD ... We talked about it before : with a sufficient number of test tones we should be able to dig out some general distortion figure. I think Paul was on the same track (well over a year ago, I think). Mind you, in the end (sufficient number of test tones in parallel), it will be equal to music - with the difference we can measure it. Peter, Our ears are not any more sensitive than precision electronic equipment. We may not be measuring the right things or using the right tools to measure in some cases, but there is absolutely no evidence that the ear can hear something that the measurement instruments cannot detect. But, I think that's the whole goal of Mani's testing with his Red/Blue pill proposal. And should his tests show that there is something he can hear that doesn't come through in the measurements, that would be a very exciting test result for me, personally. Finding the right things to compare between the original recording and the output of a test system is what DeltaWave was designed to do, and why I created it in the first place. I've since created another tool, called DISTORT, that allows any number or combination of distortions to be applied to an existing digital recording. This includes things like noise, jitter, or even 'oscillations' with a 1 minute period, IMD of any type, etc. Hopefully this can help answer the question as to what's audible to you, with your music, on your system. You can test for what matters and what doesn't and not accept someone else's study results. semente, Archimago, Arpiben and 3 others 5 1 -Paul DeltaWave, DISTORT, Earful, PKHarmonic, new: Multitone Analyzer Link to comment
pkane2001 Posted January 13, 2020 Share Posted January 13, 2020 51 minutes ago, PeterSt said: All right Paul, fair enough. Then : Could you please take distance from your own remark that some random ADC requires its time to settle and (level) trigger the recording, so that it is perfectly allowed to chop off the irregular beginnings in order to next have comparable files. I think you can. DeltaWave can chop off any mismatches at the start and even the end of the track, so no worries. It can do this automatically, or manually. The problem with mismatched data was not on ADC recordings in the first test, it was with the digital recorder. That leaves the question whether the issue was with the recorder, or the source. In the absence of additional testing, I don't know the answer. And if the error was with the source, then that could easily cause some audible differences at the start of the track and so needs to be eliminated as a possibility. 51 minutes ago, PeterSt said: Next up are the recordings already present. So what can your software do with them ? The whole point is: if that brings nothing, it most certainly does not testify of Mani cheating or anything of the sort. It would tell, however, that the test means are still too weak. The problem is that my software shows very large differences between the ADC-recorded track and the original digital one. Larger than any good DAC/ADC loop would produce. I would be very surprised if this was caused by this extremely low-level noise that you are talking about: -50dB or so difference is certainly in the audible range. Which is why I have to question the quality of the equipment and how it was set up. The other issue I find often in trying to take ADC captures is ground loops and other electrical interactions between components that are not normally part of the playback system. The result can easily be a very distorted ADC capture, while the DAC is producing a reasonably clean and accurate output. That's why I suggested a preliminary end-to-end/loop-back test using simple sine-wave signal as that will help to reveal any such issues, as well as a baseline recording using a standard music track. 51 minutes ago, PeterSt said: FYI, and just in case you didn't know it yet: I can show you and everyone the most easily discernible differences between whatever dozens of means, all leading to bit identical looped back data, those means including USB cables. It goes from, way too wide sound stages, to way to flat ones, compared to narrow and deep. Don't ask me why you and so many others have difficulties with getting that. All I know myself by guarantee, is that somehow we all (!) are not able to find test means showing it. So ... let's have it puuulease ? All not meant as teasing or anything, but I feel we are in a deadly loop. The reason this is questioned is that unless you do the test under properly controlled blind conditions, you are using your brain's interpretation of what you see in combination with what you hear. I think you'll agree that your ability to see is not what we want to test when doing proper listening tests If you can demonstrate (just like Mani is trying to do) that you can distinguish your USB cable under double-blind conditions then that will break the "deadly loop"! Even more so, if the measurements/ADC captures still don't reveal any differences. -Paul DeltaWave, DISTORT, Earful, PKHarmonic, new: Multitone Analyzer Link to comment
pkane2001 Posted January 17, 2020 Share Posted January 17, 2020 14 minutes ago, manisandher said: Paul, as you know, I've been exploring getting hold of an ADC that would be up to the job of measuring the bit-identical differences I'm hearing. Here are the loopback measurements of the ADCs I currently have to hand: (* Potential future purchase.) I suspect that in the majority of cases it's the ADC that's the limiting factor in these measurements, and not the DAC. So, it seems to me that none of these are really be up to the job at hand... and the RME is considered one of the best currently available! So where to now? Mani. The Gearslutz measurements are a little out of date DeltaWave corrects for the non-linear phase effects (which is usually caused by the filters in the DAC and the ADC). Since these are usually not audible differences, the Gearslutz results are way too low. I've had a conversation with Didier who is running the list about updating the list, at least to add the new, corrected measurements. But he's reluctant to do so, claiming that the results are already too confusing and complex and he doesn't want to add to the confusion Maybe at some point I'll just set up DeltaWave to run through all the uploaded files on that thread and produce and publish my own list of results... Let me run some of these through DW (assuming the loopback files are available) to get proper results. -Paul DeltaWave, DISTORT, Earful, PKHarmonic, new: Multitone Analyzer Link to comment
pkane2001 Posted January 17, 2020 Share Posted January 17, 2020 1 hour ago, manisandher said: Paul, as you know, I've been exploring getting hold of an ADC that would be up to the job of measuring the bit-identical differences I'm hearing. Here are the loopback measurements (most taken from the Gearslutz DA_AD thread) of the ADCs I currently have to hand: (* Potential future purchase.) I suspect that in the majority of cases it's the ADC that's the limiting factor in these measurements, and not the DAC. So, it seems to me that none of these are really be up to the job at hand... and the RME is considered one of the best currently available! So where to now? Mani. Here are the DeltaWave results for the few loop back files I could find. The others were no longer available, so I couldn't test them. Unfortunately, can't confirm that, for example, RME FF800 was not recorded through a digital loopback, similar to your first attempt with M2. In any case, Motu M2 doesn't look that bad at all. Remember that this is the combination of the DAC and ADC, so individually, ADC and/or DAC may perform even better. manisandher 1 -Paul DeltaWave, DISTORT, Earful, PKHarmonic, new: Multitone Analyzer Link to comment
pkane2001 Posted January 17, 2020 Share Posted January 17, 2020 4 minutes ago, manisandher said: Thanks Paul, that's really, really helpful. I might hang on to my new M2 after all . I've got a Fireface 800, so will do some loopbacks (once I've fixed its SMPS - second one to fail on me on an FF800!). The ADI-2 Pro FS is supposed to be the king of modern 'affordable pro' ADCs. Any idea why it performs so badly in the loopback? Mani. I'll try to dig into it, didn't have the time to look in detail. Unfortunately, it's not clear under what conditions the loopback recording was made, with what settings, etc. Let's see if DW can help to find what was causing the large error. Arpiben 1 -Paul DeltaWave, DISTORT, Earful, PKHarmonic, new: Multitone Analyzer Link to comment
pkane2001 Posted January 17, 2020 Share Posted January 17, 2020 44 minutes ago, manisandher said: Thanks Paul, that's really, really helpful. I might hang on to my new M2 after all . I've got a Fireface 800, so will do some loopbacks (once I've fixed its SMPS - second one to fail on me on an FF800!). The ADI-2 Pro FS is supposed to be the king of modern 'affordable pro' ADCs. Any idea why it performs so badly in the loopback? Mani. Hold the presses! Found another loopback recording. This one is described as using ADI-2 Pro FS with sharp filter setting for both, DAC and ADC. A much better result (see last item): -Paul DeltaWave, DISTORT, Earful, PKHarmonic, new: Multitone Analyzer Link to comment
pkane2001 Posted January 17, 2020 Share Posted January 17, 2020 58 minutes ago, manisandher said: Great! I'll go ahead and order an ADI-2 Pro FS. (The M2 seems pretty good for the price though.) Mani. To close the loop. Looking into the other ADI-2 Pro FS capture with lower scores, it appears to be using a different filter, and has an excessive amount of jitter. The first capture shows around 13μs RMS jitter over the entire recording, while the second one shows 186ns. The shape of the phase difference plot is also obviously different between the two. The first one is shaped as an inverted letter U, the second one is nearly a straight line. -Paul DeltaWave, DISTORT, Earful, PKHarmonic, new: Multitone Analyzer Link to comment
pkane2001 Posted January 17, 2020 Share Posted January 17, 2020 2 minutes ago, fas42 said: Gee whiz ... the technical, measured accuracy of a piece of hardware dramatically changes, just by altering a setting or two of how it works ... guess that proves that all one needs is dig out of a set of numbers from somewhere, to be fully assured "how good something is" ... 🙃. Good try, but in this case, there's no indication as to how the 'bad' capture was produced. There's so much jitter in it that I'd be surprised it had anything to do with settings. At least I can't imagine what setting would cause this. -Paul DeltaWave, DISTORT, Earful, PKHarmonic, new: Multitone Analyzer Link to comment
pkane2001 Posted January 18, 2020 Share Posted January 18, 2020 2 hours ago, fas42 said: Which underlies the problem with taking measurments in audio, in general - unless one has absolutely full understanding of every detail that may be pertinent to the numbers derived, then it is Russian roulette, often times, whether they have any bearing on "what it sounds like" ... Yes, if you don’t know what you’re doing, you’re likely to mess up. Surprising, right? -Paul DeltaWave, DISTORT, Earful, PKHarmonic, new: Multitone Analyzer Link to comment
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now