16/44 vs 24/192 Experiment
Entry posted by mitchco
Updated with more info on Audio DiffMaker, plus ABX listening tests.
Lots of discussion around this article: 24/192 Music Downloads...and why they make no sense http://people.xiph.org/~xiphmont/demo/neil-young.html
I decided to run a science experiment using Audio DiffMaker to compare 16/44 to 24/192 format of the same master from Soundkeeper Recordings: http://soundkeeperrecordings.com/format.htm
I have used Audio DiffMaker before to compare FLAC vs WAV and comparing two bit-perfect music players on my computer audio playback system.
Here is the result of my 16/44 vs 24/192 experiment.
First a refresher on how Audio DiffMaker works:
There are also a two papers, http://www.libinst.com/AES%20Audio%20Differencing%20Paper.pdf and http://www.libinst.com/Detecting%20Differences%20(slides).pdf The help file that comes with the program is very well documented and goes into much more detail.
Updated - I wanted to provide more with respect to how Audio DiffMaker works and why it is an important state of the art measurement tool in any Audiophiles arsenal.
Audio DiffMaker’s Differencing Process
Excerpt from the DiffMaker Help file on how the differencing process works:
While it may not be possible to show whether alteration is having effects directly on the listener, it is possible to determine whether an audio signal has been changed.
The existence of any changes to a digital recording of an audio signal can be detected by the simple process of subtraction, performed on a sample-by-sample basis. If each audio sample is the same, then subtracting one from the other leaves nothing (zero signal).
A recorded copy of the original signal (called the "Reference") can be mathematically subtracted from a recorded copy of the possibly changed signal (called the "Compared" signal). This results in a "Difference" signal recording that can be evaluated by ear or other analysis.
If the resulting Difference signal, when played as audio, is effectively silence or at least is not perceivable to a listener when played at levels in which it would occur when it was part of the "Compared" signal, then the investigator can with good confidence conclude that the change has made no audible difference.
The problems and operational, perceptual, or psychological complications about listening for whether sound is being changed are greatly reduced by transforming the task into the much simpler issue of listening for anything significant at all. The evaluation of the result is done by ear, and the user doesn't need to question hearing ability to use the tool. Audio DiffMaker test, encourages you to still "trust your ears".
Audio DiffMaker is a state of the art differencing tool that automates this workflow from 5 years ago: http://forum.audacityteam.org/viewtopic.php?f=28&t=3873#p15071 One of the reasons it is state of the art is because the software can differentiate time differences in decimal places in the parts per million (ppm): “The sample rates or speeds of player decks and soundcards are constantly drifting, if only by very small amounts. But even as little a change in sample rate as 0.01ppm (one hundredth of a part per million) can cause two otherwise identical files to leave difference sound after subtracting.”
In order to compare the two formats, I had to up sample the 16/44 to 24/192. I used http://www.voxengo.com/product/r8brainpro/ to perform the sample rate conversion:
I used the default settings. Then I used Audacity to edit the waveforms so I am just looking at the first 40 seconds of each waveform.
Then it is a matter of loading the two waveforms into Audio DiffMaker and extracting the difference.
According to DiffMaker, the difference file is -94 dB. I opened up the difference file in Audacity and here is what is left over:
Something definitely there. Here is the frequency analysis:
I have also included the difference file as an attachment to this post. Given that the majority of content is 20KHz and above, I can’t hear anything on the difference file.
Note that this is one data point. I have used Audio DiffMaker for a while now and here is one tip that will help you get consistent results if you decide to try it out.
This is the output status window from the DiffMaker progam as it is running. Note the arrow. It says that the sample rate error is low enough not to require adjustment. If the sample rate error is too high, there will be a notification as such on this line, then the program tries to automatically align the tracks. However, there seems to be a bug in the program, as noted in one of my other posts, so the track alignment does not seem to work or work very well. Therefore, I am unable to get consistent results.
If you look in the status window and see that your comparison requires sample rate adjustment, then here is what you can do. Open up the waveforms in your favorite digital audio editor and ensure that the both waveforms “start” at exactly the same time. That’s the trick. This is why I sample the first 40 seconds of the waveform, because in most cases, you do not need to line the waveforms up. Such is the case with the Soundkeeper filesas they both start at exactly the same time.
If you do need to line the waveforms up because you are recording the samples, then you can trim them later in your favorite digital audio editor. It is tedious as it may take a couple of passes before you get it lined up exactly.
Edited to add this section.
I ran another DiffMaker test, this time on Kote Moun Yo? samples from Equinox. I really enjoyed this recording as it definitely has ultrasonic information recorded (i.e. percussion instruments) and is crystal clear sound with very low noise floor. I would say state of the art recording. Great job Barry! http://soundkeeperrecordings.com/format.htm
I followed the same process as above. Again, the point in this is to either confirm or deny Monty’s claim that 16/44 is already better than our ears can hear and our sound system can reproduce. 24/192 should contain much more audio information than 16/44, so by comparing 16/44 to 24/192 using DiffMaker will show exactly how much difference there is between the two. In order for me to digitally compare the 16/44 to 24/192, I up-sampled the 16/44 to 24/192. If the R8 Brain resampler I used is doing its job proper, there should be no waveform changes as there is no information being added (or lost!), simply a (lossless) file format change.
Here is what Audio DiffMaker reports as being the difference.
-100dB difference file. It is very similar to my first test above, showing I can repeat the results, even on a completely different song/master.
Here is what the Difference waveform looks like.
And frequency analysis.
As you can see, the frequency plot shows ultrasonic energy, even though it is very low in overall level. Again, I have attached the difference file so you can listen to it. I cannot hear the ultrasonic information.
Part 2 Listening Tests
Given that the difference between 16/44 versus 24/192 is ultrasonic energy, it is important to verify that the gear used can actually reproduce ultrasonic energy. I used my Lynx L22 pro sound card that has a ruler flat frequency response out to at least 50KHz: http://i1217.photobucket.com/albums/dd381/mitchatola/lynxl22-1.jpg
I used my Sennheiser headphones with a custom Class A headphone amp that I built from the Audio Amateur from years gone by:
On the right is a toroid transformer feeding a regulated power supply and then my perf boards of the amp itself on the far left. I have measured the frequency response out to +200HKz. The headphone amp has enough clean power that you can place the headphones on the floor opened and crank it up like it was a boom box.
Next step is to verify that my gear can play ultrasonic information properly. These intermodulation test files provided by Monty’s article should be played first on your system to ensure you hear nothing at all. If you do hear tones, pops or clicks, that means the system under test is producing intermodulation distortion. http://people.xiph.org/~xiphmont/demo/neil-young.html#toc_intermod
With my particular computer system, Lynx L22 and Class A headphone amp, I did not hear any tones, clicks or pops. Ok onto step 2.
ABX testing. For listening tests that provide any level of statistical probability, double blind is the only way to go. I used Foobar2000 http://www.foobar2000.org/ and the ABX plugin http://www.foobar2000.org/components/view/foo_abx I made sure that I clicked on the Hide Results checkbox before I started the tests.
First up, 16/44 vs 24/192.
Here was the problem with this test. I could just tell by a very small delay when my DAC was switching from 16/44 to 24/192. So I was able to “game” the test:
So I resampled the 16/44 to 24/192 so I could not hear the DAC switch sample rates.
Here are the results:
Obviously I cannot hear the difference. This correlates with the DiffMaker results as well. The difference is so small that I was guessing, even though I was trying not to.
Since I cannot (significantly) measure or hear the difference between 16/44 and 24/192, I tried one more experiment where there is a known difference – MP3.
I took the 16/44 and converted it using the best MP3 codec (LAME) and encoded at 192Kbps bit rate. I used this bit rate as I listen to a lot of music on Zune and this is the default bit-rate when I download the music onto my disk for playing. As you may imagine, there is a reason that Microsoft chose this bit-rate and I will show why shortly.
Now comparing the 16/44 to the MP3 version produces the following Difference file in Audio DiffMaker:
And if I open up the waveform in Audacity:
Frequency Analysis:
I have included the Difference file again so you can hear the results. And it correlates very well with the other two other MP3 difference tests I performed here: http://www.computeraudiophile.com/blogs/FLAC-vs-WAV-Part-2-Final-Results#comment-131768
So the $64 million dollar question is, can I hear the difference in an ABX test for 16/44 and MP3?
While I did better than the 16/44 vs 24/192, it is in the territory of guessing :-) Listening closely, I thought I could hear a loss of transients on the percussion, but just barely perceptible to my ears.
Another way I can listen is to use Audio Diffmaker where I can reconstruct the comparison track by adding the difference back to the reference. By incrementally increasing the difference track level, I can easily hear the difference when the difference track is boosted by about +6dB.
I would hazard a guess this is the reason why Microsoft (and others) choose 192Kbps with MP3 as it gives the best fidelity versus file size. And likely the reason why most people don’t complain about it as most people (including me) cannot hear a quality difference, even under ABX testing conditions.
Conclusion
Well, for me, my ears, on my equipment, my test and listening results confirms Monty’s article that 16/44 is enough for my ears. This is also qualified by the science and engineering in the Digital Audio field: http://www.computeraudiophile.com/blogs/1644-vs-24192-Experiment#comment-135987
In fact, it may be that even high bitrate MP3’s is enough resolution, but that’s another debate.
Full disclosure, I am 53 years old and given the hearing loss versus age http://www.roger-russell.com/hearing/hearing.htm in the chart below, I may not be the best candidate for trying to hear ultrasonic audio information :-)
A quick hearing test from: http://www.phys.unsw.edu.au/jw/hearing.html confirms that I can hear to at least 12KHz, but down at 16KHz. It is no suprise to me why I don't hear ultrasonic audio information:
My perspective is this. If I was going to pick one cause to get behind in the world of music, it would not be over high resolution file formats. It would be the Loudness War.
Almost 30 years ago, the pop band, The Police, created a very popular album called Synchronicity: http://en.wikipedia.org/wiki/Synchronicity_(The_Police_album) With an overall Dynamic Range of 15 http://dr.loudness-war.info/details.php?id=12040 and the final cut on the album, Murder By Numbers, with a DR of 18 is an excellent example of taking the full advantage of the Red Book standard. The disc sounds fantastic. What happened since then?
The Loudness War in less than 2 minutes:
[video=youtube_share;3Gmex_4hreQ]
Given that this is CA, I would think everyone could correlate what they see in the visual representation of the waveform and what they hear. As I have discussed before, there is a direct correlation to what is measured with what is heard – it’s fundamental to the princples of audio. You can see and hear the difference, even over YouTube!
Final thoughts: All of the software used to perform both measurements and listening tests is free. Therefore, if you are curious and want to verify or deny Monty’s (and as it turns out, me too) claim, you can perform the same tests yourself.
Happy listening!<p><a href="
31 Comments
Recommended Comments