Truncating MQA files to 16 bits and the blue light still shines

[FredericV]

Today I had some fun writing a perl script, which manipulates an MQA file in such way, that I have even more serious questions about the MQA blue light.

Somewhere I read that MQA files could still be decoded if you truncate the 24 bit MQA distribution files to 16 bit, before sending it to the decoder.

So we actually tried it. Had to write some Perl code to make it work, as truncating with SOX did not work well and seems to round the value.

So this one liner does NOT work:
 

sox -V -D  input.wav -b 16 output.wav

I used @mansr's mqascan tool to verify my truncation script was correct.

So we now have a situation where:

1. Nothing new: 24 bit MQA distribution files, sent bitperfect to Mytek Brooklyn, work as expected:

If we play the 24/44.1 MQA files from 2L.no, MQA will unfold them to 24/88.2 and then upsample them to 24/352.8 in the renderer.



mqascan shows the details of the input file:

# mqascan -1 2L-050_01_stereo_DXD_WAV.mqa.flac
00000000: [5] datasync
          magic                     36: 0x11319207d
          stream_pos_flag            1: 0
          pad                        1: 1
          orig_rate                  5: 0x03 [352.8 kHz]
          src_rate                   5: 0x00 [44.1 kHz]
          render_filter              5: 8
          unknown                    5: 0x08
          unknown                    1: 0
          pad                        2: 0
          mqa_level                  8: 0x90
          items                      7: 2
          size                       8: 0x14
          size                       8: 0x0b
          type                       8: 0x00
          type                       8: 0x01
          [type 0]
          unknown                    2: 2
          unknown                    4: 0
          unknown                    7: 15
          unknown                    7: 127
          [type 1]
          unknown                   11: 0x314
          checksum                   4: 0xf

So the customer looking at his DAC display believes this is 24 bit 352.8 Khz, while the actual resolution is something like 17 bits and 88.2 Khz, upsampled with minimum phase and one cycle of postringing to the original resolution. It's misleading, but this is how MQA displays it.

 

2. MQA distribution files, sent bitperfect to Mytek Brooklyn, but with lower 8 bits blanked:

This is where it gets interesting:

 

We actually blanked the 8 LSB bits in an MQA file, so we have 16 actual bits per sample in a 24 bit file. So for each sample, 8 bits are zero/nada/rien. Mytek is smart enough to detect this zero padding, so it correctly claims it's only 16 bits.

This also happens with non-MQA files. This is why I still keep the Mytek: it's a good tool to see what's going over the USB, and what is the actual resolution of non-MQA. I can compare the ALSA stats with what the Brooklyn sees in the data.

The mqascan tool shows MQA is in the truncated file:
 

# mqascan -1 output.wav
00000000: [5] datasync
          magic                     36: 0x11319207d
          stream_pos_flag            1: 0
          pad                        1: 1
          orig_rate                  5: 0x03 [352.8 kHz]
          src_rate                   5: 0x00 [44.1 kHz]
          render_filter              5: 8
          unknown                    5: 0x08
          unknown                    1: 0
          pad                        2: 0
          mqa_level                  8: 0x90
          items                      7: 2
          size                       8: 0x14
          size                       8: 0x0b
          type                       8: 0x00
          type                       8: 0x01
          [type 0]
          unknown                    2: 2
          unknown                    4: 0
          unknown                    7: 15
          unknown                    7: 127
          [type 1]
          unknown                   11: 0x314
          checksum                   4: 0xf

So why does the MQA blue light still shines?

Now my big concern is: it's still saying the resolution is 352.8 Khz and the blue light still shines., but we have just thrown away 1/3 of the MQA input file as we blanked those bits. How in the world can such manipulation and throwing away the first unfold data, still authenticate the file.




How did we do this test?

1. take an MQA file from 2L.no:

2L-050_01_stereo_DXD_WAV.mqa.flac

http://www.2l.no/hires/index.html

2. decode the flac to input.wav

flac -d 2L-050_01_stereo_DXD_WAV.mqa.flac -o input.wav

3. then run the perl script on it, which I will release on an external site (copyright reasons, I intend to GPL this script, therefore I won't copy the code here)

./mqastripper.pl

which takes input.wav and writes 10 seconds into output.wav

# file input.wav
input.wav: RIFF (little-endian) data, WAVE audio, Microsoft PCM, 24 bit, stereo 44100 Hz
# file output.wav
output.wav: RIFF (little-endian) data, WAVE audio, Microsoft PCM, 24 bit, stereo 44100 Hz

This script can also be used to remove the encrypted non-nyquist data from MQA file, so it compresses better with flac:

input.wav: WARNING: legacy WAVE file has format type 1 but bits-per-sample=24
input.wav: wrote 30524205 bytes, ratio=0.635

output.wav: WARNING: legacy WAVE file has format type 1 but bits-per-sample=24
output.wav: wrote 608305 bytes, ratio=0.228

So this script actually implements what Jim Lesurf calls bit freezing:

http://www.audiomisc.co.uk/MQA/cool/bitfreezing.html

Script is still slow on my raspberry, but works. We'll open source it so anyone can play with it.
 

 

[FredericV]

1 minute ago, Em2016 said:

Very interesting. I wonder if the same happens with an MQA file that’s supposed to pack in more information, like this 353kHz MQA album:

 

https://shop.klicktrack.com/2l/411252/

The files that I used were sourced from 352.8 Khz (not 353Khz which is not a valid sample rate, but I get your point).

[FredericV]

5 minutes ago, hsmeets said:

 

Implementation by Mytek: perhaps it does not look at the actual encrypted bitstream but only at some value in the header of the FLAC container which says it is MQA....and turns on the 'blue light'.

 

Stick that header to a non-MQA FLAC file and see what happens......

 

Mytek does not know it's a flac of wav. All it sees is PCM. It's a WAV in this test for the 16 bits truncate test.

[FredericV]

5 minutes ago, Em2016 said:

 

Here you say it’s a 24/44 MQA file but I linked to an actual 353kHz MQA file/album. I wonder if the same happens, given this file should pack in much more info? 

 

For 24/88.2, 24/176.4, 24/352.8, .... source files, after encoding the MQA dIstribution files are always 24/44.1

For 48K multiples it's always 24/48

[FredericV]

32 minutes ago, mansr said:

This is by design. The MQA identification and authentication data is embedded in bit 8 of a 24-bit PCM stream. Dropping bits 0-7 thus leaves it untouched. The authentication works by computing a Blake2s hash over the top 15 bits (the plain PCM portion) and parts of the control stream in bit 8. This hash is then verified against a cryptographic signature extracted from the control stream. The public key is stored in the decoder. If the signature matches, the blue light goes on. The low 8 bits encoding the high-frequency content are not covered by the authentication.

I wonder if MQA CD is just the 24 bits MQA encode truncated to 16 bits. That would only work for masterings in multiples of 44.1K.
Then they can license the 24/44.1 files to Tidal and as also sell as downloads, where the customer has the illusion it's something higher like 176.4K or 352.8K after MQA decoding.
 

[FredericV]

Perl script released under GPLv2:

http://432evo.be/perl/mqastripper.pl
http://432evo.be/perl/mqastripper.txt

I also intend to add a feature where bits 0..7 are replaced by a repeating ASCII string, to see what that does with authentication.        

[FredericV]

MQA's time domain is clearly inspired by Ayre's filter:

 

I can do such filters in SOX easily. Not that I prefer them.

[FredericV]

Some updates. Our member "evangelist" was trying to attack my research as a being fake / gossip:

 

Sorry, but I still have a good old 2.2 mytek firmware which does not lie, and this firmware shows a dark grey MQA logo when non-MQA is played, and bright blue MQA logo when MQA is detected. It has no blue dot on the right.

So we updated to the latest version of Mytek's firmware, and now to my surprise the following happens:

- when playing the unstripped 24/44.1 MQA file, it shows 24/352.8 Khz and the blue MQA dot:

 

- when playing the stripped version with 8 LSB bits stripped, it will now claim it's still 24/352.8



 

The hint this is a lie, is because our 10 second loop interrupts the MQA markers for a short while, so for a fraction of a second, the DAC goes back to the actual input resolution of 16/44.1:

 

I modified my script to allow more bits to be blanked, so when stripping 9 bits, the 15 bits content in a 24 bit WAV container is still showing 16/44.1 on the Mytek. I wanted to be sure that the Mytek still detects bit blanking or padding for non-MQA, it still does work as expected for non-MQA.

So if the real MQA file, and stripped version with only 16 bits of entropy, both show decoding to 24/352.8, how does the customer know he is not being scammed?
 

[FredericV]

More fun: replacing the first unfold data with a repeating ASCII string "MQASHILL" still authenticates the file.

More experiments. Small correction: when 9 bits are stripped, the Mytek actually detects 15 bits (and not 16 bits). I looked at a distance and 15 and 16 look too similar. But here it is:




We improved the script to allow extra tampering with MQA files. We provide the 10 second sample files for each manipulation:

1. input.wav

This is 10 seconds from 2L-050_01_stereo_DXD_WAV.mqa.flac which mainly contains noise, and the first notes from the track.
This one is not manipulated.

2. mqashill.wav

 

Same file, but with the 8 LSB bits replaced by repeating ASCII string "MQASHILL". This tampered file still authenticates, and decodes to 24/352.8 on the mytek.

This was created with a newer version of my script which allows more tampering, not yet released:

./mqastripper.pl -i input.wav -o mqashill.wav -l 10 -t -d
Input file details: $VAR1 = {
          'sample_rate' => 44100,
          'total_length' => 48090204,
          'block_align' => 6,
          'length' => '181.746666666667',
          'data_start' => 44,
          'bits_sample' => 24,
          'channels' => 2,
          'wave-ex' => 0,
          'data_length' => 48090168,
          'bytes_sec' => 264600,
          'data_finish' => 48090212
        };
Samples: 8015028
Rate: 44100
Bits to shift = 8
-127,40 -> -179,77
LSB LEFT  = 77 M
LSB RIGHT = 77 M
451,304 -> 337,337
LSB LEFT  = 81 Q
LSB RIGHT = 81 Q
178,-169 -> 65,-191
LSB LEFT  = 65 A
LSB RIGHT = 65 A
89,203 -> 83,83
LSB LEFT  = 83 S
LSB RIGHT = 83 S
444,163 -> 328,72
LSB LEFT  = 72 H
LSB RIGHT = 72 H
-210,-97 -> -183,-183
LSB LEFT  = 73 I
LSB RIGHT = 73 I
493,518 -> 332,588
LSB LEFT  = 76 L
LSB RIGHT = 76 L
136,-585 -> 76,-692
LSB LEFT  = 76 L
LSB RIGHT = 76 L
139,769 -> 77,845
LSB LEFT  = 77 M
LSB RIGHT = 77 M
735,-545 -> 593,-687
LSB LEFT  = 81 Q
LSB RIGHT = 81 Q
-521,657 -> -703,577
LSB LEFT  = 65 A
LSB RIGHT = 65 A

 

3. output-stripped-8.wav

 

This is the 24 bit input.wav, with 8 LSB bits stripped per sample, in a 24 bit container. So 16 bits of entropy at best.
Authenticates and dac claims this is 24/352.8

 

4. output-stripped-9.wav

 

This is the 24 bit input.wav, with 9 LSB bits stripped per sample, in a 24 bit container.
This actually destroys the MQA control data. DAC says no-MQA and 15 bits of data.

So to conclude:

- 24 bit MQA file original authenticates (what else did you expect)
- replacing 8 LSB bits of each 24 bit sample by a repeating ASCII string "MQASHILL" and the file still authenticates and renders 24/352.8 in an MQA dac
- blanking 8 LSB bits and the file still authenticates and renders 24/352.8 in an MQA dac

I did no try to listen to the potential garbage that would come out of mqashill.wav when played on an MQA dac, as this file has new fake unfold content, and still authenticates.

[FredericV]

My biggest concern:

MQA always said tampering with the file no longer authenticates the file. We have proven the opposite.

The customer can't differentiate between the following 3 cases which all authenticate:

1. 24 bits original MQA file is playing, decoder in DAC doing 1e unfold + upsampling.
2. truncated version to 16 bits is playing, not doing 1e unfold as the origami data to do this is gone, so only upsampling
3. replacing origami data by garbage, so curious what will be in the ultrasonics, would need to measure this at the output of the Mytek

So authentication is quack on several levels:

- not always authenticated by the mastering engineer who did not give permission, but MQA claims this is the master, as encoded by request of some other party not being the mastering engineer

- fake authentication of tampered MQA files still showing the blue light

[FredericV]

1 minute ago, mansr said:

What happens if you mess with bit 9, just above the MQA control stream?

Coding as we speak.

BTW Hans Beekhuyzen claims the following:
 

Quote

However, if the file is tampered with in any way, the decoder will notice this and will not light that indicator.

Maybe he should read this post. His information is not accurate.

[FredericV]

Setting the 9th bit (starting counting from bit 0) via one of these 2 Perl operators, de-authenticates the file, and mytek says it's 24/44.1:
 

Method 1:

$ln=$l | 0b1000000000;
$rn=$r | 0b1000000000;

Method 2:

$ln=$l | 512;
$rn=$r | 512;

 

[FredericV]

34 minutes ago, mansr said:

As expected. The authentication covers only the PCM part of the stream.

 

Here's something else to try. Toggle bit 9 in one sample every few seconds.

Now it flips the 9th bit every 3 seconds.

pi@mqb:~/quack $ ./mqabit9.pl -i input.wav -o corrupt_every_3secs.wav  -l 30
Input file details: $VAR1 = {
          'bits_sample' => 24,
          'data_length' => 48090168,
          'channels' => 2,
          'wave-ex' => 0,
          'data_start' => 44,
          'length' => '181.746666666667',
          'sample_rate' => 44100,
          'bytes_sec' => 264600,
          'data_finish' => 48090212,
          'total_length' => 48090204,
          'block_align' => 6
        };
Samples: 8015028
Rate: 44100
Sample 0 corrupted
***Sample 132300 corrupted
***Sample 264600 corrupted
***Sample 396900 corrupted
***Sample 529200 corrupted
***Sample 661500 corrupted
***Sample 793800 corrupted
***Sample 926100 corrupted
***Sample 1058400 corrupted
***Sample 1190700 corrupted
***

It ignores the first corrupt sample, blue light almost immediately shines, but after 3 seconds, the light goes out and stays out. It does not recover as long as the same file is playing. I actually have to switch to another MQA file to get the blue light back.

Curious if we start to flip bits in the MQA control stream ....

[FredericV]

Corrupting the control bit (counting from 0, bit 8) is already detected when the first sample was corrupted, contrary to corrupting the bit just above it (counting from 0, bit 9). This confirm the initial part of the track is not played in MQA.

When I corrupt the control bit after 3 seconds (and not starting with the first sample), and repeat the process every 3 seconds, as soon as it detects a corrupt bit in the control stream, the MQA light goes out, and does not recover.

[FredericV]

In a Dutch Facebook group managed by a Meridian dealer, the users actually believe that they are listening to 24/192 because their NAD amp is saying so.

https://www.facebook.com/photo.php?fbid=1834948226537180&set=gm.598310543850915&type=3&theater&ifg=1

It seems on bluesound, there's also 24/192 on the display of a NAD amp, and the user believes this blindly:

 

Basically they are listening to something like 17/96 (core decoder) upsampled to 24/192 (renderer).

[FredericV]

Is MQA's compression a joke?

If we sabotage the 8 LSB bits in any way (blanking / random garbage / ascii string "MQASHILL" encoding), the spectrum changes after decoding:

 

In all four cases, the user sees 24/352.8 on his DAC display or screen / app of MQA enabled device, but does not know what he actually gets. The spectrum of the sabotaged files extends above 20 Khz, but Mans has shown this is due to aliasing.

Maybe the biggest joke is the fact that this little spectrum extention results in 50% larger PCM files (24 bits instead of 16 bits), and 100% larger flac files. Why? The 8 LSB bits in MQA don't compress well in FLAC.

First we took a new 2L.no MQA distribution file, and copied 20 secs to a wav file using sox:
 

sox  2L-053_04_stereo-DXD.mqa.flac 2L-053_04_stereo-DXD-20secs.mqa.wav trim 0 20

We stripped 8 bits using our tool, so we now have two 24/44.1 files with a duration of 20 secs, one where the 8 LSB bits have been blanked, and another which is a 20 secs copy of the real MQA file, which can both be decoded by MQA to 24/352.8.
 

$ du -h --si 2L-053_04_stereo-DXD.mqa-16in24bit.wav 2L-053_04_stereo-DXD-20secs.mqa.wav
5.3M    2L-053_04_stereo-DXD.mqa-16in24bit.wav
5.3M    2L-053_04_stereo-DXD-20secs.mqa.wav

Now if we encode both WAV files back to flac, let's observe the already impressive compression ratio difference:

 

$ flac -f --best 2L-053_04_stereo-DXD.mqa-16in24bit.wav 2L-053_04_stereo-DXD-20secs.mqa.wav

flac 1.3.0, Copyright (C) 2000-2009, 2011-2013  Josh Coalson & Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

2L-053_04_stereo-DXD.mqa-16in24bit.wav: WARNING: legacy WAVE file has format type 1 but bits-per-sample=24
2L-053_04_stereo-DXD.mqa-16in24bit.wav: wrote 1633178 bytes, ratio=0.309
2L-053_04_stereo-DXD-20secs.mqa.wav: WARNING: skipping unknown chunk 'fact' (use --keep-foreign-metadata to keep)                                                    2L-053_04_stereo-DXD-20secs.mqa.wav: wrote 3394644 bytes, ratio=0.641

Now look at the file sizes:

$ du -h --si 2L-053_04_stereo-DXD.mqa-16in24bit.flac 2L-053_04_stereo-DXD-20secs.mqa.flac
1.7M    2L-053_04_stereo-DXD.mqa-16in24bit.flac
3.4M    2L-053_04_stereo-DXD-20secs.mqa.flac

So this means that only the redbook part of the MQA distribution file compresses with flac like normal music. The secret bits don't compress and therefore the filesize goes x2. While this is nothing new, the fact that it results in such a small difference in spectrum, makes it a waste of diskspace.

Most likely this is how MQA CD gets away with it. If someone can get me an MQA cd, we can do some measurements.

Furthermore, compressing the unfolded versions (both 24/88.2 wav) back to flac, shows a small difference is file size:
 

$ du -h --si 2L-053_04_stereo-DXD.mqa-16in24bit-unfolded.flac unfold-official-20secs.flac
5.7M    2L-053_04_stereo-DXD.mqa-16in24bit-unfolded.flac
6.0M    unfold-official-20secs.flac

 

This suggests a 5% reduction in entropy in the MQA decoder output, when the decoder did not have the 8 LSB bits to work with. But as the flac version of the input file for the decoder is 50% smaller for the 16 bits truncated version, this means MQA is not very efficient.

Also note that flac can't compress the output of the decoder back to the MQA flac distribution file sizes. This suggest garbage is added which does not compress well (garbage because of aliasing and higher noise floor).



 

[FredericV]

53 minutes ago, FredericV said:

Also note that flac can't compress the output of the decoder back to the MQA flac distribution file sizes. This suggest garbage is added which does not compress well (garbage because of aliasing and higher noise floor).

The fact that the flac encode of the MQA decoder output is almost twice the size of the distribution is logical: MQA uses leaky filters which mirror the baseband into the ultrasonics. So flac has to encode the same signal twice, once at normal amplitude, and it's attenuated copy (alias), which explains the almost double file size.

[FredericV]

2 minutes ago, mansr said:

If the expected markers aren't there, it is simply ignored. It has to work this way. If some software (player/OS) has mangled the lowest bits for whatever reason, the DAC still must play something.

 

In Bob's official explication, truncating to 16 bits is supported. Some excuse about automotive and bluetooth which can truncate bit depth. But I can't find the page where this was posted. All I remember is the fact that it was once stated, so this was the reason for testing it.

[FredericV]

Let's compare fake content from SOX's leaky upsample filter settings (lower waveforms, this is certainly not a standard sox filter that normal users would use) with MQA's leaky filter (upper waveforms), when in both cases the 1e unfold data is not available:

SOX has no access to MQA's secret internals in the 8LSB bits of the distribution file, so this is just noise.
MQA's decoder has no access to 8LSB bits of the distribution file when we blank those bits, or replace them with random noise, or with a repeated string such as "MQASHILL".

The SOX filter only attenuates with 1 dB, but it looks like MQA boosts the volume a little more. See my signature how this filter works.

As you can see, we can create fake +20 Khz spectrum with SOX, starting from the MQA distribution file:

Just like with MQA's leaky filters:

 

Looking at the normal 16 bit 44.1 file, this is the "normal" spectrum of a 44.1K file, which stops at 20 Khz:

 

Conclusion:

- leaky filters can be implemented in open source, and also create the fake content just like MQA
- the extra ultrasonic spectrum that a 50% larger MQA encoded wave file is offering (24 bit MQA wave file, vs truncated 16 bit MQA wave file), is minimal

So except for a licensing scheme, what purpose does the 1e unfold offer? Most of it's audible effects due to leaky filters can also be accomplished with OSS.
 

[FredericV]

7 minutes ago, mansr said:

It undoes some of the dither applied to the PCM part as you can see in this plot of a silent bit in the file. DXD original included for reference.

 

But well below audible range ... does it matter?

 

[FredericV]

This was on public facebook, post shared with public. I first thought I needed new glasses, but no, it's in there:

 

So why not get rid of all those nasty ultrasonics? Yup he wrote it: Mr MQA aka Peter Veth aka Peter Markus, who leads a closed FB group about MQA wrote that MQA works so well because it removes ultrasonics, and as a result sounds better.

He sure must now respect me. My perl script reduces the MQA flac file size to 50% and wav file size to 2/3 and further reduce the presence of ultrasonics that nobody wants
 

[FredericV]

6 hours ago, miguelito said:

It would be quite interesting to compare an MQA CD rip with the equivalent non-MQA rip. I bet you the diff is eq.

How many MQA cd's are there ? 5...10 ?
How many MQA albums on Tidal are there? A few thousand.

While MQA could probably design a different bit allocation scheme for 16 bit distribution files, they also have stated that truncating 24 bit files to 16 bit also works, so Occam's razor dictates they have just truncated the 24 bit version for MQA CD.

It would not make much sense to do such an engineering effort for a few CD's.
 

[FredericV]

Members of the closed MQA facebook group are asking if this research has been peer reviewed.

It just proves that they are not looking for the truth, but remain in their MQA bubble.

Others have already confirmed on CA that the crippled files authenticate.

Talking to MQA members from the closed group is like talking with flat earthers. They suggested that we contact MQA. They are blind and don't want the truth.