'FeralA' decoder -- free-to-use

[John Dyson]

Current decoder status...

ETA -- maybe 1-2 more days...

 

*) It appears that the metadata transparency has been improved -- so that is close to ready, if not already ready.

*) Refactored the EQ commands for the new 2nd order modes.   My working design wasn't a 'design' but was ad-hoc.  I regularized the command structure.

*) The 250Hz IMD mitigation is a MAJOR improvement in transparency.  Gets rid of the 'electronic' sound as I perceive it.

*) In the last day, I had decided to revisit the MD mitigation.  The results of the review produced a profound improvement.

 

MD mitigation--

It consists of several tradeoffs, all of them are primarily against CPU time, but also practical limitations.  The section of the program that does the anti-MD (not primarily IMD) mitigation has two methods:

A) band splitting chops up the bands so that distortion components and signal energy causing the distortion are brute force minimized.

B) Complex Hilbert transform oriented math that exploits subtle aspects of mathematical nonlinearity -- thereby twisting the distortion to both be decreased AND less audible.

 

Each of A and B individually have tradeoffs and also there is tension between A and B above.  'A' tends not to be as effective as 'B', but also uses less CPU.

A and B have been refactored, and the complex anti-MD has replaced some band splitting -- thereby producing a neutral change in CPU utilization, but a very very very profound decrease in distortion.

 

There are numerous little tunables and tradeoffs that are not mathematically easy to calculate -- the tradeoffs depend on the statistics of the recordings,  and also just imponderable mathematics.  I tried several alternatives, when entail rather substantial changes in the code -- not just changing parameters.  After doing numerous tests -- a better distortion null has been found.  This code is EXACTLY the part that removes the DolbyA fog.

 

I have included 3 snippets...

 

nofogmin7.flac (has all of the anti-IMD enabled to a normal level)

fog-disable1level.flac (disables the anti-fog hilbert code, still has the static anti-modulation code)

diff-nofogmin7-disable1level.flac (difference between the two, time sync, +20dB)

 

The driving/calculated gain control is IDENTICAL for both versions, yet there is a noticeable difference in the sound qualities.  

 

There is more distorted 'growl' and less clean high end when the Hilbert transform code is disabled (in fog-disable1level.flac).  Even with very very careful listening -- I am often fooled -- the nofogmin7.flac doessound more clean.  The DolbyA fog  sometimes manifests worse than this example and sometimes totally unnoticeable.  I should probably have demoed an example with more extreme differences, but this is a typically noticeable example.  The most subtle distortion components are being chased!!!

 

* The disabled anti-MD code removes subtle HF filtering/spreading/softening resulting from gain control and also removes 'grinding' middle frequency distortion components.

 

This evidences only one layer of distortion management.  A direct HW implementation of a decoder would create much greater distortion - the DHNRDS FA decoder cannot even be disabled as far as the old DolbyA HW!!!

 

John

 

 

 

 

nofogmin7.flac fog-disable1level.flac diff-nofogmin7-disable1level.flac

[jabbr]

Don't forget Linux  

[John Dyson]

2 hours ago, jabbr said:

Don't forget Linux  

 

Definitely will make Linux available.   Actually, I'll upload the exact Linux code that I use for testing --  my only reason for not previously distributing the Linux version was laziness -- simple as that.  Once I realize that someone really wants it, then I am happy to make it available.

 

I REALLY wish more people used Linux -- it has a nice philosophy that 'one' program doesn't have to do it all, but programs can be connected together, producing a result that is bigger than any one piece.  This really makes the decoder MUCH easier to use for certain applications.  I do admit that the learning curve is prohibitively long for those who don't normally use a Unix clone like Linux.   Also, depending on other programs might often result in dissapointment.   Just a different way of thinking...

 

Anyway, I am at the stage where I am doing decoding tests for finding the correct parameters for various recordings in my archives.  The main reason is for examples/a weak form of documentation.

 

Also, the 'using' document has been updated over the last week or so, but it needs some more intensive work.

 

I think that the code itself is ready, except for updating the version number, etc.

 

---

About quality

 

There are massive quality improvements to the point where the previous release, and even the recent Nat King Cole demos were of lesser quality than this 'super close to final' version of the decoder can do.

 

Previously, there were 'secret' super-high-quality modes, but most of those more CPU intensive modes were spent undoing the errors in the decoder.   Now, the very highest quality mode that makes any improvement is '--xpp' or '--xtraplusplus'.  (Sorry about the naming.)  I spent lots of time today and yesterday optimizing the 'optimizations', so that time isn't wasted while doing useless things, but allowing more time to do helpful things.

 

---

About quality modes

 

Normally, --normal (the default mode) works well and is moderately fast.  However, the full anti-MD code is NOT enabled.  For better quality, either --xp (--xtraplus) or --xpp (--xtraplusplus) are amazing -- work better than any reasonable analog  HW can possibly do...  When doing differences between the outputs at different quality levels, there is a huge increase from --normal to --xtra or --xtraplus or --xtraplusplus.  The not-really-secret higher modes now make ZERO difference.

 

From --basic to --normal, there is a nice quality improvement.  Both modes are generally more clean than DolbyA HW, but I support the two different modes simply because --basic IS quite a bit more quick.  The quality is still really good, but it is by far best to use --normal of the two.

 

The '--xtra' derived modes are super good -- and these are where the vestiges of the DolbyA fog are removed.  When doing file differences, it really shows the subtle damage that compression/expansion does -- because it isn't just the damage to dynamic range, but they cause really weird and unexpected damage -- difficult to avoid, just as a matter of the math.

 

Sorry for the delays -- I am a perfectionist who panics with last minute changes...   My project partner and I call it 'stuttering' the release.  I am trying really hard for this release to be correct.

 

John

 

 

 

 

 

 

[John Dyson]

I am really hoping for today (being realistic +12Hrs - +14Hrs)  for the release.   The hard part right now is ONLY the small "usage document".  I am working on the examples, because the various options/etc can *initially*  be imponderable.  The general infromation has to be much more complete than the more usual examples at the end of the document.  Even though users might not have the same exact CDs/downloads as I do, it is still best to be accurate.

 

When producing the examples, it requires my 100% concentration to get the right corrective EQ numbers.   I suspect that most people will give a quick glance to the general usage information and scan directly to the list of decoding examples at the end.   After the first few paragraphs of general information, then boredom will direct people to the examples :-).

 

The decoder IS easy to use for someone already accommodated to command line programs -- and the choice of parameters is easier than the general usage might imply.   SO, I'm just using my very MEAGER composition skills by trying to avoid a major turn-off because of an imponderable usage document.

 

In fact, I am thinking about a better 'quick start' document, which will do the quick boostrap to avoid the worst frustration of first-usage mistakes.

 

So, the hold-up is ONLY the docs.  The decoding tests to produce the usage/examples are coming out better than I had ever hoped.  My guess is that this version is the last version of the decoding 'engine', and any subsequent improvements would be raw bugfixes or usability improvements.  IT SOUNDS VERY GOOOOD...

 

* Part of what makes the decoder sound good is knowing how to use it.  Because of the variability of techniques used in FeralA encoding, there are two general paths for the EQ instead of the previous single one.  Both paths can be intermixed, and the trick for readable docs is explaining the difference/usage for the paths.

 

The general case docs are a lot longer than what is needed 90% of the time.  All this said, most of the time, there is one 2nd order decoding EQ that works for about 1/2 of the recordings, and perhaps 2 others for most of the rest.  On top of that, it is sometimes beneficial to use one of two 1st order 'final' EQ choices.

So, quick, reasonably good decodes are easy to do, and a bit of tweaking can make noticeable improvements.

 

EDIT:  admittedly, I get distracted when listening -- forget that I need to sample and not listen & enjoy...

 

John

 

[John Dyson]

V1.4.1A of the decoder is ready.  It sounds really good, but as you know, I do botch things.  I'll fix last minute bugs as quickly as I can, and try to answer questions as long as I am awake :-).

(Hope that I didn't upload the source files instead of the binaries :-)).

 

First, the .wav file support is a little better.  Metadata should be more transparent.

Second, the quality is A LOT better, but with some caveats.  The caveats:  different decoding parameters are desirable.

 

Except for bugfixes, I am hoping that the 'decoding engine' is complete.  It is very good at avoiding modulation distortions, so good that I don't think that it is possible to be any better.  This doesn't mean that the project is 'finished', because there is ALWAYS room for usability improvements.

 

The Windows V1.4.1A version installs essentially identically to the previous.

For Linux, unpacking the ,txz file produces a V1.4.1A subdir, and the programs reside in that subdir.

 

After following the paltry install instructions (refer earlier in this forum for Windows), then the fastest way to get started is to read the 'gettingstarted-V1.4X.pdf' file.  Then, when you get somewhat comfortable, then try to read the babble in the 'using-1.4X.pdf' file.

 

I am attaching the 'guides'  for really quick reference, but everything (including the 'guides') is in the Dropbox location.

 

The Windows program file: 'da-release1.4.1A-win-2020-04-21.zip'

The Linux program file: 'da-avx-V1.4.1A-Linux.txz'

Dropbox location: https://www.dropbox.com/sh/1srzzih0qoi1k4l/AAAMNIQ47AzBe1TubxJutJADa?dl=0

 

I own the software, and you have full rights to use or redistribute as much as you want.  I'd prefer that when redistributing it -- TELL ME (not required -- I just want to have an idea about how many people using the decoder.)

The difference between two or three users total is really important to me :-).

 

The raw/true DolbyA mode itself requires a license file, so unless you use the '--fa' or '--fb' or '--fc' or '--fD' switches, then the decoder will choke without the license file.  If you want/need (unlikely), I'll  make you a license file so you can do real, undistorted DolbyA decoding.  The licensed version shouldn't be redistributed without permission.  But, for feralA -- HAVE AT IT!!!   Frankly, the FeralA decoding is more sophisticated than DolbyA anyway... :-).

 

Please have fun with this -- don't let it be frustrating or be a chore, I'll do what I can to help using the decoder.

 

John

 

install-V1.4.1A-Linux.txt using-V1.4X.pdf gettingstarted-V1.4X.pdf

[jabbr]

Wow!

 

The decoding of Supertramp's "Breakfast In America" is remarkable!

[sandyk]

2 hours ago, jabbr said:

Wow!

 

The decoding of Supertramp's "Breakfast In America" is remarkable!

John

 Has Jon heard the very latest minor correction ?

 

Alex

[John Dyson]

40 minutes ago, sandyk said:

John

 Has Jon heard the very latest minor correction ?

 

Alex

The timing seems correct that JABBR might have gotten the latest one.  Either one seems good, but you were right that the first one needed to be tamed a little bit.  (As I mentioned in the PM group, the latest version uses a full notch for the setting, instead of the 'in-between' value that I first used.)   I wasn't totally sure, so I chose a value that was either a compromise or definitely wrong, depending how you look at it.  The actual difference from old to new is about -1.5dB at about 12kHz before decoding.  That kind of change makes a difference over a much wider range than '12kHz' might imply.   That apparently minor change can actually affect the 3kHz range also because of the dynamics associated with the HF0 band.

 

John

 

[jabbr]

At the moment I am working on: 1974 The Lamb Lies Down on Broadway (VJCP-68096-097)

The intro to the first song is useful because there is a fly which flies around the soundstage during the intro -- the setting --wof 1.19 didn;'t help! Hmm maybe tone=-14.25 and I still need to try various settings ...

 

The good thing about my workstation is that I can use HQPlayer's EQ modulators at DSD128 while I run da-avx 😳

[Skip Pack]

Hello John,

After successfully converting your 820Hz wav file I immediately went to the deep end. I copied my CD sourced files of the Crime of the Century, converted the first song to wav, and got the following result when I tried to process it. As we had determined some time back that my cpu was not of a modern enough generation even though it is a 4 core I5,  I tried da-win with the following result. I'm getting the cpu generation warning and a similar stack dump. Since I'm not playing the output in real time, slowness is not a problem for me.

 

Thanks for your continued huge effort on this. 

 

 

C:\Users\Skip_Active\Music_Processing\working\STramp\Crime of the Century>da-win --info=2 --fb=4 --tone=-13.40 --xpp --input=01_-_School_-_Supertramp.wav --overwrite --output=011_-_School_-_Supertramp.wav
Audio "DHNRDS DA decoder" V1.4.1A 21Apr2020 -- Author John S. Dyson -- Copyright 2017,2018,2019,2020
License terms are in accompanying documentation

Time Out:
Required CPU type: WIN64: x86-64bit with SSE: Silvermont or greater
Input file: "01_-_School_-_Supertramp.wav"
Output file: "011_-_School_-_Supertramp.wav"
StereoImage: preA: 1.41, postA: 0.71, preB: 1.41, postB: 0.71, postFinal: 1.00
      0 [] da-win 625 cygwin_exception::open_stackdumpfile: Dumping stack trace to da-win.exe.stackdump

[jabbr]

I'm thinking 1978 And Then There Were Three... (VJCP-68102)/11 Follow You Follow Me

 

fc=X -12.85 ??

[jabbr]

So wow stuff like Supertramp & Carpenters is vastly better with decode but perhaps old Genesis isnt so vastly better — either that or I’m just not great at  using the tool, 

 

so I guess old stuff that sounds thin and aweful is the best candidate for feral A decoding ... what do people have?

[sandyk]

45 minutes ago, jabbr said:

So wow stuff like Supertramp & Carpenters is vastly better with decode but perhaps old Genesis isnt so vastly better — either that or I’m just not great at  using the tool, 

 

so I guess old stuff that sounds thin and aweful is the best candidate for feral A decoding ... what do people have?

 Exaggerated vocal sibilance is often a good guide to Dolby-A problems too.

 

 

[John Dyson]

13 hours ago, Skip Pack said:

Hello John,

After successfully converting your 820Hz wav file I immediately went to the deep end. I copied my CD sourced files of the Crime of the Century, converted the first song to wav, and got the following result when I tried to process it. As we had determined some time back that my cpu was not of a modern enough generation even though it is a 4 core I5,  I tried da-win with the following result. I'm getting the cpu generation warning and a similar stack dump. Since I'm not playing the output in real time, slowness is not a problem for me.

 

Thanks for your continued huge effort on this. 

 

 

C:\Users\Skip_Active\Music_Processing\working\STramp\Crime of the Century>da-win --info=2 --fb=4 --tone=-13.40 --xpp --input=01_-_School_-_Supertramp.wav --overwrite --output=011_-_School_-_Supertramp.wav
Audio "DHNRDS DA decoder" V1.4.1A 21Apr2020 -- Author John S. Dyson -- Copyright 2017,2018,2019,2020
License terms are in accompanying documentation

Time Out:
Required CPU type: WIN64: x86-64bit with SSE: Silvermont or greater
Input file: "01_-_School_-_Supertramp.wav"
Output file: "011_-_School_-_Supertramp.wav"
StereoImage: preA: 1.41, postA: 0.71, preB: 1.41, postB: 0.71, postFinal: 1.00
      0 [] da-win 625 cygwin_exception::open_stackdumpfile: Dumping stack trace to da-win.exe.stackdump

I will look into this.  Might have to make a version with more of a P4 type instruction set.   Give me a few days -- lots of testing to make sure that the code still holds up for that architecture.   Might be another problem also -- will keep my mind open and try to come back with an answer and hopefully a solution (If I can get a good answer.)  It really does look like an instruction set thing though.   I know that it *should* work on an i3xxx, but maybe not on an i2xxx?

 

There is probably some obscure thing that the code isn't doing right for your computer, and definitely WILL look into it.

 

If you can provide some kind of info on your CPU -- it would help point me in the right direction.

 

John

 

[John Dyson]

10 hours ago, jabbr said:

So wow stuff like Supertramp & Carpenters is vastly better with decode but perhaps old Genesis isnt so vastly better — either that or I’m just not great at  using the tool, 

 

so I guess old stuff that sounds thin and aweful is the best candidate for feral A decoding ... what do people have?

If you can send me a snippet -- I can either figure out if the decoder is missing a feature, maybe figure out if there is compression after the FeralA?  Might not even be FeralA...

 

If you privately send me a problematical selection, I'll try to figure it out.   I want to learn of any problems also.

 

John

 

[John Dyson]

About the program having problems running on certain machines.   Perhaps the best solution is to come up with a version that theoretically runs on a 32bit P4.  I found that the 32bit doesn't really run all that more slow -- there are comopensating aspects of 32 bit that make it not all that more slow.  The decoder isn't very big, but SOME of the data structures might blow-out things like maximum stack offsets on 32bits.  If it does somethings like that, I might be able to tweak down some of the more extreme capabilites, and leave the normal decoder modes intact.

 

Might take me a few days, as I havent built an SSE only version in a long time -- I might have mistakenly made some assumptions - but I think that it is worth a try.

 

For AVX2 type code, it is really advantageous to keep 64bits, but for other SIMD environments, the 32bits isnt' a killer.  This will maximally avoid excluding people.   Apple, on the other hand, does have a windows emulator -- maybe that might work (it used to work.)

 

John

 

[jabbr]

11 minutes ago, John Dyson said:

Apple, on the other hand, does have a windows emulator -- maybe that might work (it used to work.)

Apple often compiles command line stuff for *nix straightforwardly using homebrew 

[jabbr]

2 hours ago, John Dyson said:

If you can send me a snippet -- I can either figure out if the decoder is missing a feature, maybe figure out if there is compression after the FeralA?  Might not even be FeralA...

 

If you privately send me a problematical selection, I'll try to figure it out.   I want to learn of any problems also.

 

Will do, I’m not sure it’s a “problem”, per se. Maybe source not feralA ? I wrote a little script to pipe the sox and the tag the output name with the main settings

 

does a $1.f$2$3.$4.flac for output name with:

da-avx —f$2=$3 —tone=-$4

[Skip Pack]

2 hours ago, John Dyson said:

If you can provide some kind of info on your CPU -- it would help point me in the right direction.

 

Here's what I found. If there's more info that would help you, I'm happy to dig further. Thanks.

 

Skip

[rando]

To save John a few seconds looking this up

 

[John Dyson]

1 hour ago, Skip Pack said:

 

Here's what I found. If there's more info that would help you, I'm happy to dig further. Thanks.

 

Skip

It looks like it might be something I am missing on the i3000 series of processors.  I built the code for the Atom, but maybe the Atom has something that the i3000 doesn't?  I doubt that though.

 

Looks like I'll be busy tomorrow working on a few problems.  (Got some metdata problems also.)

 

:-).

 

John

 

[John Dyson]

Forthcoming release for .wav file fixes ONLY.   The audio will be identical/ZERO changes, so all of your calibrations/etc will be kept the same.   There will be further/more complete (but not total) support for .wav metadata.

 

I estimate Monday afternoon, 27Apr, we should have a new release.  Also, I'll be checking into support for a wider set of CPUs.

 

Again -- ZERO changes to the sound charactersitics.  That code is FROZEN unless there is a bug or addition.

 

John

 

[jabbr]

Here is my little script:

#!/bin/bash
sox "$1.flac" --type=wav -  | da-avx --info=1 --f$2=$3 --tone=-$4 --xpp | sox --type=wav - "$1.f$2$3t$4.flac"

and then you can generate a number of different versions to listen to:

#!/bash
for itone in 12.85 13.40 14.25
do
 daavxpipe.sh $1 a 2 $itone
 daavxpipe.sh $1 a 3 $itone
 daavxpipe.sh $1 b 3 $itone
 daavxpipe.sh $1 b 4 $itone
 daavxpipe.sh $1 c 3 $itone
 daavxpipe.sh $1 c 4 $itone
 daavxpipe.sh $1 c X $itone
done

 

[John Dyson]

4 hours ago, jabbr said:

Here is my little script:

#!/bin/bash
sox "$1.flac" --type=wav -  | da-avx --info=1 --f$2=$3 --tone=-$4 --xpp | sox --type=wav - "$1.f$2$3t$4.flac"

and then you can generate a number of different versions to listen to:

#!/bash
for itone in 12.85 13.40 14.25
do
 daavxpipe.sh $1 a 2 $itone
 daavxpipe.sh $1 a 3 $itone
 daavxpipe.sh $1 b 3 $itone
 daavxpipe.sh $1 b 4 $itone
 daavxpipe.sh $1 c 3 $itone
 daavxpipe.sh $1 c 4 $itone
 daavxpipe.sh $1 c X $itone
done

 

One suggestion for speed -- when you are just doing a choice, before a final version -- then'--xp' will run significantly more quickly or even '--normal' is quicker yet.   '--xp' will be much more similar to '--xpp' than '--normal' though.  Once you find a good final choice, then maybe rerun with '--xpp' for your 'perfect' copy.  (You might not notice quite as big a speed difference between --normal and --xp on a larger multi-core machine because the software spreads the --xp and -xpp modes across all available cores.)

 

Your idea of automating for finding the best parameters is a good one.

 

On non-difficult material, when I have run differences between the audio (I mean, real audio subtraction), sometimes --xpp and --xp are very very similar.  '--normal', however is usually different with very noticeably more distortion at the 'audiophile' level.   When I ran the tests, I did an audio sample-by-sample difference at a 20 or 30dB gain up.

 

It is amazing that for casual listening, most of the time, --normal is really good anyway.  I do think that '--basic' might sometimes be a little misleading for test-decodes when expecting to use '--xp' or '--xpp' for the final version. '--basic' is usually a little too 'basic' for me most of the time.  I made the --basic mode available for people who have very slow machines (like my Atom laptop is really, really slow.)  * Even --basic is more clean than decoding with true DolbyA, believe it or not!!!!

 

Just trying to save you some time...  I know that the decoder can seem very slow -- 'some people' with 8 core machines might not notice the speed issue :-).   I made every tradeoff with a bias towards mathematical quality&audio accuracy -- then used execution time as a practical limit for the algorithm complexities.

 

John

 

[John Dyson]

Here are some hints on selecting the best corrective EQ modes... 

 

Sometimes, if you find that --fb or --fc alone are not 'right', where it seems like the ideal is somewhere in between '--fb' and '--fc', then immediately moving to --fb=X  might be a good first choice.  (Similar concept between --fc and --fD, where --fc=X might help.)

 

Here is what the modes do:

 

--fb or --fc (or --fD) sets the rate of the LPF at approx 3k, 6k and 9k.  --fb is -3,0dB at each, --fc is -4.5dB at each and --fD is -6.0dB at each frequency range near the center.

 

When you add an =X, then the filter is extended to 12kHz also.

If you add =x, then it also extends to 12kHz, but the decrease is 1/2 of the normal decrease for the mode.

 

Example, if you use --fc=x, the resulting EQ is approx:

-4.5dB @3k, -4.5dB @6k, -4.5dB @9k, -2.25dB @12k.

 

One might think: why do you want so much LPF?  It is kind of strange, but it REALLY IS part of what undoes what the mastering/creation of FeralA does.   DolbyA compression distorts the effects of the filters using during mastering so they don't sound as bad as one might think.   The filters used during feralA creationwere designed to hide the severe effects of DolbyA multiband compression.  *There is actually a FIXED 10dB boost at 3kHz also -- it is a real mess to accurately recover the signal.

 

*  there is an entire list of add-on EQs beyond =X and =x.   There is also =y, =Y, =z, =Z which tend to have deeper and deeper amounts of in-between filtering.   Also, there is an add-on of a number from 4,3,2,1,0 - added on like '--fb=X,4', where these numbered add-ons are 1st order filters (like a simple RC rolloff) while the 'letter' filters are fairly complex combinations of 2nd order filters.   On very few recordings,  both 2nd order and 1st order add-ons are needed.

 

If, during a decoding run, you want to see the entire chain of EQ filters in full detail, the '--fd' switch will dump the filter values before starting the decoding operation.

 

Dont confuse the '--fD' switch, which is one of the main filter choices with '--fd', which is the 'dump' option.

 

John