'FeralA' decoder -- free-to-use

[lucretius]

@John Dyson

 

I ran the following:

    da-avx --outgain=-4.5 --info=2 --fa --basic --input=infile.wav --output=outfile.wav

 

but I got the following error:

    Time Out:

    Required CPU type: WIN64: x86-64bit with AVX2: (Haswell/Zen/Excavator) or greater

    ***FATAL*** open error code: 2, for input file: "infile.wav"

 

According to Intel, I do have an AVX compatible processor:

 

Product Collection	Intel® Xeon® W Processor
Code Name	Products formerly Skylake
Processor Number	W-2133
Instruction Set Extensions	Intel® SSE4.2, Intel® AVX, Intel® AVX2, Intel® AVX-512
# of AVX-512 FMA Units	2

 

What am I doing wrong?

 

Similarly, I ran:

    da-win --outgain=-4.5 --info=2 --fa --basic --input=infile.wav --output=outfile.wav

 

but got the following error:

    Time Out:

    Required CPU type: WIN64: x86-64bit with SSE: Silvermont or greater

    ***FATAL*** open error code: 2, for input file: "infile.wav"

 

 

 

[lucretius]

2 minutes ago, John Dyson said:

It seems to be having problems finding the input file 'infile.wav".  

 

The start-up message always says the kind of CPU that it wants.  If the program gets far enough along to print that message out, then at least it will run on your CPU...

 

The "Time Out:" message is a leftover vestage of the license manager, which you do not need to use.   The license manager has a timeout time for certain kinds of licenses, but instead of doing the code properly and remove all license manager references, I turned off part of the license manager for when you use the --fa switch, which enables the feralA mode.  I'll remove the rest of the chaffe in the next release.

 

Basically, I disabled the license manager for the --fa switch as an after thought, but should simply clean it up.

 

So, the major crux of the problem that you are seeing -- you need a .wav file that you use for input.   Any normal CD wav file should work for testing to see if the program runs for you.  Of course, it will only give you good results if the CD wav file is FeralA (which should be most pop stuff before 1995), and not-normalized (which is a big part of them.)

 

I should probably supply a .wav file snippet so that at least a quick startup test can be done without selecting a .wav file.  I'll create one in a hour or so, and upload/add it to the repositority.   I'll call it 'testinfile.wav'.

 

John

 

 

Thank you. I will test with your .wav file.

[lucretius]

5 hours ago, John Dyson said:

I'll fix the documentation, but I just checked using this command (similar to the one in the example), but is tuned slightly for the material that I am attaching:

 

With some luck, that the built-in play feature works, this will play the file (a truncated copy of an ONJ CD RIP):

 

~/ap/nrs/dabuild/da-avx  --input=testinfile.wav --info=2 --fa --fgh2 --tone=-12.94 --basic  --play

 

Or, to get an output file (much better tested feature, of course):

 

~/ap/nrs/dabuild/da-avx  --input=testinfile.wav --output=testoutfile.wav --info=2 --fa --fgh2 --tone=-12.94 --basic

 

Note that the 'testoutfile.wav' file MUST not exist before running the program.  It will not overwrite unless you use the

--overwrite switch before the --output command.  This feature is important because recording engineers work with master tapes and don't want to destroy files by mistake.  (It is a just in case, and paranoid/careful behavior for the software.)

 

If you don't specify the --fgh2 switch, then the material will be too tinny, it just so happens that the ONJ material likes

the --fgh2 switch.  That switch can be further tweaked, but the results without the --fhh2=xxx tweak will still be okay.

 

For the higher quality decoding, you can try the --normal switch instead of --basic, or for the super quality (and slow) decodes, you can try --xtra, or even --xpp instead of --basic.

 

The --tone value isn't very critical, and will be between -12.80 and -12.95, but errors aren't killers unless doing a precision decode.

 

--info=2 gives the second by second status.  --info=1 gives dots.  --info=0 gives no running status.

 

John

 

testinfile.wav 9.25 MB · 1 download

 

Thanks. I tried the above command line on my own file and it works -- In fact, the old command line I first tried (da-avx --outgain=-4.5 --info=2 --fa --basic --input=testinfile.wav --output=testoutfile.wav) also works -- so I cannot explain why it did not work earlier.  Now, I will have to read up on what the various switches do.

 

I note that the file is upsampled to 32 bit 88.2K and is a lot bigger.  Is there a reason for this?  What happens if  I resample it to 16bit 44.1K?

 

Also, I cannot get the following command lines to work:

da-avx  --input=testinfile.wav --info=2 --fa --fgh2 --tone=-12.94 --basic  --play

da-avx --info=2 --fa --basic --input=testinfile.wav --play

 

Probably something to do with sound drivers?

[lucretius]

38 minutes ago, John Dyson said:

The reason for the upsample is partially internal and partially avoiding lossage.   First, the decoder math doesnt' work at 44.1k when you have audio up to 20kHz -- well, it works but there are some problems.  So, it internally upsamples to 66.15kHz to do the processing, and then upsamples again on output up to 88.2kHz.   There are two reasons for the file to be bigger:  higher sample rate and also floating point.   Floating point files are huge, but are what pros like to use.   If you want to shrink the file a little, but lose very little quality, you can use the' '--intout' switch.  The --intout switch changes the decoder output to 24bits signed instead of floating point.   It is perfectly okay also to resample back down to 44.1k or 48k if you want.  For the highest quality, you might want to keep the file at 24bits, perhaps use flac for data compression without quality loss.  In reality, as long as the levels are adequate (you might want to normalize the output), then converting from the floating point to 16bits is perfectly okay.   Many of my demo versions are 48k/16bits, but instead of normalizing, I am careful to make sure that the peak audio level on the output stays between -3dB and -0.20 or -0.30 dB or so by using the --outgain=xxdB switch.   You really don't want to try to use a 16bit audio file when the signal level is -20dB, so that is why I suggest normalizing before converting to 16bits.

 

The --play switch, it worries me a little because it is new.  There might be some caveats that I don't know about yet.  There is a file called 'aoplay.exe' that should be in the same directory as the program file 'da-win.exe' or 'da-avx.exe'.   It might work also if 'aoplay.exe' is in the current directory or in the 'HOME' directory.   The problem is that your local audio system might not be set-up the way that I expected -- I have little experience in dealing with the Windows audio system (well, I do, but it is from 15yrs ago!!!)  Basically, the da-avx looks in the same place where you ran it from to also find the 'aoplay.exe' program.  That program is what actually accesses the windows audio system -- it is a separate program because of licensing issues about the library that it uses.

 

Dont be resistant about asking more questions -- I really want to help!!!

 

John

 

 

 

 

Thanks for the info. Much appreciated.

 

I have 'aoplay.exe' in the same directory as 'da-avx.exe', I made that directory the current directory and ran 'da-avx.exe' and also 'aoplay.exe' by itself. 'aoplay.exe' just hangs with no output. 

[lucretius]

19 hours ago, John Dyson said:

A minor announcement -- I have observed that there just might be some precision problems in some of the math inside of the DHNRDS DA (and FA mode also.)   The problem is that some of the operations 1) must be damned precise, 2) are separated into lots of little smaller operations that require even greater precision.

For example, normal gain control is calculated by new_signal = old_signal * gain.   Nope -- not what the DHNRDS DA does in high quality modes.  Any error in the gain can manifest directly as distortion.   Even though the math is normally single precision floating point, and even though normal gain control doesn't need any more accuracy than that -- the kind of things happening in the DA decoder are really evil wrt math accuracy.  For example, the actual gain multiplications are NOT signal*gain, but are actually signal*gain^(1/16) power.  That is, the decoder uses gain to the 1/16th power.   Think about this -- instead of a gain of -1.0dB being 0.89125, the actual multiplication is 0.99283024.  Note the loss of two full digits of accuracy!!!   That is a huge accuracy loss.  A lot of gains arent -1.0dB but sometimes -0.1dB -- think about the possible noise in such strange math!??    So, that weird 1/16th power number eventually ends up being the 0.89125 in the physical world, but the loss of precision will cause little biases and excessive dithering in the resulting signal.

 

I scoured the critical parts where the errors can be magnified as I suggest above.  So -- I moved the most critical math to double precision FP.  You might think -- why didn't I use DP to begin with?   The problem is with the monsterous amount of math being used, especially for the anti-MD and anti-IMD distortion control.   Some of the math reflects directly onto the signal, but other bits of math are time domain filtered -- therefore some of the time-errors might not be as obnoxious (causing distortion and unexpected modulation effects) as they might otherwise be.

 

This latest experiment is that the attack/release filtering, the post filtering for the Hilbert gain calculations and also the final filter for the gain control are all in double precision now.   Believe it or not, there is a VERY VERY VERY noticeable improvement in clarity.  I regret the inability to share the full decoding examples with everyone, but that is why I am making the decoder available to everyone.  Such an improvement in clarity, which worries me - because THAT IS WHAT I EXPECT.  SO, I might be fooled by my own biases.

Hi John,

 

Will you be posting an updated version of your decoder which includes the above enhancements?  And will you be releasing the linux version in addition to the Windows version? Thank you.

 

Also, is it possible to get a list of switches and values used for those albums you have decoded?  -- especially, ABBA: Gold Greatest Hits (1992) and More ABBA Gold; More ABBA HIts (1993), and of course Supertramp: Crime of the Century.   Thanks again.

[lucretius]

1 minute ago, John Dyson said:

Yes -- to all of the above.   I am probably down for about 1-2days (need another rest), but I have been exhaustively testing the new changes also.  Unless it is life or death, please give me unti about the late Monday or late Tuesday timeframe.   I work in bursts/then back off....  It is backing off time for a few days.  (I still do testing when resting, just with less intensity.)

 

Thanks a million!  Please take good care of yourself. Take all the time you need and then some.

[lucretius]

2 hours ago, John Dyson said:

The SOX play command does integrate slightly more nicely though.

 

I already had SoX on my Windows 10 computer (I needed it to correct for the pre-emphasis on CDs). But up to now, I did not create play.exe (copy of sox.exe). So I fixed that. Now when I use command play infile.wav, I get the following error:

 

    play FAIL sox: Sorry, there is no default audio device configured

 

However, the default audio device is configured in Windows 10 but SoX is not finding it.  Does anyone know how to make the default sound device available at the command line (in cmd shell)? 

[lucretius]

4 hours ago, John Dyson said:

I just uploaded new version of FA decoder, also Linux version and SOX installation.

I'll send more complete information tomorrow morning, but letting you all know ASAP.

 

Quick instructions:

The windows version works just like previous one, except maybe the --play option will work.   It also supports using sox to directly play the output by using the --splay switch instead.  For --splay to work, must install the standard SOX as included in the repository.  You can instead install the exact same version from the SOX site -- I made ZERO changes, exact same files.  --splay takes an argument like --splay=1 or somesuch to select the output device.

 

The Linux version simply unpacks into a subidrectory using tar...   SImply:

>> tar xovf filename.tar.bz2

will produce the subdir containing the decoder...  It is EXACTLY the version, same binary that I use for testing.

 

The newcommands files has updated decoding parameters for numerous sources...  Basically, --fa=2 is a good first choice for the decoding mode.

 

Here is the same directory as before....   This is probably very confusing unless you have read the previous instructions, then it is probably just confusing...

 

The FA decoding is slightly improved, the command lines are a little more simple.  The realtime play has a better chance of working.

 

Good luck (you'll need it until I update this information more in the morning!!!) :-).

https://www.dropbox.com/sh/1srzzih0qoi1k4l/AAAMNIQ47AzBe1TubxJutJADa?dl=0

 

 

 

John

 

 

 

 

Thank you.  Both --play and --splay (with no arguments) are working fine on my machine (Windows 10). I'll get around to installing the linux version later.  I cannot wait to hear what those glitches are in the native Windows version of SOX!

 

All the best!

[lucretius]

12 hours ago, John Dyson said:

Oh -- the glitches -- sorry I didn't provide more complete info last night -- I was barely awake :-).  It will stil be a few more hours before I write more about using the decoder, but it looks like you are doing great so far..  I am SO glad that both version of play worked, my Windows expertise has rotted away over the last 20yrs :-).

Okay, about sox...  The 'trick' is that SOX doesn't know which driver to use.  At first, as you know, it seems like Windows SOX almost doesn't even support a 'play' feature.  Note that I am not directly using a play command when I execute the sox command from the decoder, but I use the exact raw sox command as follows (note the paranoia in setting the I/O mode, might not be needed);

 

* actually, instead of 'sox', I am using the full path, so the sox must be installed in the normal place for the version as follows from the source code:

 

static const char *soxlocation = "c:\\Program Files (x86)\\sox-14-4-2\\sox.exe";

or for humans:

c:\Program Files (x86)\sox-14-4-2\sox.exe

 

pipe input | sox -q --type=wav - --bits=16 -twaveaudio rate 44100

or

pipe input | sox -q --type=wav - --bits=16 -twaveaudio <id> rate 44100

(<id> is an index like 0,1,2,3,4 which points to the desired output device)

 

A more normal use of sox, just to play a file, one would do:

 

sox filename.wav -twaveaudio

 

the above should work, and shouldn't really need to set the bits modes or the data rate.  I set the data rate above because the decoder writes at 88.2k/floating point, and I wasn't sure if all versions of windows drivers supported that mode.

 

Another mod that might be helpful, instead of '-twaveaudio', you might need '-t waveaudio' (note the space.)

 

Another oddity -- did you notice that I specified the INPUT data type from the pipeline?  Normally, on Unix, one doesn't need to specify the input stream type as autodection works okay on input streams.  However, on Windows, the automatic type detection seems to fail.  SO, I forced the input type to be '--type=wav'.   The type detection seems to work okay when using a normal file instead of '-' or pipe for the input file.

 

About how I run 'sox' indirectly from the DHNRDS DA...   Basically, I stay in the POSIX land instead of using direct Windows calls.  I cannot use the 'system(3)' call because 'system(3)' calls the shell, but most people aren't going to have a POSIX/cygwin 'bash' program on their systems, so I get deeper into the POSIX nitty/gritty and do the old fork(2)/execl(2) thing and manually set up the call to 'sox'.  The aoplay.exe program is called in a similar way.

 

The data stream when using SOX is a standard .wav file, but on aoplay, since I didn't want to encumber my .wav file processing subrotines (however crappy my subroutines are),  I simply created a stream that is led with a 'magic number', then a data rate and then an extra data item for the future.  The rest of the stream are the audio bytes encoded as 24 bit signed data...  Basically, the aoplay.exe uses 24bit signed .wav data, but without a normal .wav header.

 

More info coming as I 'get my engine started' later this morning or afternoon.

 

John

 

 

 

Thanks.  As long as I use '-twaveaudio', SoX play will work (whether invoked as 'sox' or 'play'). There seems to be no advantage in invoking sox as play, since all the same syntax is required (and specifying '-t' and 'waveaudio' is necessary).

[lucretius]

@John Dyson

 

Now I am trying the linux version.  I have Ubuntu 18.04 LTS installed in a VM. I already have SoX installed. Forgive my stupid question but in which linux directory should I place 'da-avx'?

 

Thank you!

[lucretius]

1 hour ago, lucretius said:

@John Dyson

 

Now I am trying the linux version.  I have Ubuntu 18.04 LTS installed in a VM. I already have SoX installed. Forgive my stupid question but in which linux directory should I place 'da-avx'?

 

Thank you!

 

Actually, I'm having a problem making the 'da-avx' executable in Ubuntu.  (I don't use Ubuntu much, LOL.)

[lucretius]

43 minutes ago, John Dyson said:

Ohhhh....  you might have to 'chmod +x da-avx'...

 

Also, you REALLY need the Intel Haswell or newer CPU (i4000 or newer), Zen or Excavator.  AVX2 is importamt, even though I called the binary da-avx.  I can make an sse3 version if needed, I don't have an sse3 machine to test on for Linux, but I can use the same settings as I use for Windows SSE3 (stuff like Atoms.)   BTW, I also don't have an AVX2 machine for Windows, but Richard does... SO, I have done so much testing through Richard that I know what I can get by with.

However, I can make an SSE3 version based upon the Windows version -- no problem, if it will help.  Just tell me the kind of CPU that you are using...  Happily, I am using a REALLY REALLY good vectorizing compiler (Clang++) so the code generated is the very best available.  g++ is much slower for some reason, even though it is a little easier to use.

 

(Sorry for the lack of CPU dispatch, but the code is so very tight because the algorithms are incredibly itntense that CPU dispatch would really slow down the code.)

 

John

 

 

I tried 'chmod +x da-avx'. However, when I try to run 'da-avx', I am still getting "command not found". [Also, my CPU has AVX2 although I am using a VMware Workstation VM.]  Maybe it's something to do with Ubuntu?

 

OTH, Sox 'play' works fine in Ubuntu.  I can just run 'play filename.wav' -- without 'waveaudio' option, the latter doesn't work -- waveaudio option must be unique to Windows?

[lucretius]

49 minutes ago, John Dyson said:

About running da-avx.   Ahhh...  the answer:  try ./da-avx...  Might have to specify the full path when in the current directory.  In normal 'Unix' installations, it has become recent (since early 1990's) tradition to keep the current directory out of the path.  That is for the early attempts at improving security.   I usually reconfigure my accounts where I do development to have the OLD style of path.   I do NOT use that for root or accounts that have priv access though.

 

About the 'waveaudio' strangeness...   Yes, that is just a windows thing.  On Linux, it appears that sox properly chases down the list of available drivers and choses the correct one.  On the normal 'Windows' build of SOX, it seems to be unable to chase down the correct driver.  The problem seems simialr WRT being able to sense the file type of a pipe input...   The cygwin version of SOX works like the Unix/Linux version, but then the cygwin version needs so many .dlls as to make it a licensing nightmare.  (I was considering making the cygwin version available, but then the user would have to install that huge morass -- don't want to do that.)

 

Let me know if ./da-avx works....  Otherwise, I'll build the SSE3 verison -- if that is the problem.

 

John

 

 

./da-avx did not work, however ...

 

Ubuntu is funny -- doesn't like giving access to root, etc (I may have to try a different distro, i.e. one that doesn't make root access really difficult)    Nonetheless, I moved 'da-avx' to /usr/bin and ran 'da-avx with the following command line:

    da-avx --input=testinfile.wav --info=2 --fa --fgh2 --tone=-12.94 --basic --play

 

The result:

 

    Audio "DHNRDS DA decoder" V1.2.2A 26Mar2020 -- Author John S. Dyson -- Copyright 2017,2018,2019,2020
    License terms are in accompanying documentation

    Time Out:
    Required CPU type: LINUX: X86-64bit with AVX2: i4xxx or greater
    Affinity entirely disabled
    Input file: "testinfile.wav"
    Output file: "<stdout>"
    totaldelay: 814
    execl: No such file or directory

 

 

I am not sure if I am getting somewhere?

 

 

[lucretius]

On 3/28/2020 at 5:25 AM, John Dyson said:

Oh, you are getting exactly what I expect...  I didn't send a copy of aoplay, which is needed for the --play switch.  Instead of a play switch,

just pipe the output to the SOX play command like this:

 

da-avx --input=testinfile.wav --info=2 --fa --fgh2 --tone=-12.94 --basic |  play -q -

 

The above is EXACTLY how I use the decoder (except I usually start with a flac file like below.)

The only time that I use 'aoplay' on Linux is for testing the code for Windows.

 

It is much nicer to be in the habit of the play/sox command, for example if you want to filter the highs a bit, just to experoment,

then you can do this:

 

da-avx stuff | play -q - treble -3.0 9k 0.707q

 

Also, if you want a flac output file, then you can do this:

 

da-avx stuff | sox - outfile.flac

 

(You know 'da-avx stuff' is whatever full da-avx command that you want to use.)

 

If you want .flac input and output, then you can do this:

 

sox infile.flac --type=wav - | da-avx stuff | sox - outfile.flac

 

Cool, huh?  SOX and da-avx mix really nicely...

 

John

 

 

 

Sorry I didn't get back to you sooner.

 

Although I can run 'play' (SoX) stand-alone in Ubuntu just fine, when I pipe the output to 'play' (SoX) via 'da-avx', it plays way too fast (like when the sound driver is broken).  Any suggestions?  OTH, this may be a problem on my end because I am running Ubuntu in a VM. [I'm going to install Fedora in a VM soon -- maybe that will work better?]

 

In any case, I can run da-avx on Windows, just fine.

 

 

 

 

 

[lucretius]

3 hours ago, John Dyson said:

Hmmm about the play...   Interesting because the da-avx provides a .wav file through the pipe, and it normally works (for me)...   So, I will definitely research it and try to figure out what I did wrong.  Most likely, I did make a mistake or assume something incorrect on Linux...

I am guessing that the version of sox on Linux that you are using might be happier with a full specification of the file type/data rate.  It is really easy for me to add the full specification to the 'secret' command line that I send sox :-).

 

I will definitely look into it, and check out what I might be doing wrong on Linux.   Thanks for the data point on Linux play though -- and I thought that was the 'easy' one :-).

 

However, thanks for the good news on Windows!!!

 

John

 

 

Hi @John Dyson,

 

I setup Fedora in a VMware VM and now the following command works fine:

     da-avx --input=testinfile.wav --info=2 --fa --fgh2 --tone=-12.94 --basic |  play -q -

 

Funny that it works in Fedora but not Ubuntu.

[lucretius]

1 hour ago, John Dyson said:

I wonder if the versions of SOX match?  I use SOX V14.4.2.

 

I'm using SoX v14.4.2 in both Fedora and Ubuntu.

[lucretius]

53 minutes ago, John Dyson said:

You know that the various versioins of Linux do patch their local versions....  I haven't done this, but I'll download the source of the specific Fedora version and then see if they patched SOX.   If they did, they might have fixed the detection problem.   Right now, I only have the source of the generic version of SOX as downloaded directly from the SOX site, so the diffs should explain what is going on.

 

BTW, the LINDA R stuff has been updated with a bit more controlled results.  That 'lisp' at the beginning of 'Its so easy' really gets on my nerves, and it is obviously an electronic artifact...  I tried to remedy it, but I think that the cure was worse than the disease.   Also, a new decode shifted the gain curve just a little bit to hide the lisp a little.

 

Everything that I demo in .mp3 snippets or full versions are the results of TESTING, and aren't really the main goal -- the decoder is the goal right now.  So, some day, other people who have better artistic ear can adjust the settings better than I can do!!!

 

John

 

 

Loaded up Linux Mint into a VMware VM with Sox v14.4.2 and tried to do this cmd:

    da-avx --input=testinfile.wav --info=2 --fa --fgh2 --tone=-12.94 --basic |  play -q -

 

It failed but here's the output:

 

play WARN alsa: can't encode 0-bit Unknown or not applicable
Audio "DHNRDS DA decoder" V1.2.2A 26Mar2020 -- Author John S. Dyson -- Copyright 2017,2018,2019,2020
License terms are in accompanying documentation

Time Out:
Required CPU type: LINUX: X86-64bit with AVX2: i4xxx or greater
Affinity entirely disabled
Input file: "testinfile.wav"
Output file: "<stdout>"
play WARN wav: wave header missing extended part of fmt chunk
totaldelay: 814
play WARN alsa: under-run
play WARN alsa: under-run
play WARN alsa: under-run
play WARN alsa: under-run
play WARN alsa: under-run
play WARN alsa: under-run
play WARN alsa: under-run
play WARN alsa: under-run
play WARN alsa: under-run
play WARN alsa: under-run
play WARN alsa: under-run
S:          1, IN(RMS:-18.67dB,PK*:-13.93dB), INL(PK*:-13.93dB), INR(PK*:-14.43dB), OUT(RMS:-19.11dB,PK:-22.47dB):
    DA LL(-90.01/ -5.89/ -1.03), M(-90.01/ -2.85/ +0.00), H(-90.01/ -9.86/ -5.68), h(-95.01/-14.61/ -9.45)
    DA RL(-90.01/ -6.02/ -0.97), M(-90.01/ -3.05/ +0.00), H(-90.01/ -9.95/ -8.12), h(-95.01/-14.76/-11.00)
play WARN alsa: under-run

 

etc.

[lucretius]

35 minutes ago, John Dyson said:

The missing 'fmt' chunk happened because it wasn't in the broadcast spec as a requirement (AFAIR.)   I can add it in but is not a fatal or necessary part of the .wav file.

 

The under-run is basically about speed issues...   FIrst, the decoder needs at least two full separate cores to run in real time.  The decoder is NOT fast, but should be fast enough to run realtime with two cores scheduled correctly.  Try the --affinity switch to see if it speeds up.   By default, the decoder doesn't work as well real-time when hyperthreading because the threads grab up the entire CPU anyway.  SO, if you have 4 cores and 8 hyperthreads, it is very possible that the decoder is scheduled by the kernel, so that there is a clash so that they are all bunched up onto the same cores/threads instead of spread throuought the 4 cores.  (The decoder has to run on the cores, not the extra hyperthreads that it might also be running on.)

 

* The decoder is probably the most CPU intensive real-time SIMD DSP program that most people ever run (maybe except games which can sort of adapt to slower CPUS.)   The decoder DEMANDS multi-core CPU power, or it simply cannot run realtime.

 

So, if you have core0/thread0 core0/thread1 core1/thread0 core1/thread1 running, that is the worst of all worlds.

What you need is core0/thread0 core1/thread0 core2/thread0 core3/thread0, where the threads are scheduled amongst all of the threads, then the world is happier.  (On each core, it doesn't make any difference which thread is running, but that it runs on only one thread on each core.)

 

So, the --affinity switch tries to modify the scheduling so that it might run faster by pointing the CPU to multiple cores instead of allowing the bunch-up.

 

Also, try piping to the 'aplay' command, with no switches -- like:

da-avx blah blah blah | aplay

It might work better, I just thought of it, and just verified that it also worked for me (no error message about missing semi-optional fmt..)

 

John

 

 

 

I didn't really notice a difference with the --affinity switch.  OTH, I increased the resources available to the Linux Mint VM and that seem to solve the problem.  I tried to do the same for the Ubuntu VM, but I still got the under-run's. (Next, I'll try 'aplay')

 

[lucretius]

Hi @John Dyson,

 

Getting back to Windows 10, I tried to run the following:

    sox infile.flac --type=wav - | da-avx --fa=2 --tone=-12.90 | sox - outfile.flac

 

It fails and gives the following output:

 

    Audio "DHNRDS DA decoder" V1.2.2A 26Mar2020 -- Author John S. Dyson -- Copyright 2017,2018,2019,2020
    License terms are in accompanying documentation

    Time Out:
    sox FAIL formats: can't open input  `-': WAVE chunk fmt not found
    sox FAIL sox: `-' error writing output file: Broken pipe

 

It's probably just a syntax thing.

 

Similarly, I tried running the following:

    sox infile.flac --type=wav - | da-avx --outgain=-4.5 --info=2 --fa --basic | sox - outfile.flac

It fails and gives the following output:

    Audio "DHNRDS DA decoder" V1.2.2A 26Mar2020 -- Author John S. Dyson -- Copyright 2017,2018,2019,2020
    License terms are in accompanying documentation

    Time Out:
    Required CPU type: WIN64: x86-64bit with AVX2: (Haswell/Zen/Excavator) or greater
    Input file: "<stdin>"
    Output file: "<stdout>"
    totaldelay: 814

     ...

    Finished

    sox FAIL formats: can't open input  `-': WAVE chunk fmt not found

 

 

[lucretius]

4 hours ago, John Dyson said:

I am wondering if the VM, if you are allocating 2 'CPUS' to it, that it is allocating two hyper threads on the same core?   That would be unproductive and would work just like it only had one core.

 

If there are actually 2 cores available, there should be zero problem running realtime on an i4xxx or better.  So, I wonder about the VM, if it is allocating the CPU for best performance.

 

I hope you are well!

 

My workstation has a single Intel Xeon W-2133 CPU. That has 6 (real) cores, 12 threads.  My host OS is Windows 10 Pro for Workstations (latest build).  I use VMware Workstation 15 Pro to run the VM's.  It allocates (virtual) "processors" based on the # of logical cores (in total) specified.  Since my CPU has 12 threads, it therefore has 12 logical cores max that can be allocated to any specific VM.  I currently have 6 of the 12 logical cores (i.e. threads) allocated for each linux VM (Ubuntu, Linux Mint, and Fedora). As well, I have allocated 16GB out of 32GB memory for each VM.

 

So far, Linux Mint and Fedora are running fine -- I have only tested the --basic switch with 'da-avx'.  OTH, Ubuntu ran into under-run's when trying to do real-time play piped from 'da-avx to SoX -- I didn't try to create a file yet.

 

Obviously, I still have much more testing to do, e.g. I need to create files using the --normal, --xtra, and --xpp switches.

[lucretius]

8 hours ago, lucretius said:

I currently have 6 of the 12 logical cores (i.e. threads) allocated for each linux VM (Ubuntu, Linux Mint, and Fedora). As well, I have allocated 16GB out of 32GB memory for each VM.

 

I'll investigate further to determine if I can associate VMs with specific logical cores (CPU affinity).

[lucretius]

3 hours ago, John Dyson said:

For an idea of what to expect -- for the 'normal' mode deocding, that would be --basic and the --normal modes, the decoder should work better with 2 cores than 1.  However, more cores won't help much.   In the higher decode modes --xtra, --xp, --xpp, the decoder pulls out all stops, and starts running lots of threads -- 8 more.  Those additional threads are 'grinders', and your  6 full cores should easily be able to run significantly faster than realtime even in --xpp mode.  The problem is the darned hyperthreading -- it only helps on memory wait states, which don't happen much on the decoder...  The decoder is careful to do linear accesses and to try to stay within the higher level cache, so the memory wait states don't happen very often.   SO, since the decoder uses up ALL of the computational resources in the SIMD portion of each core -- the scheduling 'mistakes' made because of hyperthreading slow things down.   Thank goodness for affinity.

 

However, a 6core or more machine should run really nicely in the higher quality modes, but will DEFINITELY be slower than the --basic mode.   How much slower is totally dependent on the scheduling of the hyperthreads being suppressed and how many cores it has to work with.

 

An item of note -- the higher quality modes won't be a big surprise, but is very subtle.  Normal mode now does about 50% of what --xpp mode can do, iti IS an improvement over --basic mode, which will tend to be edgy, but also almost as fuzzy as a DolbyA.   --xpp mode comes very close to removing as much modulation distortion as possible -- there is a mode that does even more, but not really worth it '--xppp=12 --xf=7', but you'll be waiting an hour for each album.

 

John

 

 

Thanks for this.

 

For 'da-dvx' running on a host OS (Windows 10 in my case), is there any chance of future versions of da-dvx taking advantage of GPU compute power (CUDA or OpenCL)?

[lucretius]

5 hours ago, John Dyson said:

New version of FA decoder is available.  I haven't prepared a Linux version yet, but will be ready tomorrow morning.  This new version has MAJOR MAJOR MAJOR quality and usability improvements.  It is almost EASY to use now.   The 'punishment' of  lower quality sound is actually less when the settings are incorrect now.

 

I think that a few things have been fixed -- I hope.   I tried to modify the internal SOX command line to make it work better -- but no guarantees.

 

Some of you know how difficult the first cut in 'Even in the Quietest Moments' has been for me to try to decode -- this version of the decoder can do it EASILY and fairly darned well.   The new version is V1.3.1B, and is in the same place as before -- the pointer is below.  Also, most of the old user docs are fairly accurate, but there is a new one:  using-V1.3X.txt.  I am including a portion of that file at the end of this post -- showing the relative consistency of the commands for each album to be decoded.

 

This is working REALLY well, if it actually works for you.  🙂 The results are actually VERY VERY pretty, and good results on tough material.

The attached example was decoded while using 3 versions of the same recording.  On the 'Give a Little Bit' example, the tonality is very similar to the other decodes (just a little fuller sounding), but the 'excessive' vocal enhancement is very precise -- showing the botched chorus effect very cleanly.   Mp3 does not show the 'botched' chorus effect nearly as cleanly as a raw decode -- there is actually a casually noticeable difference -- not worth splitting hairs on that right now. 

 

Along with the difficult snippet of 'Give a little bit', I am including a much higher quality decode of a selection from Herb Alpert.  I didn't want to only show a difficult decode -- maybe showing an underestimation of the potential quality of the decoder!!!  The 'Spanish Flea' example is actually typical, and not cherry picked.

 

Here is the repository:

https://www.dropbox.com/sh/1srzzih0qoi1k4l/AAAMNIQ47AzBe1TubxJutJADa?dl=0

 

Selections from the usage doc:

 

In many cases, on a non-normalized, raw, CD in FeralA form, a simple use directly on wav
file input and output like might be sufficient:

da-avx --input=inputfile.wav --output=outputfile.wav --fa

or

da-avx --input=inputfile.wav --output=outputfile.wav --fa=N
where N might be 0, 1, 2, 3 or 4.  Also, a 'classical' mode might be more approriate.

 

---

The --tone=-xx.xx values might seem to require 'precision', but really do not.  If the number

is within 0.02, there is almost never ANY difference in sound.  Also, the range of settnigs on non-normalized

CDs is now usually within about -13.25 to -13.40...   Even the max or min in this range still results in

typically 'good' results.

 

Carpenters:
1969    Ticket To Ride
        Basic switch: --fa

1970    Close To You
        Basic switch: --fa
        Note: the --mfm switch might be beneficial

1971    Carpenters
        Basic switch: --fa
        Note: the --mfm switch might be beneficial

1972    A Song For You
        Basic switches: --fa=4 --tone=-13.35

1973    Now and Then
        Basic switches: --fa=4 --tone=-13.30

1975    Horizon
        Basic switches: --fa=4 --tone=-13.30

1976    A Kind of Hush
        Basic switches: --fa=4 --tone=-13.30

1977    Passage
        Basic switches: --fa=3 --tone=-13.35 --mfm

1981    Made In America
        Basic switches: --fa=3 --tone=-13.35 --mfm

Linda Ronstadt:
Greatest Hits
        Basic switches: --fa=spc2 --tone=-13.35

1977    Simple Dreams
        Basic switches: --fa=spc2 --tone=-13.35

1975    Prisoner In Disguise
        Basic switches: --fa=spc2 --tone=-13.35

1989    Cry Like A Rainstorm
        Basic switches: --fa=spc2 --tone=-13.35

Nat King Cole
Nat King Cole Story/Analog Productions
        Basic switches: --fa --tone=-13.35

Carly Simon
The Very Best
        Basic switches: --fa

All 8 ABBA Albums
        Basic switches: --fa --tone=-13.35

Styx
The Grand Illusion
        Basic switches: --fa --tone=-13.35

Supertramp
Crime of the Century
        Basic switches: --fa --tone=-13.35
                * for less mellow, use --fa=spc1I
Breakfast In America
        Basic switches: --fa --tone=-13.35
Even In the Quetest Moments
        Basic switches: --fa --tone=-13.35On the 'Give a Little Bit' example, the tonality is very similar to the other decodes (just a little fuller sounding), but the 'excessive' vocal enhancement is very precise -- showing the botched chorus effect very cleanly.

The Cars
Complete Greatest Hits
        Basic switches: --fa --tone=-13.35 --mfm
                * --mfm seems important

Henry Mancini
Greatest Hits
        Basic switches: --fa --tone=-13.35

 

Bread
Bread (1973), Audio Fidelity  AFZ5 197
        Basic switches: --fa=spc1 --tone=-13.35 --mfm

Sergio Mendes, Brasil'66
The Very Best
        Basic switches: --fa --tone=-13.37 --mfm

 

Herb Alpert & Tijuana Brass
        Basic switches: --fa=spc2 --tone=-13.37 --mfm

 

 

 

 

GiveALittleBit.mp3 2.1 MB · 0 downloads 04.Spanish Flea-snippet.mp3 2.1 MB · 1 download

 

Thanks for this and thanks for simplifying.

[lucretius]

@John Dyson

 

Tried v.1.3.1B (windows).  Everything seems OK so far but I am still having trouble with the SoX pipes --

 

I run this:

sox infile.flac --type=wav - | da-avx --fa=2 --tone=-12.90 | sox - outfile.flac

 

and I get this error:

sox FAIL formats: can't open input  `-': WAVE chunk fmt not found

 

 

The same command works fine in Linux Mint so maybe this is a Windows issue?

 

Cheers!

[lucretius]

6 hours ago, John Dyson said:

Also, the range of settnigs on non-normalized

CDs is now usually within about -13.25 to -13.40

 

For previous versions, you said the --tone value will be between -12.80 and -12.95.  Why did this change?