sacd_extract DSF output problem

[mansr]

I use ISO2DSD to create a DFF Edit Master and cue file. I then convert the DFF file to one large flac file and split the file using XLD with the cue file. With this method, the only place that a click can occur is at the beginning of the first song of the album or the end of the last song of the album. But I have never heard (or perhaps noticed) a click at the start or end of an album. I think that is due to converting first to DFF, which apparently avoids the error.

 

I agree the best approach is to convert the entire album as a single unit, then split the PCM file into tracks. If there's a click at the start or end, it can be easily removed without affecting gapless playback.

[bogi]

Some CUE splitters can split DSF+CUE (see my signature) ;-)

One my customer gave me this idea that was released some time ago New Demo Policy, DFF output (Windows only), DSF CUE splitter

 

That's nice! But I had rather some freeware in mind. I counted more than $200 for that AuI functionality:

 

 

If SoX is able to trim DSD files (the post above from anoutsos), then such CUE splitter could be relatively easily implemented. Signalist DFF to DSF converter is able to add metadata to DSF, my ISO2DSF is using that functionality.

[mansr]

If SoX is able to trim DSD files (the post above from anoutsos), then such CUE splitter could be relatively easily implemented. Signalist DFF to DSF converter is able to add metadata to DSF, my ISO2DSF is using that functionality.

 

Sox (my version) can trim DSD files. It supports DSF and DFF input, DSF output. Metadata handling is incomplete, though.

[audiventory]

That's nice! But I had rather some freeware in mind. I counted more than $200 for that AuI functionality:

 

[ATTACH=CONFIG]29959[/ATTACH]

 

For solving splitting task of long DSF+CUE file pre-extracted form ISO need remove points: 3, 5 and 6 for saving money.

 

There is D64 sample rate. Why need excess modules?

 

I have no investors, who sponsor development and fast tech support.

 

What time delay provided for freeware tech support?

 

 

If SoX is able to trim DSD files (the post above from anoutsos), then such CUE splitter could be relatively easily implemented. Signalist DFF to DSF converter is able to add metadata to DSF, my ISO2DSF is using that functionality.

 

I suppose AuI have many functionality for ISO extraction / DSF conversion: True Gapless Mode, smoothDC, GainControl, AutoNormalizing, etc. ;-)

[bogi]

For solving splitting task DSF files extracted form ISO need remove points: 3, 5 and 6 for saving money.

 

There is D64 sample rate. Why need excess modules?

 

I have no investors, who sponsor development and fast tech support.

 

What time delay provided for freeware tech support?

 

Hi Yuri, I respect your work, your customer base and I understand your arguments. My contribution wasn't mean as criticism.

 

You are providing AuI as suite of tools with support. On other side, plenty of small free tools exist for different particular tasks and they function well also without higher level of support. That are different worlds. DSD world isn't yet much covered, but PCM world is.

 

It would be interesting for me to have DSD CUE splitter for some experiments , but I don't need it for regular tasks. Therefore I am not going to pay about $100 for it.

[audiventory]

It would be interesting for me to have DSD CUE splitter for some experiments , but I don't need it for regular tasks. Therefore I am not going to pay about $100 for it.

 

Bogi,

I only noted that DSF+CUE splitter exists now :-)

 

When I'm asked about long DSF splitter, I was very wondered that it is not released before (as said my customer).

[bogi]

Sox (my version) can trim DSD files. It supports DSF and DFF input, DSF output. Metadata handling is incomplete, though.

 

The following could be relatively easily implemented, if SoX would be able to output DFF as result of trimming:

1. To use SoX to split DFF edit master via CUE sheet to DFF tracks

2. To use Signalyst dff2dsf to convert DFF tracks to DSF with adding metadata from CUE sheet

 

Point 2. is already implemented (in Perl) within my ISO2DSF. If someone would be interested to implement the CUE splitter, I can provide him my code. It uses undocumented dff2dsf feature, so it would be correct yet to ask Miska (I don't expect troubles here).

 

But you wrote that only DSF output is possible (if I understand you well).

[mansr]

The following could be relatively easily implemented, if SoX would be able to output DFF as result of trimming:

1. To use SoX to split DFF edit master via CUE sheet to DFF tracks

2. To use Signalyst dff2dsf to convert DFF tracks to DSF with adding metadata from CUE sheet

 

Point 2. is already implemented (in Perl) within my ISO2DSF. If someone would be interested to implement the CUE splitter, I can provide him my code. It uses undocumented dff2dsf feature, so it would be correct yet to ask Miska (I don't expect troubles here).

 

But you wrote that only DSF output is possible (if I understand you well).

I haven't bothered implementing DFF output since there didn't seem to be much demand. Adding it wouldn't be particularly difficult.

[anoutsos]

I just want to clarify that I am not interested in the DSD streams at all. I just want to have the album tracks of the SACD as stereo PCM 88.2kHz/24bit files. The metadata that so many people are worried about can be inserted at a later stage; it's not an issue. I would also like to convert the DSD to PCM using Saracon, because it removes all the useless ultrasonic information above 88KHz.

So, I really like mansr's idea of extracting single DFFs from the ISOs, converting those to PCM and then splitting them with a cue file. I have not tried that yet, but it sounds very promising.

[lljames]

Several years ago, due to this problem, I have to output whole IOS to a single DSF file into Sony Z1ES, so it's trackless;

And for PC playing, I am using jriver importing with ISOs directly. It's been a while so far...

 

Recently, I need revisit this again am thinking split the tracks to individual DFS files, so I can use something like micoRendu or maybe Aries, so I can finally remove PC/Mac from audio chain.

 

I wonder what's the status of said_extract now? The pop/click problem has been resolved? And how is seamless playing with DFSs? Or maybe some other tools I can use? The most important thing is of course I need DSD stream to be intact, no conversion involved.

 

Thanks a lot.

[audiventory]

Several years ago, due to this problem, I have to output whole IOS to a single DSF file into Sony Z1ES, so it's trackless;

And for PC playing, I am using jriver importing with ISOs directly. It's been a while so far...

 

Recently, I need revisit this again am thinking split the tracks to individual DFS files, so I can use something like micoRendu or maybe Aries, so I can finally remove PC/Mac from audio chain.

 

I wonder what's the status of said_extract now? The pop/click problem has been resolved? And how is seamless playing with DFSs? Or maybe some other tools I can use? The most important thing is of course I need DSD stream to be intact, no conversion involved.

 

Pop/click issue have many different reasons. There is no universal advice for all cases.

If clicks exists need learn the issue reason. Reason may be out of a conversion software abilities.

Sometimes need convert extracted DSFs to WAVs for analysis.

Here my review of click issue in 4 parts: Extract ISO to DSF PCM. Clicks Issues. Part 1

[anoutsos]

Several years ago, due to this problem, I have to output whole IOS to a single DSF file into Sony Z1ES, so it's trackless;

And for PC playing, I am using jriver importing with ISOs directly. It's been a while so far...

 

Recently, I need revisit this again am thinking split the tracks to individual DFS files, so I can use something like micoRendu or maybe Aries, so I can finally remove PC/Mac from audio chain.

 

I wonder what's the status of said_extract now? The pop/click problem has been resolved? And how is seamless playing with DFSs? Or maybe some other tools I can use? The most important thing is of course I need DSD stream to be intact, no conversion involved.

 

Thanks a lot.

 

My latest attempt was a few months ago. I extracted the SACD as a single DFF (Master), together with the CUE file, using sacd_extract. Then, I converted the DFF file to AIFF 88.2/24 using Saracon, and finally I used the CUE file to split the AIFF file into tracks, using XLD. This process creates only two clicks, one at the beginning of the first track and one at the end of the last. It is very easy to remove these clicks manually, using software like Sound Studio, etc, but I did not bother since those clicks are only particularly annoying when they occur between tracks.

 

Cheers,

Aris.

[mindset]

Not sure if anybody is still interested, but it thought I would share this:

 

As some people have probably noticed, the sacd-ripper 0.3.8 release generates discontinuous .dsf because it does zero padding at the tail of each track to meet the 4096-byte/channel block size requirement of DSF while DST frame size is 4704 bytes/channel.  This discontinuity in DSF results in a pop noise at a track transition on players with gap-less playback support.

 

The following patch (against 0.3.8 release) will fix the problem (at least for gap-less playback).  Basically what it does is that it sets the "orphaned" samples at the end of a track aside and attach it to the head of the next track so no zero padding is needed.

 

The functionality has been verified by listening as well as full binary comparison of these:

1.Concatenation of all tracks (header & footer removed from .dsf's)
2. Single rip of the entire album into one .dsf (so there is no concept of tracks)

Binary comparison of raw audio data in 1. and 2. from 10 randomly selected SACDs showed absolutely zero discrepancy, which means all the tracks are truly seamless.

 

Note that for this to work, all tracks must be converted to .dsf in one shot.  In other words, the -t option for selective track extraction cannot be used.

 

No, I will not share the binary (sorry!), but compilation is very easy (at least for Linux) and steps are very well documented in the readme of the release.

diff -upr w/sacd-ripper-0.3.8/libs/libsacd/dsf.c z/sacd-ripper-0.3.8/libs/libsacd/dsf.c
--- w/sacd-ripper-0.3.8/libs/libsacd/dsf.c	2015-07-31 01:15:45.000000000 -0700
+++ z/sacd-ripper-0.3.8/libs/libsacd/dsf.c	2018-05-11 02:09:05.772839398 -0700
@@ -161,9 +161,24 @@ static int dsf_create_header(scarletbook
     return 0;
 }
 
+uint8_t buffer_prev[MAX_CHANNEL_COUNT][SACD_BLOCK_SIZE_PER_CHANNEL];
+uint8_t *buffer_ptr_prev[MAX_CHANNEL_COUNT];
+int first = 1;
 
 static int dsf_create(scarletbook_output_format_t *ft)
 {
+    int i;
+    dsf_handle_t *handle = (dsf_handle_t *) ft->priv;
+
+    // If this is not the first track, carry over the orphaned samples at the tail of the previous track.
+    if(!first && ft->dsf_gapless){
+        for(i = 0; i < MAX_CHANNEL_COUNT; i++){
+            memcpy(handle->buffer[i], buffer_prev[i], SACD_BLOCK_SIZE_PER_CHANNEL * sizeof(uint8_t));
+            handle->buffer_ptr[i] = handle->buffer[i] + (buffer_ptr_prev[i] - buffer_prev[i]);
+        }
+    }
+    first = 0;
+
     return dsf_create_header(ft);
 }
 
@@ -172,18 +187,30 @@ static int dsf_close(scarletbook_output_
     dsf_handle_t *handle = (dsf_handle_t *) ft->priv;
     int i;
 
-    // write out what was left in the ring buffers
-    for (i = 0; i < handle->channel_count; i++)
-    {
-	if (handle->buffer_ptr[i] > handle->buffer[i]) {
-	    handle->sample_count += handle->buffer_ptr[i] - handle->buffer[i];
-
-	    fwrite(handle->buffer[i], 1, SACD_BLOCK_SIZE_PER_CHANNEL, ft->fd);
-	    memset(handle->buffer[i], 0, SACD_BLOCK_SIZE_PER_CHANNEL);
-
-	    handle->buffer_ptr[i] = handle->buffer[i];
-	    handle->audio_data_size += SACD_BLOCK_SIZE_PER_CHANNEL;
-	}
+    // Save the remaining (orphaned) samples in the buffer to be attached to the beginning of the next track.
+    // This is needed for pure gapless DSF generation.  The need for this treatment stems from the 4096-byte block
+    // size per channel for DSF which is not a divisor of the DST frame size per channel of 4704 bytes.
+    if(ft->dsf_gapless){
+        for(i = 0; i < MAX_CHANNEL_COUNT; i ++){
+            memcpy(buffer_prev[i], handle->buffer[i], SACD_BLOCK_SIZE_PER_CHANNEL);
+            buffer_ptr_prev[i] = buffer_prev[i] + (handle->buffer_ptr[i] - handle->buffer[i]);
+        }
+    }
+    else{
+        // Write out what was left in the ring buffers if dsf_gapless = 0
+        // This does zero fill to make the last block size 4096 bytes which leads to pop noise (as in 0.3.8).
+        for (i = 0; i < handle->channel_count; i++)
+        {
+	        if (handle->buffer_ptr[i] > handle->buffer[i]) {
+	            handle->sample_count += handle->buffer_ptr[i] - handle->buffer[i];
+
+	            fwrite(handle->buffer[i], 1, SACD_BLOCK_SIZE_PER_CHANNEL, ft->fd);
+	            memset(handle->buffer[i], 0, SACD_BLOCK_SIZE_PER_CHANNEL);
+
+	            handle->buffer_ptr[i] = handle->buffer[i];
+	            handle->audio_data_size += SACD_BLOCK_SIZE_PER_CHANNEL;
+	        }
+        }
     }
 
     // write the footer
Only in w/sacd-ripper-0.3.8/libs/libsacd: patcc
diff -upr w/sacd-ripper-0.3.8/libs/libsacd/scarletbook_output.c z/sacd-ripper-0.3.8/libs/libsacd/scarletbook_output.c
--- w/sacd-ripper-0.3.8/libs/libsacd/scarletbook_output.c	2015-07-31 01:15:45.000000000 -0700
+++ z/sacd-ripper-0.3.8/libs/libsacd/scarletbook_output.c	2018-05-11 02:13:06.946097111 -0700
@@ -122,7 +122,7 @@ static void destroy_ripping_queue(scarle
     }
 }
 
-int scarletbook_output_enqueue_track(scarletbook_output_t *output, int area, int track, char *file_path, char *fmt, int dsd_encoded_export)
+int scarletbook_output_enqueue_track(scarletbook_output_t *output, int area, int track, char *file_path, char *fmt, int dsd_encoded_export, int dsf_gapless)
 {
     scarletbook_format_handler_t const * handler;
     scarletbook_output_format_t * output_format_ptr;
@@ -140,6 +140,7 @@ int scarletbook_output_enqueue_track(sca
         output_format_ptr->channel_count = sb_handle->area[area].area_toc->channel_count;
         output_format_ptr->dst_encoded_import = sb_handle->area[area].area_toc->frame_format == FRAME_FORMAT_DST;
         output_format_ptr->dsd_encoded_export = dsd_encoded_export;
+        output_format_ptr->dsf_gapless = dsf_gapless;
         if (handler->flags & OUTPUT_FLAG_EDIT_MASTER)
         {
             output_format_ptr->start_lsn = sb_handle->area[area].area_toc->track_start;
@@ -161,7 +162,7 @@ int scarletbook_output_enqueue_track(sca
             }
             else 
             {
-                output_format_ptr->length_lsn = sb_handle->area[area].area_toc->track_end - output_format_ptr->start_lsn;
+                output_format_ptr->length_lsn = sb_handle->area[area].area_toc->track_end - output_format_ptr->start_lsn + 1;
             }
         }
 
diff -upr w/sacd-ripper-0.3.8/libs/libsacd/scarletbook_output.h z/sacd-ripper-0.3.8/libs/libsacd/scarletbook_output.h
--- w/sacd-ripper-0.3.8/libs/libsacd/scarletbook_output.h	2015-07-31 01:15:45.000000000 -0700
+++ z/sacd-ripper-0.3.8/libs/libsacd/scarletbook_output.h	2018-05-11 02:14:24.570501092 -0700
@@ -75,6 +75,7 @@ struct scarletbook_output_format_t
 
     int                             dst_encoded_import;
     int                             dsd_encoded_export;
+    int                             dsf_gapless;
 
     scarletbook_format_handler_t    handler;
     void                           *priv;
@@ -97,7 +98,7 @@ typedef void (*stats_track_callback_t)(c
 
 scarletbook_output_t *scarletbook_output_create(scarletbook_handle_t *, stats_track_callback_t, stats_progress_callback_t, fwprintf_callback_t);
 int scarletbook_output_destroy(scarletbook_output_t *);
-int scarletbook_output_enqueue_track(scarletbook_output_t *, int, int, char *, char *, int);
+int scarletbook_output_enqueue_track(scarletbook_output_t *, int, int, char *, char *, int, int);
 int scarletbook_output_enqueue_raw_sectors(scarletbook_output_t *, int, int, char *, char *);
 int scarletbook_output_start(scarletbook_output_t *);
 void scarletbook_output_interrupt(scarletbook_output_t *);
diff -upr w/sacd-ripper-0.3.8/tools/sacd_extract/main.c z/sacd-ripper-0.3.8/tools/sacd_extract/main.c
--- w/sacd-ripper-0.3.8/tools/sacd_extract/main.c	2015-07-31 01:15:45.000000000 -0700
+++ z/sacd-ripper-0.3.8/tools/sacd_extract/main.c	2018-05-11 02:18:46.579862636 -0700
@@ -288,7 +288,7 @@ static void init(void)
 int main(int argc, char* argv[]) 
 {
     char *albumdir = 0, *musicfilename, *file_path = 0;
-    int i, area_idx;
+    int i, area_idx, dsf_gapless = 1;
     sacd_reader_t *sacd_reader;
 
 #ifdef PTW32_STATIC_LIB
@@ -367,7 +367,7 @@ int main(int argc, char* argv[])
                         file_path = make_filename(0, 0, albumdir, "dff");
 
                         scarletbook_output_enqueue_track(output, area_idx, 0, file_path, "dsdiff_edit_master", 
-                            (opts.convert_dst ? 1 : handle->area[area_idx].area_toc->frame_format != FRAME_FORMAT_DST));
+                            (opts.convert_dst ? 1 : handle->area[area_idx].area_toc->frame_format != FRAME_FORMAT_DST), 0);
                     }
                     else if (opts.output_dsf || opts.output_dsdiff)
                     {
@@ -375,6 +375,16 @@ int main(int argc, char* argv[])
                         get_unique_dir(0, &albumdir);
                         recursive_mkdir(albumdir, 0774);
 
+                        for (i = 0; i < handle->area[area_idx].area_toc->track_count; i++) 
+                        {
+                            if (opts.select_tracks && opts.selected_tracks[i] == 0){
+                                // Not all tracks are being extracted.
+                                // Disable gapless dsf generation as it requires continuous extraction of all tracks.
+                                dsf_gapless = 0;
+                                break;
+                            }
+                        }
+ 
                         // fill the queue with items to rip
                         for (i = 0; i < handle->area[area_idx].area_toc->track_count; i++) 
                         {
@@ -387,13 +397,13 @@ int main(int argc, char* argv[])
                             {
                                 file_path = make_filename(0, albumdir, musicfilename, "dsf");
                                 scarletbook_output_enqueue_track(output, area_idx, i, file_path, "dsf", 
-                                    1 /* always decode to DSD */);
+                                    1 /* always decode to DSD */, dsf_gapless);
                             }
                             else if (opts.output_dsdiff)
                             {
                                 file_path = make_filename(0, albumdir, musicfilename, "dff");
                                 scarletbook_output_enqueue_track(output, area_idx, i, file_path, "dsdiff", 
-                                    (opts.convert_dst ? 1 : handle->area[area_idx].area_toc->frame_format != FRAME_FORMAT_DST));
+                                    (opts.convert_dst ? 1 : handle->area[area_idx].area_toc->frame_format != FRAME_FORMAT_DST), 0);
                             }
 
                             free(musicfilename);

 

[audiventory]

25 minutes ago, mindset said:

This discontinuity in DSF results in a pop noise at a track transition on players with gap-less playback support.

 

Zero padding at last block is defined by DSF specification http://dsd-guide.com/sites/default/files/white-papers/DSFFileFormatSpec_E.pdf

 

DSF compatible players should know about it and play files with zero padding as gapless stream without problems.

 

I'd suggest to implement the additional command line option for this functionality.

 

Why you think, that zero padding cause clicks? Do you checked digital stream at a player output or other?

[mindset]

6 hours ago, audiventory said:

 

Zero padding at last block is defined by DSF specification http://dsd-guide.com/sites/default/files/white-papers/DSFFileFormatSpec_E.pdf

 

DSF compatible players should know about it and play files with zero padding as gapless stream without problems.

 

I'd suggest to implement the additional command line option for this functionality.

 

Why you think, that zero padding cause clicks? Do you checked digital stream at a player output or other?

Thanks.  I was not aware of zero padding defined in the DSF spec. At least my player (SONY HAP-Z1ES) stopped making pop noise with the fix I pasted.

 

One thing I noticed is that there is some data corruption at the beginning of each track  (excluding track 1) with the 0.3.8 release.  Maybe that's the real cause of pop noise.

 

[audiventory]

13 minutes ago, mindset said:

One thing I noticed is that there is some data corruption at the beginning of each track  (excluding track 1) with the 0.3.8 release.

 

It may be noise into non-skipped pause between tracks.

 

Read details about clicks:

1. https://samplerateconverter.com/content/extract-iso-dsf-pcm-clicks-issues-part-1
2. https://samplerateconverter.com/iso-converter/convert-iso-dsf-wav-flac-aiff

[mindset]

Well, what I mentioned is data corruption in the .dsf file. So It has nothing to do the player.  So I don't think the problem is related to those issues mentioned those pages.

 

I didn't spend much time to investigate the corruption since avoiding zero padding magically solved that problem as well.  I will look into this later to investigate.  Maybe I was missing something.

 

Anyways, the patch generates bit perfect dsf's and I don't hear any pop noise with gap-less playback so I am already happy

 

 

[audiventory]

1 minute ago, mindset said:

So I don't think the problem is not related to those issues mentioned those pages.

 

You mentioned sacd_extract (sacd ripper).

[mindset]

No I meant those two web pages on samplerateconvert.com.

[audiventory]

8 hours ago, mindset said:

As some people have probably noticed, the sacd-ripper 0.3.8 release generates discontinuous .dsf because it does zero padding

 

It is not sacd-ripper mentioning?

 

Here DSFs, extracted from ISO or SACD, are meant, isn't it?

[mindset]

10 hours ago, audiventory said:

Here DSFs, extracted from ISO or SACD, are meant, isn't it?

The DSFs were extracted from ISOs using sace_extract in the sacd-ripper 0.3.8 release.

 

I did further investigation, and I have to take back the data corruption at the head of tracks I mentioned earlier.  This is not happening in 0.3.8.  Therefore, my conclusion is that my player (SONY HAP-Z1ES) is not handling DSF zero padding correctly so it generates a pop noise at track transition when gap-less playback is enabled, and this can be addressed by the patch I posted.

 

Since zero padding avoidance I proposed is a lossless process, I believe this method is universally harmless.  The only thing is that up to 1 DSF block (11.6 ms) worth of audio moves to the previous track, which I don't think is a problem.

 

Now I am going to extract all ISO's over the weekend.  I am glad I have kept all ISO's in addition to DSFs.

[audiventory]

8 minutes ago, mindset said:

I believe this method is universally harmless

 

I'm agree. I'd suggest apply it by user option though.

[mindset]

2 minutes ago, audiventory said:

 

I'm agree. I'd suggest apply it by user option though.

 

Sure.  I agree that it should be a user option.  I will keep test driving this patch.  If I don't see any problem, I might propose this to mrwicked (I am not sure if he is active though).

 

[audiventory]

9 minutes ago, mindset said:

I might propose this to mrwicked

 

As I understand, you can submit it here https://github.com/sacd-ripper/sacd-ripper

[mindset]

1 hour ago, audiventory said:

As I understand, you can submit it here https://github.com/sacd-ripper/sacd-ripper

Thanks for the pointer.

 

The other thing I noticed is that the pop is gone if I include the zero padding in the sample count in the fmt section of the DSF header.  It is counter-intuitive to me. Unfortunately, whether the zero padding should be included or not in the sample count is not stated in the DSF specifications.  Intuitively it shouldn't be, of course, but SONY's interpretation may be the opposite..

 

Either way, for me, avoiding zero padding is the way to go..