paranoid about NAS music file integrity

[Søren Abildgaard]

Hi guys!

 

With all my music digitalized and on my big Synology NAS, I'm starting to get worried about file integrity (aka bit rot).

 

So I experimented on my small NAS (a well-aged Synology DS214se) with some scripts to generate and check MD5 sums. To this end, I manually installed md5deep from the ipkg feed (didn't feel like installing all of ipkg) from here:

Index of /feeds/optware/cs08q1armel/cross/stable/

I just unpacked the data.tar.gz from the package and copied opt/bin/md5deep to /opt/bin/md5deep on my Synology.

 

Then I wrote a little script to create and check the MD5 hash for a given file:

 

#!/bin/sh
echo -n "Processing $1 ... "
if [ -f "$1.md5" ]
then
 cp "$1" "$1.tmp"
 mv "$1.tmp" "$1"
 /opt/bin/md5deep -m "$1.md5" "$1" > /dev/null
 status=$?
 if [ $status -ne 0 ]
 then
   echo "FAILURE - hash value has changed"
 else
   echo "OK - hash values match"
 fi
else
 /opt/bin/md5deep -b "$1" > "$1.md5"
 echo "FIRST - stored MD5 hash"
fi

 

Being my paranoid self, I also refresh the file by copying it to a new name and moving it back to the original name. This should refresh the bits nicely

 

Finally, to check all WAV files on the "media" volume, I run the following command from my Mac Pro's terminal:

 

ssh [email protected] -t "find /volume1/media/Music/ -iname '*.wav' -exec /opt/bin/regen.sh '{}' \;" | tee regen.log

 

The log file contains FIRST for all files that were first seen and got hashed, OK for those where the computed hash matched the stored one, and FAILURE when there is bit rot (easily simulated by corrupting an .md5 file or a .wav file by hand).

 

I thought about getting a NAS with ZFS or BTRFS support, but all I would use that for is checksumming - so that can be done just as well using my above script. And I like DSM too much

 

What do you do to combat bit rot?

 

Edit: If you want to know how to do this on your NAS, I can give more detailed descriptions. Just let me know.

[wgscott]

Why would a self-proclaimed paranoid write stdout to /dev/null?

 

My paranoia led me to getting a zfs filesystem-based NAS.

 

Here are some of my paranoid ramblings:

 

bitrot: A shell script to detect changes in the audio component of ALAC files - Blogs - Computer Audiophile

 

Write-protect your iTunes library to avoid accidental deletion or damage - Blogs - Computer Audiophile

 

Review: iXsystems FreeNAS Mini: A Plug-and-Play ZFS NAS that doesn't suck to set up. - Blogs - Computer Audiophile

[wgscott]

Oh, forgot to say, "Welcome!"

[goldsdad]

I thought about getting a NAS with ZFS or BTRFS support, but all I would use that for is checksumming - so that can be done just as well using my above script.

 

BTRFS and ZFS (but not the limited implementation being introduced by Synology, AFAIK) can automatically repair bit rot, not just detect it.

[Søren Abildgaard]

BTRFS and ZFS (but not the limited implementation being introduced by Synology, AFAIK) can automatically repair bit rot, not just detect it.

 

BTRFS automatically detects corruption, but repair is by redundancy like in any other raid configuration.

 

And being paranoid, an extra level cannot hurt.

[Søren Abildgaard]

Oh, forgot to say, "Welcome!"

 

Thanks! After reading here for like 5 years, it us funny to be contributing actively

 

Your ALAC script is great, as it seems to care only for the PCM data. For my purposes this will be hard to replicate on the synology, as it requires parsing ALAC files.

 

My reason for staying in the synology is that I want to have it monitor the file integrity status regularly and notify me of any irregularities whenever they are detected.

[jabbr]

Echo what folks have said above. ZFS & BTRFS can repair disc errors/bit rot as long as you have redundancy in your pools i.e. Use mirroring or raid5/6 etc. if you don't have redundancy ZFS can't repair either. These modern file systems go waay beyond simple scripts in terms of data integrity, particularly if you also use ECC RAM in which case they can also protest against Martian invasions:)

[wgscott]

Thanks! After reading here for like 5 years, it us funny to be contributing actively

 

My shameless trolling about a band which I know almost nothing?

[Søren Abildgaard]

My shameless trolling about a band which I know almost nothing?

 

This is more down your alley ;-)

 

[goldsdad]

BTRFS automatically detects corruption, but repair is by redundancy like in any other raid configuration.

 

Repair is dependant on redundancy, yes, but unlike traditional file systems, BTRFS and ZFS automatically do that repair.

[Gregory]

Hi,

If you want to add extra redundancy to your files (if your NAS filesystem is not ZFS or BTRFS) you could use PAR2 (QuickPar for Windows) to add extra-files to each directory. This tool was widely used on binary Usenet newsgroup to help in case you missed some part of downloaded data.

It adds some extra files that can be used to check and recover lost bits.

My 2 cents...

 

Greg.