Bit rot and filesystems

[Miska]

Won't that require the use of ECC memory the way ZFS does? Won't that limit the usefulness on consumer hardware?

 

No, it is unrelated to type of memory. There's no reason why type of RAM would have anything to do with filesystem level fault tolerance implementation.

[Bob Stern]

Brilliant, Bill!

 

One great feature of Apple's afhash program is that it appears to compute the hashcode from the audio data only, excluding the tags. This is good for those of us who modify the tags from time to time, since the original hashcode should remain invariant.

 

That's an advantage over the Linux approach which that change the hashcode whenever you modify a tag embedded in an audio file.

[Jud]

Besides how can you take an article seriously by someone using the Unity interface?

 

Word.

[Miska]

That's an advantage over the Linux approach which that change the hashcode whenever you modify a tag embedded in an audio file.

 

It doesn't matter from corruption detection point of view, because checksums are compared against the latest modification point. And with log structured file systems like Btrfs you can also make snapshots and compare old and new versions of the file or revert the changes you made.

 

And of course it is not bound to any particular file type, same checks apply to all files regardless of content.

[wgscott]

You realize, I hope, that you are in danger of bringing this thread back around to being on topic?

[wgscott]

More stable link: bitrot: A shell script to detect changes in the audio component of ALAC files - Blogs - Computer Audiophile

 

Here is my first attempt to write a shell script to detect bitrot (changes) in ALAC files.

 

bitrot

 

Download, make sure the file is called bitrot with no suffix, and change the executable bit, i.e.,

 

chmod a+x bitrot

 

Now put it in your $PATH

 

cd to a directory that contains your ALAC files you want to protect.

 

Run the shell script like this to create the checksums:

 

bitrot -w

 

It will write a checksum for the audio part of the file (which should never change) to a new metadata attribute in the resource fork of each audio file (assuming you are on an Apple HFS+ filesystem). The advantage of doing this is the shell script doesn't touch your actual audio file (and it makes the shell script easier to write for me at least).

 

To check the files at a later time, simply run the same command, but with -c instead of -w.

 

bitrot -c

 

Here is some sample output, after I deliberately damaged one of the files (using vim):

 

[ATTACH=CONFIG]10390[/ATTACH]

[cmiu]

Just to be sure I took the NAS aproach - freenas.

I'm using six hdds configured as zfs raid-Z and scrub my data periodically (30 days).

and smartctl weekly checks for every hdd in order to predict hdds failures. It is not bullet proof but add another layer of checks for reliability of the hardware.

and most important I use off site back-up as well.

 

google search "bit roting zfs"

ZFS - Wikipedia, the free encyclopedia - Data integrity

http://www.freenas.org/ pretty easy to install and configure and the documentation/community support is very good. You can even create one click virtual machines for upnp server or various services.