Bit rot and filesystems

[coxhaus]

Unfortunately it's not. ECC protected solid-state memory or multiple copies and a good backup scheme are the only answers.

Btw, if your backup scheme has overwritten a good backup with a corrupted one and you have no other way to recover it then it's not a good backup scheme. Files that should never change unless they have been corrupted should be handled accordingly.

 

At some point backups are over written there is not endless storage. Of course I am talking more about the old days, I guess today you have much higher densities so maybe the cycle is longer. Drive space in the old days cost a lot more than today so we were lucky if we could keep 6 months worth of data and most of that was on tape.

[paul1970]

If a file has been backed-up and hasn't changed why would you waste space on another copy of it? If a file that shouldn't change is different from the backup then that needs investigating and not overwriting.

[goldsdad]

If a file that shouldn't change is different from the backup then that needs investigating and not overwriting.

 

+1

 

Users of Carbon Copy Cloner should ensure the archive option is enabled and they should check for unexpected archives being created when updating a clone/backup.

[wgscott]

Given all of this, wouldn't this be a compelling argument for wav files, since the metadata would be stored externally (e.g.: iTunes database) so the file wouldn't change ever, except for bit rot?

[Audio_ELF]

Given all of this, wouldn't this be a compelling argument for wav files, since the metadata would be stored externally (e.g.: iTunes database) so the file wouldn't change ever, except for bit rot?

Except (iirc) FLAC has an internal checksum so you can run a check for corrupted FLAC files.

 

Eloise

[goldsdad]

Given all of this, wouldn't this be a compelling argument for wav files, since the metadata would be stored externally (e.g.: iTunes database) so the file wouldn't change ever, except for bit rot?

 

I prefer metadata to be inside audio files so it is accessible to many programs and always travels with the files.

[wgscott]

I do, too. But then again, I just lost (and subsequently replaced from backup) 4 files from user error, but I have yet to lose one to bit-rot.

 

The FLAC internal checksum actually seems like the best idea to me. Anyone know how this works (i.e., how you can check your files for changes) and if any other formats, like ALAC, have this?

 

The ARS article BTW was focusing mainly on these new filesystem formats, not bit-rot. Apple was going to migrate to zfs from HFS+ but abandoned the project.

[wgscott]

Except (iirc) FLAC has an internal checksum so you can run a check for corrupted FLAC files.

 

Eloise

 

That actually would be a better hedge, assuming the bit-rot threat is significant.

[goldsdad]

That actually would be a better hedge, assuming the bit-rot threat is significant.

 

Bill, you might be interested in IntegrityChecker from diglloydTools

[Audio_ELF]

The FLAC internal checksum actually seems like the best idea to me. Anyone know how this works (i.e., how you can check your files for changes) and if any other formats, like ALAC, have this?

flac --test <filename(s)>

IIRC

[wgscott]

Is there a way to actually extract out the checksum from the flac file, or read the header?

[Paul R]

It is 124 bytes into the STREAMINFO header. That is a MD5 hash. Flac uses CRC Checksums on each data block when streaming though.

 

-Paul

Is there a way to actually extract out the checksum from the flac file, or read the header?

[sshd]

RAM failure is a real problem in consumer electronics.

 

Your computer may run fine and never crash for months.

 

But when you start a backup/sync routine, a calculation might produce the wrong result and a file will be transferred to the backup again. The computer will not read the file from the hard drive again, as it has it all in memory (failed memory). So the good backup is replaced with a bad one.

 

It rarely happens to other people, but I have suffered twice in 10 years.

 

The md5 checksum in FLAC is why I use FLAC. I verify the entire music collection a few times a year. Unfortunately it is not perfect. It only checksums the audio data. FLAC header and metadata corruption will not be defected by FLAC verify.

 

My future computers will all have ECC RAM, because it is so annoying when it happens.

[wgscott]

emacs

 

Well, I confess I used vim, and I tried the "strings" command, but I still couldn't find it.

[wgscott]

Here is how:

 

zsh-% metaflac --show-md5sum 04\ Journey\ Through\ the\ Past.flac

 

417ab7e2f839d1349d773ca6b09c39f2

[wgscott]

Oh, lookie here:

 

 

% afhash -h

 

Audio File Hash

Version: 2.0

Copyright 2013, Apple Inc. All Rights Reserved.

Specify -h (-help) for command options

 

usage: afhash [-w | -x] file(s)

afhash -c file1 file2

for help: afhash [-h | --help]

 

afhash computes an SHA-1 hash of the audio data in the file and

optionally writes the hash code into a chunk in the file.

A hash will only be computed for integer linear PCM or losslessly encoded data

whose sample rate is greater than or equal to 44100 Hz and bit depth is

greater than or equal to 16.

If the file already has a hash chunk, then the hash stored in

the chunk will be printed.

Writing the chunk is only supported in AIFF, AIFC, WAV, and CAF formats.

 

OPTIONS:

 

-h --help Print out this help.

 

-w Write the hash code into the file.

In order to preserve an original hash from a source file, afhash will

not overwrite a hash chunk in a file that already contains one.

 

-x Do not compute the hash of the audio data, only print out the

hash stored in the hash chunk, if present.

 

-c Print and compare the contents of the hash chunk from two files.

 

Options -c, -w and -x conflict. Only one should be specified.

 

[wgscott]

Here is my first attempt to write a shell script to detect bitrot (changes) in ALAC files.

 

bitrot

 

Download, make sure the file is called bitrot with no suffix, and change the executable bit, i.e.,

 

chmod a+x bitrot

 

Now put it in your $PATH

 

cd to a directory that contains your ALAC files you want to protect.

 

Run the shell script like this to create the checksums:

 

bitrot -w

 

It will write a checksum for the audio part of the file (which should never change) to a new metadata attribute in the resource fork of each audio file (assuming you are on an Apple HFS+ filesystem). The advantage of doing this is the shell script doesn't touch your actual audio file (and it makes the shell script easier to write for me at least).

 

To check the files at a later time, simply run the same command, but with -c instead of -w.

 

bitrot -c

 

Here is some sample output, after I deliberately damaged one of the files (using vim):

 

[Boris75]

Here is my first attempt to write a shell script to detect bitrot (changes) in ALAC files.

 

bitrot

 

Download, make sure the file is called bitrot with no suffix, and change the executable bit, i.e.,

 

chmod a+x bitrot

(...)

 

It will write a checksum for the audio part of the file (which should never change) to a new metadata attribute in the resource fork of each audio file (assuming you are on an Apple HFS+ filesystem).

 

Wow, many thanks, this is a terrific contribution for Mac/iTunes users like me.

 

I am not familiar at all with how HFS+ works and have two easy/silly questions:

 

 

1. does the resource fork of a file accompany the file if I move, copy or delete it?
   
2. how do I run "bitrot" recursively on a directory tree?
   

[Raym87]

'Bits'. A colloquialism in the UK to refer to reproductive organs. Bit rot! Could it get any worse?

[wgscott]

1. does the resource fork of a file accompany the file if I move, copy or delete it?
   
2. how do I run "bitrot" recursively on a directory tree?
   

 

1. Yes, as long as you use OS X utilities to do these things. If you move the file to a different format disk, you will see it as ._foo if the filename is foo. You can read the attributes with mdls

 

2. I am working on an improved version to do exactly that.

[Boris75]

1. Yes, as long as you use OS X utilities to do these things. If you move the file to a different format disk, you will see it as ._foo if the filename is foo. You can read the attributes with mdls

 

2. I am working on an improved version to do exactly that.

 

Many thanks!

[wgscott]

The above link now goes to version 0.1.3, which will do all this, and has some more descriptive documentation (bitrot -h prints it out for you).

[Cebolla]

'Bits'. A colloquialism in the UK to refer to reproductive organs. Bit rot! Could it get any worse?

Personally, I've got more than 1 bit to rot.

[Miska]

At least on Linux, there's a filesystem level solution, most of the distributions are slowly moving to the new Btrfs and it has checksumming support among many other features:

Btrfs - Wikipedia, the free encyclopedia

https://btrfs.wiki.kernel.org/index.php/Main_Page

[jshawn]

At least on Linux, there's a filesystem level solution, most of the distributions are slowly moving to the new Btrfs and it has checksumming support among many other features: Btrfs - Wikipedia, the free encyclopedia https://btrfs.wiki.kernel.org/index.php/Main_Page

Won't that require the use of ECC memory the way ZFS does? Won't that limit the usefulness on consumer hardware?