Bit rot and filesystems

[wgscott]

Interesting read on Arse Technica: Bitrot and atomic COWs: Inside

 

Let's talk about "bitrot," the silent corruption of data on disk or tape. One at a time, year by year, a random bit here or there gets flipped. If you have a malfunctioning drive or controller—or a loose/faulty cable—a lot of bits might get flipped. Bitrot is a real thing, and it affects you more than you probably realize. The JPEG that ended in blocky weirdness halfway down? Bitrot. The MP3 that startled you with a violent CHIRP!, and you wondered if it had always done that? No, it probably hadn't—blame bitrot.

[Boris75]

This is frightening, really, especially given that there is no easy-to-implement solution as of now.

[wgscott]

Yes, the best we can do as audiophiles is optimize our operating systems for playback by doing things like running scripts blindly that turn off disk journaling.

[WINDIANRECORDS]

Ahem ... It's "Ars" technica, not "Arse" ... unless that was on purpose.

 

Good article, though, worth a read for viewers of this site.

[wgscott]

Ahem ... It's "Ars" technica, not "Arse" ... unless that was on purpose.

 

Who, me?

[Paul R]

Actually it is most definitely a real problem. All media, disk, tape, optical, etc. has a finite lifetime? Archives and backups need to be copied to new media to avoid the media rotting out from under them.

 

This article seems to be a little - alarmist - in tone though. The issue is very well known indeed.

 

It is funny, but one of the bigger objections to CDs was that would "never wear out!" Of course, they do, but they last a whole lot longer than a vinyl album normally would.

 

-Paul

 

Haha

 

If "bitrot" was a real problem, the entire computer industry would come to a screeching stop.

[wgscott]

My cosmic ray player really burns through my digital files like a tonearm playing a warped record with a quarter on top of it weighing it down.

[Paul R]

You should know better Bill - a cosmic ray cutter is only supposed to be used when cutting through collapsium¹ armor.

 

-Paul

 

¹Collapsium

 

My cosmic ray player really burns through my digital files like a tonearm playing a warped record with a quarter on top of it weighing it down.

[Audio_ELF]

If "bitrot" was a real problem, the entire computer industry would come to a screeching stop.

That was my thought ... its only in long term archiving where there can be an issue ...

 

Actually it is most definitely a real problem. All media, disk, tape, optical, etc. has a finite lifetime? Archives and backups need to be copied to new media to avoid the media rotting out from under them.

If anyone has a copy of Dr No on Bluray (I think its on the Dr No disc - maybe one of the other earlier Bond movies) there is an interesting documentary on the digitalising of the movie from the original film and how they store the footage. Basically they have a case with a number of standard HDDs for long term storage; but these are refreshed every 6 months (might not be the actual timescale I'm working from memory) to ensure they can be read and copied to new drives.

 

This article seems to be a little - alarmist - in tone though. The issue is very well known indeed.

Yes it is alarmist. One thing to note is that if a file is subject to bit rot it is no longer bit perfect! IIRC flac files have an internal checksum and you can run a routine to check that they are not corrupted.

 

It is funny, but one of the bigger objections to CDs was that would "never wear out!" Of course, they do, but they last a whole lot longer than a vinyl album normally would.

To give it dues; I believe the concept behind the "Perfect sound forever" was that repeated playing of a CD would not degrade it - that is the first play is identical to the 10,000th so long as the CD wasn't damaged. This was in contrast to a vinyl record where every play degrades the surface a tiny amount.

 

Of course what was later discovered was that imperfect manufacturing of CDs meant that some CDs degraded a lot quicker than the 100+ years that was predicted (see Disc rot - Wikipedia, the free encyclopedia).

 

Electronic medium is unreliable - UK legislation is written on vellum for longevity :-)

 

Eloise

[Boris75]

Haha

 

If "bitrot" was a real problem, the entire computer industry would come to a screeching stop.

 

Thanks for the reassuring thought.

[PorkChop]

This sounds like a dang good reason to backup only deltas and not simply overwrite the volume.

[coxhaus]

I think this is why big business servers use parity or error correcting ram in their servers. You will see this error correcting ram as both ram and controller ram. Just in case that special sun spot happens while you are processing.

[Paul R]

Hi Lee -

 

My big computer at work not only corrects it memory, but it will (and has) noticed that the processor failed, stopped all the processing, brought a spare processor on line, and restarted the failed instruction. I didn't even notice till out SE came in to replace the processor. Which also was done while the machine was running... zero down time for my users.

 

-Paul

 

 

I think this is why big business servers use parity or error correcting ram in their servers. You will see this error correcting ram as both ram and controller ram. Just in case that special sun spot happens while you are processing.

[Boris75]

This sounds like a dang good reason to backup only deltas and not simply overwrite the volume.

 

GoodSync does just that. (I have no affiliation with Goodsync - just a happy customer).

[coxhaus]

The problem with backups fixing the bit corruption problem is by the time you figure out you have data corruption is it is too late. It may be weeks or months before the problem surfaces. Usually the backups are still running backing up the bit corrupted data and overlaying the once good data.

[esldude]

My cosmic ray player really burns through my digital files like a tonearm playing a warped record with a quarter on top of it weighing it down.

 

By warped do you mean, spun up to warp speed?

 

 

Besides how can you take an article seriously by someone using the Unity interface?

[PewterTA]

Funny that bit rot happens, yet my friend has a 10GB HD from 1990ish and it has pictures on it that come off perfect. Granted, the drive is not in use hardly at all and is more just for show (it's a 5.25" drive) as it sounds terrible, but works flawlessly.

 

So maybe that drive is just rot-blocked! lol

 

Really the article takes everything to the utmost extremes. If you do the things you should, multiple copies, backups, properly keeping hardware working (ie replace things when needed)... you don't have to worry about a whole lot about "rot."

 

The more important thing to watch out for is just overall hardware failure in your HDs... I bet most times your drive will just fail before you'd get "data corruption."

[coxhaus]

If you are really worried about bit rot the place to start is to build a machine with error correcting RAM and make sure the processor has support for ECC RAM which is error correcting RAM.

[Paul R]

Well, "bit rot" is a slang term that really doesn't have any intrinsic meaning. The problem has nothing to do with mythical digital degradation, and everything to do with media decomposition.

 

But it is more fun to call it "bit rot"- and everyone knows what one means.

 

Paul

[cjf]

Bit Rot or not this concept is all the more reason to not store your library on SSD. Speed is great and all but when your dealing with a drive subject to heavy Read/Write activity like is the case with a media library (more read then write usually) you need to take this into account when selecting a home for important data.

 

Even when using RAID on an SSD the type of RAID you choose also needs to be taken into account. RAID 1 would subject both SSD disks to the same stress and workload simultaneously so there is a high likelihood that when one disk fails the other is probably not far behind it so don't drag your feet getting a replacement disk if you find yourself in this situation.

 

In a perfect world RAID 6 would be better but that would depend on the capacity and type of disk chassis you are using. RAID 6 allows for a double disk failure in the RAID set before going belly up.

 

If you want to be real anal about it you could use RAID 5 and stagger the buildup of the RAID set over a month or two so that each drive would theoretically have a different lifespan then its partners possibly buying you more time to acquire a replacement.

 

Again, this is all referring to the use of SSD drives and not so much with spinning platters which have been around long enough to prove their reliability under heavy I/O conditions.

 

There are SSD disks on the market though that offer a 5 year guarantee under heavy Read/Write conditions but at $3K+ a piece its not something most corporations can afford to invest in, let alone us Audiophiles. I wonder how long it will be before we see the high end catch on to these fancy SSD disks and proclaim multiple veils being lifted by using them

[wgscott]

I've got a dead SSD sitting in front of me. Lasted about 8 months. It is still under warranty (OWC Mercury Pro). I never had a conventional spinning drive fail that quickly.

[coxhaus]

A hard drive with a metal case to shield it may turn out to be more secure against bit rot.

[paul1970]

A hard drive with a metal case to shield it may turn out to be more secure against bit rot.

Unfortunately it's not. ECC protected solid-state memory or multiple copies and a good backup scheme are the only answers.

Btw, if your backup scheme has overwritten a good backup with a corrupted one and you have no other way to recover it then it's not a good backup scheme. Files that should never change unless they have been corrupted should be handled accordingly.

[sandyk]

My big computer at work not only corrects it memory, but it will (and has) noticed that the processor failed, stopped all the processing, brought a spare processor on line, and restarted the failed instruction.

 

Telephone Exchanges have been doing that for more than 30 years.

They even generated alarms that earned me quite a bit of extra pay for after hours recalls. (grin)

[Paul R]

(grin) I think the network had rerouting and redundancy in it, but I don't remember even System-7 switches being able to handle failed processors like that. Certainly not back in the days of the dinosaurs when switches were actually mechanical relays... you old tyrannosaurus you!

 

I do remember having more than one go around with the little 3B based switches AT&T sold running 4ESS and up though... (brrr!)

 

-Paul

 

 

Telephone Exchanges have been doing that for more than 30 years.

They even generated alarms that earned me quite a bit of extra pay for after hours recalls. (grin)