Jump to content
IGNORED

Bit rot and filesystems


Recommended Posts

Interesting read on Arse Technica: Bitrot and atomic COWs: Inside

 

Let's talk about "bitrot," the silent corruption of data on disk or tape. One at a time, year by year, a random bit here or there gets flipped. If you have a malfunctioning drive or controller—or a loose/faulty cable—a lot of bits might get flipped. Bitrot is a real thing, and it affects you more than you probably realize. The JPEG that ended in blocky weirdness halfway down? Bitrot. The MP3 that startled you with a violent CHIRP!, and you wondered if it had always done that? No, it probably hadn't—blame bitrot.
Link to comment

Actually it is most definitely a real problem. All media, disk, tape, optical, etc. has a finite lifetime? Archives and backups need to be copied to new media to avoid the media rotting out from under them.

 

This article seems to be a little - alarmist - in tone though. The issue is very well known indeed.

 

It is funny, but one of the bigger objections to CDs was that would "never wear out!" Of course, they do, but they last a whole lot longer than a vinyl album normally would.

 

-Paul

 

Haha

 

If "bitrot" was a real problem, the entire computer industry would come to a screeching stop.

Anyone who considers protocol unimportant has never dealt with a cat DAC.

Robert A. Heinlein

Link to comment
If "bitrot" was a real problem, the entire computer industry would come to a screeching stop.

That was my thought ... its only in long term archiving where there can be an issue ...

 

Actually it is most definitely a real problem. All media, disk, tape, optical, etc. has a finite lifetime? Archives and backups need to be copied to new media to avoid the media rotting out from under them.

If anyone has a copy of Dr No on Bluray (I think its on the Dr No disc - maybe one of the other earlier Bond movies) there is an interesting documentary on the digitalising of the movie from the original film and how they store the footage. Basically they have a case with a number of standard HDDs for long term storage; but these are refreshed every 6 months (might not be the actual timescale I'm working from memory) to ensure they can be read and copied to new drives.

 

This article seems to be a little - alarmist - in tone though. The issue is very well known indeed.

Yes it is alarmist. One thing to note is that if a file is subject to bit rot it is no longer bit perfect! IIRC flac files have an internal checksum and you can run a routine to check that they are not corrupted.

 

It is funny, but one of the bigger objections to CDs was that would "never wear out!" Of course, they do, but they last a whole lot longer than a vinyl album normally would.

To give it dues; I believe the concept behind the "Perfect sound forever" was that repeated playing of a CD would not degrade it - that is the first play is identical to the 10,000th so long as the CD wasn't damaged. This was in contrast to a vinyl record where every play degrades the surface a tiny amount.

 

Of course what was later discovered was that imperfect manufacturing of CDs meant that some CDs degraded a lot quicker than the 100+ years that was predicted (see Disc rot - Wikipedia, the free encyclopedia).

 

Electronic medium is unreliable - UK legislation is written on vellum for longevity :-)

 

Eloise

Eloise

---

...in my opinion / experience...

While I agree "Everything may matter" working out what actually affects the sound is a trickier thing.

And I agree "Trust your ears" but equally don't allow them to fool you - trust them with a bit of skepticism.

keep your mind open... But mind your brain doesn't fall out.

Link to comment

I think this is why big business servers use parity or error correcting ram in their servers. You will see this error correcting ram as both ram and controller ram. Just in case that special sun spot happens while you are processing.

AMR 777 DAC, Purist Ultimate USB, PC server 4gig SOTM USB, server 2012, Audiophil Optimizer,Joule Preamp LAP150 Platinum Vcaps Bybee, Spectron Monoblocks Bybee Vcaps, Eggleston Savoy speakers, 2 REL Stentor III subwoofers, Pranawire Cosmos speaker wire, Purist Dominus Praesto cabling, Purist Anniversary (Canorus)power cables and Elrod Statement Gold power cable, VPI Aries I SDS w/Grado The Statement LP, 11kVA power isolation, 16 sound panels and bass traps TAD,RPG,GIK and Realtraps

Link to comment

Hi Lee -

 

My big computer at work not only corrects it memory, but it will (and has) noticed that the processor failed, stopped all the processing, brought a spare processor on line, and restarted the failed instruction. I didn't even notice till out SE came in to replace the processor. Which also was done while the machine was running... zero down time for my users. :)

 

-Paul

 

 

I think this is why big business servers use parity or error correcting ram in their servers. You will see this error correcting ram as both ram and controller ram. Just in case that special sun spot happens while you are processing.

Anyone who considers protocol unimportant has never dealt with a cat DAC.

Robert A. Heinlein

Link to comment

The problem with backups fixing the bit corruption problem is by the time you figure out you have data corruption is it is too late. It may be weeks or months before the problem surfaces. Usually the backups are still running backing up the bit corrupted data and overlaying the once good data.

AMR 777 DAC, Purist Ultimate USB, PC server 4gig SOTM USB, server 2012, Audiophil Optimizer,Joule Preamp LAP150 Platinum Vcaps Bybee, Spectron Monoblocks Bybee Vcaps, Eggleston Savoy speakers, 2 REL Stentor III subwoofers, Pranawire Cosmos speaker wire, Purist Dominus Praesto cabling, Purist Anniversary (Canorus)power cables and Elrod Statement Gold power cable, VPI Aries I SDS w/Grado The Statement LP, 11kVA power isolation, 16 sound panels and bass traps TAD,RPG,GIK and Realtraps

Link to comment
My cosmic ray player really burns through my digital files like a tonearm playing a warped record with a quarter on top of it weighing it down.

 

By warped do you mean, spun up to warp speed?

 

 

Besides how can you take an article seriously by someone using the Unity interface?

And always keep in mind: Cognitive biases, like seeing optical illusions are a sign of a normally functioning brain. We all have them, it’s nothing to be ashamed about, but it is something that affects our objective evaluation of reality. 

Link to comment

Funny that bit rot happens, yet my friend has a 10GB HD from 1990ish and it has pictures on it that come off perfect. Granted, the drive is not in use hardly at all and is more just for show (it's a 5.25" drive) as it sounds terrible, but works flawlessly.

 

So maybe that drive is just rot-blocked! lol

 

Really the article takes everything to the utmost extremes. If you do the things you should, multiple copies, backups, properly keeping hardware working (ie replace things when needed)... you don't have to worry about a whole lot about "rot."

 

The more important thing to watch out for is just overall hardware failure in your HDs... I bet most times your drive will just fail before you'd get "data corruption."

Link to comment

If you are really worried about bit rot the place to start is to build a machine with error correcting RAM and make sure the processor has support for ECC RAM which is error correcting RAM.

AMR 777 DAC, Purist Ultimate USB, PC server 4gig SOTM USB, server 2012, Audiophil Optimizer,Joule Preamp LAP150 Platinum Vcaps Bybee, Spectron Monoblocks Bybee Vcaps, Eggleston Savoy speakers, 2 REL Stentor III subwoofers, Pranawire Cosmos speaker wire, Purist Dominus Praesto cabling, Purist Anniversary (Canorus)power cables and Elrod Statement Gold power cable, VPI Aries I SDS w/Grado The Statement LP, 11kVA power isolation, 16 sound panels and bass traps TAD,RPG,GIK and Realtraps

Link to comment

Well, "bit rot" is a slang term that really doesn't have any intrinsic meaning. The problem has nothing to do with mythical digital degradation, and everything to do with media decomposition.

 

But it is more fun to call it "bit rot"- and everyone knows what one means. :)

 

Paul

Anyone who considers protocol unimportant has never dealt with a cat DAC.

Robert A. Heinlein

Link to comment

Bit Rot or not this concept is all the more reason to not store your library on SSD. Speed is great and all but when your dealing with a drive subject to heavy Read/Write activity like is the case with a media library (more read then write usually) you need to take this into account when selecting a home for important data.

 

Even when using RAID on an SSD the type of RAID you choose also needs to be taken into account. RAID 1 would subject both SSD disks to the same stress and workload simultaneously so there is a high likelihood that when one disk fails the other is probably not far behind it so don't drag your feet getting a replacement disk if you find yourself in this situation.

 

In a perfect world RAID 6 would be better but that would depend on the capacity and type of disk chassis you are using. RAID 6 allows for a double disk failure in the RAID set before going belly up.

 

If you want to be real anal about it you could use RAID 5 and stagger the buildup of the RAID set over a month or two so that each drive would theoretically have a different lifespan then its partners possibly buying you more time to acquire a replacement.

 

Again, this is all referring to the use of SSD drives and not so much with spinning platters which have been around long enough to prove their reliability under heavy I/O conditions.

 

There are SSD disks on the market though that offer a 5 year guarantee under heavy Read/Write conditions but at $3K+ a piece its not something most corporations can afford to invest in, let alone us Audiophiles. I wonder how long it will be before we see the high end catch on to these fancy SSD disks and proclaim multiple veils being lifted by using them :)

Link to comment

A hard drive with a metal case to shield it may turn out to be more secure against bit rot.

AMR 777 DAC, Purist Ultimate USB, PC server 4gig SOTM USB, server 2012, Audiophil Optimizer,Joule Preamp LAP150 Platinum Vcaps Bybee, Spectron Monoblocks Bybee Vcaps, Eggleston Savoy speakers, 2 REL Stentor III subwoofers, Pranawire Cosmos speaker wire, Purist Dominus Praesto cabling, Purist Anniversary (Canorus)power cables and Elrod Statement Gold power cable, VPI Aries I SDS w/Grado The Statement LP, 11kVA power isolation, 16 sound panels and bass traps TAD,RPG,GIK and Realtraps

Link to comment
A hard drive with a metal case to shield it may turn out to be more secure against bit rot.

Unfortunately it's not. ECC protected solid-state memory or multiple copies and a good backup scheme are the only answers.

Btw, if your backup scheme has overwritten a good backup with a corrupted one and you have no other way to recover it then it's not a good backup scheme. Files that should never change unless they have been corrupted should be handled accordingly.

If you see me back here chase me away.

Link to comment
My big computer at work not only corrects it memory, but it will (and has) noticed that the processor failed, stopped all the processing, brought a spare processor on line, and restarted the failed instruction.

 

Telephone Exchanges have been doing that for more than 30 years.

They even generated alarms that earned me quite a bit of extra pay for after hours recalls. (grin)

 

How a Digital Audio file sounds, or a Digital Video file looks, is governed to a large extent by the Power Supply area. All that Identical Checksums gives is the possibility of REGENERATING the file to close to that of the original file.

PROFILE UPDATED 13-11-2020

Link to comment

(grin) I think the network had rerouting and redundancy in it, but I don't remember even System-7 switches being able to handle failed processors like that. Certainly not back in the days of the dinosaurs when switches were actually mechanical relays... you old tyrannosaurus you!

 

I do remember having more than one go around with the little 3B based switches AT&T sold running 4ESS and up though... (brrr!)

 

-Paul

 

 

Telephone Exchanges have been doing that for more than 30 years.

They even generated alarms that earned me quite a bit of extra pay for after hours recalls. (grin)

Anyone who considers protocol unimportant has never dealt with a cat DAC.

Robert A. Heinlein

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×
×
  • Create New...