Music file corruption without a change in checksum

[Teresa]

I have a question, not about checksums but occasional loss of bytes on previously saved backups.

 

I exclude my music files from Time Machine backups and have then in separate folders on my Seagate Slim Mac drive, so the music can be on both my Mac Mini and my Gateway PC laptop. This is because the Gateway PC Laptop can't download from Mac Time Machine backups but can if in normal folders.

 

When I plug in the Seagate Slim Mac I always check to make sure the music files on my computer and backups match items, size and bytes. Occasionally while the items and size match there will be a loss of some bytes on the backup and I delete the backup file and re-save the file. I am curious what causes this loss of bytes? I don't know how to do checksums and also wonder if this loss of bytes could change it?

 

I finally had an example of what I am talking about:

 

The folder: Dave Brubeck Quartet (downloaded from Acoustic Sounds Super Hi-rez May 2, 2014)

 

On the hard drive of my Mac Mini: 1,706,691,239 bytes (1.71 GB on disk) for 12 items

 

On my Seagate Slim Mac backup: 1,706,689,191 bytes (1.71 GB on disk) for 12 items

 

I deleted the folder on the Seagate and re-saved it from my Mac Mini and they both match again.

 

I backup to the Seagate whenever I download something new, usually many times per month. About once every two months or so one of my backup files losses bytes, is this normal or is there something wrong with the Seagate drive?

 

It's different files each time and it has effected all my file formats: dff, dsf and aiff.

[wgscott]

Well my drive where this was happening was failing (still is). Fortunately I have multiple backups. I am storing everything in yet another backup off-site on a zsf NAS filesystem; copying as I type...

 

This is why having checksums is so important. You are guessing (probably correctly) that the larger file is the uncorrupted one. But what if it wasn't -- and you propagated the damaged file? What if you didn't happen to catch it?

[Don Hills]

That behaviour could be quite normal.

I think it likely that it is the size on the PC that is changing over time, not the size on the backup. I'm not very familiar with Mac filesystems, but I suspect the resource fork is being updated. You need to save the size information at the time you make the backup. On Windows, there's a simple command-line command to write a full list of folders and file sizes into a text file. Hopefully some Mac expert can give you a similar syntax for Mac.

[dalethorn]

I finally had an example of what I am talking about:

 

The folder: Dave Brubeck Quartet (downloaded from Acoustic Sounds Super Hi-rez May 2, 2014)

 

On the hard drive of my Mac Mini: 1,706,691,239 bytes (1.71 GB on disk) for 12 items

 

On my Seagate Slim Mac backup: 1,706,689,191 bytes (1.71 GB on disk) for 12 items

 

I deleted the folder on the Seagate and re-saved it from my Mac Mini and they both match again.

 

I backup to the Seagate whenever I download something new, usually many times per month. About once every two months or so one of my backup files losses bytes, is this normal or is there something wrong with the Seagate drive?

 

It's different files each time and it has effected all my file formats: dff, dsf and aiff.

 

Macs put lots of temporary hidden files into folders with the visible files. This usually happens in 1-to-1 file correspondence only when the files are individually accessed, or in folders only (without creating temps for each file in the folder) when the folder is accessed. This is an extreme nuisance with plug-in USB drives, when their capacity is limited.

 

Windows PC's, at least up to Windows 7, did not create these nuisance files.

[kumakuma]

I finally had an example of what I am talking about:

 

The folder: Dave Brubeck Quartet (downloaded from Acoustic Sounds Super Hi-rez May 2, 2014)

 

On the hard drive of my Mac Mini: 1,706,691,239 bytes (1.71 GB on disk) for 12 items

 

On my Seagate Slim Mac backup: 1,706,689,191 bytes (1.71 GB on disk) for 12 items

 

I deleted the folder on the Seagate and re-saved it from my Mac Mini and they both match again.

 

I backup to the Seagate whenever I download something new, usually many times per month. About once every two months or so one of my backup files losses bytes, is this normal or is there something wrong with the Seagate drive?

 

It's different files each time and it has effected all my file formats: dff, dsf and aiff.

 

My guess is that one folder has a .DS_Store file (hidden file) in it and the other doesn't. I don't think it's anything to worry about. You can find out more about it here:

 

https://en.wikipedia.org/wiki/.DS_Store

[Teresa]

Thanks for the link Tom, that would be a possible explanation if it effected all music files, it does not.

 

It is one or two music files out of over a thousand that lose bytes about once every two months or so, and different ones each time. They are usually older music files that have matched exactly, sometimes for over a year and then shazam the backup loses a few thousand bytes. I'm afraid it is what Wgscott suggests, my Seagate backup drive is slowly beginning to fail.

 

Thanks Wgscott, Tom, Don, and Dale.

[goldsdad]

Thanks for the link Tom, that would be a possible explanation if it effected all music files, it does not.

 

It is one or two music files out of over a thousand that lose bytes about once every two months or so, and different ones each time. They are usually older music files that have matched exactly, sometimes for over a year and then shazam the backup loses a few thousand bytes. I'm afraid it is what Wgscott suggests, my Seagate backup drive is slowly beginning to fail.

 

Thanks Wgscott, Tom, Don, and Dale.

 

Your earlier example was of differing folder sizes, which would very likely be due to hidden .DS_Store files being created or changed, and that certainly can be the explanation for a given folder changing size while others don't change. But now you're stating that some actual music files have changed size. Have you been comparing the size of the music files themselves on the two drives, or just the size of the folders?

[mansr]

Thanks for the link Tom, that would be a possible explanation if it effected all music files, it does not.

 

It is one or two music files out of over a thousand that lose bytes about once every two months or so, and different ones each time. They are usually older music files that have matched exactly, sometimes for over a year and then shazam the backup loses a few thousand bytes. I'm afraid it is what Wgscott suggests, my Seagate backup drive is slowly beginning to fail.

 

That is an extremely unlikely failure mode for a hard drive. My money is on something benign such as resource forks or hidden files.

[kumakuma]

Thanks for the link Tom, that would be a possible explanation if it effected all music files, it does not.

 

I have been using a Mac for years but I'm still not clear why OS X creates a .DS_Store file in some directories but not others. I'm sure I could figure it out but I can't be bothered.

 

 

 

It is one or two music files out of over a thousand that lose bytes about once every two months or so, and different ones each time. They are usually older music files that have matched exactly, sometimes for over a year and then shazam the backup loses a few thousand bytes. I'm afraid it is what Wgscott suggests, my Seagate backup drive is slowly beginning to fail.

 

Please see goldsdad's response above. If you have files (not folders) with different sizes, you have a problem.

[Audio_ELF]

I have been using a Mac for years but I'm still not clear why OS X creates a .DS_Store file in some directories but not others. I'm sure I could figure it out but I can't be bothered.

AFAIK the .DS_Store files are created on non-OSX native drives as required. It's required if you have files where thumb nails are created along with other extended attributes.

 

You can also disable the creation of .DS_Store files on external drives but if you copy and paste a folder it will create / copy .DS_Store files.

[Bob Stern]

Teresa:

 

Since the folder in question only has 12 files, it should be trivial for you to compare the size of each individual file. A quick method is to open an Inspector window in the Finder by pressing option-command-I (instead of the usual command-I for Get Info). Then select one file at a time to view its properties.

 

In addition, to inspect whether there is a difference between the .DS_Store files or any other hidden files in the two folders, download a copy of the program FileXaminer ("donation-ware").

 

(Eloise: I believe a .DS_Store file also would be created if you changed any of the Finder view properties of a folder, such as which column it was sorted by.)

[wgscott]

To turn it off:

 

defaults write com.apple.desktopservices DSDontWriteNetworkStores true

[jabbr]

Well my drive where this was happening was failing (still is). Fortunately I have multiple backups. I am storing everything in yet another backup off-site on a zsf NAS filesystem; copying as I type...

 

This is why having checksums is so important. You are guessing (probably correctly) that the larger file is the uncorrupted one. But what if it wasn't -- and you propagated the damaged file? What if you didn't happen to catch it?

 

yep

 

zfs will often tell you that it is repairing a drive, and for me this is an early sign that the drive is starting to fail, so I replace them when this starts occuring. It tell you that you have a certain number of checksum errors during a scrub (the scrub does the repair).

[Teresa]

...Have you been comparing the size of the music files themselves on the two drives, or just the size of the folders?

 

I did it the beginning, however it's a lot of work.

 

I have two main folders (DSD with 70 folders) and (PCM with 98 folders)

 

Here is what I do:

 

1) I check to see if the two folders on my Mac Mini matches exactly the same two folders on Seagate Slim Mac backup drive. If they do, I'm done I breath a sigh of relief.

 

2) If one main folder does not match, then I have to individually open each subfolder in the effected main folder on the Seagate Slim Mac backup drive until I find the one with the missing bytes. Rather than comparing each individual music file in the folder I just replace the folder on the backup.

 

Teresa:

 

Since the folder in question only has 12 files, it should be trivial for you to compare the size of each individual file.

 

The Dave Brubeck Quartet folder that lost bytes I reported on the 5th was re-saved that day, so I can't recheck that one.

 

I saved some new downloads yesterday to my Seagate Slim Mac backup drive and my DSD and PCM folders matched exactly. I will have to wait until the next time one of my back-up files losses a few thousand bytes. At the current rate, it could easily be two or three months from now before it happens again. But if I remember correctly it was usually one or two files within the folder that has less bytes than the ones on my Mac Mini. It just was easier and faster to replace the entire folder with the less bytes.

[dalethorn]

If I were backing up on a Mac, I would not bother to check, because the Mac will always hide the '.' files even when you tell the system not to hide files. On a PC, you can make the computer show all files, but not on a Mac.

[Paul R]

If I were backing up on a Mac, I would not bother to check, because the Mac will always hide the '.' files even when you tell the system not to hide files. On a PC, you can make the computer show all files, but not on a Mac.

 

ls -a works just as well on a Mac as on any other Unix based computer, and will show you all the files in a particular directory.

 

What Theresa is apparently seeing is a change in the reported size of a folder. Without a full listing of the folder's contents, there is no way to tell what really happened. However, the conjecture that MacOS created a ds_store folder for some reason is probably correct.

 

-Paul

[dalethorn]

ls -a works just as well on a Mac as on any other Unix based computer, and will show you all the files in a particular directory.

What Theresa is apparently seeing is a change in the reported size of a folder. Without a full listing of the folder's contents, there is no way to tell what really happened. However, the conjecture that MacOS created a ds_store folder for some reason is probably correct.

-Paul

 

You may indeed have to go to the Unix command line in Mac, but you don't have to go to DOS on a PC, to see the system files. At least up to Win7.

[kumakuma]

You may indeed have to go to the Unix command line in Mac, but you don't have to go to DOS on a PC, to see the system files. At least up to Win7.

 

The current default in OS X is to hide the hidden files but it is really easy to change things so they show up all the time:

 

How to show hidden files in Mac OS X Finder - How to - Macworld UK

[wgscott]

If I were backing up on a Mac, I would not bother to check, because the Mac will always hide the '.' files even when you tell the system not to hide files. On a PC, you can make the computer show all files, but not on a Mac.

 

All unix systems behave this way. That is the whole point of the dot file.

[kumakuma]

I did it the beginning, however it's a lot of work.

 

I have two main folders (DSD with 70 folders) and (PCM with 98 folders)

 

Here is what I do:

 

1) I check to see if the two folders on my Mac Mini matches exactly the same two folders on Seagate Slim Mac backup drive. If they do, I'm done I breath a sigh of relief.

 

2) If one main folder does not match, then I have to individually open each subfolder in the effected main folder on the Seagate Slim Mac backup drive until I find the one with the missing bytes. Rather than comparing each individual music file in the folder I just replace the folder on the backup.

 

This seems like a lot of work. This is the tool I use for tasks like this:

 

SyncTwoFolders

 

One thing I especially like about this utility is that it has a simulation mode so you can see what will be changed before any changes are actually made.

[kumakuma]

All unix systems behave this way. That is the whole point of the dot file.

 

.wgscott

[kumakuma]

.wgscott

 

Sorry Bill, I was channeling Alex there for a moment...

[dalethorn]

The current default in OS X is to hide the hidden files but it is really easy to change things so they show up all the time:

How to show hidden files in Mac OS X Finder - How to - Macworld UK

 

Bad enough that you have to enter commands in Terminal, but worse is that the Mac keeps making tons of useless hidden files with use, unlike a PC. To get photos onto my Mac to load onto a iPhone or iPad, it makes thousands of hidden files. What would be the benefit of unhiding thousands of files that keep cropping up again and again? It's an O/S designed to sabotage users' efforts to have a clean system. But then again, users who know the difference between a personal computer O/S and a corporate O/S (WinNT and up unfortunately) already have a clue.

[dalethorn]

All unix systems behave this way. That is the whole point of the dot file.

 

Sure - "it's supposed to be that way". Uh, yeah, right.

[kumakuma]

Bad enough that you have to enter commands in Terminal, but worse is that the Mac keeps making tons of useless hidden files with use, unlike a PC. To get photos onto my Mac to load onto a iPhone or iPad, it makes thousands of hidden files. What would be the benefit of unhiding thousands of files that keep cropping up again and again? It's an O/S designed to sabotage users' efforts to have a clean system. But then again, users who know the difference between a personal computer O/S and a corporate O/S (WinNT and up unfortunately) already have a clue.

 

.dalethorn