Peanuts to Storage Space cost?

[jabbr]

So does this mean you have personally witnessed ZFS fix a corrupted file in your music library?

 

 

Yes.

 

Snapshots can be a dangerous option if your intention is to try and run from said Snapshot for long periods of time. If Bit-Rot occurs in the Snapshot you usually need to be willing to discard whatever changes have been made to the Snapshot since it was taken. Unless of course ZFS has the ability to allow you to pick and choose which files to discard and which ones to keep while still allowing you to commit everything else that hasn't fallen victim to Bit-Rot back to the base image. I've not seen such a capability but that’s not to say one doesn't exist.

 

That isn't how it works. Basically you issue a "scrub" command, and zfs heals the pool. Take it from the experts:

https://blogs.oracle.com/bonwick/entry/zfs_end_to_end_data

A Conversation with Jeff Bonwick and Bill Moore - ACM Queue

http://www.enterprisetech.com/2014/05/22/emcs-dssd-looks-like-flash-array-runs-like-memory/

[andersole]

You could use a motherboard like this: ASRock E3C224D4I-14S Extended mini ITX Server Motherboard LGA 1150 Intel C224 DDR3 1600/1333 - Newegg.com along with a case like this: LIAN LI PC-Q26B Black Aluminum Computer Case - Newegg.com

 

Not "hot swap", but aluminum. I use aluminum hot swap bays like this: Amazon.com: iStarUSA BPN-DE350SS-RED 3x5.25 to 5x3.5 Trayless Red: Computers & Accessories but that involves more cost and building. All depends on your cost constraints.

 

Regarding drives, I like the Hitachi SAS (not SATA): HGST Ultrastar 7K4000 HUS724040ALS640 (0B26885) 4TB 7200 RPM 64MB Cache SAS 6Gb/s 3.5" Enterprise Hard Drive Bare Drive - Newegg.com and the drives are formatted as "ZFS", so just copy the old NTFS directories onto the new using "rsync" or using the file broweser, drag & drop ... which will use SMB.

 

What you do is basically create a zfs mirror device using 2 drives and then share it using SMB:

 

$ zpool create pool mirror /dev/sda /dev/sdb

$ zfs create pool/music

$ zfs set compression=on pool/music

$ zfs set sharesmb=name=music pool/music

 

then copy your music files in.

 

I think FreeNAS will help you do this with a GUI.

 

I advocate simply mirroring the drives using ZFS as a filesystem. Use FreeNAS or Ubuntu (install ZFS). You don't need a special "RAID" controller. ZFS is much much better than hardware RAID. For hard drives, SAS2 is really fine. There are LSI 9200 series controllers for $<100 which will handle 8 internal SAS drives. Get some RAM, and you are good to go.

 

I'm in the middle of a new NAS build (15 bay) which I'll post (really the only way to back up a 16Tb NAS is with another NAS well there's tape ...

 

Hi

 

Sorry some very newbie questions about ZFS.

 

Once ZFS partitioned drives (I'll say start off with 4 drives, for 1:1 mirroring) is setup with FreeNAS with one of the hardware controllers you recommended you can simply write to it over the network? Even if the CDs are ripped to an NTFS drive on another computer? Like wise for recovery the ZFS drive files can be written back to a NTFS formatted drive?

 

To use ZFS does it have to be a SAS/SAS2 drive or will SATA work as well? The latter are significantly cheaper and since ZFS has error checking/correction what is the downside to SATA (besides fewer years on warranty)?

 

Cheers

 

-Anders

[jabbr]

Hi

 

Sorry some very newbie questions about ZFS.

 

Once ZFS partitioned drives (I'll say start off with 4 drives, for 1:1 mirroring) is setup with FreeNAS with one of the hardware controllers you recommended you can simply write to it over the network? Even if the CDs are ripped to an NTFS drive on another computer? Like wise for recovery the ZFS drive files can be written back to a NTFS formatted drive?

 

No problem. You will use the SMB protocol over the network to copy files between machines regardless of the file system (ZFS vs NTFS) on each machine.

 

 

To use ZFS does it have to be a SAS/SAS2 drive or will SATA work as well? The latter are significantly cheaper and since ZFS has error checking/correction what is the downside to SATA (besides fewer years on warranty)?

 

You can use SATA as well. Best to use only either all SAS or all SATA per controller because some controllers get confused if SAS and SATA drives are used at the same time. SAS tends to have a bit more performance and a bit more reliably and cost just a bit more so either way:)

[andersole]

No problem. You will use the SMB protocol over the network to copy files between machines regardless of the file system (ZFS vs NTFS) on each machine.

 

 

 

You can use SATA as well. Best to use only either all SAS or all SATA per controller because some controllers get confused if SAS and SATA drives are used at the same time. SAS tends to have a bit more performance and a bit more reliably and cost just a bit more so either way:)

 

Great info, thank you!

[gnan]

I have no such issues with the Synology I am using which obtains copy speeds from my local SSD drive over a 1gb link to the NAS at rather impressive transfer rates.

 

I wont argue that the software RAID the majority of NAS's utilize is garbage. But for a home user its very cost prohibitive to purchase are real hardware RAID solution for the home that also has enough capacity. Were talking about adding three zeros to the end of the price tag in most cases.

 

 

What I took from the comment is not that the syno NAS boxes use "garbage RAID." But that they are just low spec computers running versions of what amounts to linux software raid.

 

You've mentioned monitoring for bit rot and the need to catch it before you sync it to your backups. How do you do this with your Synology?

[cjf]

What I took from the comment is not that the syno NAS boxes use "garbage RAID." But that they are just low spec computers running versions of what amounts to linux software raid.

 

You've mentioned monitoring for bit rot and the need to catch it before you sync it to your backups. How do you do this with your Synology?

 

Personaly, I just stick to a fairly long overwrite schedule to my other archive NAS drives in hopes that if BitRot sets in on the primay copy of data I will know before I overwrite the good copy.

 

Worst case scenerio would be that I lose an album or two that I've burned since the last Sync. I don't buy online albums very often and mainly use Redbook hard copies that I've purchased used. In this case I would just re-rip the CD.

[One and a half]

To make backups for now, I use Beyond Compare to copy files over the network or to local drives. When the program analyses the files to start the comparison, it detects corrupted files, even for FLAC, flags and won't copy these but will copy the files that are OK. Beyond Compare could be scripted, but I'm not that smart to make them, so the discovery of faulty files is a manual affair.

 

NTFS will certainly not let you now if the headers of FLAC files are damaged.

 

Is there any other backup software that a consumer can use that can:

 

- Create backups to multiple local drives and network shares

- Report/email that corrupted files exist

- Create regular backup jobs on automated basis

 

By consumer I mean software that doesn't need a Server OS with the accompanying high price tag.

 

Going to a ZFS solution doesn't work when I could drop an NTFS volume into another computer and it works fine and I can recover the data (what's left of it). If it was ZFS, I could not read anything on a Windows platform, I would need a ZFS compatible OS, like Ubuntu, OSX(?).

[DavidL]

Not sure about your second requirement as I do not backup email on my NAS but I use ChronoSync for your other requirements. I run it on a headless Mac mini to back up files from 3 computers to my NAS.

[gnan]

Personaly, I just stick to a fairly long overwrite schedule to my other archive NAS drives in hopes that if BitRot sets in on the primay copy of data I will know before I overwrite the good copy.

 

But how do you detect the errors? Do you run checksums against the files or use an automated tool? Not trying to be pedantic, genuinely curious.

[One and a half]

Not sure about your second requirement as I do not backup email on my NAS but I use ChronoSync for your other requirements. I run it on a headless Mac mini to back up files from 3 computers to my NAS.

 

Yes item 2 is critical, since a corrupted file can overwrite a good file in the backup, how's one to know?

[gnan]

Yes item 2 is critical, since a corrupted file can overwrite a good file in the backup, how's one to know?

 

If you are willing to go outside windows there are many free OSs that will do the job. Whether you can read the data disks in Windows will depend on how you set up your disks and your technical ability.

[jabbr]

Yes item 2 is critical, since a corrupted file can overwrite a good file in the backup, how's one to know?

 

That is precisely why you need a either a filesystem that will prevent this (zfs/btrfs stores checksums for each block along with the block) or another mechanism to store checksums for each file along with the file, and a software backup system that would first check and not overwrite if the checksums fail.

 

Alternatively don't worry about this because as time goes on and bitrot occurs, your hearing also deteriorates and you are less likely to notice :)

[jabbr]

 

By consumer I mean software that doesn't need a Server OS with the accompanying high price tag.

There's the rub.

 

Going to a ZFS solution doesn't work when I could drop an NTFS volume into another computer and it works fine and I can recover the data (what's left of it). If it was ZFS, I could not read anything on a Windows platform, I would need a ZFS compatible OS, like Ubuntu, OSX(?).

 

Consider installing Ubuntu on a USB drive and booting into this for recovery purposes.

[sm31]

FWIW, Synology's DSM 6.0 will have BTRFS support, for higher end machines.

 

https://m.reddit.com/r/synology/comments/3lh0cc/synology_dsm_6_new_features/

[Sal1950]

For backup I prefer the old fashion ways. I have two separate banks of storage and just run a manual rsync command from a terminal about once a week, or whenever I know I've added a bunch of data.

My only issue is I'd like to find some affordable off-site storage for backup in case of fire, flood, natural disasters, etc. Who's got the cheapest cloud storage for no more than 2t.