ZFS for data (i.e., music) storage?

[wgscott]

Anyone using ZFS for music file storage?

 

I was thinking of trying this (OS X): http://www.macupdate.com/app/mac/38847/maczfs

 

 

 

[Happy]

I use ZFS for everything in the house including music. I have 18TB of space organised as RAIDZ2. I snapshot every 60 mins and hold the snapshots for 2 weeks.

 

Regular RAID is really not very good. RAID5 has serious flaws - most people won't see the problems until they get a failure.

 

I currently use FreeNAS. MacZFS will be very good when it comes out. ZFS is exceptional technology.

 

/Paul

 

[AudioDoctor]

how would I go about utilizing this technology?

 

[wgscott]

Get an external drive. Install the software linked above, and this adds a ZFS option to Disk Utility's list of available filesystem formats for reformatting. Select ZFS and reformat the external drive, and then put your library on it and you should see fewer instances of bit rot. Not that I have ever seen even a single bit rot.

 

[Happy]

MACZFS will make it easier.

 

http://www.freenas.org/ is a server based version of ZFS. To use it you will need to be comfortable with unix/admin operations.

 

/Paul

 

[Lorenzo]

Sounds like a fun project! But given the requirements of ZFS, I wouldn’t run it on the main music player Mac… Rather I’d build a NAS which then exports shares via iSCSI or so…

 

Paul, what is your exact setup?

 

When building my current system, I considered shortly using OpenIndiana with ZFS, but mostly for budgetary reasons I then opted for a QNAP NAS…

 

[wgscott]

Yesterday I bought a 1.5 TB USB Seagate bus-powered drive at Costco, because they were $100 instead of $130. I have a 750 GB one currently formatted HFS+ connected to my airport extreme, and I do wireless backups of my music library (manually using rsync, thanks) and time-machine backups to it from my wife's laptop.

 

I figured we could use the extra space of the larger drive. I formatted the new one with zfs, just the simplest way with one pool. I just wanted to try the new file system. Unfortunately, the airport extreme doesn't recognize it, and when it was plugged directly into my mac mini, it trigged a kernel panic a couple of times (so much for their claims that it is a rare event). So I reformatted it as hfs+ (took a bit of coaxing). I'll try again when it is a bit more user-friendly, or maybe when I get a new firewire disk, where I won't have to worry about airport express incompatibility issues ...

 

 

 

[Happy]

Well there are a couple of ways to set this up.

 

This is my setup.

 

I purchased a Supermicro 16 Disk, rack mountable, PC enclosure. I installed a motherboard (I had spare), processor, 4M memory. I added 5 * 2T disks and 9 * 1Tbyte disks (I had lying around - from different manufactures).

 

I downloaded FreeNAS 8 - it resides/boots on/from a USB stick (not on a hard drive). It executes in memory.

 

Config: The 5 * 2T disks are configured as a RaidZ2 ZFS pool, the 9*1Tbyte disks are configured as another RAIDZ2 pool.

 

FreeNAS8 is now very stable - previous versions were not that lucky.

 

In ZFS one can assign several filesystems to a pool. Each filesystem can be used/exports as NFS, iSCSI, afp, windows share CIFS, FTP, SSH, TFTP. When this happens you gain the underlying reliability of ZFS.

 

For the music stuff I use afp. The MACs around the house just mount the drive as a afp folder - then you can operate as if the files were local (I have a wired GE network).

 

 

I use a ZFS filesystem for TimeMachine.

 

I also use various filesystems for work related stuff e.g. web hosting.

 

The filesytems are snapshotted every 60 mins - for a duration of 2 weeks. Each weekly snapshot is kept for 1 year. I run bitchecks every day, smart is also enabled.

 

One can use a smaller system e.g. standard PC with 3/4/5 drive bays - a nice feature of FreeNAS is that the OS and App is kept on a USB stick - no need to lose a drive bay to the OS - best kept for storage.

 

I have tested ZFS - removed drives, inserted drives, swapped drives created bit errors etc.. it works like a dream.

 

/Paul

 

 

 

 

[Happy]

I used rsync for years, I now use Goodsync - it is very good/inexpensive and preserves those apple idiosyncratic things that Unix doesn't like. I recommend it highly.

 

/Paul

 

[wgscott]

Do you know if there is a zfs analogue to hfscompression (i.e., storing data in the resource fork)?

 

It is not only a cool party trick, but saves considerable disk space, especially with ascii files.

 

[Happy]

ZFS supports explicit compression independently for each 'filesystem'.

 

The choices are:

 

 

 

Original Data Size : 412MB

 

lzjb : 312MB Compression ratio : 1.32

 

gzip : 293MB Compression ratio : 1.41

 

gzip9 : 292MB Compression ratio : 1.41

 

On office like data sets. You can switch on and off compression for the filesystem whenever you like.

 

I'm not sure how hfscompression would work over afp.

 

ZFS also supports Deduplication although I have never used it - it is the big feature in Enterprise Storage. FreeNAS 8 I don't believe supports dedup at the moment - encryption and dedup are rumored in release 8.1

 

 

/Paul

 

[Lorenzo]

Hi wgscott,

 

have you tried to do "zpool export " before removing the device? Maybe unmounting in Finder is not enough…

(And don’t forget to do zpool import when reattaching the drive)

Haven’t tried it yet, just skimming the man pages…

 

btw, your zsh-templates save me a lot of time every day, thanks!

 

@Paul: nice!

 

[wgscott]

Next time I need to add a disk directly to the computer, rather than my airport extreme, I'll try it again. The export command is probably what I would need. Maybe I could write a launchd or at least and ejection script...