ZFS file system again, but this time with usage experience.

[AudioDoctor]

I have tried every single OS that there is, within reason, on my music server. AudioLinux, Euphony, Windows (ugh) Arch variants of Linux, and finally Ubuntu. I came to the decision to try Ubuntu because the JCAT cards do not work in Euphony, yet. That support is coming, says Euphony, in the next version which will be based on a newer Linux kernel. I like Euphony, it made my server sound amazing. I think that once the JCAT cards work with it, that will be a great OS to use.

 

Enter Ubuntu and ZFS. ZFS is, very simply, a file system that uses checksums and snapshots to ensure data integrity by checking the files against either a mirror or parity disks and if it finds an error, checks the other and compares it to snapshots to find the right file, and fixes the error. This is a very elementary description of it. But when it comes to a music server, which holds large amounts of data over long periods of time, you can see the benefits of such a file system. The file system is fault tolerant, it can warn you of impending disk failure, it can be expanded, and it can use any disks. Best part, with just one command entered via command line it works with very minimal intervention from me.

 

So far, both Roon server and HQPlayer work on it with no problems that aren't caused by me. Remotely managing an ubuntu server via SSH and command line is new to me and I make mistakes. With a ZFS file system, rolling back to a previous snapshot, which it takes automatically when I make a change, is easy as well. I am using Ubuntu 20.04 LTS (Long Term Support) with ZFS on Root which gives the OS and all programs on it the same benefits as the zpool my music files are on.

 

How does all this file integrity goodness sound? Great. Fantastic. Natural. Beautiful. I have no idea how much of that is the bare bones Ubuntu installl, the JCAT Network XE card, or the ZFS file system, or the fact that I can relax and enjoy the music knowing my files are not being degraded over time by invisible bit rot.

 

important bits of the picture below. 

 

rpool is the root of the OS
bpool is the boot section of the disk

mypool is where my music files reside on two identical 10TB Seagate Ironwolf drives in a mirror.

 

You can see that sometime overnight the OS did a "scrub" to check for any data errors and fix them. It found none. If it had found some it would have fixed them, and told me about them.

 

I am considering ditching my NAS (iosafe 1019) and building one that uses ZFS instead of the BTRFS of the Synology Disk Manager, because I can't think of a reason not to use ZFS for that purpose as well.

 

Link for people like me for whom this is all new information:

https://itsfoss.com/what-is-zfs/

 

one last edit: ZFS is not for a low powered NUC. It requires some horsepower and lots of RAM.

[Jud]

15 minutes ago, AudioDoctor said:

I am considering ditching my NAS (iosafe 1019) and building one that uses ZFS instead of the BTRFS of the Synology Disk Manager, because I can't think of a reason not to use ZFS for that purpose as well.

 

Be a shame to waste that fireproof, virtually theftproof goodness.  Wonder if you can install ZFS on the ioSafe?

[AudioDoctor]

Just now, Jud said:

 

Be a shame to waste that fireproof, virtually theftproof goodness.  Wonder if you can install ZFS on the ioSafe?

 

I agree completely, those features are why I bought it in the first place. If I can, it will be without the Synology software as that is BTRFS only.

[AudioDoctor]

While I will admit this isn't a sexy topic like many others here, I think this one is just as important if we hope to maintain a healthy collection of locally stored music over long periods of time.

[AudioDoctor]

@Jud Turns out, iosafe makes a solution to the problem.

 

https://iosafe.com/products/server-5/

[Jud]

9 hours ago, AudioDoctor said:

@Jud Turns out, iosafe makes a solution to the problem.

 

https://iosafe.com/products/server-5/

 

Of course they do! Cheap at twice the price!

[AudioDoctor]

I should add that some googling about ZFS can make it look supremely complicated and hard to understand. And it can be, if you have ridiculously large pools of data with tons of read and write going on constantly. A simple music server, with enough RAM, is stupid simple and doesn't require all the complications.

 

If I were to make a change to my set up, I would get another nvme drive and mirror my boot drive as right now that's on one drive and therefore a failure point.

[Jud]

13 hours ago, AudioDoctor said:

I should add that some googling about ZFS can make it look supremely complicated and hard to understand. And it can be, if you have ridiculously large pools of data with tons of read and write going on constantly. A simple music server, with enough RAM, is stupid simple and doesn't require all the complications.

 

If I were to make a change to my set up, I would get another nvme drive and mirror my boot drive as right now that's on one drive and therefore a failure point.

 

There's probably a simple answer to this, but if you mirror a boot drive that gets corrupted, won't the mirror fail too?

[AudioDoctor]

3 hours ago, Jud said:

 

There's probably a simple answer to this, but if you mirror a boot drive that gets corrupted, won't the mirror fail too?

 

Corrupted how?  Right now the single drive gets checked for errors just like the pool, it has checksums and will correct any errors. Its partitioning scheme is odd compared to the file systems we are used to as it creates a bunch of different partitions that allow snapshotting, rolling back of the OS without touching user data, parity (I think) etc... I think it would warn me of impending disk failure just like any of the disks in the music pool. So no, I don't think so, I think it would self correct and warn rather than just fail if the drive failed.

[Jud]

54 minutes ago, AudioDoctor said:

 

Corrupted how?  Right now the single drive gets checked for errors just like the pool, it has checksums and will correct any errors. Its partitioning scheme is odd compared to the file systems we are used to as it creates a bunch of different partitions that allow snapshotting, rolling back of the OS without touching user data, parity (I think) etc... I think it would warn me of impending disk failure just like any of the disks in the music pool. So no, I don't think so, I think it would self correct and warn rather than just fail if the drive failed.

 

Reason I ask is that though I supposedly have SMART drives, warnings of impending failure typically come only after sectors have already experienced errors, and things go downhill rapidly from there. I understand you're talking about solid state rather than spinning drives, and don't know how those begin to fail (thankfully).  So I wonder how ZFS copes with physical drive errors (such as sectors on an NVme that can no longer be written to, just to take a hypothetical).

[AudioDoctor]

1 hour ago, Jud said:

 

Reason I ask is that though I supposedly have SMART drives, warnings of impending failure typically come only after sectors have already experienced errors, and things go downhill rapidly from there. I understand you're talking about solid state rather than spinning drives, and don't know how those begin to fail (thankfully).  So I wonder how ZFS copes with physical drive errors (such as sectors on an NVme that can no longer be written to, just to take a hypothetical).

 

yes, and ZFS is engineered to eliminate those errors and keep your system running without data loss until you can replace the drive. If you see my screenshot above, both the r(oot) and b(oot) pool have checksums. If it does a "scrub" and finds the checksum doesn't match, I believe in a single drive scenario it compares the value to the snapshots to decide which one is the correct one and fixes it from the parity partition. If the boot drives were mirrored it would compare them to each other and the snapshots, decided which is correct, and fix the errors, and then alert you the user that your disk is giving errors. Then you can decide what to do about it. If the drive just up and dies one day with no warning, then in a mirror the system would still boot.

 

For a music server, ZFS is probably massive overkill as its designed for absolutely massive amounts of data exponentially larger than a music server will handle. Think Qobuz's internal libraries level of data. Or Amazon. It is as fault tolerant as a file system can be these days.

[jabbr]

My home photos are certainly important enough to use ZFS and mirrored drives. Storage is cheap, the cost is negligible, its built into Ubuntu so why not?

[AudioDoctor]

41 minutes ago, jabbr said:

My home photos are certainly important enough to use ZFS and mirrored drives. Storage is cheap, the cost is negligible, its built into Ubuntu so why not?

 

And that is another good point. On my NAS I have a lot more than music file backups as well as information that I am legally required to keep for a charitable org.

[jabbr]

23 hours ago, AudioDoctor said:

I should add that some googling about ZFS can make it look supremely complicated and hard to understand. And it can be, if you have ridiculously large pools of data with tons of read and write going on constantly. A simple music server, with enough RAM, is stupid simple and doesn't require all the complications.

 

If I were to make a change to my set up, I would get another nvme drive and mirror my boot drive as right now that's on one drive and therefore a failure point.

All the writing may make is seem complicated but its bulletproof and easy to use. I have literally 15 year pools from back in the openindiana/opensolaris days that I've migrated to ubuntu and are still running ... I swap in new 4TB SAS drives as needed.

[AudioDoctor]

2 minutes ago, jabbr said:

All the writing may make is seem complicated but its bulletproof and easy to use. I have literally 15 year pools from back in the openindiana/opensolaris days that I've migrated to ubuntu and are still running ... I swap in new 4TB SAS drives as needed.

 

I tested it on my 2011 mini for a while before switching my music server to it, so far not a single problem.

[AudioDoctor]

A few more links:

https://arstechnica.com/information-technology/2019/10/a-detailed-look-at-ubuntus-new-experimental-zfs-installer/

 

https://wiki.ubuntu.com/Kernel/Reference/ZFS

[Jud]

I used to run FreeBSD a lot before the lack of audio drivers started to wear on me, and they switched to ZFS some time ago.

[AudioDoctor]

1 minute ago, Jud said:

I used to run FreeBSD a lot before the lack of audio drivers started to wear on me, and they switched to ZFS some time ago.

 

My Ubuntu machine, running ZFS on Root, is also running HQPlayer and Roon Server, and doing it very well.

[Jud]

7 hours ago, AudioDoctor said:

If the boot drives were mirrored it would compare them to each other and the snapshots, decided which is correct, and fix the errors, and then alert you the user that your disk is giving errors.

 

This the part I'm most curious about, particularly how the determination is made of which drive is correct. I imagine it's not difficult, I'm just not familiar with the concepts behind the algorithms.

[AudioDoctor]

4 minutes ago, Jud said:

 

This the part I'm most curious about, particularly how the determination is made of which drive is correct. I imagine it's not difficult, I'm just not familiar with the concepts behind the algorithms.

 

This is my understanding, and I think it assumes that you have a history of snapshots over a long period of time before a drive fails, but by checking either the other drive, or a parity drive if you have more than two, and then comparing them to the history of snapshots it can decide which is the new change and which is the historically accurate version. It's not a random pick, there is some logic behind it. It takes snapshots much like Apple's Time Machine does, and keeps them for a period of time.

[AudioDoctor]

Oh, and CUDA works as well on ZFS on Root.

[AudioDoctor]

I have had my music server up and running on ZFS on Root for about a week now. I just checked a list of snapshots and I scrolled and scrolled and scrolled through the terminal until my scroll finger got sore. It keeps hourly, daily, weekly, and eventually, monthly and yearly.

[jabbr]

18 hours ago, AudioDoctor said:

 

This is my understanding, and I think it assumes that you have a history of snapshots over a long period of time before a drive fails, but by checking either the other drive, or a parity drive if you have more than two, and then comparing them to the history of snapshots it can decide which is the new change and which is the historically accurate version. It's not a random pick, there is some logic behind it. It takes snapshots much like Apple's Time Machine does, and keeps them for a period of time.

snapshots don't protect against drive failure, rather they allow you to go back and retrieve a file that you've accidentally deleted or modified. mirroring protects against drive failure.

[jabbr]

18 hours ago, Jud said:

 

This the part I'm most curious about, particularly how the determination is made of which drive is correct. I imagine it's not difficult, I'm just not familiar with the concepts behind the algorithms.

 

forget snapshots. when two drives are mirrored, zfs stores the hash of each block. when a block is read into memory, the hashes are compared. if the hashes are different, and the hash from one drive doesn't match the stored hash, then there is a fault. when you scrub a pool, every block is checked.

[AudioDoctor]

33 minutes ago, jabbr said:

snapshots don't protect against drive failure, rather they allow you to go back and retrieve a file that you've accidentally deleted or modified. mirroring protects against drive failure.

 

yes I am aware of that, nor did I say that. I said that the system uses them to compare as well as the mirror or parity drives to decide which is correct.