Article: Reevaluating My Music Storage - Update

[The Computer Audiophile]

View full article

[DuckToller]

Wow, 10GigE in the private environemnt  and you speak of saturating. Let that SINK in !
I remember upgrading our European network of IXPs into 10GE between 2004 (planning) and 2006 (starting the rollout) for IP interconnection. It served hundreds of thousands of eyeballs back then ...
You hint of saturation tells me that 100GE won't be too far away in your timeline ... ;-)

Did you take in consideration to use RAID 10 ? Or would that considered only as a solution for the paranoid?

[The Computer Audiophile]

4 minutes ago, DuckToller said:

Wow, 10GigE in the private environemnt  and you speak of saturating. Let that SINK in !
I remember upgrading our European network of IXPs into 10GE between 2004 (planning) and 2006 (starting the rollout) for IP interconnection. It served hundreds of thousands of eyeballs back then ...
You hint of saturation tells me that 100GE won't be too far away in your timeline ... ;-)

Did you take in condideration to use RAID 10 ? Or would that considered only as a solution for the paranoid?

Hi Tom, I hear you! It's crazy. 

 

The 100GbE MikroTik CRS504-4XG-IN might be in my future (thanks to @jabbr telling everyone about it).

 

RAID10 would certainly work well, but cut down on my usabel storage space of course. I'm OK with replacing a failed drive and runnign a sync to restore from the attached USB drives. I think this is a good trade off for the gain in speed in everyday use. 

 

 

[Account Closed]

Might I suggest that you make two of the USB backups and rotate one of them to a bank safety deposit box on a periodic basis. This would give you triple redundancy with an off-site  physical component. 

[The Computer Audiophile]

19 minutes ago, bobflood said:

Might I suggest

Yes, you may suggest :~)

[Account Closed]

3 minutes ago, The Computer Audiophile said:

Yes, you may suggest :~)

Well, even if you don't do it, I'll feel better for trying. :)

[The Computer Audiophile]

Just now, bobflood said:

Well, even if you don't do it, I'll feel better for trying. :)

I'm with you :~)

[bobfa]

I think I see why you went down the NAS master path.  It seems to fit how you have been working for a while and this gives you the availability you need. Very nice.

 

I am interested in the reasoning for using ZFS, I have not studied it at all.  Also, more discussion on 10Ge would have a lot of value to some of us.  We are really running mini data centers in our homes.  That is part of the reason I went back to UnFi on mine.  Stability and performance at the core.

 

Bob

 

And still you keep finding great new music!

 

 

[jabbr]

Personally I mirror all my of my ZFS drives. My photos are just mirrored. Music is mirrored and the mirrors are striped. I also backup to the cloud.

 

ZFS uses RAM as read cache. An NVME cache is useful when doing lots of fast writes. 
 

100Gbe approaches the speed of local PCIe and there is 200,400,800 of your PCIe bus can handle it.

 

NVME arrays are the next thing but pricey — think $4-6K but that’s where 100Gbe hits the mainstream (NVMEoF) … the network becomes an extension of the PCIe bus. SmartNICs do it all behind the scenes and the data looks/is local to the CPU.

 

One reason I build my own NAS is that I can put 16-64 Gb RAM and a SAS3 card talks to the external SAS3 arrays which can be daisychained. 
 

When drives start to fail I yank them and replace, the mirror self heals.

[AudioDoctor]

I am curious to see what you did to get 10G ethernet to the MacBook.

[MFJG]

10Gb for Apple, did you use Atto or Sonnet? I’ve had success with both companies on Apples M chips. I’m not at 10Gb with Ravenna yet, that’s very interesting. 

[Bertel]

Many thanks for mentioning / pointing me to iDrive. Prices for online storage / backup space and solutions have really come down even further recently, and iDrive‘s backup features/functionalities make it very easy.

 

I use a Synology DS1821+, have subscribed to iDrive and have just downloaded the iDrive app right onto my Synology NAS - love to manage backup right there directly. Although I‘m just at ~9TB yet, the calculated backup upload duration atm is 62 days and 3 hours though 🤣

[Jud]

Quote

If my local music files are unavailable for a couple days, while the restore is running, I'm OK with that.

 

Heh, I winced when I saw that sentence. 🙂 But good that you found the solution that works best for you. Also, the sheer volume of music you've got relative to me makes it much easier for me than for you to consider options that multiply storage needs.

 

Adding myself to the list of folks interested in reading the next article about 10Gbps networking to the Mac.

[firedog]

Started a trial of Backblaze. 

I have 5TB to backup, most is music files.

So far 4 days have gone by and only about 200GB have uploaded. At this pace, the trial will end before my upload is even finished. 

I only have an ADSL line, but this is still incredibly slow.

[The Computer Audiophile]

3 hours ago, firedog said:

Started a trial of Backblaze. 

I have 5TB to backup, most is music files.

So far 4 days have gone by and only about 200GB have uploaded. At this pace, the trial will end before my upload is even finished. 

I only have an ADSL line, but this is still incredibly slow.

I uploaded 16TB to iDrive in about 3 days. 

[botrytis]

This is only useful if one doesn't have a data cap on their internet connection. Comcast is 1.2 TB cap. Others are more or less. For many, cloud storage is not an option because of that.

[jrobbins50]

I have 43Tb to backup, 40Tb of which are music files. The cloud is just not feasible with that quantity of data and the Comcast data cap. I’m using USB3.0 to transfer from my NAS through another PC to 16Tb drives, which I will store offsite. Got through the music files in 9 days.  I did it in 1.5-2.5Tb chunks and lost about 100Mb of data each time, allegedly to a file name that was too long and thus skipped over by Windows. It seems they were artwork files in virtually each case, so not a big issue — but keeping data integrity over such a large amount of data is truly difficult. JCR 

[Mike Rubin]

22 hours ago, Bertel said:

Many thanks for mentioning / pointing me to iDrive. Prices for online storage / backup space and solutions have really come down even further recently, and iDrive‘s backup features/functionalities make it very easy.

 

I use a Synology DS1821+, have subscribed to iDrive and have just downloaded the iDrive app right onto my Synology NAS - love to manage backup right there directly. Although I‘m just at ~9TB yet, the calculated backup upload duration atm is 62 days and 3 hours though 🤣

Idrive will ship you a USB drive and a return mailer. They then will upload your files to their servers locally, which takes about a week after they get the drive back. That works well - as it must, because that really is the only way that cloud storage possibly works with a large collection, a data cap, and asymmetric internet, which most of us have. 

[Jud]

9 hours ago, firedog said:

Started a trial of Backblaze. 

I have 5TB to backup, most is music files.

So far 4 days have gone by and only about 200GB have uploaded. At this pace, the trial will end before my upload is even finished. 

I only have an ADSL line, but this is still incredibly slow.

 

Interesting. For me, upload on another service brought the rest of my network speed to a screeching halt, while uploading to Backblaze went fast and didn't interfere with the rest of the network.

 

Do you have QoS settings or anything else that may be limiting speed to less than the maximum your ADSL is capable of?
 

I believe Backblaze offers to ship a USB drive for your files, at least in the US, though I don't remember what the cost was.

[Bertel]

3 hours ago, Mike Rubin said:

Idrive will ship you a USB drive and a return mailer.

Thanks @Mike Rubin ! I had seen this option, but had assumed that it wasn‘t valid for Europe (I live in Germany). Now have just requested iDrive Express from the German version of the website, and it seems to work. Will give it a try, and see how it goes - we‘ll see, will be happy to report here.

 

EDIT: If someone knows about a similar provider / service in Europe, please let me know.

 

EDIT 2: Will be interesting whether I can then do an incremental / differential update over internet then once the bulk upload through iDrive Express is up, else the whole thing is kinda moot…

[miguelito]

I am brutally basic:

• Main library on my macpro - I download purchases, rip, manage metadata, and curate Roon db here (AIFF+DSF files)
• This library gets backed up on TimeMachine as well as four copies in sequence to external drives (with Chronosync)
• Library gets copied with Chronosync to a NUC with ROCK where Roon Core runs

 

So library is sourced for playback from the SSD in the NUC/ROCK, and there are 7 full copies of it (some can get as old as ~1month as I cycle over drives).

[Jud]

12 hours ago, firedog said:

Started a trial of Backblaze. 

I have 5TB to backup, most is music files.

So far 4 days have gone by and only about 200GB have uploaded. At this pace, the trial will end before my upload is even finished. 

I only have an ADSL line, but this is still incredibly slow.

 

3 hours ago, Jud said:

 

Interesting. For me, upload on another service brought the rest of my network speed to a screeching halt, while uploading to Backblaze went fast and didn't interfere with the rest of the network.

 

Do you have QoS settings or anything else that may be limiting speed to less than the maximum your ADSL is capable of?
 

I believe Backblaze offers to ship a USB drive for your files, at least in the US, though I don't remember what the cost was.

I checked and no, unfortunately Backblaze doesn't offer a hard drive for initial backup, only for restoring if you've lost a bunch of files.

[Bertel]

On 2/6/2023 at 9:56 PM, Bertel said:

Thanks @Mike Rubin ! I had seen this option, but had assumed that it wasn‘t valid for Europe (I live in Germany). Now have just requested iDrive Express from the German version of the website, and it seems to work. Will give it a try, and see how it goes - we‘ll see, will be happy to report here.

 

Just as an fyi, in case anybody else was interested: iDrive just notified me that the USB drive for iDrive Express was shipped today. It seems that for Europe they ship from Ireland, which should make it both fast and convenient. Good!

[Jud]

General information:

 

- Ars Technica likes iDrive. https://arstechnica.com/information-technology/2023/02/ars-archivum-top-cloud-backup-services-worth-your-money/  While Backblaze, their second choice, has faster upload/download, they aren't pleased with the Backblaze app UI. (It's adequate for my purposes, though I can understand their complaints about it.) iDrive is offering large discounts on 1 year of service (at least here in the US). After that, iDrive is very price competitive up to 10 TB for personal backup (no fancy S3 compatibility), and even for the S3 compatible plan that opens up capabilities for lots of nice 3rd party apps it's reasonably price competitive up to 2 TB. After that, the price difference with Backblaze's unlimited personal backup plan is probably too great to ignore.

 

- I'm very, very intrigued by Storj, which seems reasonably price competitive up to a couple of TB and has a uniquely resilient and secure method of storing your backups.  https://www.storj.io/pricing  https://www.storj.io/how-it-works

[The Computer Audiophile]

Hi Guys, iDrive saved me last night, BIG TIME!

 

I backed up 16TB to iDrive. I also backed up 16TB to local drives, using QNAP's Hybrid Backup Sync app. The app works great and I can schedule it or run it manually etc... One the backups were done, I added four more 6TB drives to the QNAP, deleted the storage pool, and recrated it as RAID0 with 8 drives. Bazing speed now. 

 

I then copied all the data from my local storage, back to the QNAP. I checked the folder sizes and file numbers etc... at backup and restore times, and thought it was all good. The number is never identical because of hidden file variables. 

 

When I opened Roon, I checked my Three Blind Mice albums. The Supreme 1500 collection should have 100 albums. There were only 98. After a long troubleshooting session, I discovered that tracks, albums, or folders with quotation marks in the name, were not backed up to the local storage. Then I looked as some other albums in my Atmos WAV file collection, because I know some of the classical albums have "crazy" titles. I found more albums and tracks that were missing!

 

I opened the iDrive cloud backup, and found everything was there! I initiated a restore of the missing albums and tracks, and I had everything back to normal by the time I went to bed. Fortunately, I'm pretty certain which albums and tracks needed to be restored from the cloud. Yes, I realize that sounds crazy in a 16TB collection, but some of us are a bit crazy with our music collections :~)

 

I guess the moral of the story is, check your backups, use different methods of possible, and check your backups again.