Reevaluating My Music Storage - Update

Wow, 10GigE in the private environemnt  and you speak of saturating. Let that SINK in !
I remember upgrading our European network of IXPs into 10GE between 2004 (planning) and 2006 (starting the rollout) for IP interconnection. It served hundreds of thousands of eyeballs back then ...
You hint of saturation tells me that 100GE won't be too far away in your timeline ... ;-)

Did you take in consideration to use RAID 10 ? Or would that considered only as a solution for the paranoid?

4 minutes ago, DuckToller said:

Wow, 10GigE in the private environemnt  and you speak of saturating. Let that SINK in !
I remember upgrading our European network of IXPs into 10GE between 2004 (planning) and 2006 (starting the rollout) for IP interconnection. It served hundreds of thousands of eyeballs back then ...
You hint of saturation tells me that 100GE won't be too far away in your timeline ... ;-)

Did you take in condideration to use RAID 10 ? Or would that considered only as a solution for the paranoid?

Hi Tom, I hear you! It's crazy. 

The 100GbE MikroTik CRS504-4XG-IN might be in my future (thanks to @jabbr telling everyone about it).

RAID10 would certainly work well, but cut down on my usabel storage space of course. I'm OK with replacing a failed drive and runnign a sync to restore from the attached USB drives. I think this is a good trade off for the gain in speed in everyday use. 

Might I suggest that you make two of the USB backups and rotate one of them to a bank safety deposit box on a periodic basis. This would give you triple redundancy with an off-site  physical component. 

19 minutes ago, bobflood said:

Might I suggest

Yes, you may suggest :~)

3 minutes ago, The Computer Audiophile said:

Yes, you may suggest :~)

Well, even if you don't do it, I'll feel better for trying. :)

Just now, bobflood said:

Well, even if you don't do it, I'll feel better for trying. :)

I'm with you :~)

I think I see why you went down the NAS master path.  It seems to fit how you have been working for a while and this gives you the availability you need. Very nice.

I am interested in the reasoning for using ZFS, I have not studied it at all.  Also, more discussion on 10Ge would have a lot of value to some of us.  We are really running mini data centers in our homes.  That is part of the reason I went back to UnFi on mine.  Stability and performance at the core.

Bob

And still you keep finding great new music!

I am curious to see what you did to get 10G ethernet to the MacBook.

10Gb for Apple, did you use Atto or Sonnet? I’ve had success with both companies on Apples M chips. I’m not at 10Gb with Ravenna yet, that’s very interesting. 

Many thanks for mentioning / pointing me to iDrive. Prices for online storage / backup space and solutions have really come down even further recently, and iDrive‘s backup features/functionalities make it very easy.

I use a Synology DS1821+, have subscribed to iDrive and have just downloaded the iDrive app right onto my Synology NAS - love to manage backup right there directly. Although I‘m just at ~9TB yet, the calculated backup upload duration atm is 62 days and 3 hours though 🤣

Quote

If my local music files are unavailable for a couple days, while the restore is running, I'm OK with that.

Heh, I winced when I saw that sentence. 🙂 But good that you found the solution that works best for you. Also, the sheer volume of music you've got relative to me makes it much easier for me than for you to consider options that multiply storage needs.

Adding myself to the list of folks interested in reading the next article about 10Gbps networking to the Mac.

Started a trial of Backblaze. 

I have 5TB to backup, most is music files.

So far 4 days have gone by and only about 200GB have uploaded. At this pace, the trial will end before my upload is even finished. 

I only have an ADSL line, but this is still incredibly slow.

3 hours ago, firedog said:

Started a trial of Backblaze. 

I have 5TB to backup, most is music files.

So far 4 days have gone by and only about 200GB have uploaded. At this pace, the trial will end before my upload is even finished. 

I only have an ADSL line, but this is still incredibly slow.

I uploaded 16TB to iDrive in about 3 days. 

This is only useful if one doesn't have a data cap on their internet connection. Comcast is 1.2 TB cap. Others are more or less. For many, cloud storage is not an option because of that.

I have 43Tb to backup, 40Tb of which are music files. The cloud is just not feasible with that quantity of data and the Comcast data cap. I’m using USB3.0 to transfer from my NAS through another PC to 16Tb drives, which I will store offsite. Got through the music files in 9 days.  I did it in 1.5-2.5Tb chunks and lost about 100Mb of data each time, allegedly to a file name that was too long and thus skipped over by Windows. It seems they were artwork files in virtually each case, so not a big issue — but keeping data integrity over such a large amount of data is truly difficult. JCR 

22 hours ago, Bertel said:

Many thanks for mentioning / pointing me to iDrive. Prices for online storage / backup space and solutions have really come down even further recently, and iDrive‘s backup features/functionalities make it very easy.

 

I use a Synology DS1821+, have subscribed to iDrive and have just downloaded the iDrive app right onto my Synology NAS - love to manage backup right there directly. Although I‘m just at ~9TB yet, the calculated backup upload duration atm is 62 days and 3 hours though 🤣

Idrive will ship you a USB drive and a return mailer. They then will upload your files to their servers locally, which takes about a week after they get the drive back. That works well - as it must, because that really is the only way that cloud storage possibly works with a large collection, a data cap, and asymmetric internet, which most of us have. 

9 hours ago, firedog said:

Started a trial of Backblaze. 

I have 5TB to backup, most is music files.

So far 4 days have gone by and only about 200GB have uploaded. At this pace, the trial will end before my upload is even finished. 

I only have an ADSL line, but this is still incredibly slow.

Interesting. For me, upload on another service brought the rest of my network speed to a screeching halt, while uploading to Backblaze went fast and didn't interfere with the rest of the network.

Do you have QoS settings or anything else that may be limiting speed to less than the maximum your ADSL is capable of?
 

I believe Backblaze offers to ship a USB drive for your files, at least in the US, though I don't remember what the cost was.

3 hours ago, Mike Rubin said:

Idrive will ship you a USB drive and a return mailer.

Thanks @Mike Rubin ! I had seen this option, but had assumed that it wasn‘t valid for Europe (I live in Germany). Now have just requested iDrive Express from the German version of the website, and it seems to work. Will give it a try, and see how it goes - we‘ll see, will be happy to report here.

EDIT: If someone knows about a similar provider / service in Europe, please let me know.

EDIT 2: Will be interesting whether I can then do an incremental / differential update over internet then once the bulk upload through iDrive Express is up, else the whole thing is kinda moot…

I am brutally basic:

• Main library on my macpro - I download purchases, rip, manage metadata, and curate Roon db here (AIFF+DSF files)
• This library gets backed up on TimeMachine as well as four copies in sequence to external drives (with Chronosync)
• Library gets copied with Chronosync to a NUC with ROCK where Roon Core runs

So library is sourced for playback from the SSD in the NUC/ROCK, and there are 7 full copies of it (some can get as old as ~1month as I cycle over drives).

12 hours ago, firedog said:

Started a trial of Backblaze. 

I have 5TB to backup, most is music files.

So far 4 days have gone by and only about 200GB have uploaded. At this pace, the trial will end before my upload is even finished. 

I only have an ADSL line, but this is still incredibly slow.

3 hours ago, Jud said:

 

Interesting. For me, upload on another service brought the rest of my network speed to a screeching halt, while uploading to Backblaze went fast and didn't interfere with the rest of the network.

 

Do you have QoS settings or anything else that may be limiting speed to less than the maximum your ADSL is capable of?
 

I believe Backblaze offers to ship a USB drive for your files, at least in the US, though I don't remember what the cost was.

I checked and no, unfortunately Backblaze doesn't offer a hard drive for initial backup, only for restoring if you've lost a bunch of files.

On 2/6/2023 at 9:56 PM, Bertel said:

Thanks @Mike Rubin ! I had seen this option, but had assumed that it wasn‘t valid for Europe (I live in Germany). Now have just requested iDrive Express from the German version of the website, and it seems to work. Will give it a try, and see how it goes - we‘ll see, will be happy to report here.

Just as an fyi, in case anybody else was interested: iDrive just notified me that the USB drive for iDrive Express was shipped today. It seems that for Europe they ship from Ireland, which should make it both fast and convenient. Good!

Chris did you see anything in the logs for Hybrid Backup???  This is scary.  I know a lot of folks that use that tool!