Article: Cloud Music Library Backup

[Dan Gravell]

View full article

[The Computer Audiophile]

Thanks so much @Dan Gravell. This is excellent information for everyone who doesn't eat, sleep, and breathe the fine details of owning a music collection. 

[jrobbins50]

Thanks, @Dan Gravell.  As Chris knows, I have a quite large (over 35 Tb) music collection on my Synology NAS which has fault tolerance for two hard drives to fail simultaneously, but no offsite storage. I’m sure offloading a copy of my library over the cloud would take days if not weeks, but that’s not my real concern. I’m guessing the cost to maintain a cloud-based backup of a library this size must be out of control. How would you approach such a beast?  Thanks. JCR 

[Qhwoeprktiyns]

Nice article!

 

I find using the "sync storage services", without actually "synching" to be a good option for those who have relatively small music libraries (full upload < 1 day) with files that are not constantly being changed (i.e. tags modified). The obvious drawback is that you need to remember to manually add your new files/changed files regularly, the advantage is that you don't risk accidentally replicating file deletions in both your local storage and cloud storage (as you pointed out). For "smallish" libraries, you could regularly do a full backup. For large libraries, especially when files are frequently updated, this options is probably not workable. You can also deactivate "synching" and reactivate it when needed.

 

These types of issues, by the way, highlight the fundamental disadvantage of having music metadata managed within music files (as tags) - but that's an entirely different topic :)

 

 

[Dan Gravell]

Thanks for posting Chris!

[Dan Gravell]

1 hour ago, jrobbins50 said:

Thanks, @Dan Gravell.  As Chris knows, I have a quite large (over 35 Tb) music collection on my Synology NAS which has fault tolerance for two hard drives to fail simultaneously, but no offsite storage. I’m sure offloading a copy of my library over the cloud would take days if not weeks, but that’s not my real concern. I’m guessing the cost to maintain a cloud-based backup of a library this size must be out of control. How would you approach such a beast?

 

It comes down to your constraints. First of all - if you were to lose the library, or part of it, is that actually an issue? Maybe a strange question to ask, but, if not, there's not necessarily much point in spending money and time on a solution.

 

Backblaze Personal Backup offers unlimited backup storage: https://www.backblaze.com/cloud-backup.html. They are a reputable company - no bait and switch here.

[The Computer Audiophile]

Also, the Backblaze B2 service is likely integrated into your NAS @jrobbins50. You can see pricing here for B2.

 

https://www.backblaze.com/b2/cloud-storage-pricing.html

[Dan Gravell]

20 minutes ago, hopkins said:

I find using the "sync storage services", without actually "synching" to be a good option for those who have relatively small music libraries (full upload < 1 day) with files that are not constantly being changed (i.e. tags modified). The obvious drawback is that you need to remember to manually add your new files/changed files regularly, the advantage is that you don't risk accidentally replicating file deletions in both your local storage and cloud storage (as you pointed out). For "smallish" libraries, you could regularly do a full backup. For large libraries, especially when files are frequently updated, this options is probably not workable. You can also deactivate "synching" and reactivate it when needed.

 

You have to be careful with that though. If you don't automate this there's a good chance you'll forget, or something else will get in the way. Humans are best at creativity, not routine - we get easily bored - "I'll just skip it this month, it'll be fine". I know, I know, some of us are better than others but...

 

Put another way, it depends on the value of the library.

[Qhwoeprktiyns]

Just now, Dan Gravell said:

 

You have to be careful with that though. If you don't automate this there's a good chance you'll forget, or something else will get in the way. Humans are best at creativity, not routine - we get easily bored - "I'll just skip it this month, it'll be fine". I know, I know, some of us are better than others but...

 

Put another way, it depends on the value of the library.

 

Absolutely - and I just started an other full upload after reading your article !

[DuckToller]

Excellent article, @Dan Gravell, something which guides the reader fluently through a decision path/process.
 

I have been there lately and decided against online backup at the moment, but I may look for a selection of albums and playlists to be backed up in the cloud in the future.

I am a far shot away from the collection size of @The Computer Audiophile or @jrobbins50, however including all other data incl. photos and videos as a driver demand is surely above 10TB, but the rate of data growth has been slowed down in the last 3 years, mostly by fiber connection and corresponding streaming services for video and music, while having arrived concomitant at a fairly complete collection for my needs at the other hand. growth drivers have been rather photos & mobilke phone backups  lately ...

After re-visiting some of the helpful threads in the forums, I chose to upgrade my live NAS and to redress my 2012 desktop machine into a NAS using any disk above 1TB I had acquired since 2009 as backup. Plan is to do an automated backup once a week. Number two will be a mini-ITX-tower using 3 oversized HDDs. This one will be stored at a different location and only synced every other month.
 

As I feel that cloud data services (perhaps except for the Norwegian ones) have proved unreliable in terms of price, quantity, customer data protection and life cycle, I may consider using them only for selected items which are the core of the music I am usually listening to. Which is far  less than 5 % of the collection and would be more playlist oriented than complete albums.

During the process of analyzing and planning I have learned how much data our family's digital life has assembled in the last 25 years. There are obviously some parts more than doubled by older incoherent backup strategies, however losing them incidentally would be something great of a drama and imaging that scenario also convinced the treasury that an investment into back up tech is a wise transfer. Far easier to negotiate than buying a new pair of loudspeakers. ... cheaper too ...

One important driver for taking care of the data available in our household and in the cloud, has been the example of Tidal (suddenly but not unexpected) changing the format of the streaming files into lossy and MQA's attempt into dominating the consumers' choices. Being able to rely on the lossless format of my music in the future and to pass it on to my sons, imho,  is the most serious driver for me and demands a coherent back up planning.

Cheers, Tom

 

Cheers, Tom

[Dan Gravell]

1 minute ago, DuckToller said:

During the process of analyzing and planning I have learned how much data our family's digital life has assembled in the last 25 years. There are obviously some parts more than doubled by older incoherent backup strategies, however losing them incidentally would be something great of a drama and imaging that scenario also convinced the treasury that an investment into back up tech is a wise transfer. Far easier to negotiate than buying a new pair of loudspeakers. ... cheaper too ...

 

 

Very much so - when I think of all our family photos and videos... and, of course, the "sync" and "backup" services can be used for non-audio media.

[Danny Kaey]

the reference guide for today's digital libraries!!! wow!

[Qhwoeprktiyns]

11 minutes ago, DuckToller said:

As I feel that cloud data services (perhaps except for the Norwegian ones) have proved unreliable in terms of price, quantity, customer data protection and life cycle, I may consider using them only for selected items which are the core of the music I am usually listening to. Which is far  less than 5 % of the collection and would be more playlist oriented than complete albums.

 

Deciding on a backup strategy some time ago also led me to re-organize my music collection and store in a separate folder/location those albums which I had accumulated over the years but practically never listened to. I have a local back up of all my albums, but only do a cloud backup of my "favorites" (1TB of Flac files at the moment, but growing...).

[jrobbins50]

@Dan Gravelland @The Computer Audiophile, thanks for the info on Backblaze. Way less expensive than what I had otherwise seen. Thanks for the tip!  JCR 

[AudioDoctor]

I find this to generally be a good article. Perhaps my question is too technical and beyond the scope of this article, but... How do we address bit rot? Is that something that can be transferred between devices in such a way that the backup file would also be harmed somehow? I have been thinking about this as drives I have are aging to beyond a decade.

[jventer]

Great article, thanks @Dan Gravell

I learned a few things re acronyms too 😁!

e.g.: 

• PEBKAC ("problem exists between keyboard and chair")
• identity error or ID-10T/1D-10T error ("idiot error"), 
• PICNIC ("problem in chair, not in computer"), 
• IBM error ("idiot behind machine error") and other similar phrases are also used as slang in technical circles with derogatory meaning

 

I am currently rebuilding a 1TB drive at a significant $ and time cost,  that was subject to my IBM error, so very timely article.

 

[jhwalker]

11 hours ago, The Computer Audiophile said:

Also, the Backblaze B2 service is likely integrated into your NAS @jrobbins50. You can see pricing here for B2.

 

https://www.backblaze.com/b2/cloud-storage-pricing.html

 

I started with Backblaze Personal but switched to Backblaze B2 recently because it's so easy to automate from my Synology Diskstation.  Very fast uploads, reasonable prices.

[Johnseye]

Thanks for the writeup. It looks like prices have come down and stabilized with multiple offerings. My preference is to backup to a platform like Backblaze or Carbonite, but I haven't done this yet. Does anyone know if it's possible to map a drive to these services or connect a player like Roon to that cloud storage? It would be great if I could use it as a source and not just a backup space. 

[The Computer Audiophile]

1 hour ago, Johnseye said:

Does anyone know if it's possible to map a drive to these services or connect a player like Roon to that cloud storage? It would be great if I could use it as a source and not just a backup space. 

I’ve done this but the speed and latency sucks. It isn’t a design that equivalent to a streaming service. 

[Johnseye]

7 minutes ago, The Computer Audiophile said:

I’ve done this but the speed and latency sucks. It isn’t a design that equivalent to a streaming service. 

 

Even with buffering is it unlistenable? Or are you better off with a tunnel and using something like JRiver to connect to your home NAS?

 

[bluesman]

Nice!!!  Beautifully written and well thought out, this is essential knowledge for any audiophile with digital library content.  I went to cloud backup after I accidentally deleted about 500 photos that I shot between weekly incremental backups of my NAS drives (swapping with the prior week’s drives in my safe deposit box).

 

I use a synch storage service.  With my meager 5 mbps upload speed at the time, it took about a week to backup my photo and music files.  But, like most of life, patience is a virtue with vast rewards.  Once the files are on the cloud server, incremental updates are fast.  And the service I chose about 10 years ago (Livedrive) has been outstanding, from customer service to easy file availability - and at a very fair price.  Versioning is “limited” to the 30 most recent, which is fine with me because I’ve never needed or used it.

 

I solved the rapid recovery issue with local backup on an SSD, along with RAID mirroring on my NAS for continuity.  This scheme is structured along the hierarchy of probabilities.  One HD in a NAS is more likely to suffer premature death than both. And because accession is limited to incremental B/U, the local SSD is less likely to fail than the NAS drives from which I play most of my music and store & edit my photos (of which I now have about 100,000, many of which are in a RAW format).

 

I hope none of you ever knows the dread that squeezes your heart when you realize you’ve permanently lost files you really wanted and can never get again.  Once was enough for me!  So thanks for a critical wake up call, Dan - I hope everyone takes your advice to heart and uses the information you provided.

 

PS:  Be proactive to minimize loss. Check your drives with whatever monitoring utility you like.  Replace them when the first indicator of impending failure pops up.  Monitor drive temps along with processor temps, and check overall computer performance regularly.  Run benchmark tests.  If R/W speeds drop, get on it ASAP.  If a baseline or operating temp goes up, check it out. Vacuum and clean all fans, ducts, and internal surfaces. If that doesn’t bring down the elevation, find out what’s wrong and fix or replace the offender(s) before a failure.

[Account Closed]

OraStream used to offer a service for this purpose. I haven't kept up with them in recent years but they may still have it. They would also stream your library to you as well.

[Johnseye]

39 minutes ago, bobflood said:

OraStream used to offer a service for this purpose. I haven't kept up with them in recent years but they may still have it. They would also stream your library to you as well.

 

Thanks, that would be along the lines of what I'm looking for. If only their cloud storage was more than 1 TB.

[Dan Gravell]

On 4/16/2021 at 10:13 PM, AudioDoctor said:

How do we address bit rot? Is that something that can be transferred between devices in such a way that the backup file would also be harmed somehow? I have been thinking about this as drives I have are aging to beyond a decade.

This is a fair question. If bit-rot is manifest as a change in the byte stream for any given file then yes, it could be transferred. This is precisely why snapshotting, as supported by the backup services (as opposed to most of the more generic and music focused services) is important. If you can find the last time the file(s) were good, you can roll back to that time. There may be many files affected.

 

In addition, managing the integrity of your files, i.e. detecting bit-rot, is likely to be a separate management task dealt with by separate services.

[Dan Gravell]

On 4/18/2021 at 1:26 AM, bobflood said:

OraStream used to offer a service for this purpose. I haven't kept up with them in recent years but they may still have it. They would also stream your library to you as well.

 

If you mean BRIO, that would be a type of music locker - it's a useful redundant copy, but as I have said it is not a full backup service. Depending on the value of your collection it may or may not be appropriate.