Jump to content
  • Dan Gravell
    Dan Gravell

    Cloud Music Library Backup

     

    Editor's Note: While researching cloud music backup solutions I checked the Bliss music library management blog from Bliss creator Dan Gravell. I found that Dan had recently written a similar article, so I asked him to create something similar for Audiophile Style. Fortunately, Dan agreed to write the following article about backing up music libraries to the cloud. Dan is a great resource in this area and I encourage people to ask follow up questions and check out his app Bliss, that makes music collections look as good as they sound. 

     

     

    Cloud Music Library Backup

     

    As humans we're great at a lot of things, but those things tend to be addressing short term stuff. What's the funnest thing I can do right now? I've got a headache - where's the paracetamol?

     

    Most of the time I'm thinking "what music should I add to my collection". Rarely do I think "what would I do if I accidentally deleted my entire music library". But the reality is that the impact of the latter far outweighs the former.

     

    We all know 'backups are important'. It's almost a cliche; it's something told to us but also something many of us don't do. Depending on how complex you want to get, backups are not trivial. This complexity means we immediately see an upcoming cost, mostly in terms of our time.

     

    The emergence of the cloud has made this even more overwhelming. Now there are different types of backup services with different pricing structures and different capabilities. When should you choose this service, and when should you choose a different service?

     

    The result is that backups are, for many people, The Job That Gets Put Off. Again and again, we have good intentions to eventually implement backups, but we never quite get around to it.

     

    So I'm here to cut through that and outline a plan for you to implement music library backups. I've surveyed multiple cloud services for storing music, both generic and music-oriented. I've been storing, managing and organizing my own library for 25 years. By the end of this article you'll have decided the best route for you to take in backing up your precious music collection. You'll thank me one day...

     

    ... admittedly not today, but one day. ;-)

     

    What's the worst that can happen?

    Music library disasters come in different forms. It's easy to think of the obvious ones: fire, theft, computer viruses, PEBKAC. But there are more insidious ones too. Bit rot and automation-gone-mad are types of changes to your library that can sometimes go unnoticed, but are damaging nonetheless.

     

    Different types of problem can be fixed in different ways. But they all share one solution: backups. Backups are simple: they are redundant copies of your data. To fix a problem, simply replace your affected files with a previous copy. These redundant copies of your music library are your backstop against pretty much any disaster that can befall your music library.

     

    But there are backups and there are backups.

     

    Backups, as redundant copies of data, can range from a simple drag-and-drop copy of your music library to a different location right through to a best-practice, off-site, automated, scheduled and fully versioned history of file changes.

     

    If you agree you need to do something about backups, you need to decide how you're going to create them. That's difficult, because there are many ways of creating backups, and there are many technical solutions and services that help you do that. What matters is choosing the right solution for your context. There's no one-true-way.

     

    As an overview, these are the broad constraints you should consider when deciding how to backup your music:

     

    • What is the cost of not being able to access your music library? Think in terms of both financial cost and loss of enjoyment.
    • In the event of a problem, how quickly do you need access to your affected music to be reinstated?
    • How often do you make changes to your library?
    • What is the size of your music library (generally expressed in a data storage requirement, i.e. in giga- or terabytes)?
    • What are the technical capabilities of the site where the library is stored, e.g. Internet upload speed?
    • What is your own technical understanding?
    • How much money can you commit to a solution?

     

    We've completed the first step in our plan: to understand our backup constraints.

     

    A mental model for backups

    What we've identified so far are a lot of potential problems and a lot of variety in the impact of those problems. Later I'll also be introducing a number of potential varied solutions to backup.

     

    This is confusing. With so many variables how do you pick the right approach for your backups? The resulting multitude of questions lead to procrastination.

     

    This is the most important part of this guide. A mental model for music library backups. Follow this model, and you can choose the best way of backing up your music library. This model takes the constraints we've outlined to decide the best approach to backups. Later we'll apply this approach to the available cloud services.

     

    The most important place to start is with the impact of loss. To start with, if the impact of music library loss can be measured in dollars it's almost definitely worth investing money on a more complete backup strategy. Usefully, having an approximation of how much the loss of a music library might cost gives a good measure by which you can judge the value of solutions.

     

    If the loss of your music library is more about a loss of enjoyment or even sentimental reasons then it's more difficult to judge how much effort you should place on backup. Consider how you would feel if you lost your music library. Different people will have different opinions. The answer might range from utter devastation at the loss of the collection you have curated your entire life, to a "blow it, I'll just switch to a streaming service". Either response is valid.

     

    Continuing with the same mindset, if the loss of music can be measured financially then it's likely fast recovery is required. However, if not, a slower form of recovery could be permitted. For example, physical media is its own form of backup. If you lost a music library which contained vinyl rips that you now have in long term offsite storage the recovery will be slow, but it might be fast enough.

     

    The next major constraints are around the library itself. This has a specific resonance with audiophiles, who are more likely to store music in higher-fidelity formats. If the library is large (1TB or over, say) then even with a fast consumer Internet connection it will likely take a number of days to upload the collection in its entirety. Furthermore, many online backup services are metered by storage, meaning it will cost more to store more music. At one extreme, if you live in a cabin in the woods without an Internet connection you know online backups are not going to work.

     

    If you have a library that is often updated, by adding new music or changing the music that is in the library (including by changing music tags, or file and folder structures) then you might want backups to be scheduled more often; the longer the gap between backups the more likely one tranche of those changes will be missed.

     

    Finally, let's point the finger at… you. Your own technical understanding can be a blocker to some backup methods. Fortunately, most backup services try to make this as straightforward as possible, but in the case of generic storage services coupled with backup routines created by yourself, you need to be honest that you have both the skills and time to manage these things.

     

    The model in summary:

     

    • If you attach a financial cost to the loss of your music library, best practice, automated and offsite backup should be considered.
    • If a streaming service could replace your library, maybe backups aren't required at all.
    • If recovery from disaster must be fast, consider a service that offers speedy recovery.
    • If recovery from disaster would otherwise be too slow, would choosing a different backup approach make it faster?
    • Compare your library size to your Internet connection. If you have no or limited Internet connection, local backups are the best way. However, if best practice backup is required, you might also need to upgrade your Internet connection.
    • If you don't have the time or skills to work on backup, choose a pre-packaged service.

     

    Status quo bias suggests people may overestimate the chances of current conditions (such as: having a pristine music library) continuing in the future, thus they do not plan for the conditions to change.

     

    By at least considering your constraints and working through this mental model you're doing something, even if that's concluding you don't need a music library anyway because you're going to move to streaming.

     

    By using the mental model we have a general idea of our backup requirements. Knowing these gives us the security to guard us from loss. Our plan has evolved from constraints to requirements.Now it's time to implement a solution to those requirements!

     

    Cloud backup solutions

    This is where it becomes clear that applying our own constraints is important; there are multiple services that provide backups in the form of redundant copies of your music library, but they vary in terms of the security they provide.

    I'm not going to talk about on-site backup services here. There are plenty of options for on-site backups. This article focuses on what the cloud can provide.

    Broadly, there are three types of cloud backup service for your music library. These three types provide a convenient way to slice up the potential services we can use and make choosing easier.

    For all of these examples, be aware that cloud services are a very evolving landscape. Some services come and some go. Some change to solve a new problem, and some change to only serve certain markets.

    This makes it difficult to list examples with any guarantee they will stick around, let alone suggest specific recommendations as to which service you should pick.

    You can use tools like alternativeTo to find the right service for you. Start with services named here, for example iDrive. From there you can filter the results to find the right service. This is useful because, being crowd sourced, this directory is updated often. However note that, being crowd sourced, some of the entries can be misleading so always double up with research once you've found an alternative.

    Sync storage services

    The first type is sync storage. These services are built to provide synchronization of files between devices and people. Backup is a special case of synchronization; in most cases backup is one-way and typically there's only one source of data. As backup is a type of synchronization that means these sync storage services can still be used for backup.

     

    However, you have to be aware the features of these services are oriented to a different problem than backup specifically. This means some more advanced backup features are not provided out-of-the-box. Examples of such features are snapshotting; the ability to download your entire library from a single point in time. Implementing these features yourself is sometimes possible by installing self hosted software, but consider the time cost in implementation and maintenance.

     

     

    Pros of sync storage are:

     

    • Provided by the largest tech firms, so you may already have an account with the service.
    • Support for a wider variety of third-party software, such as music players and best-practice backup software.

     

    Cons are:

    • Any problems (or deletions) introduced to your library are mirrored. File history is often available but a lot of files might have been affected, so sorting through them all is a lot of work.
    • Change history works on a file level, but rarely a library level. This makes recovery much more difficult (third party software can sometimes solve this).

     

    Examples of sync storage are:

     

    Backup storage services

    The next type of cloud storage service are backup services. These are oriented specifically toward backup and solving disaster recovery issues.

     

    These services boast features that are especially useful for backup. Scheduling, automation, file versioning and snapshot restores are all standard. In addition, some services even offer the ability to receive your music library on an external hard drive - which might make recovery faster.

     

     

    Pros of backup services are:

    • Easy to setup best practice backup with the supplied software.
    • Multiple options for recovery, including physical storage shipment.
    • Point-in-time recovery is available giving you an instant "snapshot" of your library.

     

    Cons:

    • Less flexible in some ways; the services are more constrained and designed to work with the proprietary software shipped with the service.

     

    Examples of backup storage are:

     

    Cloud music lockers

    The final type of cloud storage worth considering are music lockers. Music lockers are cloud storage services with added services for music libraries. They allow music to be uploaded to their storage service and then offer extra tools to stream the music to your devices.

     

    They can be thought of as sync storage but with various music-library-oriented additional features. In most cases they are built to solve the problem of remote playback rather than backup. However, as they allow the upload of your music library they do offer a redundant copy and so do offer a small level of protection.

     

    That said, I would not recommend these services as a full backup solution. If you place much value on your library, look to the other categories.

     

     

    Pros:

    • An easy way of making your music library available remotely and allowing it to be streamed.

     

    Cons:

    • Most services don't automate synchronisation of your library, so you have to keep backups up to date yourself.
    • Tricky recovery; there's no support for snapshots of your library.
    • Exporting data can be variable; some services have been known to wipe tags from music files.

     

    Examples of music lockers are:

     

    Now we can apply what we learnt about our constraints and backup requirements to the actual solutions.

     

     

    Choose a backup service if:

     

    • You can attach a financial cost to losing your music library.
    • Losing your library is too terrible to bear.
    • You have no way of recovering your music library in the event of loss.
    • You frequently update your music library.
    • You have an Internet connection capable of uploading your music library.
    • You are less technically inclined, or simply don't want to spend time managing the backup process.

     

     

    Choose a sync service if:

     

    • It's not that important if it takes a few days or weeks to get your music back.
    • It doesn't matter if you can't easily download your entire library from a point in time.
    • Some parts of your library are just not that important, and losing them permanently is not a big deal.
    • You have an Internet connection capable of uploading your music library.
    • You don't update your music library frequently.
    • You are prepared to put more time into adapting the service for use as a more formal backup approach.

     

     

    Or choose a music locker service if:

    • You also want a solution for playing your music remotely.

     

    Our plan is almost complete: we have narrowed down the services to those that most meet our requirements.

     

    The final hurdle

    Backups are necessary for self-stored music libraries. Music library disasters are rarely front of mind and this, combined with the complexity of the different solutions, can lead to inaction.

     

    In this article I've tried to ask the questions you should be asking yourself about the best way of securing your music library. What makes sense to you given your constraints and requirements.

     

    You now understand the type of cloud storage service you need for your backups. The final step in the plan is to choose which service is the one you should implement backups with. It's now easier to progress with implementing backups having articulated our requirements and narrowed the possible solutions.

     

     

    So here's your next step.

     

    Take the shortlist of your chosen storage type and filter each service you want to try, given the cost and the fit to your constraints. Then try the one at the top of your list. Sign up, install, try a backup. By just doing something you've built momentum to having a secure music library.

     

     

     

     

    photo_portrait_dan_75x75.jpg Dan Gravell is the founder and programmer of Bliss. He wrote bliss to solve his own problems with his digital music collection.



    User Feedback

    Recommended Comments



    Thanks, @Dan Gravell.  As Chris knows, I have a quite large (over 35 Tb) music collection on my Synology NAS which has fault tolerance for two hard drives to fail simultaneously, but no offsite storage. I’m sure offloading a copy of my library over the cloud would take days if not weeks, but that’s not my real concern. I’m guessing the cost to maintain a cloud-based backup of a library this size must be out of control. How would you approach such a beast?  Thanks. JCR 

    Share this comment


    Link to comment
    Share on other sites

    Nice article!

     

    I find using the "sync storage services", without actually "synching" to be a good option for those who have relatively small music libraries (full upload < 1 day) with files that are not constantly being changed (i.e. tags modified). The obvious drawback is that you need to remember to manually add your new files/changed files regularly, the advantage is that you don't risk accidentally replicating file deletions in both your local storage and cloud storage (as you pointed out). For "smallish" libraries, you could regularly do a full backup. For large libraries, especially when files are frequently updated, this options is probably not workable. You can also deactivate "synching" and reactivate it when needed.

     

    These types of issues, by the way, highlight the fundamental disadvantage of having music metadata managed within music files (as tags) - but that's an entirely different topic :)

     

     

    Share this comment


    Link to comment
    Share on other sites
    20 minutes ago, hopkins said:

    I find using the "sync storage services", without actually "synching" to be a good option for those who have relatively small music libraries (full upload < 1 day) with files that are not constantly being changed (i.e. tags modified). The obvious drawback is that you need to remember to manually add your new files/changed files regularly, the advantage is that you don't risk accidentally replicating file deletions in both your local storage and cloud storage (as you pointed out). For "smallish" libraries, you could regularly do a full backup. For large libraries, especially when files are frequently updated, this options is probably not workable. You can also deactivate "synching" and reactivate it when needed.

     

    You have to be careful with that though. If you don't automate this there's a good chance you'll forget, or something else will get in the way. Humans are best at creativity, not routine - we get easily bored - "I'll just skip it this month, it'll be fine". I know, I know, some of us are better than others but...

     

    Put another way, it depends on the value of the library.

    Share this comment


    Link to comment
    Share on other sites
    Just now, Dan Gravell said:

     

    You have to be careful with that though. If you don't automate this there's a good chance you'll forget, or something else will get in the way. Humans are best at creativity, not routine - we get easily bored - "I'll just skip it this month, it'll be fine". I know, I know, some of us are better than others but...

     

    Put another way, it depends on the value of the library.

     

    Absolutely - and I just started an other full upload after reading your article !

    Share this comment


    Link to comment
    Share on other sites

    Excellent article, @Dan Gravell, something which guides the reader fluently through a decision path/process.
     

    I have been there lately and decided against online backup at the moment, but I may look for a selection of albums and playlists to be backed up in the cloud in the future.

    I am a far shot away from the collection size of @The Computer Audiophile or @jrobbins50, however including all other data incl. photos and videos as a driver demand is surely above 10TB, but the rate of data growth has been slowed down in the last 3 years, mostly by fiber connection and corresponding streaming services for video and music, while having arrived concomitant at a fairly complete collection for my needs at the other hand. growth drivers have been rather photos & mobilke phone backups  lately ...


    After re-visiting some of the helpful threads in the forums, I chose to upgrade my live NAS and to redress my 2012 desktop machine into a NAS using any disk above 1TB I had acquired since 2009 as backup. Plan is to do an automated backup once a week. Number two will be a mini-ITX-tower using 3 oversized HDDs. This one will be stored at a different location and only synced every other month.
     

    As I feel that cloud data services (perhaps except for the Norwegian ones) have proved unreliable in terms of price, quantity, customer data protection and life cycle, I may consider using them only for selected items which are the core of the music I am usually listening to. Which is far  less than 5 % of the collection and would be more playlist oriented than complete albums.

    During the process of analyzing and planning I have learned how much data our family's digital life has assembled in the last 25 years. There are obviously some parts more than doubled by older incoherent backup strategies, however losing them incidentally would be something great of a drama and imaging that scenario also convinced the treasury that an investment into back up tech is a wise transfer. Far easier to negotiate than buying a new pair of loudspeakers. ... cheaper too ...

    One important driver for taking care of the data available in our household and in the cloud, has been the example of Tidal (suddenly but not unexpected) changing the format of the streaming files into lossy and MQA's attempt into dominating the consumers' choices. Being able to rely on the lossless format of my music in the future and to pass it on to my sons, imho,  is the most serious driver for me and demands a coherent back up planning.

    Cheers, Tom

     

    Cheers, Tom

    Share this comment


    Link to comment
    Share on other sites
    1 minute ago, DuckToller said:

    During the process of analyzing and planning I have learned how much data our family's digital life has assembled in the last 25 years. There are obviously some parts more than doubled by older incoherent backup strategies, however losing them incidentally would be something great of a drama and imaging that scenario also convinced the treasury that an investment into back up tech is a wise transfer. Far easier to negotiate than buying a new pair of loudspeakers. ... cheaper too ...

     

     

    Very much so - when I think of all our family photos and videos... and, of course, the "sync" and "backup" services can be used for non-audio media.

    Share this comment


    Link to comment
    Share on other sites

    I find this to generally be a good article. Perhaps my question is too technical and beyond the scope of this article, but... How do we address bit rot? Is that something that can be transferred between devices in such a way that the backup file would also be harmed somehow? I have been thinking about this as drives I have are aging to beyond a decade.

    Share this comment


    Link to comment
    Share on other sites

    Great article, thanks @Dan Gravell

    I learned a few things re acronyms too 😁!

    e.g.: 

    • PEBKAC ("problem exists between keyboard and chair")
    • identity error or ID-10T/1D-10T error ("idiot error"), 
    • PICNIC ("problem in chair, not in computer"), 
    • IBM error ("idiot behind machine error") and other similar phrases are also used as slang in technical circles with derogatory meaning

     

    I am currently rebuilding a 1TB drive at a significant $ and time cost,  that was subject to my IBM error, so very timely article.

     

    Share this comment


    Link to comment
    Share on other sites
    11 hours ago, The Computer Audiophile said:

    Also, the Backblaze B2 service is likely integrated into your NAS @jrobbins50. You can see pricing here for B2.

     

    https://www.backblaze.com/b2/cloud-storage-pricing.html

     

    I started with Backblaze Personal but switched to Backblaze B2 recently because it's so easy to automate from my Synology Diskstation.  Very fast uploads, reasonable prices.

    Share this comment


    Link to comment
    Share on other sites

    Thanks for the writeup. It looks like prices have come down and stabilized with multiple offerings. My preference is to backup to a platform like Backblaze or Carbonite, but I haven't done this yet. Does anyone know if it's possible to map a drive to these services or connect a player like Roon to that cloud storage? It would be great if I could use it as a source and not just a backup space. 

    Share this comment


    Link to comment
    Share on other sites
    1 hour ago, Johnseye said:

    Does anyone know if it's possible to map a drive to these services or connect a player like Roon to that cloud storage? It would be great if I could use it as a source and not just a backup space. 

    I’ve done this but the speed and latency sucks. It isn’t a design that equivalent to a streaming service. 

    Share this comment


    Link to comment
    Share on other sites
    7 minutes ago, The Computer Audiophile said:

    I’ve done this but the speed and latency sucks. It isn’t a design that equivalent to a streaming service. 

     

    Even with buffering is it unlistenable? Or are you better off with a tunnel and using something like JRiver to connect to your home NAS?

     

    Share this comment


    Link to comment
    Share on other sites

    OraStream used to offer a service for this purpose. I haven't kept up with them in recent years but they may still have it. They would also stream your library to you as well.

    Share this comment


    Link to comment
    Share on other sites
    39 minutes ago, bobflood said:

    OraStream used to offer a service for this purpose. I haven't kept up with them in recent years but they may still have it. They would also stream your library to you as well.

     

    Thanks, that would be along the lines of what I'm looking for. If only their cloud storage was more than 1 TB.

    Share this comment


    Link to comment
    Share on other sites
    On 4/16/2021 at 10:13 PM, AudioDoctor said:

    How do we address bit rot? Is that something that can be transferred between devices in such a way that the backup file would also be harmed somehow? I have been thinking about this as drives I have are aging to beyond a decade.


    This is a fair question. If bit-rot is manifest as a change in the byte stream for any given file then yes, it could be transferred. This is precisely why snapshotting, as supported by the backup services (as opposed to most of the more generic and music focused services) is important. If you can find the last time the file(s) were good, you can roll back to that time. There may be many files affected.

     

    In addition, managing the integrity of your files, i.e. detecting bit-rot, is likely to be a separate management task dealt with by separate services.

    Share this comment


    Link to comment
    Share on other sites
    On 4/18/2021 at 1:26 AM, bobflood said:

    OraStream used to offer a service for this purpose. I haven't kept up with them in recent years but they may still have it. They would also stream your library to you as well.

     

    If you mean BRIO, that would be a type of music locker - it's a useful redundant copy, but as I have said it is not a full backup service. Depending on the value of your collection it may or may not be appropriate.

    Share this comment


    Link to comment
    Share on other sites
    On 4/16/2021 at 12:02 PM, The Computer Audiophile said:

    Also, the Backblaze B2 service is likely integrated into your NAS @jrobbins50. You can see pricing here for B2.

     

    https://www.backblaze.com/b2/cloud-storage-pricing.html

    I have to look into this! 
    I’ve been using Google’s business level Google Workplace as Cloud backup in addition to a Synology NAS in Raid 5 with local sync and backup from my Aurender and also local external drives connected to a local desktop for my “core listening collection” and then also having that Synology NAS auto backup to and from my Google Workplace cloud. 
    However, I can tell you from my extensive experience of downloading albums and collections from the cloud to my local “core collection”, that it takes a very long time to download from Google cloud and also there are a lot of errors often, so it requires double checking album by album and often re-downloading 2-4 tracks individually to fix these errors. 
    I couldn’t begin to fathom what a Herculean task it would be if I ever actually needed to rely on a complete download of my Google cloud backup to my local storage and/or “core collection”, it would be a nightmare! 
    Seems like it may be time for me to check out BackBlaze B2! 
    I wonder if BackBlaze also has the ability to transfer / backup/ sync with Google (cloud to cloud)? 
    I currently use MultCloud to sync and backup to and from Google Drive, MS Onedrive, Apple iCloud, and Dropbox automatically and it works great. 
    My music collection is ENORMOUS, so my issues and situation is probably different than most others. 
    The last I checked, I have about 560 TB’s of music in my collection and I have tried my best not to add much to that collection for at least the past 1-2 years since it’s become so expensive and time consuming to try and store and manage and tag all of it!! 
    I only have about 80+ TB’s of this digital collection at home in my NAS (aka “core collection”) and it’s all backed up (but not on an automated or regular schedule) to a bunch of local external HDD’s. 
    One of these days I really need to make an additional backup of it all and bring to a friend or relatives house for an offsite backup).  
    Also, as others have mentioned, I have the top level subscriptions to Qobuz and Tidal, and as they’ve continued to get better and better and as my home internet has become Fiber and 1+Gbps, I’ve been listening to more and more HiRes streaming that sounds really good. 
    Between these streaming services and the ability to download so many great and high quality recordings for your absolute favorite and best recorded albums from Qobuz, HDTracks, and many other online sources, I think if I suddenly lost my entire collection, I’d be able to “live with it”. 

    Share this comment


    Link to comment
    Share on other sites



    Create an account or sign in to comment

    You need to be a member in order to leave a comment

    Create an account

    Sign up for a new account in our community. It's easy!

    Register a new account

    Sign in

    Already have an account? Sign in here.

    Sign In Now



×
×
  • Create New...