Jump to content
  • Dan Gravell
    Dan Gravell

    Cloud Music Library Backup

     

    Editor's Note: While researching cloud music backup solutions I checked the Bliss music library management blog from Bliss creator Dan Gravell. I found that Dan had recently written a similar article, so I asked him to create something similar for Audiophile Style. Fortunately, Dan agreed to write the following article about backing up music libraries to the cloud. Dan is a great resource in this area and I encourage people to ask follow up questions and check out his app Bliss, that makes music collections look as good as they sound. 

     

     

    Cloud Music Library Backup

     

    As humans we're great at a lot of things, but those things tend to be addressing short term stuff. What's the funnest thing I can do right now? I've got a headache - where's the paracetamol?

     

    Most of the time I'm thinking "what music should I add to my collection". Rarely do I think "what would I do if I accidentally deleted my entire music library". But the reality is that the impact of the latter far outweighs the former.

     

    We all know 'backups are important'. It's almost a cliche; it's something told to us but also something many of us don't do. Depending on how complex you want to get, backups are not trivial. This complexity means we immediately see an upcoming cost, mostly in terms of our time.

     

    The emergence of the cloud has made this even more overwhelming. Now there are different types of backup services with different pricing structures and different capabilities. When should you choose this service, and when should you choose a different service?

     

    The result is that backups are, for many people, The Job That Gets Put Off. Again and again, we have good intentions to eventually implement backups, but we never quite get around to it.

     

    So I'm here to cut through that and outline a plan for you to implement music library backups. I've surveyed multiple cloud services for storing music, both generic and music-oriented. I've been storing, managing and organizing my own library for 25 years. By the end of this article you'll have decided the best route for you to take in backing up your precious music collection. You'll thank me one day...

     

    ... admittedly not today, but one day. ;-)

     

    What's the worst that can happen?

    Music library disasters come in different forms. It's easy to think of the obvious ones: fire, theft, computer viruses, PEBKAC. But there are more insidious ones too. Bit rot and automation-gone-mad are types of changes to your library that can sometimes go unnoticed, but are damaging nonetheless.

     

    Different types of problem can be fixed in different ways. But they all share one solution: backups. Backups are simple: they are redundant copies of your data. To fix a problem, simply replace your affected files with a previous copy. These redundant copies of your music library are your backstop against pretty much any disaster that can befall your music library.

     

    But there are backups and there are backups.

     

    Backups, as redundant copies of data, can range from a simple drag-and-drop copy of your music library to a different location right through to a best-practice, off-site, automated, scheduled and fully versioned history of file changes.

     

    If you agree you need to do something about backups, you need to decide how you're going to create them. That's difficult, because there are many ways of creating backups, and there are many technical solutions and services that help you do that. What matters is choosing the right solution for your context. There's no one-true-way.

     

    As an overview, these are the broad constraints you should consider when deciding how to backup your music:

     

    • What is the cost of not being able to access your music library? Think in terms of both financial cost and loss of enjoyment.
    • In the event of a problem, how quickly do you need access to your affected music to be reinstated?
    • How often do you make changes to your library?
    • What is the size of your music library (generally expressed in a data storage requirement, i.e. in giga- or terabytes)?
    • What are the technical capabilities of the site where the library is stored, e.g. Internet upload speed?
    • What is your own technical understanding?
    • How much money can you commit to a solution?

     

    We've completed the first step in our plan: to understand our backup constraints.

     

    A mental model for backups

    What we've identified so far are a lot of potential problems and a lot of variety in the impact of those problems. Later I'll also be introducing a number of potential varied solutions to backup.

     

    This is confusing. With so many variables how do you pick the right approach for your backups? The resulting multitude of questions lead to procrastination.

     

    This is the most important part of this guide. A mental model for music library backups. Follow this model, and you can choose the best way of backing up your music library. This model takes the constraints we've outlined to decide the best approach to backups. Later we'll apply this approach to the available cloud services.

     

    The most important place to start is with the impact of loss. To start with, if the impact of music library loss can be measured in dollars it's almost definitely worth investing money on a more complete backup strategy. Usefully, having an approximation of how much the loss of a music library might cost gives a good measure by which you can judge the value of solutions.

     

    If the loss of your music library is more about a loss of enjoyment or even sentimental reasons then it's more difficult to judge how much effort you should place on backup. Consider how you would feel if you lost your music library. Different people will have different opinions. The answer might range from utter devastation at the loss of the collection you have curated your entire life, to a "blow it, I'll just switch to a streaming service". Either response is valid.

     

    Continuing with the same mindset, if the loss of music can be measured financially then it's likely fast recovery is required. However, if not, a slower form of recovery could be permitted. For example, physical media is its own form of backup. If you lost a music library which contained vinyl rips that you now have in long term offsite storage the recovery will be slow, but it might be fast enough.

     

    The next major constraints are around the library itself. This has a specific resonance with audiophiles, who are more likely to store music in higher-fidelity formats. If the library is large (1TB or over, say) then even with a fast consumer Internet connection it will likely take a number of days to upload the collection in its entirety. Furthermore, many online backup services are metered by storage, meaning it will cost more to store more music. At one extreme, if you live in a cabin in the woods without an Internet connection you know online backups are not going to work.

     

    If you have a library that is often updated, by adding new music or changing the music that is in the library (including by changing music tags, or file and folder structures) then you might want backups to be scheduled more often; the longer the gap between backups the more likely one tranche of those changes will be missed.

     

    Finally, let's point the finger at… you. Your own technical understanding can be a blocker to some backup methods. Fortunately, most backup services try to make this as straightforward as possible, but in the case of generic storage services coupled with backup routines created by yourself, you need to be honest that you have both the skills and time to manage these things.

     

    The model in summary:

     

    • If you attach a financial cost to the loss of your music library, best practice, automated and offsite backup should be considered.
    • If a streaming service could replace your library, maybe backups aren't required at all.
    • If recovery from disaster must be fast, consider a service that offers speedy recovery.
    • If recovery from disaster would otherwise be too slow, would choosing a different backup approach make it faster?
    • Compare your library size to your Internet connection. If you have no or limited Internet connection, local backups are the best way. However, if best practice backup is required, you might also need to upgrade your Internet connection.
    • If you don't have the time or skills to work on backup, choose a pre-packaged service.

     

    Status quo bias suggests people may overestimate the chances of current conditions (such as: having a pristine music library) continuing in the future, thus they do not plan for the conditions to change.

     

    By at least considering your constraints and working through this mental model you're doing something, even if that's concluding you don't need a music library anyway because you're going to move to streaming.

     

    By using the mental model we have a general idea of our backup requirements. Knowing these gives us the security to guard us from loss. Our plan has evolved from constraints to requirements.Now it's time to implement a solution to those requirements!

     

    Cloud backup solutions

    This is where it becomes clear that applying our own constraints is important; there are multiple services that provide backups in the form of redundant copies of your music library, but they vary in terms of the security they provide.

    I'm not going to talk about on-site backup services here. There are plenty of options for on-site backups. This article focuses on what the cloud can provide.

    Broadly, there are three types of cloud backup service for your music library. These three types provide a convenient way to slice up the potential services we can use and make choosing easier.

    For all of these examples, be aware that cloud services are a very evolving landscape. Some services come and some go. Some change to solve a new problem, and some change to only serve certain markets.

    This makes it difficult to list examples with any guarantee they will stick around, let alone suggest specific recommendations as to which service you should pick.

    You can use tools like alternativeTo to find the right service for you. Start with services named here, for example iDrive. From there you can filter the results to find the right service. This is useful because, being crowd sourced, this directory is updated often. However note that, being crowd sourced, some of the entries can be misleading so always double up with research once you've found an alternative.

    Sync storage services

    The first type is sync storage. These services are built to provide synchronization of files between devices and people. Backup is a special case of synchronization; in most cases backup is one-way and typically there's only one source of data. As backup is a type of synchronization that means these sync storage services can still be used for backup.

     

    However, you have to be aware the features of these services are oriented to a different problem than backup specifically. This means some more advanced backup features are not provided out-of-the-box. Examples of such features are snapshotting; the ability to download your entire library from a single point in time. Implementing these features yourself is sometimes possible by installing self hosted software, but consider the time cost in implementation and maintenance.

     

     

    Pros of sync storage are:

     

    • Provided by the largest tech firms, so you may already have an account with the service.
    • Support for a wider variety of third-party software, such as music players and best-practice backup software.

     

    Cons are:

    • Any problems (or deletions) introduced to your library are mirrored. File history is often available but a lot of files might have been affected, so sorting through them all is a lot of work.
    • Change history works on a file level, but rarely a library level. This makes recovery much more difficult (third party software can sometimes solve this).

     

    Examples of sync storage are:

     

    Backup storage services

    The next type of cloud storage service are backup services. These are oriented specifically toward backup and solving disaster recovery issues.

     

    These services boast features that are especially useful for backup. Scheduling, automation, file versioning and snapshot restores are all standard. In addition, some services even offer the ability to receive your music library on an external hard drive - which might make recovery faster.

     

     

    Pros of backup services are:

    • Easy to setup best practice backup with the supplied software.
    • Multiple options for recovery, including physical storage shipment.
    • Point-in-time recovery is available giving you an instant "snapshot" of your library.

     

    Cons:

    • Less flexible in some ways; the services are more constrained and designed to work with the proprietary software shipped with the service.

     

    Examples of backup storage are:

     

    Cloud music lockers

    The final type of cloud storage worth considering are music lockers. Music lockers are cloud storage services with added services for music libraries. They allow music to be uploaded to their storage service and then offer extra tools to stream the music to your devices.

     

    They can be thought of as sync storage but with various music-library-oriented additional features. In most cases they are built to solve the problem of remote playback rather than backup. However, as they allow the upload of your music library they do offer a redundant copy and so do offer a small level of protection.

     

    That said, I would not recommend these services as a full backup solution. If you place much value on your library, look to the other categories.

     

     

    Pros:

    • An easy way of making your music library available remotely and allowing it to be streamed.

     

    Cons:

    • Most services don't automate synchronisation of your library, so you have to keep backups up to date yourself.
    • Tricky recovery; there's no support for snapshots of your library.
    • Exporting data can be variable; some services have been known to wipe tags from music files.

     

    Examples of music lockers are:

     

    Now we can apply what we learnt about our constraints and backup requirements to the actual solutions.

     

     

    Choose a backup service if:

     

    • You can attach a financial cost to losing your music library.
    • Losing your library is too terrible to bear.
    • You have no way of recovering your music library in the event of loss.
    • You frequently update your music library.
    • You have an Internet connection capable of uploading your music library.
    • You are less technically inclined, or simply don't want to spend time managing the backup process.

     

     

    Choose a sync service if:

     

    • It's not that important if it takes a few days or weeks to get your music back.
    • It doesn't matter if you can't easily download your entire library from a point in time.
    • Some parts of your library are just not that important, and losing them permanently is not a big deal.
    • You have an Internet connection capable of uploading your music library.
    • You don't update your music library frequently.
    • You are prepared to put more time into adapting the service for use as a more formal backup approach.

     

     

    Or choose a music locker service if:

    • You also want a solution for playing your music remotely.

     

    Our plan is almost complete: we have narrowed down the services to those that most meet our requirements.

     

    The final hurdle

    Backups are necessary for self-stored music libraries. Music library disasters are rarely front of mind and this, combined with the complexity of the different solutions, can lead to inaction.

     

    In this article I've tried to ask the questions you should be asking yourself about the best way of securing your music library. What makes sense to you given your constraints and requirements.

     

    You now understand the type of cloud storage service you need for your backups. The final step in the plan is to choose which service is the one you should implement backups with. It's now easier to progress with implementing backups having articulated our requirements and narrowed the possible solutions.

     

     

    So here's your next step.

     

    Take the shortlist of your chosen storage type and filter each service you want to try, given the cost and the fit to your constraints. Then try the one at the top of your list. Sign up, install, try a backup. By just doing something you've built momentum to having a secure music library.

     

     

     

     

    photo_portrait_dan_75x75.jpg Dan Gravell is the founder and programmer of Bliss. He wrote bliss to solve his own problems with his digital music collection.




    User Feedback

    Recommended Comments



    One caveat to the Backblaze option I've found is that you can not backup a mapped drive (to your NAS) or from your NAS directly without the B2 package.  Going to B2 can increase the cost from $6/mo substantially depending on your storage and usage.  Of course a direct sync with the NAS has its own benefits

     

    It does appear that Carbonite will allow backing up mapped drives from a Windows desktop.  So if you're looking to keep costs down that might be the best option.

     

    Does this align with what others have found?

    Share this comment


    Link to comment
    Share on other sites

    I take that back.  It looks like once you start getting into Safe Backup Pro level for Carbonite you don't get unlimited storage.

    Carbonite Support Knowledge Base

     

    So the Backblaze B2 solution is likely the most cost effective and feature rich.

    Share this comment


    Link to comment
    Share on other sites

    9 hours ago, Dan Gravell said:


    This is a fair question. If bit-rot is manifest as a change in the byte stream for any given file then yes, it could be transferred. This is precisely why snapshotting, as supported by the backup services (as opposed to most of the more generic and music focused services) is important. If you can find the last time the file(s) were good, you can roll back to that time. There may be many files affected.

     

    In addition, managing the integrity of your files, i.e. detecting bit-rot, is likely to be a separate management task dealt with by separate services.

     

    I have versioned backups on my NAS, and those are backed up to an identical off-site NAS. Would bit-rot become evident when in the course of snapshotting, files that I have not recently changed start transferring to the NAS as changed files?

    Share this comment


    Link to comment
    Share on other sites

    8 hours ago, agladstone said:

    The last I checked, I have about 560 TB’s of music in my collection

     

    Is that in a lossless compressed format such as FLAC, or uncompressed?

    Share this comment


    Link to comment
    Share on other sites

    21 minutes ago, AudioDoctor said:

    I have versioned backups on my NAS, and those are backed up to an identical off-site NAS. Would bit-rot become evident when in the course of snapshotting, files that I have not recently changed start transferring to the NAS as changed files?

    If your NAS units are all running simple storage, any corruption on the main NAS will be backed up off-site exactly as read.  If you're using automatic backup software that updates the off-site unit every time the main one changes, corruption will immediately trigger an update of the off-site unit to reflect the change (whether good or bad).  I suspect (but don't know for certain) that if your disks have proper error correction, bit corruption could trigger a remote update but autocorrection would do so again and restore the file to its correct form.  It probably depends on how fast your system is and what triggers the read that detects the error and fixes it.  I've never seen data on these processes, so I can't speculate on how reliable they are.

     

    A simple RAID mirror will also see corrupt bits as a change and make the same change in the mirror disc.  As I recall, only RAID 6 will protect against "rebuilding" a good drive with corrupt data from the affected one.  But there are a few basics to keep in mind:

     

    First, data decay is not very likely in the relatively small HDDS used for home NAS until they start closing in on their end of life.  Use SMART data to check on your drives, and replace them at the first sign of error correction.  I replace HDDs every 5 years even if they're working because the risk of data decay starts going up as they age.

     

    Second, modern drives and the software that controls them use error correction code to identify flipped bits and remediate data decay.  Your SMART drive monitoring software will show you the rate of corrected errors - if that rate is too high (per the maunfacturer's instructions) or it's going up each time you check it over 3 consecutive times, replace the disk.  You can check your discs with utilities built into your OS or use 3rd party software like CrystalDiskInfo.  Scan your disks regularly and hope for this:

     

          image.png.cd4ed7eadf5626dca43a467705015e20.png

     

    Third, SSDs have a different cause of data decay.  The insulating layer that keeps charged electrons where they belong degrades over time, and the bits flip.  So you need to follow the same replacement schedule for SSDs that you do for HDDs.  They both have finite data integrity periods, but for different reasons.

     

    Fourth, heat accelerates this and most other memory, storage, and performance problems.  Run performance benchmarks every few months to be sure your computer isn't slowing down from undetected problems.  Make sure your computers, NAS units, etc are all in well ventilated spaces with good air flow around them.  Dust is a killer because it educes heat transfer to air - vacuum it all off of and out of everything.  Keep all fans clean and make sure they're functioning properly.  Monitor your drive temps just as you do (or should do) with your CPU and GPU temps. Use monitoring software that will alert you to potential problems.

     

    conky_loading_JRMC_library.jpg.d861240d32a74b6070ef053a5c098764.jpg

    Share this comment


    Link to comment
    Share on other sites

    I spent a good chunk of the day thinking about this and looking into all the set up. I have the "Master" music library here on my desktop on a large HDD. That gets synced to the music servers internal drive, and backed up to my local NAS, which then backs up to my NAS at my brothers house across town. They are versioned backups, so in theory I have multiple backups and should have good files...  I also checked drives on the 2011 Mac Mini that runs as a whole house Roon Server and while the drives passed (I think) it also looks like they need to be replaced. The thing still works though, so I may put a new SSD in it and plug a new external drive in as well. Or just use a new M1 mini, but that seems like massive overkill. I thought it would be a good idea to get checksums of the music files on the master as I am not sure that happens right now. I use Forklift to keep things in sync on my network and am still looking into whether or not it has that capability. Can anyone suggest a program that runs on an M1 mac that can generate checksums for files in a folder, with sub folders?

     

    regarding checking the temps and general operational health of the various systems around my house, I am almost obsessive about that. Within a minute I can tell you the temps of any aspect of my computer, my CAPS, the mini, my M1 mini, and all the Raspberry Pi's in the house.

     

    edit: When building the house I had what I consider to be a genius idea regarding the heat generated from the small room/closet that has the NAS and network gear in it. I ducted the heat from a duct in the ceiling, through a wall up above, and up into the heat exchanger. I can remotely check the temp in that room as well.

    Share this comment


    Link to comment
    Share on other sites

    On 4/19/2021 at 3:38 PM, AudioDoctor said:

     

    Is that in a lossless compressed format such as FLAC, or uncompressed?

    I have a lot of DSD, a lot of original vinyl rips in DSD and 24/96 FLAC, a ton of 24 bit digital downloads from Qobuz and HDTracks,etc. and the remainder is mostly 16/44 FLAC. 
    Ive gotten to the point where I feel like I basically have almost everything ever recorded!! 
    Lots and lots of rare live recordings and bootlegs too. 

    Share this comment


    Link to comment
    Share on other sites

    43 minutes ago, agladstone said:

    I have a lot of DSD, a lot of original vinyl rips in DSD and 24/96 FLAC, a ton of 24 bit digital downloads from Qobuz and HDTracks,etc. and the remainder is mostly 16/44 FLAC. 
    Ive gotten to the point where I feel like I basically have almost everything ever recorded!! 
    Lots and lots of rare live recordings and bootlegs too. 

     

    Wow, I'm impressed.

    Share this comment


    Link to comment
    Share on other sites

    “...some services have been known to wipe tags from music files.”

     

    Wait, what?

     

    I’m well past the point I should have implemented some rigorous backup process. Swapping hard drives with my brother is one thing, but I’m better at accumulating things than organizing them. Metadata is just one part of the headache. I need revision control. I may have a half dozen versions of, say, an LP dub. Raw, clicks & noise repaired, an archival version and maybe one with a little EQ for general listening. Some random category that applies only to those files, perhaps. I may or may not want to keep all the intermediate versions. Keeping track via tags is difficult enough without something deleting them!

     

    Yet I am not really surprised. Having just skimmed this article, I’ll dive in further soon. Thanks!

    Share this comment


    Link to comment
    Share on other sites

    @Mike27 Yes, this is a thing. I should've linked to some examples actually because as it reads right now it seems like I've just attempted to insert a bit of FUD...

     

    Exhibit A: YouTube Music https://www.reddit.com/r/musichoarder/comments/o1purr/recovering_metadata_from_youtube_music_library/

     

    Exhibit B: Apple Music (when Match replaces with a lossless version; it appears possible to get the old version back) https://www.reddit.com/r/AppleMusic/comments/nwtyh4/apple_lossless_desktop_follow_up/

    Share this comment


    Link to comment
    Share on other sites

    16 hours ago, Mike27 said:

    “...some services have been known to wipe tags from music files.”

     

    Wait, what?

     

    I’m well past the point I should have implemented some rigorous backup process. Swapping hard drives with my brother is one thing, but I’m better at accumulating things than organizing them. Metadata is just one part of the headache. I need revision control. I may have a half dozen versions of, say, an LP dub. Raw, clicks & noise repaired, an archival version and maybe one with a little EQ for general listening. Some random category that applies only to those files, perhaps. I may or may not want to keep all the intermediate versions. Keeping track via tags is difficult enough without something deleting them!

     

    Yet I am not really surprised. Having just skimmed this article, I’ll dive in further soon. Thanks!

    Google Drive leaves all metadata the same as what is uploaded (what I’m using for cloud backup). 
    The only thing is that it does take “forever” to download backed up copies and it’s best to download one album at a time for FLAC/WAV/ ALAC, and I’ve discovered it’s best to download one track at a time for DSD or 24 bit FLAC/ALAC if it’s a large box set or similar. 

    Share this comment


    Link to comment
    Share on other sites

    @Dan Gravell: I was peripherally aware of the Apple “Match” hijinx, but not how convoluted. Hideous! Never considered YouTube because yuck. 
     

    Bliss looks interesting, but I’m a notorious cheapskate. Historically, though, if I find something I really like, and can squeeze into the budget, who knows? Certainly some interesting info in that blog, thanks!

     

    @agladstone: I have looked at Google Drive. Might be my best solution. They already know what I’m doing before I do. Or maybe Dropbox; I’ve used the free allotments for file transfer with no problem, except once someone invited me to share their space, only to find they put me on their employer’s corporate server, which then declared they owned my data before I even uploaded any... that didn’t last long. 
     

    Meanwhile Windows networking somehow decided my Ethernet crossover cable is a public network, and blocked “rover” from “mothership” but not the other way round. Argh!!

     

    Thanks for the replies!

    Share this comment


    Link to comment
    Share on other sites

    I suspect the generic storage services will be much better in terms of not overwriting metadata - after all, they should just be storing data, not overwriting it as they see fit.

     

    The music lockers, on the other hand, may have more convoluted approaches to managing your library, and this might be where issues creep in. I don't think they should but I find it unsurprising they do... if that makes sense.

    Share this comment


    Link to comment
    Share on other sites

    On 6/24/2021 at 5:52 AM, Dan Gravell said:

    I suspect the generic storage services will be much better in terms of not overwriting metadata - after all, they should just be storing data, not overwriting it as they see fit.

     

    The music lockers, on the other hand, may have more convoluted approaches to managing your library, and this might be where issues creep in. I don't think they should but I find it unsurprising they do... if that makes sense.

    Dan:

    Curious if you know anything or have any experience with SongKong ? 
    I came across them in an article yesterday mentioning that it was being used with great success transferring music library from a server that stored all the meta data in a separate folder, and as a result when transferred to another brand music server it doesn’t bring over any of the metadata or album art. 
    Apparently SongKong can scan the entire music library automatically after you transfer or download your music collection to a new server and it will re-tag and add album art automatically? 
    Seems like this may actually be another solution to the situation mentioned here?  

    Share this comment


    Link to comment
    Share on other sites

    On 6/25/2021 at 11:33 PM, agladstone said:

    Curious if you know anything or have any experience with SongKong ? 

    I came across them in an article yesterday mentioning that it was being used with great success transferring music library from a server that stored all the meta data in a separate folder, and as a result when transferred to another brand music server it doesn’t bring over any of the metadata or album art. 
    Apparently SongKong can scan the entire music library automatically after you transfer or download your music collection to a new server and it will re-tag and add album art automatically? 
    Seems like this may actually be another solution to the situation mentioned here?  

     

    Yep - it is - and I know the author!

    Share this comment


    Link to comment
    Share on other sites

    3 hours ago, Dan Gravell said:

     

    Yep - it is - and I know the author!

    Thanks! Maybe I’ll give it a try! I have about 80 TB’s of downloads on a series of external drives that I suspect are full of duplicates that I’ve been too lazy to attempt to look through for 1+ years now, should be the perfect place to experiment with it! (It apparently can manage duplicates automatically in addition to auto-tagging meta data and auto adding album art). 

    Share this comment


    Link to comment
    Share on other sites




    Create an account or sign in to comment

    You need to be a member in order to leave a comment

    Create an account

    Sign up for a new account in our community. It's easy!

    Register a new account

    Sign in

    Already have an account? Sign in here.

    Sign In Now




×
×
  • Create New...