Music library (backup) in the cloud?

[Jud]

3 hours ago, bodiebill said:

 

IBackblaze does not work with Windows Server 😐

 

Just Mac and consumer versions of Windows.

 

Since I run these operating systems along with a couple of Linuxes, this works for me. Running the server version of Windows or modifying it to eliminate services can sometimes produce errors with the players I use, so as long as I'm running Windows (or MacOS), I leave it as the player developer expects to find it.

[bodiebill]

I just subscribed to Jottacloud for a one-month account and am currently uploading my library. With some luck this will complete within that month. Initial impressions are good in terms of speed (especially download is very fast, upload is approx. 20MB/s in my case) and intuitive UI.

 

€7,50 per month unlimited; beyond 5TB upload speed goes down...

[Dan Gravell]

23 hours ago, Iving said:

ewww ... sounds like Apple ["we know what you want before you know what you want"]

 

Thanks for the feedback, it's very useful - although I don't recall saying "I know what you want" - album art could easily be optional, in fact the tagging options in general could be optional.

 

What are the problems with online backup for you at the moment?

 

Personally I just use S3 still (used to use rsync.net who had excellent customer service, but the price increase was just too much).

[Iving]

31 minutes ago, Dan Gravell said:

 

Thanks for the feedback, it's very useful - although I don't recall saying "I know what you want" - album art could easily be optional, in fact the tagging options in general could be optional.

 

What are the problems with online backup for you at the moment?

 

Personally I just use S3 still (used to use rsync.net who had excellent customer service, but the price increase was just too much).

 

The "We know what you want before you know what you want" thing is just a euphemism for a certain kind of philosophy of computing epitomised in my mind (and uncomfortable experience) by Apple. All computing systems (mostly software of course) present frustrating obstacles to me (I know I am my own victim) - and I have found Apple the worst environment of all. You can see from my post that I am left with a bit of "If you want something done properly you have to do it yourself" belief-script. I am not responding to your Bliss service directly - only your post, "What if ... etc".

 

I have never warmed to Cloud storage. I imagine it can be a fitting solution in corporate environments and for some personal users - I guess. I don't like the idea of handing over my data voluntarily to third parties - we condone things by participating in them. But I could get over this in relation to my music files. If I use third party software to backup and update my music library, could I be 100% sure that no snooping of my other files (or browsing activity etc) ever happened. That was a rhetoric question no matter the facts. Pragmatically I can imagine all sorts of upload problems and hiccoughs compared with USB transfer to another HDD. Obviously I don't want to pay unless of value to me. And what if I ever stopped using a paid-for service for any reason - financial hardship, forgetting etc. I would have to anticipate an alternative solution in advance - the one I now prefer i.e. simple off-site HDD. Probably ultimately for me I prefer to work in my own shed than somebody else's - that's all.

 

I have no experience of S3.

 

I wasn't and am not now remarking on Bliss. Just joining the conversation started by the OP.

[TJtennispro]

On 2/16/2020 at 5:02 PM, bodiebill said:

 

Ah confusing. Google say unlimited only starts when you pay for at least 5 users. But they have not enforced the 1TB for the last few years. See https://servergems.com/unlimited-google-drive-storage/ When they start enforcing the 1TB the cost would be times 5. 

Okay but I’m not using that service so that’s not applicable to me. I’m using G Suite Business. Unlimited data storage for one user so no need for me to worry. The main reason I have to collect my own lossless copies is the music I listen to mainly won’t EVER be on Qbuoz or Tidal. Some streaming services do have some Dave Matthews Band stuff but I can’t find a streaming service besides nugs.net that offfers all the Widespread Panic shows in

lossless but then I can’t listen to more mainstream music on that service. 
 

Chris CarPlay article is what got me interested in getting my NAS in the cloud to play in my Genesis G 70 Sport plus at work. 

[TJtennispro]

On 2/17/2020 at 12:29 PM, bodiebill said:

I just subscribed to Jottacloud for a one-month account and am currently uploading my library. With some luck this will complete within that month. Initial impressions are good in terms of speed (especially download is very fast, upload is approx. 20MB/s in my case) and intuitive UI.

 

€7,50 per month unlimited; beyond 5TB upload speed goes down...

 

To each their own but I’ll never understand people that ask about suggestions about where to store music and when presented with suggestions does one that nobody has heard of or suggested. Jottacloud? Really? 

[bodiebill]

Yesterday I started uploading my music library from a Windows machine using Jottacloud's automated uploader. After about one day I reached 170GB. So uploading a 5TB library would take approx. 24 days.

 

However, this afternoon I started to do the same, this time using the rclone tool to sync into my Jottacloud. 45 minutes later I was already at 280GB, so 110 added in that time. So now it seems the library can be uploaded in less than 2 days.

 

The improvement in speed is almost incredible. Fingers crossed...

And I no longer have to use the somewhat buggy Jottacloud desktop app.

 

For more info see: https://rclone.org/ 

It is a free tool based on rsync that works with many cloud storage providers.

 

 

 

[bodiebill]

45 minutes ago, TJtennispro said:

To each their own but I’ll never understand people that ask about suggestions about where to store music and when presented with suggestions does one that nobody has heard of or suggested. Jottacloud? Really? 

 

I tried five providers, including some suggested here. Jottacloud for me felt best in terms of speed, user experience, cost and data center location. It is a well known service in Europe that started in 2008.

 

I would have preferred G Suite Business as recommended by TJtennispro, however was advised elsewhere that this could be risky:

 

Google write: "With G Suite Business, each user in your organization can store unlimited Gmail messages, Google Photos, and files in Drive. Or if there are 4 or fewer users in your organization, each user gets 1 TB of storage."

 

The unlimited in the first part refers to Google files that are unlimited in all Google storage products, so this does not apply to music files.

The second sentence means that if you pay for one user (which is less than 4 users), the official max is 1TB. They just do not enforce that yet. I did not want to take the risk of later enforcement of that rule given the hassle to get all the data up there.

 

For now happy with Jottacloud with rclone (see previous post). Hereby suggested by me a service. 😉 

 

And actually the advice here really helped to make that choice, so thanks all.

 

[kirkmc]

I have multiple backups of all my media. Three are local, and one is in the cloud with Backblaze. On a Mac, it's a good idea to use Time Machine, because of its versioning; every now and then I find files missing in my iTunes/Music library, and I can recover them easily from my Time Machine backups. 

[Qhwoeprktiyns]

I use google drive to back up my library (in addition to a backup on local drives).

I like the fact that file content is automatically indexed (either text files or images/pdf through OCR)

The automatic sync software on Windows works pretty well, but takes up a lot of resources on my computer when syncing. As mentioned by others, the initial upload of all the files is slow (takes a day for 1 terabytes). With automatic sync one has to be extra careful not to modify/delete content on Google Drive accidentally...

 

I did not find a good application to browse and play my music collection using Google Drive, so I developped my own.

 

I designed a web page to browse my music collection and play files either locally (using LMS) or through the cloud when I am not at home. I initially used the copy of the files on Google Drive, but soon found out that Google Drive is not the best platform for this, as access to the files is quite slow.

 

In addition to having a Google Drive backup, I store all album covers and album information in "buckets" on Google Cloud Storage, and selectively upload music files to Google Cloud Storage as well when I want to play an album "on the road". Here is an example: https://gentle-ridge-34867.herokuapp.com/album#1574857707026 (you cannot play the album - copyright issues... so I embedded the spotify album at the bottom). When run locally the web page allows you to edit the content of the page (which is simply a text file in markdown format stored in the album's folder). The content of the text file is parsed when saved and the credits are indexed and searchable (ex: find all albums were Jimmy Forrest played...).

Here is another example: https://gentle-ridge-34867.herokuapp.com/album#1567008961828

The composers are missing and plan on adding them.

 

[bodiebill]

1 hour ago, hopkins said:

I designed a web page to browse my music collection and play files either locally (using LMS) or through the cloud when I am not at home.

 

Nice design!

[Jud]

1 hour ago, hopkins said:

I use google drive to back up my library (in addition to a backup on local drives).

I like the fact that file content is automatically indexed (either text files or images/pdf through OCR)

The automatic sync software on Windows works pretty well, but takes up a lot of resources on my computer when syncing. As mentioned by others, the initial upload of all the files is slow (takes a day for 1 terabytes). With automatic sync one has to be extra careful not to modify/delete content on Google Drive accidentally...

 

I did not find a good application to browse and play my music collection using Google Drive, so I developped my own.

 

I designed a web page to browse my music collection and play files either locally (using LMS) or through the cloud when I am not at home. I initially used the copy of the files on Google Drive, but soon found out that Google Drive is not the best platform for this, as access to the files is quite slow.

 

In addition to having a Google Drive backup, I store all album covers and album information in "buckets" on Google Cloud Storage, and selectively upload music files to Google Cloud Storage as well when I want to play an album "on the road". Here is an example: https://gentle-ridge-34867.herokuapp.com/album#1574857707026 (you cannot play the album - copyright issues... so I embedded the spotify album at the bottom). When run locally the web page allows you to edit the content of the page (which is simply a text file in markdown format stored in the album's folder). The content of the text file is parsed when saved and the credits are indexed and searchable (ex: find all albums were Jimmy Forrest played...).

Here is another example: https://gentle-ridge-34867.herokuapp.com/album#1567008961828

The composers are missing and plan on adding them.

 

 

I've divided the functions of backup (Backblaze) and access to my music at full resolution anywhere with an internet connection (Subsonic and the app AVSub on my phone). This doesn't have the speed of access problem your (impressive) home brew solution does.

[Qhwoeprktiyns]

I also wanted to avoid having an incoming internet connection to my home, and the need for a server running 24/7, so I developped this "hosted" solution... There are some solutions available to have a full cloud-based access to your library, which i looked into, but none of them were satisfactory in my opinion. Cloud computing is in its "infancy"...  

 

This is getting off topic, as it goes beyond the "backup" aspects, but maybe i will open another thread to discuss it. 

[mfaoro]

I don’t play from the cloud or want to.

 

I backup my 4.6TB music collection with iDrive. I chose them because my monthly upload capacity is limited, I didn’t want to wait weeks or months for a full backup and, more importantly, they will send you a hard drive that is used to seed your backup.
 

Put another way, I copied my music collection to the drive they sent me, sent it back and voila, I have a backup in the cloud that I can add to.

 

Backblaze is cheaper. I use it for my photographs. I started with a small backup so seeding wasn’t issue.

 

[Dan Gravell]

On 2/21/2020 at 12:27 PM, hopkins said:

This is getting off topic, as it goes beyond the "backup" aspects, but maybe i will open another thread to discuss it. 

 

I'd be interested to learn more about what you are doing - I think "hybrid cloud" or "self hosting" in the cloud might be a growing phenomena.

[Qhwoeprktiyns]

3 minutes ago, Dan Gravell said:

 

I'd be interested to learn more about what you are doing - I think "hybrid cloud" or "self hosting" in the cloud might be a growing phenomena.

 

Will open up a thread to describe what I am doing.

[bodiebill]

On 2/27/2020 at 11:03 AM, hopkins said:

Will open up a thread to describe what I am doing.

 

@hopkins Did you ever start this thread about self-hosting?

[Qhwoeprktiyns]

1 hour ago, bodiebill said:

 

@hopkins Did you ever start this thread about self-hosting?

 

Sorry, i had not taken the time to explain in detail.

 

Unfortunately what I am doing is a "custom" solution and I do not really see how it could be extended to others without investing significant time and effort into it. So i don't think a seperate thread is required. I'll explain here briefly.

 

I developped everything from scratch. My program scans automatically my folders. 

I store  album covers and associated artwork/pdf  on Google Cloud Storage. The "tags" and comments on my albums are stored in a PostgresQL database hosted on Heroku. 

 

The tracks themselves are for the time being only stored locally, but i plan on storing them as well on Google Cloud Storage, selectively for the time being, to allow me to  play them remotely.  For the time being, I play my files through LMS by sending the URLs of the files from my app. Works well, but it ain't Roon... 

 

That's about it

 

I put a few examples here: 

The example illustrate why i did all this. They are individual albums. I also have an "album grid" to browse my entire collection, but it is not finalized. I plan on adding artist bios/pictures as well - it all takes time. I spend more time "collecting" information on my ever-growing collection than on actually developping the solution. 

 

By the way, i am the author of this thread: https://community.roonlabs.com/t/bye-roon-why-i-am-not-renewing-my-subscription/67012

 

I followed through on my ideas and am very happy to have done so  I mostly listen to Jazz, and as a Jazz "collector", there are in my opinion many aspects which are just not handled by any music application today. Unfortunately,  there is no "collaborative" database that address these requirements. A shame, but that's the way it is.

 

 

[AudioDoctor]

1 hour ago, bodiebill said:

 

@hopkins Did you ever start this thread about self-hosting?

 

You really need only three things to self host your own music backup in the cloud.

 

1) Time, because doing this yourself is going to take some set up time

 

2) A NAS, with real NAS software. I use an iosafe NAS that runs Synology Disk Station Manager

 

3) a place to put it with a great internet connection, up and down. My Brother and I have identical iosafe NAS units at each others houses.

 

That's it.

 

https://iosafe.com/products/1019-nas/

 

https://www.synology.com/en-us

 

[Qhwoeprktiyns]

Yes, and my solution is "hybrid' as some data is stored on the cloud and some locally. The app itself can be run either locally or from a remote server (on heroku) - the code itself is identical. 

[The Computer Audiophile]

I still can't believe HiFi companies haven't built cloud backup into their products yet. The Vox app (https://vox.rocks/) has had it for a long time. We pay premium prices for Roon, Aurender, Lumin, Auralic, etc... One would think the companies would just work with Backblaze or Amazon behind the scenes to white label the storage or just hook into the API and call it done. 

[Qhwoeprktiyns]

11 minutes ago, The Computer Audiophile said:

I still can't believe HiFi companies haven't built cloud backup into their products yet. The Vox app (https://vox.rocks/) has had it for a long time. We pay premium prices for Roon, Aurender, Lumin, Auralic, etc... One would think the companies would just work with Backblaze or Amazon behind the scenes to white label the storage or just hook into the API and call it done. 

 

Agreed ! It really is not rocket science to develop a system that syncs your files to cloud storage. Having your associated metada (what Roon calls your "database") stored on the cloud as well is not an issue, given the bandwidth that most of us enjoy at home. For those who don't, it could be optional.

[Jud]

I am sorry to say the robustness of various backup solutions (multiple locations, remote governance and maintenance, etc.) will likely be tested in the coming weeks and months.

[Qhwoeprktiyns]

24 minutes ago, Jud said:

I am sorry to say the robustness of various backup solutions (multiple locations, remote governance and maintenance, etc.) will likely be tested in the coming weeks and months.

 

Cloud storage/apps arise happening everywhere else... Corporations are moving to cloud based solutions.

 

Having to run a home server 24/7 to listen to your music files (at home or remotely) seems "archaic" to me. A chromebook, a backup solution (local or remote) - that's all we should need.

[AudioDoctor]

I don't actually use mine to listen to music, just to backup. I could however.