Article: Reevaluating My Music Storage - Update

[The Computer Audiophile]

View full article

[The Computer Audiophile]

4 minutes ago, DuckToller said:

Wow, 10GigE in the private environemnt  and you speak of saturating. Let that SINK in !
I remember upgrading our European network of IXPs into 10GE between 2004 (planning) and 2006 (starting the rollout) for IP interconnection. It served hundreds of thousands of eyeballs back then ...
You hint of saturation tells me that 100GE won't be too far away in your timeline ... ;-)

Did you take in condideration to use RAID 10 ? Or would that considered only as a solution for the paranoid?

Hi Tom, I hear you! It's crazy. 

 

The 100GbE MikroTik CRS504-4XG-IN might be in my future (thanks to @jabbr telling everyone about it).

 

RAID10 would certainly work well, but cut down on my usabel storage space of course. I'm OK with replacing a failed drive and runnign a sync to restore from the attached USB drives. I think this is a good trade off for the gain in speed in everyday use. 

 

 

[The Computer Audiophile]

19 minutes ago, bobflood said:

Might I suggest

Yes, you may suggest :~)

[The Computer Audiophile]

Just now, bobflood said:

Well, even if you don't do it, I'll feel better for trying. :)

I'm with you :~)

[The Computer Audiophile]

3 hours ago, firedog said:

Started a trial of Backblaze. 

I have 5TB to backup, most is music files.

So far 4 days have gone by and only about 200GB have uploaded. At this pace, the trial will end before my upload is even finished. 

I only have an ADSL line, but this is still incredibly slow.

I uploaded 16TB to iDrive in about 3 days. 

[The Computer Audiophile]

Hi Guys, iDrive saved me last night, BIG TIME!

 

I backed up 16TB to iDrive. I also backed up 16TB to local drives, using QNAP's Hybrid Backup Sync app. The app works great and I can schedule it or run it manually etc... One the backups were done, I added four more 6TB drives to the QNAP, deleted the storage pool, and recrated it as RAID0 with 8 drives. Bazing speed now. 

 

I then copied all the data from my local storage, back to the QNAP. I checked the folder sizes and file numbers etc... at backup and restore times, and thought it was all good. The number is never identical because of hidden file variables. 

 

When I opened Roon, I checked my Three Blind Mice albums. The Supreme 1500 collection should have 100 albums. There were only 98. After a long troubleshooting session, I discovered that tracks, albums, or folders with quotation marks in the name, were not backed up to the local storage. Then I looked as some other albums in my Atmos WAV file collection, because I know some of the classical albums have "crazy" titles. I found more albums and tracks that were missing!

 

I opened the iDrive cloud backup, and found everything was there! I initiated a restore of the missing albums and tracks, and I had everything back to normal by the time I went to bed. Fortunately, I'm pretty certain which albums and tracks needed to be restored from the cloud. Yes, I realize that sounds crazy in a 16TB collection, but some of us are a bit crazy with our music collections :~)

 

I guess the moral of the story is, check your backups, use different methods of possible, and check your backups again. 

[The Computer Audiophile]

20 minutes ago, bobfa said:

Chris did you see anything in the logs for Hybrid Backup???  This is scary.  I know a lot of folks that use that tool!

 

NOTHING! 

 

Check this out. I ran a test last night on one folder to make sure I wasn't losing my mind. It says no files skipped. 

 

[The Computer Audiophile]

@bobfa I think this has something to do with the perfect combination of sync type, file system, and filename. A lot of people have has issues with double quotes (") in the name over the years. now I know, after having the issue and looking it up. 

[The Computer Audiophile]

2 minutes ago, Jud said:

It might be more surprising if the algorithm “realized” it hadn’t backed up the files it failed to detect needed to be backed up.

HA! So true.

[The Computer Audiophile]

OK, I re-ran the iDrive backup from scratch and have some information about number of files, size of backup and length of time it took to run.

 

Backup Start Time: Sun Feb 12 22:04:05 2023

Files considered for backup: 336482

Files backed up now: 336482

Backup Set Size: 15.04 TB

Backup End Time: Wed Feb 15 09:36:24 2023

 

That's roughly 15TB uploaded in 2.5 days if my math is correct.

[The Computer Audiophile]

6 minutes ago, Jud said:

 

What's your max upload speed from your ISP if you know? Did you check during the upload to try to see actual speed?

 

Here we've got 1.2Gbps download but only 40Mbps upload.

I have 1Gbps up/down, without a cap, for $65 per month. 
 

The upload was frequently around 80-108Mbps when I checked. The CPU of the QNAP was frequently around 60-90%. 

[The Computer Audiophile]

On 2/26/2023 at 12:57 PM, robbbby said:

Hey Chris, how often are you rebooting that NAS for firmware updates?

 

I update the firmware whenever a new version is released, and reboot. I've had this NAS die on me once, and be replaced. I feel your pain :~)