Jump to content
  • The Computer Audiophile
    The Computer Audiophile

    Recovering 20,000 Albums From A NAS Failure Without Backup

     

     

        

        Audio: Listen to this article.

     

     


    As The Computer Audiophile I often get contacted by audiophiles in a technological pinch. This was the case over the weekend, and I've been working to resolve it for the last couple days. The issue and solution are more tech related than audio related, but when one's unstreamable music is stored on a NAS, it becomes an audio issue. 


    A great friend of mine had been using a QNAP TS-853 Pro eight bay NAS for several years without an issue. He has a Roon core running on a NUC and roughly 12 Terabytes of music on the TS-853 Pro, on eight 6TB drives in a RAID6 configuration. This configuration allows for two of the eight drives to fail, without losing any of his music. He had no backup USB drives connected to the NAS, no second NAS with a synchronized copy, or even a cloud backup. The time bomb was ticking.

     

    All was fine until I received a text on Saturday morning, while I was on the way to pickup fresh picked strawberries from Little Hill Berry Farm in Northfield, MN. The text stated, "Sometime last night, QNAP went from being operational to "System Booting" and does not recognize any of the drives as indicated by the solid red lights for each HDD.  Are you able to remotely connect?"

     

    I wasn't able to connect to it via VPN or the QNAP Cloud because the NAS wouldn't boot up to the operating system. It was stuck on System Booting. I had my friend pull the power cable, remove all the drives, wait a minute or so, then power it back up without the drives. This method can work in some instances, but it wasn't meant to be in this case. Nothing changed. Still stuck at System Booting.

     

    After thinking about it for a few more miles in the car, I hatched a plan to attempt to recover his music. There were many unknowns and plenty of contradictory information available online, but the other option was to do nothing and lose a lot of music. I had him send me the QNAP NAS, so I could start the process. 

     

    My idea was to pull the eight spinning drives and one NVMe out of my QNAP TVS-872XT, put his drives into the chassis, cross my fingers, and hope I could jump over the technical hurdles that appeared upon booting my NAS. 

     

    Fast forward to Thursday afternoon. I was working on my review of the Schiit Audio Urd CD transport, when UPS delivered the nonfunctioning QNAP TS-853 Pro. I thought I'd check it out and who knows, maybe it would just boot up. I experienced the same System Booting message. My inner geek kicked in, forcing me to set the Schiit Urd aside and focus on solving the riddle of music file recovery. 

     

    I removed my drives from my QNAP TVS-872XT, placed his drives in the chassis and hit the power button. Things were looking good, the unit booted up, showed me the IP address on the front display, and didn't offer any warning beeps in the process. I connected to the NAS with a web browser and saw the message that consume my entire day, "Detected Incompatible Disks."

     

    incompatible disks.png

     

     

     

    That's when it dawned on me. My NAS runs on the QNAP QuTS Hero operating system and his NAS ran on the QTS operating system. The two systems aren't compatible. Switching from one to the other requires all drives be reformatted. The solution was to remove all the drives from my QNAP and reboot it. Upon powering up, the QuTS hero Smart Installation screen appears, enabling me to select Switch OS QTS. 

     

    My NAS then installed the QTS operating system, pulling down the newest version from QNAP automatically, and rebooted without any disks. With the unit running, I put each of his disks into my chassis and crossed my fingers. This was a step in the right direction as I could see a large hard drive volume with many Terabytes used. However, I couldn't access any of the folders that were on his NAS previously because they weren't there. The data seemed to all be there, but I couldn't see it. 

     

    When the NAS booted up after the OS switch, it asked me a question about reinitializing it or resetting it. One of those options left all the data intact, the other wiped all the drives. I went with the KEEP MY DATA option, but that's why the unit was in a default state, without any shared folders I needed. 

     

    To get the folders back, with all the music files, I went into the QNAP Control Panel > Shared Folders > Create > Shared Folder. This brought up the new shared folder window. I selected Enter Path Manually, clicked the dropdown arrow, and saw all the previous shared folders listed by name. I selected Music Files, then named the new shared folder Music Files to avoid any possible confusion down the road. Once this was done, I had access to all the music files by browsing the QNAP File Station app or by browsing the NAS over my network. It was as if I had his NAS back up and running. I could now recover the data. 

     

    manually select folder.png

     

     

     

    I went to Microcenter and purchased a 20TB USB drive. After formatting as exFAT I had roughly 18TB usable space. I didn't want to screw around with a smaller drive that may have matched the space I needed perfectly, only to find out I needed more space for some unforeseen reason. 20TB was the larges drive I could pickup in 20 minutes, so I went with it. 

     

    I connected the 20TB drive to the QNAP's USB 3 port, installed QNAP's HBS 3 Hybrid Backup Sync, and setup a one-way sync of all the folders on the NAS to identically named folder on the USB drive, then clicked Start. This utility is really nice for backing up data, but isn't perfect. I've discovered it won't copy files with quotation marks in the file names, but for this sync I told it to convert reserved characters, not knowing if there are any songs with quotes in the names or if that would even resolve the issue. It was worth a shot anyway. 

     

    HBS Backup.png

     

     


    Looking forward and Recommendations


    TS-i410X.jpgWhile the backup sync to USB was running, I talked to my friend about his music storage needs. He said the TS-853 Pro was a little noisy when the hard drives churned for no apparent reason, so replacing it with an SSD based NAS was preferred. I looked at TrueNAS, but couldn't guarantee a trueNAS Mini would be quieter given its use of a fan. 

     

    I told him to order the QNAP TS-i410X four bay fanless NAS with four 8TB Samsung 2.5" QVO drives. This will be more than enough space for now and guarantee silent operation. When the NAS and drives arrive at my house I will get the unit setup, then create a one-way sync job to copy the backed up data from the USB drive to the TS-i410X NAS. Once the data is copied to the NAS, I will reverse the direction of the sync job and create a schedule that automatically backs up everything on the NAS to this attached USB drive every night. 

     

    The attached USB drive will remain formatted as exFAT. This way, if the NAS dies, a Mac, Linux, or Windows machine can at least read the data on the drive if needed, without jumping through any hoops. Replacing this new NAS would also be pretty straight forward as one would run the sync from the USB drive to a new NAS, in the identical way I did when setting it up. 

     

    I will also recommend my friend use iDrive for offsite cloud storage. The iDrive app runs on QNAP and can use any schedule one wants to backup data to the cloud. iDrive is incredibly cheap. I think I'm paying less than $200 per year for around 20TB and I love it. Data backed up to the cloud could come in handy if for whatever reason, both the new QNAP and USB drives are unavailable (fire, theft, etc...).

     

    One finally note, there is talk online about other QNAP units with this system booting message and fixing it by adding a resistor to the motherboard. Such a repair is beyond my skillset and a bit sketchy for me because I have no clue if the repair would last or cause other issues. For all I know it could be perfect or start a fire. I wasn't willing to take a risk or possible waste more time, when I had a perfect solution going forward. 

     

     

     




    User Feedback

    Recommended Comments

    That's exactly what I would have done. First thing that occured to me.......

     

    After all, it's so obvious a solution 🙃

    Share this comment


    Link to comment
    Share on other sites

    I have a 453Pro and had the same issue.  Online is information that the "failure" is due to a bad celeron processor and the voltage degrades over time.  Eventually it will not operate.  The "fix" is to solder in a resistor and bypass one on the pcb.  I got mine working again but it is a temporary fix.  The affected circuit will continue to degrade and eventually be non-repairable...at least the last time I checked.

     

    I have external dr backups.   I have not decided if I want another QNAP or just use external HDs...  Note that I love the QLMS app for the QNAP.  Seems like all the other NAS vendor have stopped LMS support.

    Share this comment


    Link to comment
    Share on other sites

    My only question is how did he ship the NAS to you? I would have been scared to death something happened to the hard drives during shipment. It seemed like you figured out pretty quickly the issue was probably going to be related to the NAS itself and not the hard drives, so why risk having anything happen to them? Not that I would ever doubt your technical abilities, just my own curiosity. 

    Share this comment


    Link to comment
    Share on other sites

    4 minutes ago, Jivatma said:

    My only question is how did he ship the NAS to you? I would have been scared to death something happened to the hard drives during shipment. It seemed like you figured out pretty quickly the issue was probably going to be related to the NAS itself and not the hard drives, so why risk having anything happen to them? Not that I would ever doubt your technical abilities, just my own curiosity. 

    The other option was to personally drive them from California to Minnesota :~)

     

    He shipped the NAS with drives inside as if it was unplugged, packaged, and shipped. 

    Share this comment


    Link to comment
    Share on other sites

    As good a tale about the pluses and minuses of technology as any ... when it works, it most likely will be brilliant; when it goes wrong, it can be almost instantly a nightmare. Something like the saying, "The price of freedom is eternal vigilance." Trouble is, "eternal vigilance" gets pretty wearing - and it's so, so easy to get lulled into a false security about the stability, and reliability of some system.

     

    Well done. As many backups as possible, in as different a way as possible, is usually the best defence. But the trouble quite often is that the backup mechanism itself has flaws, which you never find out until it's needed for a recovery - and it in turn fails, or becomes a massive headache to sort out ...

    Share this comment


    Link to comment
    Share on other sites

    There's a reason iDrive is cheap, not the least that they can access all your data. Don't store anything personal on there. 😀

     

    Ideally, before working on the drives, you would have bit-for-bit cloned them, in case something went wrong during the recovery, though I realise that this was not practical to do. 

     

    I've considered using a NAS for storage, however the very thing I'm afraid of -- the NAS failing, has stopped me.

    Share this comment


    Link to comment
    Share on other sites

    4 hours ago, Currawong said:

    There's a reason iDrive is cheap, not the least that they can access all your data. Don't store anything personal on there. 😀

     

    Ideally, before working on the drives, you would have bit-for-bit cloned them, in case something went wrong during the recovery, though I realise that this was not practical to do. 

     

    I've considered using a NAS for storage, however the very thing I'm afraid of -- the NAS failing, has stopped me.

     

    Granted, its the most expensive way, but two NAS + Offsite backup of those works pretty well. TrueNAS is super simple to back up to Backblaze as well as TrueNAS has taken advantage of the Backblaze APIs and integrated it very well.

    Share this comment


    Link to comment
    Share on other sites

    2 minutes ago, Jud said:

    I would never need this type of help because I would just have a heart attack if this happened to me.

    Ha! 

    Share this comment


    Link to comment
    Share on other sites

    I also had an issue with my QNAP NAS, a twin bay fanless HS251.  I got a virus with a system update about two years ago . Qnap became aware, too late for me, but the issued fix was so complicated and laborious AND I only have Apple OS it seemed insurmountable.  Two terrabityes of a mix of CD and purchased downloads.   In the end I only steam now and find the playlist algorithms make my listening more varied than before and therefore the experience better.

    I'm for  sure the shoemaker with worn out soles.   I had always set up my clients with a 3-2-1 back up because I know drives fail and virus exist.  I label their drives with a date of purchase and replace every two years when failure rate gets to 10 percent. But for some reason for myself I felt invincible in a fire walled system run by a Mac in Raid-1 .  I hadn't  experienced a virus within a OS  system update before.    

    The fix in this article was clearly by a very literate user and a dash of luck.  Well done.

     

    https://www.seagate.com/ca/en/blog/what-is-a-3-2-1-backup-strategy/

     

    Cheers

    Steve

    Share this comment


    Link to comment
    Share on other sites

    I have a qnap nas ts-451 with red wd  4 bay x4tb ea.  I have only had 1 hd failure in last 3900 days.  I run benchmark tests weekly and extensive monthly to check hdd health.  Cool thing is you can setup all these tests run on a schedule and send you notifications.  It’s always good to keep extra hdd drives on hand just in case for a hot swap.  I thought about moving to enterprise level hdd any experience with enterprise server level drives are they worth it?  I was only interested for the write speeds.

    Share this comment


    Link to comment
    Share on other sites

    10 minutes ago, kurb1980 said:

    I have a qnap nas ts-451 with red wd  4 bay x4tb ea.  I have only had 1 hd failure in last 3900 days.  I run benchmark tests weekly and extensive monthly to check hdd health.  Cool thing is you can setup all these tests run on a schedule and send you notifications.  It’s always good to keep extra hdd drives on hand just in case for a hot swap.  I thought about moving to enterprise level hdd any experience with enterprise server level drives are they worth it?  I was only interested for the write speeds.

    Really depends on the drive. Some are only more expensive because they have longer warranties without a performance benefit. Others are really better. 

    Share this comment


    Link to comment
    Share on other sites

    I use TrueNAS on a DIY Supermicro server. I love the Backblaze backup builtin. So far no issues whatsoever. It's very important to have a separate drive for OS and TrueNAS with separate backup of that drive. I use NVMe on the motherboard for the OS etc. One thing I do with every computer that is somewhat critical is use ECC memory. Of course you need a processor and motherboard that support it, Supermicro does a superb job with that. Random upsets (mostly from muons produced by cosmic rays) DO happen and ECC is one of the best ways to protect against that.`

     

    John S.

    Share this comment


    Link to comment
    Share on other sites

    Wow. You're a good friend to do all that. I confess I could only follow along in a general way, as I have no great knowledge of NAS systems.

    I've pondered setting up a NAS on a spare pc for a long time. But I wonder if this route is worth it. For the foreseeable future I will not being buying a commercial-made NAS unit. After looking at much software I considered mostly the openmediavault option.

     

    My pc has 16 internal drives, 12 HDD and 4 SSD's all packed into my top "pedestal" where there is a rack that holds 3 HDD cages of 4 drives each. I have over 100TB of storage, but many disparate drive sizes. Still, my entire ripped surround music collection resides on these drives with backups of folders on separate drives, save most BD's which are on smaller drives sitting boxed up.

     

    Drives are controlled through a LSI SAS 9211 board and a Lenovo extender board.

     

    My backup otherwise is simple. Rip it, (.iso when possible)

    copy to the folder and the backup folder.  Not elegant, not automatic, but just by  habit I suppose.

    I only buy NAS drives as they have held up well.

     

    99.9% of playback is the pc or pc gpu > hdmi > AVR, > 7.1.4 Atmos system.

     

    I wish I had the knowledge of NAS systems some of you seem to have.

    Share this comment


    Link to comment
    Share on other sites

    Yet another instance where people need to realize that RAID is NOT a backup!!!!!

    Share this comment


    Link to comment
    Share on other sites

    10 minutes ago, rhmmmm said:

    Yet another instance where people need to realize that RAID is NOT a backup!!!!!

    It was well known going into this by all involved. 

    Share this comment


    Link to comment
    Share on other sites

    2 minutes ago, The Computer Audiophile said:

    It was well known going into this by all involved. 

    I’m talking about other people. So many folks I talk to think a NAS or RAID is functionally a backup. The word needs to be spread better so others stop making this mistake. 

    Share this comment


    Link to comment
    Share on other sites



    Create an account or sign in to comment

    You need to be a member in order to leave a comment

    Create an account

    Sign up for a new account in our community. It's easy!

    Register a new account

    Sign in

    Already have an account? Sign in here.

    Sign In Now




×
×
  • Create New...