Network Recovery After Outage

[audiobomber]

We had a series of short power outages two days ago. The modem, router and NAS are on UPS, but the outage caused several of devices to drop off my network. I have had to do a lot of booting and rebooting of the NAS, router and devices. The worst was Raspberry Pi. Volumio mobile couldn't find it, so I could not turn it off with the software command. I had to unplug it and when it came back, there was only a blank page. I had to re-image the SD card.

 

Is there a recommended procedure for attacking this issue? Should I turn everything off from upstream to down, then start up from the other end?

[The Computer Audiophile]

If I see something like this I shut everything off. Then I start at the modem and work backward powering things on. I wait for each device to come up before moving on. 

[ipeverywhere]

1 hour ago, bluesman said:

Using a combination of fixed & dynamic IPs carries the risk that your DHCP server will assign an IP address already in fixed use.  I think the most common reason is that whatever device is managing DHCP comes up after your networked devices are already up. 

 

This is actually a massive bug/flaw in most "consumer" grade DHCP servers which run on routers. Within the DHCP configuration there is a subnet declaration. This tells the DHCP server when it should respond (it will "hear" a request on the broadcast domain of that subnet). Within that declaration there is a pool, or range, of addresses that the server is allowed to lease, or assign, to the requester. Most consumer grade stuff allows the entire subnet into the pool excluding only its local IP. Some implementations let you explicitly exclude individual addresses. Some let you exclude a range. But... that's a little backwards to the way it is supposed to work if you cannot configure the pool.

 

In my network my pool is always set at 50% or 75% of the entire subnet range. I exclude the higher addresses from my pool. That way if I need to assign a static IP I know the DHCP server has not leased an address from the block that lives outside the pool. Stuff that needs to be configured via a web GUI I generally assign an IP so I can find it easily.

 

Address duplication should never happen if you're able to limit the size of your pool and/or at least start excluding the addresses you want to statically configure in the DHCP server itself. 

 

There is, of course, nothing wrong with statically assigning everything but that can be a bit of a PITA if you're having to re-image devices and start over. In that case I'd highly recommend mapping ethernet MACs to static addresses in your DHCP server. This lets the device run DHCP but you configure it with the same address every time at the DHCP server. It centralizes your address assignments and makes your devices just use DHCP as default. How to do this varies from server/router vendor. Some make it as easy as just clicking on the device list and telling it to make that assignment static. 

 

One other thing I did not explicitly point out in my previous post. If your DHCP server restarts it is worth turning _everything_ off and starting at the top of my list. This includes things like mobile phones and tablets. Most DHCP servers do not maintain the lease DB between restarts. However, a lot of devices will hold onto their addresses even if their lease expires and cannot be refreshed. What can happen is that a device will hold address X, the DHCP server will come up without any lease state, a request comes in for an address, and the DHCP server assigns address X because it didn't know the other device held onto it. Only safe way around this is _force_ everything using DHCP to request a new address. This will properly build the lease DB at the server/router. 

 

 

[audiobomber]

18 hours ago, The Computer Audiophile said:

If I see something like this I shut everything off. Then I start at the modem and work backward powering things on. I wait for each device to come up before moving on. 

I know one is not supposed to unplug a computer without powering down. With the Pi this time, and the sms-200 at other times, I seem to have no choice. The device is not showing up in the network, and doesn't have a power button. The sms has a reset button, but I assume that would do a lot more than power down. 

[The Computer Audiophile]

9 minutes ago, audiobomber said:

I know one is not supposed to unplug a computer without powering down. With the Pi this time, and the sms-200 at other times, I seem to have no choice. The device is not showing up in the network, and doesn't have a power button. The sms has a reset button, but I assume that would do a lot more than power down. 

I wouldn't worry about powering off either of those devices without a power button.

[ipeverywhere]

I work up the "stack" when powering back on. Basically.. less "smart" to most "smart". For me that is:

 

1) Switches (wait for full boot)

2) Routers 

3) Cable modem (can be turned on while router is booting)

*At this point, in my network, the DHCP server is up (runs on router), the Internet is up, and the switch will allow everything to communicate to each other once those devices come up. If you run a separate DHCP server then continue to 4 and make sure that is functional before continuing. 

 

4) Power servers and any network storage that is hardwired to the switch(es). (I define "server" as any device that is providing a service to the rest of the network. For example I run a RADIUS server that provides authentication for Wifi clients).

5) Power the wireless Access Points (APs) or Mesh Wireless if you've got it. (wait for it to fully come up)

6) Power any servers or network storage that attaches to the network via Wifi. 

7) Power everything else including computers, tablets, etc..  

 

 

Now test by pinging gateways, testing services, verifying Internet connectivity, and run a speedtest both from hardwired and wireless. Do this from a hardwired device and wireless device. If this works but you still have specific devices that didn't come back onto the network you can be pretty confident it is an issue with that device and not the network itself. Troubleshoot as if you are installing that device for the first time. 

 

If your Raspberry Pi is running Linux it should still be shutdown properly and not simply powered off. For a whole host of reasons, removing power from a Pi is a lot safer and less risky than your general purpose computer but without letting the system flush cache and unmount the disks there is still a risk of data corruption (although the way most people use their Pi's minimizes this risk). *Full disclaimer: Even though I just said "should" I will admit that I do not properly shutdown my Pi. I'm fully prepared to re-image if I damage the filesystem and it does not come back. 

 

 

[bluesman]

Been there, done that - it's a bummer any way you view it! But I've found a few things that help.  DHCP conflicts and failures are not rare after power surge &/or loss.  Using a combination of fixed & dynamic IPs carries the risk that your DHCP server will assign an IP address already in fixed use.  I think the most common reason is that whatever device is managing DHCP comes up after your networked devices are already up.  So I've been disabling DHCP & running fixed IP addresses on all devices on my networks (LAN AND WLAN) for years.  I also keep a running record of all device IP addresses, after discovering an occasional reversion to DHCP when a device comes back up after a power glitch. I just screen capture the "connected devices" chart whenever I add or change anything.

 

I'm currently running 4 Pi 3B+s, a Pi 3, a Pi Zero W, a Beaglebone Black, and a Beagleboard xM (I'm evaluating and comparing several OSes, players, and "minimal devices" for review articles).  I've never hesitated to unplug any of them if an installation fails, I can't SSH into it, or I lose connectivity to it for any other reason.  I've probably unplugged the same BBB a hundred+ times since Chris's MPD player how-to 6 or 7 years ago, and it doesn't seem to have affected anything adversely.  I do the same thing when a Pi runs away from its network home for any reason, again with no apparent bad effect.

 

My router and switches come up first whenever we have an outage.  Peripherals may come back on their own or require powering up. With 30-some devices on my LAN, I can't get around to them all in time - and it doesn't seem to matter. I credit the fixed IP addresses for most of the stability.

[audiobomber]

Thanks guys, I did have address duplication after the event and wondered about assigning more static addresses. I'm still having trouble maintaining the connection to my TV's and my main laptop. I will investigate how to assign static IP to these and the Pi.

 

I have Fing which has been invaluable in troubleshooting.   

[audiobomber]

We had a two-hour power outage yesterday. I suspect they're having trouble with ground water from the spring melt. While power was off, I unplugged all devices. When power returned, I restarted the modem, router, NAS, extenders and 11 receiving devices in that order. No IP conflicts or trouble with wi-fi on any of the devices for the first time ever, but the procedure was bothersome, especially since the NAS takes so damned long to shut down and start up. There are times when we have a few short power interruptions in a row. I'd hate to go through this every time. 

 

My router assigns IP addresses in the 100-199 range. Of the receiving devices, only the sms-200 has a static IP now (#57), but I intend to assign static IP's to the more problematic devices. Thanks again for the replies.

[Kal Rubinson]

1 hour ago, audiobomber said:

There are times when we have a few short power interruptions in a row. I'd hate to go through this every time. 

That's why many of us keep them on battery-backups.

[bluesman]

1 hour ago, audiobomber said:

I intend to assign static IP's to the more problematic devices. Thanks again for the replies.

 

I urge you to assign static IP addresses to every device that’s part of your network infrastructure. Beyond reducing risk of mishap, it’s a lot easier to replace and upgrade devices without disrupting function - you just assign the same address to your replacement, and everything will usually continue to function without interruption. Only if you also use secondary controls like MAC filtering will you have to reset or change anything.

 

I keep screen shots of my router tables and settings, as well as all device settings. You can’t track too much info - you can only regret what you didn’t save.

 

Keep a clear and current record of your network device map with IP & MAC addresses, so you can easily find and fix problems without creating new ones. Hope you never need it, but sleep better because you have it.

[audiobomber]

55 minutes ago, Kal Rubinson said:

That's why many of us keep them on battery-backups.

My modem, NAS and router are on a UPS. I have wireless devices all over the house, battery backups for all would be impossible, especially the TV's.

[Kal Rubinson]

1 hour ago, audiobomber said:

My modem, NAS and router are on a UPS. I have wireless devices all over the house, battery backups for all would be impossible, especially the TV's.

OK.  I was really speaking about only those devices.  Mebbe you want bigger batteries. 😊

[audiobomber]

1 hour ago, Kal Rubinson said:

OK.  I was really speaking about only those devices.  Mebbe you want bigger batteries. 😊

My understanding of the purpose of the UPS is to allow for a safe and orderly shut down. At some point, the batteries will run out of juice and the connected gear will crash. I shut the system down before that can happen.

 

I'm not sure how long the UPS would keep going, I've never tried. I think the NAS may be able to detect low voltage and power down on its own. My practice so far has been to shut down if the power is out for more than about 15 minutes, which is very safe. Maybe I"m being too cautious.

[Kal Rubinson]

1 hour ago, audiobomber said:

My understanding of the purpose of the UPS is to allow for a safe and orderly shut down. At some point, the batteries will run out of juice and the connected gear will crash. I shut the system down before that can happen.

 

I'm not sure how long the UPS would keep going, I've never tried. I think the NAS may be able to detect low voltage and power down on its own. My practice so far has been to shut down if the power is out for more than about 15 minutes, which is very safe. Maybe I"m being too cautious.

There are setups where the BBU can signal the NAS (or other devices) at a particular point in its capacity, say 30mins left.  That is more convenient than shutting down immediately or waiting for BBU to expire.

[ipeverywhere]

2 hours ago, audiobomber said:

My understanding of the purpose of the UPS is to allow for a safe and orderly shut down. At some point, the batteries will run out of juice and the connected gear will crash. I shut the system down before that can happen.

 

I'm not sure how long the UPS would keep going, I've never tried. I think the NAS may be able to detect low voltage and power down on its own. My practice so far has been to shut down if the power is out for more than about 15 minutes, which is very safe. Maybe I"m being too cautious.

 

It can be used for both. I have an excessively large UPS in my closet which can stay up for ~90 minutes with the load I currently have on it. The UPS at my desk is about ~8 min. At my desk I power everything down within 3 minutes but my server / network gear I'll let go for at least an hour in the hope that the power company can get over to the fault before my batteries run out. 

 

If you don't have a display on your UPS then there are calculations to determine the run time. The inputs require knowing a bunch of info which is easier to look-up worst case scenario (max wattage draw) but not as easy, would require a meter, to get actual wattage of your gear plus the AH rating of the UPS. You're doing the safe thing by shutting things off before the UPS runs out. Also, the lead acid batteries in the UPS don't really like to run completely down so best to get the UPS turned off before that occurs. 

 

Some UPS have network or serial ports where they can report their status. Gear that can use this status will then trigger a controlled shutdown as the UPS gets low on power. That requires support of both the UPS and software on the device plus set-up. Nothing will detect and shut itself down by monitoring the AC voltage. The UPS will shut itself down well before the AC voltage drops to a "I'm in trouble" level. Basically, the UPS should be outputting 110v (220v or whatever in your country) right up until the point the batteries in the UPS drop to a voltage the UPS is no longer able to invert DC to AC at the load being demanded by the connected equipment. At which point the UPS just shuts down.