In the last few weeks I’ve been playing around a lot with servers running as VMs on a test instance setup at home on some old hardware.¬†Recently I experienced an issue where my home router, which was configured to send alert emails whenever anything above ‘info’ level was logged, attempted to offer a DHCP lease to a client. On this occasion the email service failed to send the email which then generated another ‘error’ log message which in turn tried to send another email, leading to a cascade which overloaded the router’s systems. In the first instance this caused the entirety of my home network to go down for a few moments whilst the router reset itself. However this reboot cycle continued once the router tried to reissue the same static lease again to the same failing client. At this point one of my family took it upon themselves to try and fix the problem and in doing so performed a physical reset of the bare metal that was hosting the instance of Xen Server which all my VMs run on. This had the effect of performing an unsafe shutdown of all the virtual machine as well as an unsafe shutdown of the hypervisor itself. Although the Xen Server host restarted, none of the VMs automatically restarted which meant the LDAP server that my VPN authenticates against wouldn’t accept any new connections. So I couldn’t remote in to my home network to fix the problem and just so happened to be in Japan at the time!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>