Tag Archive for 'downtime'

DNS, DNS, DNS……..

Anyone notice some downtime earlier this morning? First of all let me apologise, this was due to our main nameserver crashing after an automated system update. We were first made aware of the situation around 04:00 this morning, and we managed to rebuild the server by 06:00 however during this time, DNS servers around the world and cached invalid IP addresses for many of the websites on our system. Once our nameservers were sending out the correct IP addresses the cache was updated and the websites should have magically reappeared! Most of the websites were up and running by 10:00.

Our sytems administrators with the help of an external company are trying to find out exactly what caused the crash, however in the meantime, we have moved our nameservers onto new hardware and moved NS3 and NS4 onto new clusters to ensure this DOES NOT happen again.

I hope any customer that has been with us for a while knows that this sort of thing does not happen often, and as for our new customers, I cannot apologise enough and I hope we have not put you off.

I want to take this time to thank all our customers, and to wish you all a happy and successful 2009! Good luck with all your ventures!

Power Outages…

On the morning and afternoon of the 2nd of July we had two power failures. The first knocking out our VPS’s for 4 hours, and our shared platform for 5 minutes, the other in the afternoon knocking out all platforms for 5 minutes. Ok, 5 minutes is not long but it caused a number of problems for us.

Better yet, everyone else’s racks where still running! What went wrong?

Well according to the staff at BlueSquare, the nightshift staff where working on some nearby racks and managed to ‘dislodge’ out some power cables! After getting everything back up, we raised a ticket and got a reply with an apology which said:

we’ve moved the other customer’s cabling to avoid this from re-occuring.

Well then why on earth did this happen again! We are let to find out. We have also been looking into the new BlueSquare MK Datacentre. This would help us keep our costs low (even with the increased cost in power) as well us giving us much more room to expand.

Apologies for any inconveniences caused, and as usual we will keep you updated with any updates.

In the mean time we are fitting our own UPS units in all our racks to ensure if something like this happens again, your data will be available.