Disaster Preparation Tips from a CEO’s Perspective

A few days after Ike hit us, I was dealing with several decisions we needed to make as a result of the impact of the storm.  I decided to make a quick note to my self (using my handy dandy Jott-RTM-ToDo combo on my iPhone) so that I could come back later and create a more detailed summary of those lessons.  I created an entry titled simply “Lessons from Ike” and below it, I wrote the following:

1. Always choose a co-location facility that is completely self-contained
2. In a lease agreement, always look at the time the landlord has to restore the building before you can walk away from the lease
3. Always take a hurricane seriously

So, here I am more than two months after the storm – still at a temporary location and not quite certain when we will be back in our space. The landlord is telling us mid-January, but that seems hard to believe.  Our team has been awesome and has made a huge effort to adapt to this crazy, no-office scenario, but it’s getting old!  I want to expand on those 3 quick initial thoughts that came to mind right after the storm so that the lessons become part of our corporate memory … Plus, I told Chris Pitre I would send him this to post on our blog, and he will drive me crazy until I do it!

1.  Always choose a co-location facility that is completely self-contained

We have used the same co-location facility for almost 10 years (it is now owned by Data Foundry).  We have never had a problem before… But, then again, we had never been hit this hard by a storm since we moved our servers there.  The facility lost power, like everybody else, their back-up power kicked in, BUT the chiller that allows for the cooling of the space broke.  Data Foundry told us it was not their fault.  That particular point of failure resided with the building itself and it was out of their control!  I want a co-location provider that controls everything in that facility.  Considering the seriousness of what was going on around us, we were not down for a long time. BUT, this scenario was exactly why we pay the cost of co-location year-round… For that one day when a hurricane hits.  The bottom line is Data Foundry failed us… And I wanted to kick myself for having overlooked what seems like such a logical requirement.
If you are going to co-locate your servers somewhere, make sure your co-location provider cannot say that they are down because of a failure they do not control.  Things will fail eventually, and you want one entity to be accountable for everything.

2.  In a lease agreement, always look at the time the landlord is allowed to restore the building before you can walk away from the lease

Lease agreements are huge, and because they are supposed to be pretty standard, I have tended to not use a fine-tooth comb when it comes to reviewing them.  WELL… I should have… It turns out our landlord has 12 months to fix our space before we can walk away from the lease.  60 or 90 days would be acceptable, but 12 months?!!!  Lawyers are paid to think of worst-case scenarios and build contracts that address those.  A tornado hitting our building during Ike, the roof of our building flying off, our office located on the top floor… I really needed to read that contract from a worst-case scenario perspective to have concluded that particular clause mattered.

3.  Always take a hurricane seriously

It seems obvious, right?  But, living where we do, we tend to assume the storm will never hit us and get lazy with the preparations.  We prepared, but we did it all with the unspoken assumption that Ike would not affect us that much.   Believe me… We all need to prepare as if the storm is hitting us for sure… It’s worth the hassle.

There are some important post-Ike lessons I have jotted down and will need to expand on and save for a later post.  Our company continues to operate, serving clients and advancing the developments of our products, thanks to the great attitude and dedication of our team members… We were hurt, but not injured.