Those words are all to familiar to the Staff at Northwoods who survived the Great Crash of 2006, but now those words are being heard… by customers of Charter Communications.
Fox news reports that Charter Communications was performing maintenance Monday night to their systems and inadvertently deleted over 14,000 email accounts that were used by their active customers. And oh yea there was no way to recover the data… but they are giving the affected customers $50 credit.
My favorite quote in the article “It’s never happened before. They are taking steps to make sure it never happens again.” Duh, you think? Talk about a credibility killer.
Maybe its my total lack of compassion and I should feel sorry for the poor staff at Charter Communications…. for whom I dare assume this was a resume generating maintenance night, but I have to admit it makes me chuckle when it happens to the big guys.
I should have some minor compassion especially when we experienced the crash of 2006 the second week I was on campus at Northwoods. But while there wasn’t a backup of the data we were able to recover the mailbox store we thought was a complete loss…
Note: We don’t use that Backup Strategy anymore…
Learning Moment: Is your backup plan configured in a way that you can recover from any dumb admin error any of us could make during a maintenance window at 2 am? Not to mention have you tested the plan?
We learned our strategy is sufficient to have a complete failure and not lose data, but we are still tweaking the system for the future… We learned in our recovery we had covered the basics but we need to improve the recovery window. So we to need to test some more. (Note I use the word ‘test’ assuming a test is not a recovery of data in a actual failure )
We all should take note.. this could happen to any of us… we all make mistakes maybe not on this grand of scale, but we all make mistakes. We should have systems in place that can tolerate not just our users but US too.
Just because we don’t support as many users but a proportionally sized event in our organizations would be as catastrophic and equally as much a ‘credibility killer’.. what say you?