If you have been in the technology field for more than a few weeks, you know that people make mistakes. Someone with administrator-level access can make big mistakes. One wrong line of code can either make a program behave in strange and unexpected ways, but that program might also do some serious damage if the user account running the program has elevated privileges.
There is a story that details how a system administrator incorrectly ran a line of code to delete all the files on his server. While the story is sad (and kind of funny at the same time) it can also be a lesson for the IT crowd reading about this now. This person did some basic things that led to this mistake being so very serious:
- Making untested changes on production servers
- Storing backups on production servers
- Not having a written Incident Recovery or Business Continuity plan
This individual has now probably destroyed his business with a simple mistake. He might be able to retain is business partners and rebuild his business from scratch, but most of these businesses will probably not trust him enough to continue to pay him for professional services.
As you reflect on this person’s pain, remember these important lessons:
- Create a written Incident Recovery Plan
- Test your Incident Recovery Plan once per calendar year
- Create frequent backups and store them off-site, separate from any production servers
- Always use a test environment to create and test changes
- Only run tested and verified code on production servers
- Always plan for the worst and hope for the best