How Good Is Your Rollback Strategy? – 5 Ingredients For A Successful Rollback

Have you ever found yourself in the middle of a change and all of a sudden the decision to rollback the change is taken? Did you have a plan to follow? Or did you wing it? If the latter was the rollback successful? Or did you miss a step, that came back the next day to bite you? I’ve been there, done that. And because I learn from my mistakes, I don’t wing rollbacks anymore.

Why do I need a ROLLBACK Plan?

You need a rollback plan because:
– deployment could go South,
– you need to know how to restore the database or application to the initial state, the one it was pior to the change.
– steps got missed in the deployment
– change management dictates that rollback instructions be provided
– to sleep better at night

When the decision is taken to rollback, it is a critical moment, stress levels are high, panic sets in, and even worst, it is the middle of the night. Thinking clearly and sequentially is hard. This is when most mistakes are made without a proper plan.

Before defining what the rollback plan is, let me clarify what a rollback plan is NOT (based on personal experience):

– it is not a conversation in the lunch room about what can be done if a change fails.
– it is not a scribble on a post-it.
– it is not a “Oh-I-know-what-I-need-to-do-no-need-to-write-it-down” idea.
– it is a general statement, such as “recover database from backup”
– it is not Google search at the last minute, or Oracle Support

A rollback exists only in the context of a change.

If you work in the IT world, most likely you heard of SOX or CSOX compliance. In companies that have to be SOX or CSOX compliant there must be CHANGE MANAGEMENT in place. This means that any changes that go into a production system, have to go through proper testing and approvals. Even in companies that don’t have these compliance needs, common sense dictates to have some sort of Change Management. (If you don’t have it in your shop, you can start now…it’s never too late)

What is a CHANGE?

I consider change, any type of new code deployment, parameter changes, upgrades, patching, outages, even some password changes. Some of these changes will not require testing all the time, such as an outage (you test once, record the steps and you are done). Some of these changes will require extensive testing and multiple deployments.

In a perfect world, the developers develop and test their code in dev and test, the DBA deploys the code to pre-production system. Once the pre-production deployment is successful and the change is tested, the DBA deploys the change to production environment. In order for the pre-prod and prod deployements to be successful, there must be a MAGIC INGREDIENT, without which the deployment is doomed to failure. Can you guess what that ingredient is? I will not leave you guessing…
The MAGIC INGREDIENT is called: THE DEPLOYMENT PLAN.

Remember this:
Without A DEPLOYMENT PLAN, a CHANGE or DEPLOYMENT is doomed to fail!

What is a good DEPLOYMENT PLAN?

In my view, a good deployment plan is a document that outlines the following, and it is not limited to:
– people involved in the deployment
– preparation steps, I call these pre-deployment steps
– actual deployment steps, with peoples’ name attached (who is performing the actual step)
– post-deployment steps
– rollback plan

You get the picture where the rollback plan fits in.

What is a ROLLBACK PLAN?

– it is a WRITTEN DOCUMENT that outlines (just like a deployment plan), all the steps that are required to recover the database or application to the point before you started the deployment.
– it is a PLAN, with specific steps for each participant in the deployment.
– it is a TESTED PLAN. This is a very important distinction to make. The Rollback needs to be tested, just like the deployment, in order to determine all the components of the plan.
– it contains the location of scripts that need to run
– it contains the database name and username that needs to run the scripts

The 5 key ingredients for a successful rollback are: (1) written, (2) tested, (3) detailed step by step plan, that contains the following (4) location of the scripts, (5) database name and user who runs the scripts.
Because a picture is worth a thousand words, let me show you the difference between a deployment plan without rollback, wannabe rollback, and an actual rollback plan.

Remember this, A Good DBA always has a rollback plan!

–Diana