Journey to Database Change Automation

Mabuti Ng’andu Director of IT, Data Management, Database & Middleware at MedImpact Healthcare Systems, walked us through his team’s journey to database deployment automation. Here are the key takeaways from his DevOps.com webinar.

MedImpact’s journey to improve their database deployment throughput and quality required three main pieces: people, updated processes, and tooling changes. But to even start the journey, it also required a justification process to enable the investment. Here’s an overview of the challenges MedImpact faced and how they overcame them to get to implementation.

Treating database code differently

Many – if not most – organizations are guilty of treating database code differently than other code stacks. There are a couple of reasons for this.

1. Very few organizations have dedicated database developers, unlike they do for other specialized code for Java, Python, Swift, etc.
2. Databases and database code have not traditionally been easily managed by commonly used software deployment processes or toolsets.

These differences can lead to occasional deployment mishaps resulting in slow deployments or costly mistakes that are difficult to roll back. Ironically, these mishaps, though costly, are simply viewed as process failures due to resource or team discipline challenges. This makes the case to justify an investment in database deployment automation a tough task.

First, let’s get into why MedImpact was looking to improve its database deployment process in the first place.

Why database deployment automation is important to MedImpact

• Their value proposition dictates unique and agile views of data to support the reporting and analytics behind their service offerings
• Data objects need to be created and/or modified quickly, with reliable quality
• They guarantee high availability and quality-related performance to customers
• They began the process of adopting Agile software development methodologies to be even more responsive to market needs

All of the above points mandate reliable database change management practices that minimize or eliminate errors.

Database Deployment Challenges

There were three types of challenges to overcome on their journey: process, team, and outcome challenges.

Process challenges

Like most companies, MedImpact did not treat database deployments with the same rigor as software deployments. Although database source code lived in Git, the traceability of the code from checkout to deployment was manual and limited. Database deployment components were not part of the core build package. In fact, at times database changes were deployed days or even weeks before the software deployments.

Deployment processes were mostly manual, save for some scripted verifications/validations. This caused a number of problems:

• Long, labor-intensive deployment processes (not Agile!)
• Inconsistent deployments across the enterprise
• Incomplete deployments within databases
• Limited ability to audit when changes occurred in a given environment
• Limited visibility into, and limited ability to review, deployment failure points
• Labor-intensive, manual validation/verification of all of the above

Team challenges

One major challenge, from a people-perspective, is that three sets of teams had the ability to deploy DML or DDL and each team had their own set of deployment processes:

• The DBA team deployed all DDL with the exception of application-specific PL/SQL
• Software engineering teams deployed application-specific DDL via configuration management processes
• The DBA team, software engineering teams and even some business teams could deploy DML through manually controlled configuration management processes.

This variation of deployment processes MedImpact had limited the visibility and traceability of deployments.

Outcome challenges

Even though MedImpact employed rigorous manual verifications (unit, QA, test and validation, as well as UAT), they had a few instances of deployment failure. In those cases, additional challenges were introduced, including:

• Long, arduous debugging processes for the DBA, Dev, QA and testing & validation (business) teams
• Unexpected post-deployment incidents/problems in QA, UAT and production environments
• Longer software delivery timelines
• More rework

The Solution: Database Deployment Automation

There were two primary objectives for MedImpact’s Database Deployment Automation initiative:

1. Create a process that provided a consistent framework for all database deployments including DDL, PL/SQL and DML to all non-COTS Oracle databases using the same rigor for DB deployments that was expected in software deployments
2. Identify and acquire a tool that would streamline database deployments with the following capabilities:
3. Enforce automated deployments from the source code repository (Git)
4. Deploy predictably, repeatably, and consistently to individual or group targets (one QA database, or all QA databases, etc.)
5. Log all deployment script outcomes for immediate or future review
6. Integrate with standard DevOps tools, including the Atlassian stack
7. Audit deployments

Getting Executive Buy-in for Database Deployment Automation

Now that the MedImpact team knew what they were looking for, it was time to present it to executives for approval. They thought, “this is the obvious answer to our problems. This is a no-brainer.” Well, making this kind of investment isn’t always easy to make to C-suite approvers.

First attempt at executive justification

The first-round justification described the potential value of adding database deployment automation.

• Improved deployment quality
• More Agile turnaround

These points weren’t enough to sway the executive team. Mabuti and his team were asked why this same value couldn’t be achieved with more efficient processes and better ‘people management’. It’s not always inherently obvious to executive decision-makers that you can only manage fatigue to a point. Manual processes always carry a certain error rate (to err is human, after all) and throwing more people at a broken process doesn’t make it better.

Second attempt at executive justification

Mabuti had to go back to the drawing board to get executive buy-in. In the next round, he described the potential value of automated database deployments.

• Improved deployment quality
• More Agile turnaround
• Reduction in deployment costs
• Reduction in rework, associated costs, and schedule impact
• Seamless integration with planned CI/CD pipelines

MedImpact’s second justification summary included the following components:

• An opportunity cost calculation including:
• Labor cost of slow deployments through all environments (DEV, QA, PRD)
• Labor cost of manual verification
• Cost of deployment-related rework
• A roadmap to achieving automated deployment value, including:
• Tool evaluation and selection process
• Pilot (POV) of tool to prove it could deliver on value
• Alignment on success criteria
• Reference calls with enterprises who have implemented selected tool

Why MedImpact Chose Liquibase Enterprise

MedImpact evaluated four tools and agreed that Liquibase Enterprise was the best fit for their needs. The team executed a Proof of Value, meeting all success criteria. All executive leaders agreed to make the investment.

The Implementation Process with Liquibase Enterprise

The MedImpact team decided to pilot the Liquibase Enterprise implementation on two applications: one that was simple and one that was a bit more complex so that they could modify their database deployment processes and train people.

Today, MedImpact is integrating Liquibase Enterprise into its broader continuous delivery ecosystem.

Key Takeaways

• Be very clear on what your objectives are
• Invest time in articulating the value that automated database deployments provide. It may be “obvious” to the technical team, but it must still be well articulated to executive leaders Implementation
• If you are not already actively Agile, plan time to work through process questions (e.g., Who owns deployments in which environments? What is the process to trigger a deployment? Who needs to approve it? etc.)
• Solicit input from and stay well aligned with all deployment stakeholders – DevOps, software engineering, database administration/data management, configuration management, release management, etc.
• Never forget that this is a CHANGE MANAGEMENT activity.
• Identify the advocates and utilize them to drive change.
• Communicate 7 times, 7 ways.
• Prioritize those who understand the vision and the rest will follow

Check out the webinar presentation by MedImpact: MedImpact’s Journey to Database Change Automation