Dealing with Database Drift
It’s almost midnight on a Saturday evening. By this point, you know that sleep and relaxation are not on the agenda for the weekend. The production deployment of a major end-user facing application has gone horribly wrong. Your phone is blowing up, your inbox is blowing up, and with bleary eyes, you take a moment to consider your career choice. This should have been a smooth and painless deployment. But, since it’s your turn to take a weekend away from the family, something has gone wrong – and it’s with the database.
This all too familiar scenario is often because the target schema of the production database has drifted from what it was supposed to be. Database drift happens because hotfixes and patches are often applied to staging and production systems in panicked efforts to rapidly restore functionality or address critical system flaws. Often, these late-night, last-minute changes are not properly documented. Nor are these changes applied to lower environments and are often at the heart of show-stopping surprises during later production deployments. In the worst case, database drift can result in data loss. In the best case, it is a delay and time lost reworking a failed execution.
So, how does one get rid of or mitigate database drift? It’s the timeless, basic recipe of organizational change, process change, and proper tooling.
Solution #1: Create Transparent Organizational Structures
Organizationally, teams need to be more transparent and cooperative as changes flow from development to production. Database administrators (DBAs) might sit on a separate team and act as a shared service to different application teams that all rely on the same database. But you can’t just fire off help tickets and expect that everything will turn out okay. Instead, you have to create open, transparent communication between development and operational teams. You have to build a shared sense of responsibility for database change deployments in all environments – production or otherwise.
Solution #2: Address “Quick Fix” Processes
In addition to organizational habits, you have to adjust processes. Hotfixes and patches are often made in a hurry to higher environments because of a mis-match between application and database code changes or because of issues that were discovered very late (often in production). To prevent such scrambling, organizations need to invest in a more robust testing process. Teams need a unified and transparent path for application and database code as it moves from development to production. So long as application and database code take separate paths to production, there’s a risk of mismatch or error. Separate paths lead to the conditions that produce database drift and service outages.
Solution #3: Automate Systems Where Possible
Database drift can be addressed with appropriate automation solutions. Manual Db deployment processes yield poorly documented, custom, hard-to-reproduce deployments that contribute to chaos and confusion. By investing in automation tools that can verify database code changes and package verified changes into an immutable artifact for downstream deployment. Database deployment automation can also perform transparent, auditable, and repeatable deployment, ending most of the database drift. And, with tools that can snapshot and compare databases as part of an automated and regular process, you can quickly catch any “out-of-process” changes and fix problems fast.
Ultimately, market expectations have evolved. This makes it necessary for teams to deliver new capabilities more quickly and at higher quality. It is critical to ensure that database drift doesn’t creep up while attempting to speed up software releases. Database drift can do major damage in the form of service outages, brand and reputation damage, data loss, and increased time to resolution. Without making the necessary changes in culture, process, and tooling, attempts to speed up software delivery will fail.