Dealing with Database Drift
It’s almost midnight on a Saturday evening, and you know that sleep and relaxation are not on the agenda for the weekend because the production deployment of a major end-user facing application has gone horribly wrong. Your phone is blowing up, your inbox is blowing up, and with bleary eyes, you pause to take a moment to consider your career choice. This should have been a smooth and painless deployment, but of course, since it’s your turn to take a weekend away from the family, something has gone wrong – and it’s with the database.
This all too familiar scenario is often because the schema of the production database has drifted from what it was supposed to be. This commonly happens because hotfixes and patches are often applied in haste to staging and production systems in panicked efforts to rapidly restore functionality or address critical system flaws. In many cases, these late-night, last-minute changes are not properly documented nor applied to lower environments and are often at the heart of show-stopping surprises during later production deployments. In the worst case, database drift can result in data loss, and in the best case, it is a delay and time lost reworking a failed execution.
So, how does one eliminate or mitigate database drift? It’s the timeless, basic recipe of organizational change, process change, and proper tooling. Organizationally, teams need to be more transparent and cooperative as changes flow from development to production. While database administrators (DBAs) might sit on a separate team and act as a shared service to different application teams that all rely on the same database, it’s important to not simply fire-off help tickets and expect that everything will turn out okay. Instead, a critical element of the solution is establishing more open, transparent communication between development and operational teams, and building a shared sense of responsibility for database change deployments in all environments – production or otherwise.
In additional to organizational habits, it’s important to adjust process. Hotfixes and patches are often made in a hurry to higher environments because of a mis-match between application code changes and database code or because of issues that were discovered very late (often in production). To prevent such scrambling, organizations need to invest in a more robust testing process and need to have a more unified and transparent path for application and database code as it moves from development through to production. So long as application and database code take separate paths to production, there’s a risk of mismatch or error that leads to the conditions that produce database drift and service outages.
Lastly, database drift can be addressed with appropriate automation solutions. Manual database deployment processes yield poorly documented, custom, hard-to-reproduce deployments that contribute to chaos and confusion. By investing in automation tools that can verify database code changes, package verified changes into an immutable artifact for downstream deployment, and preform transparent, auditable, and repeatable deployments, you can eliminate most of sources of database drift. Furthermore, with tools that can snapshot and compare databases as part of an automated and regular process, you can quickly catch any “out-of-process” changes and reconcile accordingly.
Ultimately, as market expectations have evolved, making it necessary for organizations to deliver new capabilities more quickly and at higher quality, it’s critical to ensure that database drift doesn’t creep up while attempting to accelerate software releases. In the long term, database drift can do substantial damage in the form of service outages, brand and reputation damage, data loss, and increased time to resolution. Without making the necessary changes in culture, process, and tooling, attempts to accelerate software delivery can backfire.