Automate BigQuery schema change and version control with database DevOps

Google’s BigQuery is a fully managed, serverless cloud data warehouse, or database as a service (DBaaS), that brings unparalleled scalability and convenience to data analytics. With built-in functionality for high throughput streaming ingestion, machine learning, geospatial analysis, data sharing, and log analytics, it sets the benchmark for data management. 

BigQuery is integrated natively with various Google Cloud services like PubSub, Dataflow, Vertex AI, and Looker, making it a comprehensive solution for organizations looking to leverage the power of their data without the hassles of traditional data warehouses.

Yet, with all these capabilities and connections, managing BigQuery schema and application data changes can still be tedious, manual, and error-prone. Hence, the need for Liquibase Pro’s streamlined approach to BigQuery database schema management and version control. 

The Liquibase BigQuery extension enables efficient version control and database change management for BigQuery schema and application data. This extension is the latest addition to our 59+ supported databases and gives Google BigQuery users a smooth, streamlined approach to database change management and deployment, fitting effortlessly into Agile development and CI/CD automation practices. 

Manage BigQuery in the CI/CD pipeline with Liquibase

With the Liquibase BigQuery extension, you can bring database schema and data changes into your CI/CD pipelines. This fusion allows for the automatic validation and propagation of database changes into specified environments, offering a new level of automation and precision to your database change management process.

As your application code is deployed across development, test, and production environments, so are the corresponding BigQuery database changes, in lock-step, ensuring consistent and complete application deployments.

Liquibase Pro helps with BigQuery change management so data professionals can bring database DevOps to their teams. By automating database change management, Liquibase Pro enables: 

• Faster, consistent validation
• Faster, more compact, and more frequent deployments
• Fewer delays
• Lower overall failure rates
• Reductions in time spent on deployments
• Increased deployment success
• Faster audits

Get started with BigQuery and Liquibase or check out a database release automation demo to see how Liquibase can help you deliver faster and safer with database DevOps.

Why database DevOps and BigQuery?

While DevOps practices are the norm for application teams, data professionals aren’t consistently included until much later – if at all. 

The DevOps you know

Most development and operations teams define and automate a CI/CD process that allows small teams of developers to:

• Independently implement their features
• Validate their correctness in production-like environments
• Have their code deployed into production quickly, safely, and securely

This ensures code deployments are routine and predictable.

They achieve this with changes committed to version control that trigger fast, automated tests. Quick feedback loops at every step of the process provide validation results so developers can see the effects of their actions and fix mistakes quickly (usually within minutes), which enables faster fixes as well as genuine learning. 

This continually assures that the code and environments operate as designed and are always in a secure and deployable state. Production telemetry ensures that operational problems get detected and corrected quickly, confirming that everything works as intended and that customers are getting value from the software created.

Teams that only apply DevOps practices for their software code quickly realize that they still have a manual, error-prone bottleneck in the process: updates to their databases.

The (database) DevOps you need

Data professionals, including both those who work on databases for traditional web applications as well as those who work on data warehouses and analytics “data projects,” are often left out of traditional DevOps. When the database doesn’t participate in the build, validation, and deployment automation that exists for the application, database change management is either manual or teams have to build their own solution to fill the gap. To complicate matters further, databases have state. Many of the methods that worked in the stateless world of code don’t work for databases.

Application teams often have a DevOps practice and infrastructure that data professionals can integrate their database change management process into. However, data warehousing and analytics teams focused on delivering data projects (instead of traditional web applications) often have little or no DevOps standards to streamline operations and handle changes in an automated way.

Regardless of the situation, database DevOps aligns data professionals and operations teams in the same way developers and operations became aligned through DevOps.

The problems faced before DevOps and the solutions addressed by DevOps are not just required for code – they are also required for the database.

DevOps and its related benefits can only be fully realized by including the database.

Database DevOps in action: Moving from a legacy data warehouse to BigQuery

A critical application team at a large financial services company was moving its operations from a legacy data warehouse to BigQuery. While the application teams liked the new database, the project was in jeopardy due to the unacceptably high change lead time. The problem wasn’t with the BigQuery platform itself, it was a process, tooling, and collaboration problem for the implementation team that stood in the way of successful migrations.

Deployments took too long – even simple changes could take many days and developer hours to complete and five full-time engineers to support. Managing concurrent changes to shared objects created overhead and contributed to a high-friction developer experience. The total cost of ownership was unacceptable.

It was clear that the team needed to dramatically simplify the developer experience and reduce deployment times to make the new solution viable.

The customer team came prepared with their use cases and key stakeholders from the development and database teams who were actively engaged and aligned on their shared problems. This was the perfect starting point that allowed the Liquibase team to quickly define an implementation that worked for everyone.

The discussion started with defining a branching strategy that worked for the development and database teams. With this information and the rest of the plan, the combined team implemented the core CI/CD solution. 

The team next implemented quality checks to ensure errors were detected and could be fixed early in the development process. To make it easy to correct anything that made it past the automated validation, the team added rollback capabilities. 

For enablement, developers were thoroughly trained via self-paced onboarding courses on Liquibase Academy. Over the first few weeks of usage, the operations team continued their work with the Liquibase team to integrate the data produced from Liquibase operations into a Splunk custom dashboard giving them real-time database observability into their team’s performance via the four DORA metrics.

Post-adoption resulted in

• Deployment times reduced from hours to minutes 
• 98% reduction in deployment time
• 90% reduction in lead time to delivery
• Significant improvements in developer productivity
• Speedier delivery of new capabilities

Achieving Database DevOps

Liquibase helps BigQuery teams accelerate the path to value with an onboarding and adoption process that minimizes risk and ramps up productivity rapidly. Customer Success leads the customer team through planning, implementation, go-live, and then into a continuous improvement cycle leveraging database observability logs.

Planning

To ensure a quick start, Liquibase Customer Success is available to help teams define their database DevOps approach. Once the plan is well defined, the teams can work together to start implementation.

Implementation

During implementation, teams integrate Liquibase into their CI/CD pipelines so that:

• Database changes are checked into a BigQuery version control to provide traceability and governance
• Checked-in database changes trigger automated validation with alerts for any problems detected
• Validated application code and database changes are packaged and available for deployment in subsequent environments

Enablement & go live

Once the implementation is complete, Liquibase helps teams set a target date for using Liquibase in day-to-day operations. Development teams go through a short, self-paced enablement training in Liquibase Academy then start managing database changes with Liquibase Pro.

Observability & continuous improvement

As teams are using Liquibase Pro to manage their changes, the operations team builds the final part of the implementation to bring visibility and insights to the team.

The data produced from Liquibase Pro’s operations feeds structured log data into your company’s dashboards to deliver database observability into process and team performance, generating insights for continuous improvement.

Get started with BigQuery and Liquibase or check out a database release automation demo to see how Liquibase can help you deliver faster and safer with database DevOps, including change management and version control for BigQuery.