Sponsor: Using RabbitMQ or Azure Service Bus in your .NET systems? Well, you could just use their SDKs and roll your own serialization, routing, outbox, retries, and telemetry. I mean, seriously, how hard could it be?
Database migrations gone wrong. You deploy a new feature in your app and suddenly half of it breaks. You start digging through your logs and realize it’s a database schema change that’s causing all the issues. Now you’re scrambling, wondering if you can roll back your code changes. Nope, the damage is done. You have to roll forward and fix everything as fast as possible. This whole mess could be managed better if you handle your database migration correctly and understand that backwards compatibility is key.
YouTube
Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.
Keeping Your App Code and Database in Sync
The goal is to keep your app code and database schema in sync, meaning your app understands exactly what the database schema looks like. So when you need a schema change, the typical approach is to make the schema change first, then deploy your new app code that matches that schema and understands it.
Now, I say it’s typical to make the schema change first, but that really depends on the type of database you’re using. For example, if you’re using a relational database and need to add a column, you have options like making it nullable or giving it a default value. If you do that, your existing application doesn’t care about the new column, it just works the way it did before.
So if you deploy your new schema changes and the current version of your app is fine with them, then when you deploy the new version of your app, if something goes wrong that’s unrelated to the schema change, like a bug in your code, you can roll back. Because you were already running on that schema, everything stays stable.
Backwards Compatibility in Database Migrations
Let’s say you’ve deployed your schema changes and your app is fully in sync. But what if the column you added was nullable, and you actually want it to be NOT NULL? You can’t just switch it immediately, so you might create a backfill script to populate the column with real values.
Then, as a next step, you make another schema change that marks the column as NOT NULL and backfills the data in the same transaction. Your code changes can now assume the column is never null and stop dealing with null values altogether.
This process doesn’t always happen in a single step. Sometimes it’s a multi-step cycle where you first make a backwards-compatible change, stabilize, and then make further changes to clean things up.
Applying These Concepts to Event Streams and Document Stores
If you’re using an event stream or event store, the same principle applies. There’s no rigid schema like in a relational database, it’s all defined in code. So you make your changes backwards compatible by allowing new fields to be nullable or by upcasting events to fill in missing data at runtime.
For example, if you have an order shipment event and want to start capturing the carrier (UPS, USPS, FedEx, etc.), you’d add a nullable carrier ID field. Your code has to handle the case where that field might or might not be there when deserializing events.
The same goes for document stores. Old documents might not have the new property, so your code needs to treat it as nullable or optionally present. You may or may not want to backfill the data, but keeping it backwards compatible lets you deploy without breaking existing functionality.
Why Backwards Compatibility Matters in Scaled-Out Environments
This becomes especially important when you’re running multiple instances of your app and doing rolling deployments. Imagine you deploy a new version of your app to one instance, but other instances are still running the old version. During this transition, you have both old and new versions running against your new schema.
Until all instances are updated, your schema changes and app code must be backwards compatible to avoid breaking functionality.
When to Make Schema Changes During Deployment
We know it’s often best to make schema changes before deploying new app code, but when exactly that happens depends on your database type and the changes you’re making.
With event streams or event stores, you often don’t have a schema change per se, it’s all about deploying backwards-compatible code that can handle existing and new event formats, possibly with upcasting.
For relational databases, one approach is to have your app make schema changes at startup before handling requests. This means your deployment includes deploying the app, which then runs the migration on startup.
The downside is in scaled-out environments: multiple instances might try to apply the same schema change concurrently, and every app startup has to check if the schema is up to date, which adds overhead.
Depending on your context, how often you deploy, how many instances you run, this overhead may or may not be acceptable. You might even use health checks to prevent the app from starting if the schema isn’t at the expected version.
Another approach is to separate schema changes from app deployment. You run your database migrations independently as a distinct step in your deployment pipeline. Once the schema change succeeds, you deploy your app code.
This approach works regardless of database type. For example, if you want to transform or migrate event streams, you’d do it as part of your deployment pipeline, not tied directly to app startup.
Using Flyway for Database Migrations
For tooling, I’ve been using Flyway for over 10 years. It’s served all my needs related to relational database changes.
Here’s a quick example of steps:
- First, initialize Flyway and set the baseline of your current schema using the Flyway CLI.
- Running the baseline command creates a schema history table that tracks which migration scripts have been applied.
- Next, create a new migration script, for example, to add a new column to a table.
- Run the migrate command to apply the migration script as part of your deployment process.
- Flyway records the migration in the schema history table and updates your database schema.
When developing locally, you make your schema changes and create migration scripts that will be run during deployment. The key to database migrations is maintaining backwards compatibility.
Backwards Compatibility is the Key
Database migrations can be a multi-step process. You start by making non-breaking changes that are backwards compatible, then later do cleanup steps like backfilling data or tightening constraints.
The fundamental idea is to look closely at your code and schema changes and ensure backwards compatibility at every step. This allows you to deploy new versions safely, roll back if needed, and keep your app running smoothly throughout the process.
Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.