Sponsor: Do you build complex software systems? See how NServiceBus makes it easier to design, build, and manage software systems that use message queues to achieve loose coupling. Get started for free.
The moment you retrieve data, it’s potentially already stale. And I’m not talking about eventual consistency or reading from a cache, which is what people typically think of when they think of stale data. No, I’m talking about your primary database.
YouTube
Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.
Developers often make the assumption that we can just query the database, get some data back, and return it to the client. Maybe we’re showing something in a UI where the end user can make some type of decision about the action they want to perform, or what options we make available to them. Then, based on that information, they perform an action.
If we zoom in a little bit, the command might be doing the same thing. The client makes a request, the command fetches that data again, performs some type of business logic validation, and then tries to execute our state change.
But here’s the question:
What guarantees that information is still accurate when you perform the command to make that state change?
You’re Looking at Stale Data
Here’s an example from my logistics shipment application. We have Dispatcher A looking to assign a carrier to a shipment. Carrier A has one slot available. That’s the one we’d probably like to pick, and I should be able to, right? The UI says it’s available.
But we’re actually in an environment with multiple users. We have another dispatcher who, guess what, just assigned that particular carrier to a different shipment. Now that carrier is no longer available.
If we go back to Dispatcher A, guess what? They’re looking at stale data. The UI still shows one slot available, even though that capacity has already been allocated to another shipment.
Now you might be thinking this is just concurrency control and you could use optimistic locking, but that’s not actually the case here. I’m not trying to update the same shipment. I’m trying to reserve and allocate a carrier, which is a different resource.
Stale Data Isn’t Just Eventual Consistency
What’s interesting is that this isn’t really a new problem. My example involves two different services, shipments and carriers, trying to determine what carriers are available.
But this goes back to a post from 2009 by Udi Dahan of Particular Software called Clarified CQRS. What’s really interesting is that he’s coming from the angle of collaboration and stale data. Go check out that post in the context of CQRS.
I want to go back to stale data because people often think of it in different ways.
Developers typically associate stale data with caches that haven’t been invalidated yet, read replicas that are eventually consistent because of log shipping, materialized view projections, event sourcing projections, or simply calling another service.
But stale data isn’t limited to any of those scenarios. Even when we’re querying our primary database, the information can become stale the moment we retrieve it.
This relates to Udi’s post because often we need that data, whether we realize it’s stale or not, to make some type of decision and apply business logic to determine if we can actually proceed and successfully execute a command.
Validation and Business Rules Aren’t the Same Thing
It’s also worth pointing out the difference between validation and business rules because these often get conflated, and they’re not really the same thing.
If we’re talking about assigning a carrier to a particular shipment, that’s part of our command. The carrier ID we’re passing along may need some type of validation. Is it an integer? Is it a UUID? Whatever we’re expecting.
But I don’t necessarily know whether that particular carrier exists or can actually perform the work. That’s where our business rules come into play. Do we actually have capacity with that carrier to assign them to that shipment?
I always like to say that sometimes we get caught up thinking business rules are trivial validation. The superficial stuff. Often, it’s the business rules that actually need state.
When we have a UI showing available carriers, we’re really guiding our dispatcher by showing them what options might be available. This is what the state looked like at that given time when we presented it to them.
This is really nothing more than guidance on making a decision.
Moving the Query Closer to the Command Doesn’t Solve It
However, once they hit “Assign Carrier,” what are we really doing?
Because we know the UI could be stale, maybe we do something like this when the command actually executes. We call our carrier service to see if we actually have capacity. If it returns that we do have capacity, then we assign that carrier to our shipment.
Here’s the problem: These are two separate operations, and that doesn’t solve our problem.
Sure, it makes the window of time smaller because now the query is directly next to our state change where we’re executing our command. But that doesn’t actually solve the problem.
It’s not just about where that query is. It’s that the query and the state change aren’t part of an atomic decision making process.
Between checking whether the carrier has capacity and actually assigning that carrier, another dispatcher could have reserved that same capacity. We’re still making a decision based on information that could already be stale.
Reserving Capacity Is the Actual Business Operation
What we’re actually trying to do is reserve capacity. That’s the business operation.
If you think about the UI, it was showing slots available. What we’re actually trying to do is reserve one of those slots. It’s not necessarily about assigning a carrier within a shipment. It’s about the capacity of particular carriers.
This also relates to something I’ve discussed before about generating IDs.
In the reservation, I also have the shipment ID. I may not even have persisted or created the shipment in my database yet, but I already know what the ID is. That means I can pass it along to my reservation.
Since reserving capacity is one operation, it can be atomic. Whether that involves a database transaction, optimistic concurrency, or whatever the implementation happens to be, we’re performing an atomic operation within one boundary.
Instead of querying whether capacity exists and then separately trying to assign it, we’re asking the carrier service to reserve that capacity. It can determine whether capacity is available and reserve it as part of the same operation.
That’s a very different approach because we’re now enforcing the actual business rule within the boundary responsible for that capacity.
What About Asynchronous Processing?
To add another dimension, let’s say we have asynchronous processing.
Our client makes a call to dispatch to perform the reservation with carriers. If that’s successful, maybe we then place a message on our queue to dispatch that shipment and create it.
Since we’ve already generated the shipment ID on our client, we know what it is and can include it in the message.
Our worker then asynchronously processes that message to create our shipment and assign the carrier we’ve already reserved.
It’s worth pointing out that reliable messaging absolutely matters here. I’ve covered this in a lot of previous videos.
What I’m showing is really the reservation pattern, where typically you’d have a timeout on that reservation because you need to acknowledge it.
When processing the message asynchronously, the worker would contact the reservation to acknowledge it. If that acknowledgment doesn’t happen, the reservation would eventually time out and release the capacity.
It’s essentially like a lock on that reservation slot for that carrier.
This allows us to reserve capacity within one consistency boundary while still processing the rest of the workflow asynchronously. The reservation protects the capacity while the shipment is being created, and the timeout prevents that capacity from being held indefinitely if the workflow never completes.
Stale Data Is a Design Problem
Looking back at the UI, when you think about stale data and immediately jump to solutions like WebSockets or real time asynchronous updates, that’s really not the answer.
It’s actually about design and following what the actual business operation is.
It’s not simply assigning a carrier. It’s reserving one.
Queries are there to help make decisions, but they’re generally immediately stale the moment you fetch that data. Commands, on the other hand, are there to enforce the decision you’re trying to make and check whether you can actually execute it.
We’re always dealing with stale data. If you can acknowledge that, then you can look at your consistency requirements to guide the design around your boundaries. What consistency boundary do you need, and what patterns should you use?
If it’s a simple single database and not distributed, keep it simple. But if you have boundaries you need to cross, like reserving a carrier, realize what the actual business operation is instead of thinking superficially about stale data.
We’re not going to eliminate stale data by making our queries faster, moving them closer to our commands, or constantly refreshing our UI. We need to design our commands and consistency boundaries around the business operations we’re actually trying to perform.
Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.