Sponsor: Using RabbitMQ or Azure Service Bus in your .NET systems? Well, you could just use their SDKs and roll your own serialization, routing, outbox, retries, and telemetry. I mean, seriously, how hard could it be?
One of the most common and very overlooked issues when writing a distributed system is consistency. You have one thing happening in one part of your system that triggers something else to happen in another part of the system, except it doesn’t happen. And that can be a nightmare to deal with.
This problem is incredibly overlooked but incredibly common. Let’s dive into an example to illustrate what I mean.
YouTube
Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.
Example: Asynchronous Processing
Here’s a simple example using NServiceBus. I’ve got an ASP.NET Core controller where I generate an ID using a GUID, then I use EF Core to create an entity. The entity is really simple—it just has an ID and a Process property that starts as false.
I save that record to my database, then I send a message that will be processed asynchronously. This means the controller doesn’t wait for the message to be processed; it immediately returns to the client.
Meanwhile, NServiceBus invokes a message handler that processes the message we sent. The handler fetches the entity from the database and sets the Process property to true, showing the asynchronous nature of this operation.
In real-world applications, this pattern is very common. For example, in e-commerce, when an order is placed, you might want to process payment asynchronously. Or when a user signs up, sending a confirmation email is often done asynchronously. Basically, any time you have a trigger that causes some other action—especially involving third-party integrations—you’re likely doing this kind of asynchronous processing.
The Glaring Flaw: Two Separate Operations
Now, while this example is simple, it has a glaring flaw that’s very common and often overlooked. The problem lies between these two lines:
- Saving changes to the database
- Sending the message
The assumption is that these happen one after the other and always succeed, as if they were a single atomic operation. But they’re not. This oversight can seriously impact the consistency of your system.
Imagine a real system with much more complex logic, state changes, and messages. If the message isn’t sent, the other parts of your system don’t know the state changed. For example, in an e-commerce workflow, if the message to process payment isn’t sent, the payment will never be processed.
Illustrating the Problem with an Exception
To demonstrate, I threw an exception right after saving the database changes. This simulates a scenario where some other code might fail, like a null reference exception—something we’ve all seen.
When running this, the record is saved with Process = false, but the exception prevents the message from being sent. As a result, the asynchronous handler never runs, and the system remains inconsistent.
Why Changing the Order Doesn’t Fix It
You might think, “Why not just send the message first, then save the database changes?” But this just creates a different problem. Now, you risk sending a message before the state change actually exists.
This can cause a race condition where the handler is invoked before the data is saved. I tested this by throwing an exception before saving the data after sending the message. The handler runs and tries to process data that doesn’t exist, leading to errors.
The Solution: One Atomic Operation Using the Outbox Pattern
The key is to treat saving the state and sending messages as one atomic operation. This means both happen together or neither happens. You can achieve this using database transactions.
Here’s how it works:
- Within the same transaction, save your state changes and persist the message you want to send.
- Commit the transaction, ensuring both the data and the message are saved atomically.
- Separately, a process reads those persisted messages and sends them to the message queue or topic.
The important part is that if sending the message fails, you haven’t lost it—it’s safely stored in your database and can be retried.
Transactional Session
NServiceBus provides exactly this with its transactional session and outbox pattern. In my example, I switched my controller route to use ITransactionalSession from NServiceBus (instead of IMessageSession in the prior example). This works like middleware or a filter in ASP.NET Core.
The transactional session opens a session before the request is processed, and if no exceptions occur, it commits the transaction at the end. This means:
- Your state changes and outgoing messages are saved in one atomic transaction.
- Messages are sent separately after the transaction commits.
- You gain consistency without worrying about lost messages or partial updates.
Because of this, I removed explicit calls to SaveChanges in my controller route; the transactional session handles that for me.
Testing the Solution
To test failure handling, I throw an exception after adding data like my earlier example. The transaction rolls back, so no data or messages are saved. Nothing happens asynchronously, which is exactly what we want.
I also tested the reverse—throwing before adding data but after sending the message. Again, nothing happens asynchronously, and no inconsistent state occurs.
Why This Matters
One of the biggest issues developers face when using messaging is thinking the state change and message sending are one atomic operation when they’re not. In reality, a lot can go wrong:
- More complex logic between state changes and message sends
- Message brokers or queues being unavailable
- Unexpected exceptions causing partial updates
Without a proper solution, you end up with data in your database that no one knows about or messages sent about data that doesn’t exist. Neither scenario is good.
The solution is using the tooling available. Any good messaging library like NServiceBus, Wolverine, MassTransit, Brighter, will support the outbox pattern to make these operations atomic and consistent.
Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.