Outbox Pattern: Reliably Save State & Publish Events

What’s the Outbox Pattern? A reliable way of saving state to your database and publishing a message/event to a message broker.

Why do you need it? When you get into messaging, you will often find code that needs to save state to your database, and then publish a message/event to a message broker. Unfortunately, because they are two different resources, you need a distributed transaction to make these atomic.

There is another option to use the Outbox Pattern which does not require a distributed transaction and most messaging libraries support it.

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

Unreliable

To illustrate the issue, the first step is saving/changing the state in your primary database. It doesn’t matter if this is an RDMBS or a NoSQL Store.

Subsequently, since you have changed state, you need to now publish a message/event to a queue or message broker to let other systems know of the state change.

Since each step is independent, if there is a failure in publishing the event to the queue, then you’ve made a state change without letting other systems know of that change.

Outbox Pattern

Why is this a problem? If you’re using messaging in an event-driven architecture, you likely rely on events being published. An event not being published could have all sorts of implications.

For example, if you’re using events to invalidate a cache, you’ll now have stale data. Or worst, if you’re using events apart of a Saga (long-running process), the next portion/step of the saga will possibly never occur.

You need atomicity. All or nothing.

Outbox Pattern

The Outbox pattern solves this by using the transaction from your primary database to store your state changes along with the messages/events your publishing.

Outbox Pattern

This means that messages your publishing are initially stored in the same database alongside your other application data. Each messaging library may implement this slightly differently in terms of the structure of the data and messages as well as how their API looks, but this is the overall idea.

The order of saving state or publishing an event doesn’t really matter anymore as they are saved to the database in the same transaction.

Once we commit our transaction, a secondary process/thread (Publisher) will pull the unpublished events from the primary database.

Outbox Pattern

Then the Publisher will publish the events it pulled to the queue or message broker.

Finally, the Publisher will update/delete the records back to the database, so it knows that the events have been successfully published to the queue.

Code Example

In this example, I’m using CAP, but as mentioned, different messaging libraries will have different ways they implement this in their API.

CAP’s is really straight forward. It provides an extension method on the Entity Framework Core DatabaseFacade to begin a database transaction. You simply pass along the ICapPublisher when starting the transaction. This tells CAP to save the published event to the database, rather than directly to the message broker, in my case RabbitMQ.

It’s really that straightforward. I’ve left out the configuration of CAP, which is also dead simple, it’s just a matter of using the correct data storage provider and specifying the configuration string in the ASP.NET Core Startup.ConfigureServices()

What this looks like in our primary database is a table that was created by CAP automatically called cap.Published. This is where it’s storing the published messages. Initially, the StatusName is null, and after successful publish to the queue, it updates it with Succeeded.

At Least Once Messaging

You may have noticed in the original diagrams, that Publisher that pulls from the database, publishes to the queue, then marks the event published in the database, ultimately has the same problem we started out with.

Really we just moved the problem but have a different outcome. If we publish the message to the queue, but for some reason, CAP cannot update the StatusName of the record, then we will ultimately publish the same event again.

In a lot of situations, this is a better problem to have. We are now in an “At Least Once” scenario. Meaning, events will get published once or possibly more. In the original scenario, we were in an “At Most Once”, which also implies only once or possibly none.

I’ll cover handling duplicate messaging in another blog/video! Stay Tuned!

Follow @CodeOpinion on Twitter

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.

Links

The Complexity of Caching

Caching ain’t easy! There are many factors that add to the complexity of caching. My general recommendation is to avoid caching if you can.

However, caching can bring performance and scaling which you might need. If you’re starting to use a cache in your system here are some things to think about. Adding a cache isn’t that trivial and requires some thought about caching strategies, how to invalidate, and fallbacks to your database. Caching can improve performance and scalability, but can also bring your entire system down if it’s failing.

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

Strategy

The first thing to think about is the caching strategy. The two most common methods that I’ve noticed in code-bases are the write-through and cache aside methods.

The write-through method is when your application writes to its primary database, and then immediately updates the cached value. Meaning if you add a new record to your database, you immediately add the equivalent value to the cache. If you were to update a record, you would immediately update the cached value.

The Complexity of Caching
The Complexity of Caching

The second method most often used is the Cache Aside (Lazy Loading) method. This can be used in-conjunction with Write-Through or can be used by itself.

When the application needs something from cache, it first tries to retrieve it. If it does not exist (cache miss), it will then hit the primary database. Then it will write the value to the cache. Essentially you’re lazy loading the cache when data is requested for values that are not in the cache.

Invaliding the Cache

There are only two hard things in Computer Science: cache invalidation and naming things. -Phil Karlton

https://www.karlton.org/2017/12/naming-things-hard/

If you’re not using the write-through method, then that means your cache is stale when data gets updated in your primary database.

There are a couple methods I’ve used to invalidate the cache (remove the value from cache) and let the cache aside (lazy loading) method do it’s job.

Cache Expiry (TTL)

Most caches have the ability to expire a cached value after a period of time (time to live). When this occurs by the cache, the next call for an expired item will have to go through the 3 steps of the lazy loading method to re-populate the cache.

Async Messaging

The second method is using asynchronous messaging to notify another process that data has changed and to invalidate the cache.

This requires you to already be using messaging (events) and have a well defined API on where data is mutated in your system. If you have any external system modifying data within your database, you will not be able to emit an event everytime data is changed.

If you’re using something like Entity Framework, you could override the SaveChangesAsync to look at the ChangeTracker to determine which entities have changed and publish events.

Failures

One benefit to the Cache Aside (Lazy Loading) method is that if for whatever reason, you cannot reach the cache, you can fallback to using your database. This would work exactly like a cache miss. You would need to handle the appropriate Exceptions and Timeouts from the cache client to determine the Cache is unavailable, and then go directly to the database and return the value.

The Complexity of Caching

The one thing to very aware of, if you cache is unavailable, that all requests are now going to be fulfilled by the database. This could have a significant performance impact on your primary database. Depending on how many requests are normally handled by your cache are now adding all that extra load to your database.

Complexity of Caching

The complexity of caching isn’t trivial.

Avoid caching if you can.

First, look at the queries to your primary database before going down the path of adding a cache. There are many more complexities that you introduce when adding a cache. Avoid it if you can.

Follow @CodeOpinion on Twitter

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.

Links

Talking C# Performance with Steve Gordon

We’re talking C# Performance! I sat down and chatted with Steve Gordon to talk about writing performant C# code. When should you be concerned about performance? Does your application code need to be performant? At what cost to readability? How do you measure and test that you’re changes are useful? Steve and I cover all of this in this video.

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

When to think about optimizing?

As Steve mentions, it’s easy to get really deep into writing performant code. It’s fun! However, it’s probably a good idea to have a discussion with stakeholders to figure out how much performance matters.

This answer can vary wildly.

If you’re writing framework or library code, I think performance matters. As an application developer, I want framework or library code to get out of my way. Meaning it has as little overhead as possible.

When writing application code, this goes back to figuring where performance matters.

Once you do want to get deep into performance, here are some resources and blog posts that I think would be useful.

BenchmarkDotNet

BenchmarkDotNet helps you to transform methods into benchmarks, track their performance, and share reproducible measurement experiments.

dotMemory

dotMemory allows you to analyze memory usage in a variety of .NET and .NET Core applications: desktop applications, Windows services, ASP.NET web applications, IIS, IIS Express, arbitrary .NET processes, and more.

Follow @CodeOpinion on Twitter

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.

Links