CAP: Event Bus & Outbox Pattern

Outbox Pattern

If you’re thinking of building or already are implementing a system using async messaging (SOA or Microservices) then you need to start thinking about what type of messaging library you want to use in front of a message broker. CAP is an Event Bus that implements the Outbox Pattern to deal with distributed transactions.

YouTube

I did a live stream exploring CAP that is over on my YouTube Channel.

Distributed Transaction

When you’re using messages (events) to communicate between systems you will run into the situation where you need to save data to your database, then publish an event to a message broker. These events will then be received up by other systems for their own internal use.

The problem arises when two situations occur:

  • You save data to your database but there is a failure when publishing the event to your message broker
  • You publish an event to your message broker that something occurred in your system, but then when trying to save to your database, it fails.

In both situations, there is no consistency between what your database has saved and what you have published to the message broker.

What you want in this situation is one atomic transaction that can save data to your database and publish the event. If either fails, the other is rolled back. Basically, a distributed transaction.

Outbox Pattern

The Outbox pattern solves this issue by using a single transaction to perform both actions. What this involves is rather than publishing the event directly to the message broker, it serializes the event and saves it to a table in the same database using the same transaction for persisting your application data.

This is a diagram of the outbox pattern described in Microsoft’s eShopOnContainers reference application

Outbox Pattern

Once the event is persisted to the table within the database, it will then be published to the message broker. If the message broker is unavailable or there is a failure publish to the message broker, the library you’re using will retry to publish. It provides reliability that you are not losing messages that you need to publish.

CAP

I stumbled upon the CAP project as I was looking for references for how other libraries have implemented the outbox pattern.

The following just illustrates the simplicity of using the CAP library and it’s API. My example uses MySQL as the database and RabbitMQ as the message broker.

First step is to include the relevant packages to your project

Next is to configure the Startup. In the ConfigureServices we need to use the AppCap() with various options to configure our database and message broker connections. Also in the Configure I’ve added the UseCapDashboard() which provides a little web-based UI for showing the messages and event subscriptions.

Now that we have CAP configured, the first step is going to be publishing an event. You do this by using the ICapPublisher. The parameters are a name and the contents of the event.

The key thing to point out here is that BeginTransaction() is an extension method from CAP. This extension method starts the MySQL transaction but also passes it along to the CapPublisher so they are using the same transaction. CapPublisher needs this transaction because it is going to write the published events to a table within your database.

CAP automatically creates this table. There is no setup required on your end.

Finally, we can create a subscriber/receiver for this event. To do so is simply a matter of implementing ICapSubscribe and add the CapSubscribe attribute to a method with the appropriate arguments that match what was published.

That’s it!

That’s the simplest example. And it’s pretty simple and hopefully gives you an idea of how CAP works with implementing the outbox pattern. There are various message brokers it supports such as Kafka, RabbitMQ, and Azure Service Bus. On the database side it supports SQL Server, MySQL, PostgreSQL, and MongoDB.

My entire sample application is available on GitHub.

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.

Why use DTOs (Data Transfer Objects)?

Data Transfer Objects

Should you really use DTOs (Data Transfer Objects)? Seem like a lot of work mapping your database entities to another object? Why Bother? The simple answer is coupling.

Data Transfer Objects

First, what are DTOs? When people refer to Data Transfer Objects, what they mean are objects that represent data structures that generally do not contain any business logic or behavior. If they do contain behavior, it’s generally trivial.

Data Transfer Objects are often used to be serialized by the producer and then deserialized by the consumer. Often times these consumers may live in another process being used by an entirely different language and/or platform.

YouTube

I’ve recorded a short video explaining this with some sample code. Check out the video and make sure to subscribe to my YouTube Channel.

Crossing Boundaries

The most common example of this is creating an HTTP API using ASP.NET Core that returns an object from its Controller actions that are ultimately serialized to JSON. The consumer is oftentimes JavaScript that uses those JSON responses in displaying the browser using a Component or SPA Framework.

Internals

If you’re not using DTOs, then you’re likely exposing internal data structures.

The biggest culprit of this is simple TODO demo applications that expose the database entities directly. Meaning they output a serialized list of TODOs to the javascript frontend. And when you want to create a new record, they often times take the TODO object to insert directly into the database. This is leaking internals.

This is my biggest complaint with simple demo applications are they often don’t implement or follow some practices, because rightly so, they aren’t applicable to a simple TODO application. However, people take the example of a simple TODO and use the same patterns into a much large application.

The problem is when internal data objects are serialized and consumed by a client you either down own or cannot change easily. Or, which happens more often, the application itself gets very large.

Contracts

The moment you want change internal data objects, you now have to update the clients.

Take this simple example of a Customer that is an internal data structure we use through the system and likely use to persist using an ORM.

If we are serializing this structure and clients are consuming this, if we change this structure, we’re likely going to break out clients.

This change would require a change to all of our clients. We could easily make this change in our own codebase and have all our own usages be correct, but we would be breaking all of our decoupled clients that get a serialized representation.

Representation

When creating an HTTP API, it’s all about representations. Most often times clients need a rich representation of a resource, not a just serialized version of a database entity. They often times need related data.

Having your API return rich representations means you must do some level of composition to create an object, not just a database entity, that will get serialized. This is where a DTO comes into play.

I actually don’t often use the term DTO, but rather use the word Representation or ViewModel. The purpose is still the same, it’s a data structure that is a contract between the producer and the consumer. That contract should remain stable (through backwards compatibility) or have a versioning strategy.

Coupling

The reason you want to use DTOs is that you want clients to couple to that contract, not to your internal data structures. This allows you to modify and evolve your internals freely without breaking clients.

Links

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.

CQRS: Refactoring Queries without Repositories

CQRS: Refactoring Queries without Repositories

This question posted on Twitter (and the subsequent thread) caught my attention. I had to pipe in with my own opinion on having queries without repositories.

To summarize, the question is about using Aggregate Roots with an ORM like Entity Framework. Should you eager load all navigation properties or alternatively I guess, use lazy loading?

My answer is if you’re only using an aggregate root for commands (to change state) and you generally have a higher read to write ratio, then eager load the navigation properties. As a general rule of thumb.

eShopOnWeb

I recently heard about eShipOnWeb from Steve Smith. It’s a reference application using ASP.NET Core 3.1 using a layered monolithic architecture.

I decided to take a look around and try and refactor a spot that was using a repository for a query.

There is a query to return all of the orders for a signed-in user. It’s pretty simple but is a good example of it using the OrderRepository along with a specification to fetch out the relevant orders. From there it creates a ViewModel that is used in a Razor view.

Here’s the razor view for reference

Looking at the Razor view, you can see that it doesn’t use some of the properties on OrderViewModel. The OrderItems or ShippingAddress properties are never used.

The Repository with the specification is retrieving the entire Order Aggregate Root and eager loading the OrderItems. We’re fetching in a lot of data that we simply aren’t using.

Refactoring Queries without Repositories

The refactor is rather simple. Just remove the repository and instead inject the Entity Framework DbContext (CatalogContext). The second part is creating a separate ViewModel (MyOrdersViewModel ) that only contains the data we need for our Razor view. Then use this new ViewModel as a projection in our EF Query.

Side Notes

In many situations, I prefer to create a SQL Database View that I will create a new class to represent and use that as a DbSet in my DbContext. And most often I have a completely different DbContext for writes vs reads. My Reads DbContext is what will have a model that represents the SQL view. This allows you to do that projection as an SQL View rather than doing the projection in your code.

If you were using a SQL Database View and created an OrderSummary class that was on your DbContext, that now might look like this:

Live Coding Session

I did a live coding session on my YouTube and Twitch channels. You can check out the recording here of doing this exact refactor.

GitHub

If you want to take a look a the full sample, you can also find my refactor changes on my fork of eShipOnWeb on GitHub.

Links

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.