Getting Started: Apache Kafka with .NET Core

Getting Started with Apache Kafka & .NET Core

If you’re interested in playing around with Apache Kafka with .NET Core, this post contains everything you need to get started.

I’ve been interested in Kafka for awhile and finally sat down and got everything configured using Docker, then created a .NET console app that contained a Producer and a Consumer.

Here’s my complete process of what that involved.

YouTube

Check out my YouTube channel where I created a video that accompanies this blog post.

Docker

The first thing you need is to pull down the latest Docker images of both Zookeeper and Kafka.

Before we create any contains, first create a new network that both contains are going to use.

Now you can create both Zookeeper and Kafka containers. Kafka needs to communicate with Zookeeper. All the port mappings are the standard ports listed in the Zookeeper and Kafka docs.

Producer

Now that we have Zookeeper and Kafka containers running, I created an empty .net core console app. I’m going to create a hosted service for both Producer and Consumer.

For the producer in this demo, I’m using the Confluent.Kafka NuGet Package.

For this example, I’m going to iterate 100 times and produce a string of “Hello World {i}” that is going to be sent to a topic called “demo”.

In the constructor I’m creating a new producer and specifying that Kafka is running on localhost:9092, which is our container.

The other thing to note is that this “demo” topic I’ve created manually ahead of time, not within this code sample.

Kafka UI

If you need to create a topic, you can do so using the command line tools, or if you prefer something visual you can check out Conduktor.

Consumer

For the consumer, I decided to try the kafka-sharp NuGet Package after I saw this tweet.

The consumer follows a similar type of pattern where in the Constructor I’m creating the connection to localhost:9092. In the StartAsync is where the magic happens of subscribing to the demo topic. The MessageReceived lambda is what will handle all messages received, where I’m just logging the message value.

Full Sample

For completeness, here’s the generic host and both Producer and Consumer hosted services above.

You can find the entire sample on my GitHub.

If you’re using Kafka, which library are you using? Are you using a event/message dispatcher on top of Kafka or using it directly. Let me know in the comments or on Twitter.

Related Links

Follow @CodeOpinion on Twitter

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.

CAP: Event Bus & Outbox Pattern

Outbox Pattern

If you’re thinking of building or already are implementing a system using async messaging (SOA or Microservices) then you need to start thinking about what type of messaging library you want to use in front of a message broker. CAP is an Event Bus that implements the Outbox Pattern to deal with distributed transactions.

YouTube

I did a live stream exploring CAP that is over on my YouTube Channel.

Distributed Transaction

When you’re using messages (events) to communicate between systems you will run into the situation where you need to save data to your database, then publish an event to a message broker. These events will then be received up by other systems for their own internal use.

The problem arises when two situations occur:

  • You save data to your database but there is a failure when publishing the event to your message broker
  • You publish an event to your message broker that something occurred in your system, but then when trying to save to your database, it fails.

In both situations, there is no consistency between what your database has saved and what you have published to the message broker.

What you want in this situation is one atomic transaction that can save data to your database and publish the event. If either fails, the other is rolled back. Basically, a distributed transaction.

Outbox Pattern

The Outbox pattern solves this issue by using a single transaction to perform both actions. What this involves is rather than publishing the event directly to the message broker, it serializes the event and saves it to a table in the same database using the same transaction for persisting your application data.

This is a diagram of the outbox pattern described in Microsoft’s eShopOnContainers reference application

Outbox Pattern

Once the event is persisted to the table within the database, it will then be published to the message broker. If the message broker is unavailable or there is a failure publish to the message broker, the library you’re using will retry to publish. It provides reliability that you are not losing messages that you need to publish.

CAP

I stumbled upon the CAP project as I was looking for references for how other libraries have implemented the outbox pattern.

The following just illustrates the simplicity of using the CAP library and it’s API. My example uses MySQL as the database and RabbitMQ as the message broker.

First step is to include the relevant packages to your project

Next is to configure the Startup. In the ConfigureServices we need to use the AppCap() with various options to configure our database and message broker connections. Also in the Configure I’ve added the UseCapDashboard() which provides a little web-based UI for showing the messages and event subscriptions.

Now that we have CAP configured, the first step is going to be publishing an event. You do this by using the ICapPublisher. The parameters are a name and the contents of the event.

The key thing to point out here is that BeginTransaction() is an extension method from CAP. This extension method starts the MySQL transaction but also passes it along to the CapPublisher so they are using the same transaction. CapPublisher needs this transaction because it is going to write the published events to a table within your database.

CAP automatically creates this table. There is no setup required on your end.

Finally, we can create a subscriber/receiver for this event. To do so is simply a matter of implementing ICapSubscribe and add the CapSubscribe attribute to a method with the appropriate arguments that match what was published.

That’s it!

That’s the simplest example. And it’s pretty simple and hopefully gives you an idea of how CAP works with implementing the outbox pattern. There are various message brokers it supports such as Kafka, RabbitMQ, and Azure Service Bus. On the database side it supports SQL Server, MySQL, PostgreSQL, and MongoDB.

My entire sample application is available on GitHub.

Follow @CodeOpinion on Twitter

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.

Why use DTOs (Data Transfer Objects)?

Data Transfer Objects

Should you really use DTOs (Data Transfer Objects)? Seem like a lot of work mapping your database entities to another object? Why Bother? The simple answer is coupling.

Data Transfer Objects

First, what are DTOs? When people refer to Data Transfer Objects, what they mean are objects that represent data structures that generally do not contain any business logic or behavior. If they do contain behavior, it’s generally trivial.

Data Transfer Objects are often used to be serialized by the producer and then deserialized by the consumer. Often times these consumers may live in another process being used by an entirely different language and/or platform.

YouTube

I’ve recorded a short video explaining this with some sample code. Check out the video and make sure to subscribe to my YouTube Channel.

Crossing Boundaries

The most common example of this is creating an HTTP API using ASP.NET Core that returns an object from its Controller actions that are ultimately serialized to JSON. The consumer is oftentimes JavaScript that uses those JSON responses in displaying the browser using a Component or SPA Framework.

Internals

If you’re not using DTOs, then you’re likely exposing internal data structures.

The biggest culprit of this is simple TODO demo applications that expose the database entities directly. Meaning they output a serialized list of TODOs to the javascript frontend. And when you want to create a new record, they often times take the TODO object to insert directly into the database. This is leaking internals.

This is my biggest complaint with simple demo applications are they often don’t implement or follow some practices, because rightly so, they aren’t applicable to a simple TODO application. However, people take the example of a simple TODO and use the same patterns into a much large application.

The problem is when internal data objects are serialized and consumed by a client you either down own or cannot change easily. Or, which happens more often, the application itself gets very large.

Contracts

The moment you want change internal data objects, you now have to update the clients.

Take this simple example of a Customer that is an internal data structure we use through the system and likely use to persist using an ORM.

If we are serializing this structure and clients are consuming this, if we change this structure, we’re likely going to break out clients.

This change would require a change to all of our clients. We could easily make this change in our own codebase and have all our own usages be correct, but we would be breaking all of our decoupled clients that get a serialized representation.

Representation

When creating an HTTP API, it’s all about representations. Most often times clients need a rich representation of a resource, not a just serialized version of a database entity. They often times need related data.

Having your API return rich representations means you must do some level of composition to create an object, not just a database entity, that will get serialized. This is where a DTO comes into play.

I actually don’t often use the term DTO, but rather use the word Representation or ViewModel. The purpose is still the same, it’s a data structure that is a contract between the producer and the consumer. That contract should remain stable (through backwards compatibility) or have a versioning strategy.

Coupling

The reason you want to use DTOs is that you want clients to couple to that contract, not to your internal data structures. This allows you to modify and evolve your internals freely without breaking clients.

Links

Follow @CodeOpinion on Twitter

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.