Sponsor: Using RabbitMQ or Azure Service Bus in your .NET systems? Well, you could just use their SDKs and roll your own serialization, routing, outbox, retries, and telemetry. I mean, seriously, how hard could it be?
If you’re looking at using .NET Channels as a replacement for a MediatR, or if you’re not in .NET but searching for an in-memory producer-consumer API, hang on a second—because you might be in for a pile of pain. That’s not actually what you want.
YouTube
Check out my YouTube channel, where I post all kinds of content accompanying my posts, including this video showing everything in this post.
Channels in .NET: A Simple Example
Let me give you a really simple example of channels in C#. If you’re not using .NET, you’ll totally understand the gist of this because it’s pretty straightforward. Think about your own platform and how you can do producer-consumer in-memory communication within the same process, but between threads.
Here’s the main idea: I’m creating dependency injection (DI) by registering a channel. I have a class—a record actually—that I created called Ping. It just has an input message and a GUID that’s going to be new every time. You’ll see why that’s important in a second.
So, I’ve created an unbounded channel, meaning I can add as many items as I want. Then, I have a minimal API endpoint called /test. When this endpoint is called, it appends a new Ping message to the channel. That’s the producer side.
On the flip side, I have a consumer running separately as a background service. It just waits for new items on the channel. When it receives one, it reads it and outputs the date/time, the GUID, and the message to the console.
To illustrate this better, I added a one-second delay in the consumer. When running this example, if you send a message, you’ll see the request write to the channel, and the background service will process the message one at a time, outputting the details.
Reliability and Durability: The Biggest Issue
Now, here’s the big problem: this is all done in memory, which means it’s not reliable. If you expect your system to be reliable—meaning you want the messages you produce to actually be consumed—you need durability. When messages live just in memory, if the application crashes or restarts, you lose those messages.
Imagine this scenario: you’re using this setup for an order system. You place an order, which puts a message on the channel. Then, asynchronously, a background service sends out a confirmation email. What if the email never gets sent because the application crashed or an exception occurs during the process? The order was placed, but the email didn’t go out. That might be a big deal depending on your context.
If you need reliability, that means you want durable messages. Meaning persisting messages in durable storage.
That way, you don’t lose messages or the processing of messages.
The Slippery Slope of Reliability Patterns
When you start needing reliability, you’ll find yourself implementing a whole bunch of patterns you didn’t plan on. Using channels or any lightweight in-memory producer-consumer solution makes you responsible for handling these patterns yourself.
Failures and Dead Letter Queues
What happens when a consumer fails? Let’s say one consumer is responsible for sending the confirmation email, and it fails. Do you just log the error and move on? Probably not. You need a way to handle failures, which usually means implementing a dead letter queue—a separate storage where you put failed messages so you can deal with them later.
Retries
Along with dead letter queues, you’ll want retries. Sometimes failures are transient, and retrying immediately or with exponential backoff can fix the problem. You might have to implement retries yourself or use a third-party library like Polly in .NET to handle retries gracefully.
Timeouts and Visibility Timeout
What if a consumer takes too long to process a message? This is where visibility timeouts come in. The idea is that a consumer must acknowledge processing a message within a certain time frame. If it doesn’t, the message becomes visible again and can be reprocessed. This pattern leads to at-least-once delivery, meaning your consumer needs to be idempotent—it must handle the same message more than once without negative side effects.
Bonus Patterns You Might Encounter
There are a couple of other patterns you might run into that aren’t necessarily about reliability but are common in producer-consumer scenarios.
Delayed Delivery (Scheduling)
Sometimes you want to publish a message but not have it consumed right away. You want it to be consumed after a certain period or at a specific date/time in the future. This is called delayed delivery or scheduling. The message is there but not visible to consumers until the specified time arrives.
The Claim Check Pattern
When your messages are large—like including a big file or image—you don’t want to put all that data directly in the message (for various reasons). Instead, you store the large data somewhere else, like blob storage, and send a message that contains a reference or identifier to that data. This is the claim check pattern.
What Should You Use?
If you’re using an in-memory producer-consumer like MediatR or using Channels API, and you don’t need reliability, that’s great. These tools can work well for you.
But if you do need reliability, you’re going to slide down this slippery slope of implementing all the patterns I just mentioned: dead letter queues, retries, timeouts, idempotent consumers, and more. Before you know it, you’ve basically built your own messaging library.
So take a step back, look at your requirements, and figure out what you really need. If you do need a messaging tool with reliability, there are plenty of great options in the .NET ecosystem that implement almost everything talked about and are built for this purpose.
.NET Channels
I think the .NET Channels API is really nice, but I see it more as a tool for library authors rather than application developers for communication inside your app—unless you specifically need it for that.
Stay out of the plumbing world unless you have to dive in. Think carefully about your messaging needs and what trade-offs you’re willing to make regarding reliability and resiliency.
Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.