Separating Concerns with Pipes & Filters

How do you separate concerns when processing a request? Typical concerns such as Logging, Validation, Exception Handling, Retries, and many more. One way is to build a request pipeline for separating concerns by using the Pipes and Filters pattern. You can also build a pipeline using the Russian Doll model that allows you to short circuit at any point throughout the pipeline.

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

Pipes & Filters

Often time when processing a request you need to handle various concerns. The Pipes & Filters pattern allows you to break up these various concerns into a series of steps that create a pipeline. A request can be an HTTP request but can also be a message that’s being processed. If you’re using ASP.NET Core, you’re already using the Pipes & Filters pattern!

Pipes & Filters

In the diagram above, the sender is sending a request to the receiver, however, there are filters in between. These filters in between are what can handle various cross-cutting concerns.

Pipes & Filters

The request will pass through the various filters until it finally reaches the Receiver. Both Sender and Receiver are unaware that the request passed through the filters.

Pipes & Filters

As mentioned, the filters can be cross-cutting concerns such as logging, validation, caching, etc.

Pipes & Filters

One important note is that filters are independent. They should be composable with various types of requests. As you can assume a logging filter could be used in many different types of requests.

You should be able to decide per request type, which filters you want to use. They should be plug-and-play.

This allows the receiver of the request to really focus on the behavior that it needs to perform and not other concerns.

A good example of where you may have used this is with ASP.NET Core Action Filters.

Russian Doll

Another way of implementing Pipes & Filters is with the Russian Doll Model. The concept is the same that you have a Sender and a Receiver with Filters in between. The difference however is that each filter calls the next filter in the pipeline.

Each filter is still independent but because the pipeline is using a uniform interface, each filter can be provided the next filter to be invoked.

Russian Doll

Since each filter is calling the next filter, this means that you can short circuit the pipeline at any point.

Russian Doll

In the diagram above, the Validation filter may choose to short circuit the request if validation fails. Because it’s not calling the next filter in the pipeline the Receiver will never get executed to handle the request.

A good example of where you maybe have used this is with ASP.NET Core Middleware.

Messaging

Processing a message is often very similar to processing any other type of request, except it often doesn’t have an in-process response. There are still various concerns when processing a message that creating a pipeline helps with.

As an example using the Brighter messaging library, which supports Pipes & Filters using the Russian Doll model using attributes. This feels very similar to action filters in ASP.NET Core.

In the example, the PlaceOrderHandler has 3 different filters that will be executed before it is executed. Logging, Retry, Validation. I’ve implemented logging as a generic filter that can be used in any type of request. While Retry and Validation are specific to the PlaceOrderCommand.

Pipes & Filters

Creating a pipeline can be a powerful way to separate various concerns when processing a request. It allows you to focus on each individual task as a task (filter) within the pipeline.

You can create generic filters that can be used with any type of request or create very specific filters that are only used for a very specific request Also leverage the Russian Doll model to short circuit your request pipeline.

Source Code

Developer-level members of my CodeOpinion YouTube channel get access to the full source for any working demo application that I post on my blog or YouTube. Check out the membership for more info.

Related Posts

Follow @CodeOpinion on Twitter

Software Architeture & Design

Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design

Restructuring to a Vertical Slice Architecture

What is vertical slice architecture and how does it compare to clean architecture? Ultimately it’s about coupling and dependencies. With vertical slice architecture, you’re organizing your code across a feature rather than layers. The focus is on features and capabilities rather than technical concerns. Coupling is limited by the scope of the feature and the size of the vertical slice.

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

Clean Architecture

Before jumping into Vertical Slice Architecture let me first talk about Clean Architecture and use it as a comparison. In Clean Architecture (or Onion, Ports & Adaptors) the main aspect to point out is how dependencies point inward.

https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html

The core of your application described as “entities” in the diagram above is what contains your core domain and business logic. It has zero dependencies on any web or application frameworks or logic. The focus is purely on business logic or domain model.

As you go outwards the “uses cases” layer contains your application logic and will invoke your core domain or “entities”. It doesn’t matter how you term these layers or what they contain really, the gist is that dependencies point inwards. The outer layer will depend on the more inner layers.

Put simply, it’s about coupling. Check out my post on coupling and I cover Afferent and Efferent Coupling.

Vertical Slice Architecture

There are many different strategies for dealing with coupling. Vertical Slice Architecture is just another but handled differently. Instead of separating based on technical concerns like Clean Architecture, Vertical Slices are about focusing on features.

This means you’ll take all the technical concerns related to a feature and organize that code together.

Think about a cake that has multiple layers. Now cut a piece out. It’s a vertical slice.

By doing this you’re dealing with coupling in a different way because slices are for the most part, self-contained. They don’t couple to other slices.

I said that they are self-contained, however, features obviously are going to relate to the same underlying domain. In this case, features may share an underlying rich domain model. They may also share an entry point such as a web framework and host, for example, ASP.NET Core. However, everything you’d typically think of being in separate layers in a Clean Architecture (all data access, validation, authorization) is kept together in a single feature. They aren’t separated by different projects.

To illustrate this more, maybe some features are invoked by a web framework and some are invoked by a message processor picking up messages off a queue. There can be some features that share a domain and some features that just have straight data access and use simple data models in more of a transaction script style.

Structure

What this looks like in code is having files for various features live alongside each other. Again organizing code by feature, not technical concerns. In the YouTube video, I re-structure a Clean Architecture towards Vertical Slice Architecture. Ultimately you end up with an organization similar to this.

Here are the contents of the GetMyOrders.cs. It contains everything required to get the list of orders.

Coupling

You may be asking: “why aren’t you using a Repository or why isn’t the data access abstracted”. The reason is that there’s no need to abstract it into anything else. This feature is about returning this specific result set and shape of data. In the prior version, a repository was being used to return an Aggregate, which ultimately was fetching more data than what was required. Simply go to the source and get the data required to build a result.

You may also be asking, “But what if you need to replace Entity Framework”. Then change the dependency in this feature. You’re defining dependencies for a single slice. If the overall schema changes that end up affecting multiple features, then you’ll end up changing those features together. In my experience that won’t be a large number if you actually needed to change the dependency.

Things that change together (features) belong together. That’s what Vertical Slice Architecture is about. Stop focusing and organizing by technical concern but rather start focusing and organize by the features and capabilities of your system.

Source Code

Developer-level members of my CodeOpinion YouTube channel get access to the full source for any working demo application that I post on my blog or YouTube. Check out the membership for more info.

Related Posts

Follow @CodeOpinion on Twitter

Software Architeture & Design

Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design

Software Architecture Q&A: Microservices, CQRS & More!

You have questions, I have answers… well mainly opinions! Here are the top questions I was asked for this Software Architecture Q&A. As you can expect they are around Microservices, Messaging, CQRS & Event Sourcing.

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

What is the difference between SOA and Microservices?

I get this question often and I was expecting it for this Software Architecture Q&A. I believe it’s because of how I present Event Driven Architecture in relation to Microservices. The answer to this question really depends on your definition of both SOA and Microservices. The definition of Microservices that I generally use is from Adrian Cockcroft:

Loosely coupled service oriented architecture with bounded contexts.

https://www.slideshare.net/adriancockcroft/dockercon-state-of-the-art-in-microservices

If I dissect that definition, Service Oriented Architecture that’s loosely coupled. Loosely coupled implying Event Driven Architecture.

In my mind, SOA was always about being loosely coupled, however that may not have been how the general developer community understood it at the time.

The Bounded Context portion of the definition is from Domain Driven Design. Bounded Contexts are about defining boundaries. Services should own a set of business capabilities and the data behind those capabilities. You aren’t sharing behaviors or data between services. Services are independent. And because they are independent communication is done in a loosely coupled fashion via an Event or Message Driven Architecture.

So my answer to this question is nothing is different between the two.

Is the saga an anti-pattern since you have a transaction spanning multiple bounded contexts?

No, a saga is not an anti-pattern. It’s responsible for coordinating a long-running business process between multiple services where there are no distributed transactions. Since there can be failures, the saga is responsible for handling failures by sending compensating actions to the appropriate services. The saga is a centralized place to handle the workflow of a long-running business process. And when I say long-running business process, that could mean milliseconds to weeks.

An alternative to a Saga is using Event Choreography, which removes the centralized logic of orchestration. Event Choreography is having all services consume and produce events that ultimately fulfill the long-running business process.

Check out my post on Event Choregrophay and Orchestration for more.

When should I choose CQRS over CRUD based “RESTful” endpoints?

Choose CQRS over CRUD when you want to be explicit.

With CRUD, when you’re making state changes via Create, Update, Delete, you aren’t capturing explicitly why it’s occurring. For example, if you’re performing an Update to a customer why is that happening? Did their address change? Did their discount rate change? With CRUD based approach, you don’t know exactly, you’d have to imply it based on the change being made.

With CQRS, all your commands are making those state changes explicit. You’re not “updating a customer” but rather you’re using explicit commands such as ChangeCustomerAddress or IncreaseDiscountRate. Because commands are explicit, you can then infer various events to be published from those commands.

If you wanted to publish an event based on CRUD, you would have very generic events such as CustomerUpdated. If you’re using CQRS, you’d have events such as CustomerAdresssChanged and CustomerDiscountRateIncreased.

How to handle versioning when Event Sourcing?

No Software Architecture Q&A on this Blog/YouTube could happen without a few questions about Event Sourcing.

The definitive answer to versioning is Greg Young’s Versioning in an Event Sourced System eBook.

I often like to compare this to a relational database. If you were adding a new column to a table, you would either need to make the column nullable or give it a default value and then possibly backfill existing data.

With event sourcing, making a new property on an existing event nullable is exactly the same. You’d need to deal with the value being null. If it’s not null and has a default value the situation is still the same as a relational database. The difference is that you cannot backfill and update old events in an existing stream. What you can do in this situation is make sure any old event can be upconverted at runtime to the new event.

Best Practices and Pitfalls when Event Sourcing?

The biggest pitfall is how different the concept can be to developers who have only ever done CRUD and only ever recorded the current state.

While I don’t think the concept of Event Sourcing is difficult, how you deal with an immutable log (event store) is very different than just recording the current state.

An example of this is when using a relational database if you needed to “fix” data, you’d simply go and write an UPDATE statement to update records in your database. With Event Sourcing, since the events are in an immutable log (event stream), you can’t just “update” an existing event. Nor would that actually even make much sense. You generally have to create compensating events to “undo” something.

In the stereotypical example, if you deposited $10 into a bank account, however, that deposit was recorded as $100 by mistake. The bank would actually create a withdrawal of $100 and then a deposit of $10. Meaning they would do a full reversal of your original incorrect deposit.

It’s just different than I believe most developers are used to. As mentioned in the previous question, versioning is another topic that’s just something very different from what most people are used to.

How to handle eventual consistency from eventually consistent projections (read models)

In most event-sourced systems, projections are used to create different read models used oftentimes for specific use cases. If you’re unfamiliar with Projections, check out my post on Projections in Event Sourcing: Build ANY model you want!

However this question doesn’t even need to be specific to event sourcing, but any data source that you’re querying that is eventually consistent. For example, a read replica that has lagged behind the primary.

I think the pitfall here is how you set users’ expectations. If a user performs an action that they know is changing the state of the system, however after they perform the action, you don’t provide them with the data that shows the updated state, they will not have a good experience. They have an expectation of consistency.

From a technical perspective, using versions (numbers). When you perform a query, return a version number in the response to the caller. When the caller needs to re-query, they will know if the update has occurred because the version will have changed. If it hasn’t you can wait and retry to get the data again.

How and when did you start using DDD & CQRS?

I can’t remember the exact year, but I bought the Domain-Driven Design book sometime around 2009. I can’t remember exactly how I was introduced to it but I do know at the time I stumbled upon some blog posts and conference talks by Greg Young and Udi Dahan.

Both Greg and Udi were/are a large influence on how I think about designing and developing software.

Source Code

Developer-level members of my CodeOpinion YouTube channel get access to the full source for any working demo application that I post on my blog or YouTube. Check out the membership for more info.

Related Posts

Follow @CodeOpinion on Twitter

Software Architeture & Design

Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design