Event Sourcing vs Event Driven Architecture

Event Sourcing is seemingly constantly being confused with Event Driven Architecture. In this blog/video I’m going through a popular blog post that explains various points that are very valid, however, they are conflating Event Sourcing with Event Driven Architecture. Event Sourcing is about using events as the state. Event Driven Architecture is about using events to communicate between service boundaries.

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

Event Sourcing vs Event Driven Architecture

There is a blog post that keeps making its round over various news or social media sites every year or so that gets a lot of attention. The issue is it conflates Event Sourcing and Event Driven Architecture.

To be clear, Event Sourcing is about using events to represent state. In Event Driven Architecture, events are used to communicate with other service boundaries.

Event Sourcing

Event Sourcing is a different approach to storing data. Instead of storing the current state, you’re instead going to be storing events. Events represent the state transitions of things that have occurred in your system. If you want more details on exactly what Event Sourcing is, check out another post I’ve written Event Sourcing Example & Explained in plain English

Event Sourcing vs Event Driven Architecture

Event Driven Architecture

Event Driven Architecture is about using events as a way to communicate with other service boundaries. Generally, leveraging a message broker (or event log) to use the Publish/Subscriber pattern. Publish events and consume events asynchronously within other boundaries. When publishing an event, there may be zero or many consumers. The publisher is unaware of who is consuming an event. Consumers are unaware of each other. Event Driven Architecture is a way of loose coupling between service boundaries.

Event Sourcing vs Event Driven Architecture

Purely based on these two definitions, you might already start to see why Event Sourcing vs Event Driven Architecture isn’t even a valid comparison.

Counterpoints

The blog post illustrates having multiple services both publish and consume events from an event log (I’m going to assume Kafka).

The idea of a keeping a central log against which multiple services can subscribe and publish is insane. You wouldn’t let two separate services reach directly into each other’s data storage when not event sourcing

Yes, that’s insane. Don’t do that. If you’re using the event log for communication between service boundaries AND to represent state within boundaries, that’s a terrible idea. But that’s has nothing to do with Event Sourcing but rather mixing two concepts: state persistence and communication.

Under “normal” development flows, you operate within the safe, cozy little walls which make up your service. You’re free to make choices about implementation and storage and then, when you’re ready, deal with how those things get exposed to the outside world.

Exactly! This is exactly what you should be doing. A service boundary owns its data and defines how it exposes it. If you’re conflating Event Sourcing (state) with Event Event Architecture (communication) you can see how this would violate that.

For one, you’re probably going to be building the core components from scratch. Frameworks in this area tend to be heavy weight, overly prescriptive, and inflexible in terms of tech stacks.

Now it depends on which side of the question they were referring to. If we’re talking about Event Sourcing, there really isn’t any “framework” required. For example, if you’re using EventStoreDB, you can use their SDK directly. In many of the examples I use, I’m creating a Repository that is reading from an event stream and replaying the events to build up an Aggregate Root to the current state. From there the Aggregate Root generates new events and those are persisted back to the Event Stream from the Repository. There isn’t any framework or library code required other than the SDK Client from the Event Store.

With Event Driven Architecture, you absolutely want to use a library for messaging. I say this because there has a lot of patterns and concepts that come with using Event Driven Architecture. Messaging libraries provide these patterns and concepts for you so that you don’t have to implement them yourself. In the .NET space, libraries like NServiceBus, MassTransit, Brighter all come to mind that handles things like the Outbox Pattern, Process Managers, Fault Tolerance, Retries, Dead Letter Queues, and more.

If you have a UI, it generally needs to play along with the event driven aspect of the back end. Meaning, it should be task based.

Absolutely. Task-Based UIs will guide the end-user to perform specific tasks usually in some type of workflow. Tasks (or actions, commands) are explicit and allow you to then derive what intend to do where you can then generate the appropriate event from that task. CRUD isn’t explicit. You do knot know the intent of the end-user when you provide them with a CRUD-based UI. For more on task-based UIs, check out my post Decomposing CRUD to a Task Based UI.

A super common piece of advice in the ES world is that you don’t event source everywhere *. This is all well and good at the conceptual level, but actually figuring out where and when to draw those architectural boundaries through your system is quite tough in practice.

Defining boundaries is one of the most important things to do when developing and designing a system. Yet it’s one of the hardest things to do. In my experience, the real core of your solution space is where the complexity lies. On the outer edges, there are boundaries that are often in a supporting role. These supporting boundaries may either be very generic and can be something you buy off the shelf and integrate with, while others you may want to develop but can simply be CRUD. Each boundary defines how it persists state best on the requirements and what may fit best. Some boundaries might be best suited for a relational database, others a document database, or some an Event Store. But again that is about persistence and state. Each boundary decides its persistence. For communication, you can still leverage an Event Driven Architecture even though some boundaries are using a relational database. Again, don’t conflate needing to do Event Sourcing for persisting state in order to communicate via events.

We made it about a month before a shift in focus caused us to hit our first “oh, so these events are no longer relevant, at all?” situation. Once you hit this point, you’ve got a decision to make: what to do with the irrelevant / wrong / outdated events.

With Event Sourcing, the concern about versioning events is done within a single boundary that owns and uses those events for state. There are different strategies for versioning with event sourcing and how you want to handle “old” events.

One thing I’ve noticed however is that “no longer relevant” doesn’t actually happen all that often. If you’re using established business concepts, they don’t often become irrelevant. Usually, events that become irrelevant are because the developers defined the events and are generally based more on technical concerns rather than business concepts.

Once your data grows to the point where you can no longer materialize from the ledger in a reasonable amount of time, you’ll be forced to offload the reads to your materialized projections. And with this step comes materialization lag and the loss of read-after-write consistency.

This is a valid concern when you’re using Event Sourcing and creating Projections (a read model) that is generated asynchronously. There are various strategies to handle this that I’ve talked about in a video about Event Consistency. However, to point out, this isn’t specifically about Event Sourcing Projections but any type of system where you don’t have a read-after-write consistency. This can include using a database that is eventually consistent and has replication lag.

Event Sourcing vs Event Driven Architecture

Hopefully, this clarifies the differences of Event Sourcing vs Event Driven Architecture. While the original blog post has some valid points, the issue is it’s pointing those problems at Event Sourcing, when in many cases it’s because it’s conflating the two.

Event Sourcing is about using events as state. Event Driven Architecture is about using events to communicate. That’s not to say that Event Sourcing or Event Driven Architecture don’t have their difficulties, they do. However, if you treat them for what they are, you eliminate a whole set of problems the original post had because you’re not conflating the two. They are completely orthogonal from each other.

Source Code

Developer-level members of my YouTube channel or Patreon get access to the full source for any working demo application that I post on my blog or YouTube. Check out the YouTube Membership or Patreon for more info.

Follow @CodeOpinion on Twitter

Software Architecture & Design

Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design

Securing Sensitive Data in an Event Driven Architecture

You secure sensitive data when stored in a database, but are you securing sensitive data in an event or message driven architecture? For example, if you need to include Credit Card information in a message for a queue to be processed. But need to be compliant with PCI-DSS! You can encrypt the actual message or a portion of the message, or if the broker supports it, use server-side encryption.

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

Transfering Senstive Data

When not using a message broker and doing everything in a synchronous request/response, we can simply use a secure transport such as HTTPS/TLS. As an example, if we had a checkout process where we needed to process a credit card, we would have the browser send the data over HTTPS/TLS to our HTTP API service, and then it would, in turn, make an HTTPS/TLS request to the Payment Gateway we’re using to process credit card transactions.

Our HTTP API just becomes a proxy to relay that information to our payment gateway. At no point are we storing that data on any type of durable storage where it will be left unencrypted in plain text. Everything is secured transferring it from our Client/Browser to our HTTP and then finally to the payment gateway all over HTTPS/TLS.

Securing Sensitive Data At Rest

This works fine until you actually need to persist sensitive data, or in this example, credit card info. It may not be ideal to be bound to process the credit card via the Payment Gateway when we place an order. If the payment gateway is unavailable or causes an error, we will lose the order.

Rather what we can do is place the order and then send a message to a queue to process the credit card asynchronously in a separate process.

This decouples the two actions of placing an order and processing the credit card.

So what’s the issue? It may not seem obvious, because we are well aware of securing sensitive data in a database, however, in order to do this, we must place the credit card information inside our message onto the queue, which is likely using durable storage. Now we have sensitive data that are unencrypted in plain text. Not ideal.

Encrypted Message (or Properties)

One solution to this is to encrypt the message or properties within the message before they are actually sent to the broker. When the client sends the data over HTTPS/TLS, we then create our message encrypt it (or properties within the message)

Securing Sensitive Data in an Event Driven Architecture

Then our consumer can pick up the encrypted message from the message broker, decrypt it locally in memory and get out the values.

Securing Sensitive Data in an Event Driven Architecture

This requires the producer and consumer to share a key that is used for encryption and decryption.

As an example, NServiceBus enables securing sensitive data by encrypting properties within a message. You can configure this by using the EnableMessagePropertyEncryption on the endpoint configuration.

Then it’s as seamless as using the EncryptedString type in any message. NServiceBus will handle the encryption and decryption automatically.

Here’s a screenshot of a PlaceOrder command using RabbitMQ. You can see the CreditCard Number has an EncryptedValue. Everything is stored in rabbit as an encrypted value and nothing is in plain text.

Server Side Encryption

Another option is using a message broker that supports server-side encryption (AWS SQS, Azure Service Bus). This means that all messages are encrypted at rest by the broker itself.

This means that nothing has to change in your application in terms of encrypting messages. We send the message broker our messages in plain text.

Securing Sensitive Data in an Event Driven Architecture

However, the broker itself is encrypting the messages while at rest.

Securing Sensitive Data in an Event Driven Architecture

When a consumer receives a message, the broker will unencrypt it and send the consumer the message in plain text.

Securing Sensitive Data in an Event Driven Architecture

All communication is done over secure channels.

Securing Sensitive Data

Often times messages are overlooked or not thought of as persistent storage or somewhere to think about having to secure sensitive data. If you have requirements to store sensitive data for various compliance reasons, make sure to secure messages either by encrypting the message at self prior to sending it to the broker or by leveraging a broker that supports encryption at rest.

Source Code

Developer-level members of my YouTube channel or Patreon get access to the full source for any working demo application that I post on my blog or YouTube. Check out the YouTube Membership or Patreon for more info.

Related Links

Follow @CodeOpinion on Twitter

Software Architecture & Design

Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design

When NOT to write an Abstraction Layer

Common advice is to abstract dependencies. Often this is because people have been burned by depending directly upon 3rd party dependencies that they are highly coupled to. If they need to change the dependency, which they are highly coupled with, it can be pretty painful. This is why people say to create an abstraction layer around any dependency to isolate it so that your application code is coupled to yourself rather than a 3rd party. While this is generally true, it depends, and you don’t want to create an abstraction that is more work in the short and long run by creating the wrong abstraction. Here’s a concrete example of when you shouldn’t abstract.

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

Abstraction Layer

So why is the general advice to abstract 3rd party dependencies? Well if you’re going to be highly coupled to the dependency, and it has breaking changes (via new releases/versions), then you’re going to have to deal with all the breaking changes everywhere you’re using that dependency in your application code.

To alleviate that issue, one solution is to create your own abstraction layer that your application code depends on instead. This way if there is any change to the underlying dependency, you are changing it within the implementation of your abstraction.

Generally, this abstraction is a wrapper (facade) that hides the complexity of the 3rd party and often times simplifies its usage for your specific use case(s).

A common example to think of is the Repository Pattern. You’re creating a repository to abstract the native interactions with the underlying database. These interactions could be from a 3rd Party ORM or even using the native SDK for the database you’re using.

Abstraction Layer

In this case, your application code is not directly coupled to the ORM (dependency) but rather your application code is coupled to your repository. If anything changes with the underlying ORM (or SDK) then you must make changes to the repository rather than in any application code.

I’ve talked about the repository pattern before, you might want to check out my post Should you use the Repository Pattern? With CQRS, Yes and No!

Messaging

While creating abstractions around 3rd parties is generally a good idea in a lot of situations, it’s not a very good idea in others. A concrete example of this is with a messaging library.

If you’re using a specific message broker directly with their SDK, then you likely will create for the same reason create an abstraction around it.

Abstraction Layer

However, once you get into the deep end with messaging and are using an Event (Message) Driven Architecture, you’ll start implementing common patterns and concepts.

Common patterns and concepts such as the Outbox Pattern, Fault Tolerance, Scheduled (Delayed) Delivery, Claim Check, Encryption, Stateful(less) Process Managers, and more.

However, there are messaging libraries that already do this. They are themselves abstractions over various queuing technology. Meaning you can use a messaging library that would support RabbitMQ, Azure Service Bus, Amazon SQS, and others.

Messaging libraries also implement and expose ways to use those patterns and concepts that are very common when in an Event (Message) Driven Architecture.

So the question becomes, should you abstract a messaging library?

Abstraction Layer

Opinionated

The problem is these types of libraries are very opinionated on how they implement common patterns and concepts. This means that if you create your own abstraction over a messaging library, you’ll inherently create something that is very specific to the actual messaging library.

If you need to expose something like the outbox pattern in your own abstraction, then depending on how the underlying messaging library accomplishes that, you’ll have to expose that in a very similar way.

Meaning you’ll likely leak some details that are not implemented the same way within another messaging library.

Your abstraction will have the same similar API surface and opinions about implementation as the underlying messaging library. This means your abstraction will be very similar to the point of adding no additional value other than hiding the dependency.

Abstraction Layer

Now if you want to change the messaging library dependency with another, you might think that having your abstraction would be useful. But since it’s very opinionated, they don’t really line up in terms of how functionality is exposed. I purposely made the two Messaging Library boxes in the diagrams different shapes for this reason.

You won’t be able to fit your abstraction over a different dependency because both dependencies are different in how they accomplish the same patterns and concepts.

I’m not against abstractions or creating your own abstraction layers over dependencies. I do think in general rule is a decent approach in a lot of situations, just not all situations. Creating your own abstraction so you depend on your own types, often facades that isolate and simplify the underlying dependency for your use cases.

Complexity

In the concrete example of a messaging library, it doesn’t make sense to create an abstraction layer. A messaging library is itself an abstraction. It will be core to a system that is built around Event Driven Architecture and messaging. There will be little value gained by creating your own abstraction but rather more complexity added by adding indirection.

Source Code

Developer-level members of my YouTube channel or Patreon get access to the full source for any working demo application that I post on my blog or YouTube. Check out the YouTube Membership or Patreon for more info.

Related Links

Follow @CodeOpinion on Twitter

Software Architecture & Design

Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design