Category: Software Design

How do you share data between services? This is one of the most common questions when you have a system composed of many different services, and each service owns a data set. For example, you may need data from another service for validation or to send data to another service as part of a workflow. I will explain a few situations where you could share data between services, but more importantly, when you shouldn’t.

YouTube

Check out my YouTube channel, where I post all kinds of content accompanying my posts, including this video showing everything in this post.

https://www.youtube.com/watch?v=anL-RGKz_Ak

Data Ownership

A common situation is thinking you need to call another service to perform some action because you need to provide it with the required data. This is because data is being passed around services rather than living with the service that owns it.

For example, an order is placed through an e-commerce checkout process. The assumption is the Ordering service owns the entire checkout process and needs to call the shipping service with the order’s shipping details.

We need data in the correct place to avoid this service-to-service communication between Ordering and Shipping services. Instead, the checkout process involves each service as a part of the workflow.

The first part of the checkout process would be the client calling the Ordering service to start the checkout process. This could include the ordering service returning an identifier.

Next, the client would send the shipping service the address and other required details. This information would correlate with the identifier used in the first step.

Finally, the client would complete the checkout process and place the order with the Ordering Service. At this point, the ordering service persists the order and can publish an OrderPlaced event with the checkout process identifier.

Now we’re temporally decoupled, and the shipping and ordering services aren’t communicating directly. We simply are using publish-subscribe (or a queue).

The shipping service will then consume that OrderPlaced event. Since it contains the identifier used in the checkout process, the shipping service already has all the information in its own local database.

So if you need data from another service or you need to send data to another service, ask yourself why a single service owns the entire workflow.

Reference Data

Sometimes, you need data from another service for validation purposes or to persist with your own service state. I’m not referring to reporting or query purposes. I’ve discussed that in my post on The Challenge of Microservices: UI Composition.

You have an action/command to perform and must call another service to get data. It could be for validation, or you must save that data with your own state.

The important aspect is what type of data you request from another service. If it’s reference data from a supporting service boundary that’s non-volatile, then you have options. If it’s transactional data, I suspect you might have some service boundaries that aren’t entirely aligned with the business processes.

I mention non-volatile because the frequency of the data changing and the validity of the reference data play a role in whether it is plausible to have data from another service.

For example, our Sales service needs currency exchange rates when placing orders.

I mentioned earlier the validity of the data. In the case of currency exchange, it may be acceptable for Sales to use the exchange rate given for the entirety of a day. We only need to get the exchange rate between two currencies once within a day.

We then can cache currency exchange rates within the sales boundary.

Because the validity is a day, old exchange rates are still valid for orders for the same day. It’s historical data that isn’t going to change.

The exchange rate service owns exchange rate data. However, because its data is valid for a window of time (a day), this limits the number of calls we need to make to it, which also helps the availability of our Sales service.

Join!

Developer-level members of my YouTube channel or Patreon get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Microservices gets it WRONG defining Service Boundaries
• The Challenge of Microservices: UI Composition

Follow @CodeOpinion on Twitter

Software Architecture & Design

Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design

Leave this field empty if you're human:

A common approach people take with testing is mocking. Specifically defining interfaces for dependencies which are then typically mocked so you can test in isolation. While interfaces can be helpful for mocking as well as fakes and stubs, there can be other approaches taken. Meaning you don’t need to create an interface for everything.

YouTube

Check out my YouTube channel, where I post all kinds of content accompanying my posts, including this video showing everything in this post.

https://www.youtube.com/watch?v=2oRofOH2NT4

Deterministic

Let’s start with a method in an OrderService for creating an Order.

To test this method, there are a few dependencies involved. The OrderRepository, ItemRepository, URIComposer. Here’s what the test might look like by using a mocking library.

The problem is, as I’ve described above, is this test is flaky. That’s because the OrderDate is being set by DateTime.UtcNow. That’s non-deterministic.

Sure, we could be more lenient on our assert by maybe using a small range/window, but ultimately we want the result to be deterministic.

Interfaces

You could jump directly to an interface, which I’ve seen quite a bit of for this exact usage case with DateTime.

With the appropriate registration with the ServiceCollection, we can now inject a ISystemDateTime instead of calling DateTime.UtcNow.

We can be deterministic within our test by returning a specific DateTime for UtcNow.

Function

If you have a class/interface with one method, you have a function. The ISystemDateTime is exactly that. We have more options than interfaces when it comes to abstractions. In this case, using a delegate is also an option.

Again, by registering this delegate and the static method implementation, we can inject that delegate rather than an interface.

Our test becomes less cumbersome as we can easily create a stub for returning a deterministic DateTime without additional libraries or dependencies.

Values

As we break this down, you might also wonder why we even inject some abstraction when creating the order but instead pass the value of the DateTime to the order creation—basically moving up the call stack.

Now the caller is responsible for passing a DateTime to the CreateOrder, simplifying the test even more as we no longer have a dependency to pass to the OrderService

Abstract Classes

Have you ever needed to test an implementation that needed to use HttpClient? If so, you’re faced with the same issue where you want a deterministic result.

First, here’s an example of an ExchangeRateClient for getting currency exchange rates.

HttpClient doesn’t have an interface. So how do you test? Well, it does support providing your implementation of an HttpMessageHandler, where you must implement the SendAsync method.

Abstractions

You don’t always need to default to interfaces. You have other abstractions like delegates and abstract classes, and changing your design to move non-deterministic calls up the call stack so you can pass values instead.

Join!

Developer-level members of my YouTube channel or Patreon get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Testing your Domain when Event Sourcing
• Why is Clean Architecture so Popular?

Follow @CodeOpinion on Twitter

Software Architecture & Design

Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design

Leave this field empty if you're human:

One of the most challenging aspects of distributed applications like microservices is UI Composition. How to combine all the data from various services for your UI. I’m going over a few different solutions and their pros and cons for UI Composition and ViewModel Composition.

YouTube

Check out my YouTube channel, where I post all kinds of content accompanying my posts, including this video showing everything in this post.

https://www.youtube.com/watch?v=ILbjKR1FXoc

Multiple Sources

When you’re working within a monolith and a single database, we don’t have the issue of compositing data. We get all the data we need from the single source database.

As an example, here’s a typical e-commerce website with a listing of products.

The client could be a browser, and the app could be doing server-side rendering, or the client could be a JS/SPA frontend, and our App is an HTTP API.

Either way, our app can get all the data from a single database and compose and return the data as needed. It can get the product names, images, and prices from the same database.

The challenge is when multiple services are involved, each with its own database. Each service owns its own set of business capabilities and the data behind it. For example, if we had a Catalog service that owns all the product names and images. The Sales service owns the product price. How do you compose all that data together from two different services?

There are a few different options. The first is doing UI Composition.

UI Composition

UI Composition is about having a logical boundary own UI components that are composed to make your UI. A component (or set of components) would be responsible for getting the relevant data it requires.

The catalog service would own a Brand and Type filtering component. It would also own a Product display component to show the image and product name.

However, The price is owned by the Sales service and it would own a component for displaying the price.

With some pseudo-code, this is what the UI might look like.

We’re iterating over a list of products and then displaying the image, and name and also using the product price component.

This can work in a lot of situations however, in a grid/list, as you might have guessed, we just turned this into an N+1 problem. For each product, our price component is going to have to make an HTTP request to the sales service to get the price for that individual product.

This is not ideal at all. As mentioned, this can work in different situations, but not when we have a lot of components, nested components, or N+1 situations.

ViewModel Composition

Another option is using ViewModel composition. Typically this is done with a Backend-for-Frontend (BFF), which will call the relevant services and compose the data to be returned to the client.

The BFF would call the catalog service to get all the products to display.

Then it would make a single call to Sales requesting the price for only the products it received from the catalog. We’ve removed the N+1 issue as we’re now requesting all the prices in a single request so that we can compose all the relevant data and return it to the client.

Here’s an example of what the composed data being returned to our client would look like.

You’re not limited to doing UI Composition or ViewModel composition. You can do both. Use the method that makes the most sense in a given situation.

Sorting/Paging/Filtering

The biggest issue people run into with ViewModel composition is needing the ability to sort, page, and filter a listing of items.

In the example of our product listing, if we were sorting by name, it would work totally fine. However, if we’re sorting by price, our current flow would not work. We were making our initial call to the Catalog service, then getting the prices for all the items. Now we would have to reverse that flow so that got all the SKUs from the Sales service are sorted by price, then call the catalog service to get the rest of the information.

While this isn’t overly complicated in this exact example, if you have more than two services and/or you want to sort and filter. For example, you want to filter by product type and sort by price.

Event Carried State Transfer

If you have all the data required in a single service, then you don’t have to worry about these types of sorting/paging/filtering issues with multiple services.

This is why event-carried state transfer is popular to distribute data as a local cache to other services.

Before I explain how it works, I must say that I’m not a fan of distributing data to other services. If you’re doing so with reference data, such as my example for query purposes (not commands), it can work. However, I do not recommend distributing transactional data around. Check out my post Event Carried State Transfer: Keep a local cache!

The way event-carried transfer works is by publishing events of state changes so other services can consume these events and keep a local cache copy of the data from another service. Again this should be non-volatile reference data.

When a catalog item is updated, let’s say the name changed, a ProductChanged event would be published.

The sales service would consume this message, which contains the ProductId (SKU) and new product name. It updates its local database of this catalog information. It’s essentially a local cache of catalog data.

With this local cache, the sales service has all the data required to compose the product listing and perform any filtering, paging, and sorting.

Join!

Developer-level members of my YouTube channel or Patreon get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Do you need a Distributed Transaction? Maybe not!
• Event Carried State Transfer: Keep a local cache!
• UI composition – the blind spot of distributed systems
• Designing a UI for Microservices

Follow @CodeOpinion on Twitter

Software Architecture & Design

Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design

Leave this field empty if you're human: