How your “Sr.” Devs incurred Technical Debt

Are you overwhelmed by technical debt? Taking the path of least resistance when implementing new features in a large existing codebase will ultimately turn it into a difficult-to-change turd pile. It’s a vicious circle. Making the “quick change” constantly makes it harder to make future changes. So what’s the solution? Being aware of technical debt, stop solely thinking about data, and give yourself options in your architecture.

YouTube

Check out my YouTube channel, where I post all kinds of content accompanying my posts, including this video showing everything in this post.

Path of Least Resistance

One common reason for a system growing over time and becoming unmaintainable is developers choosing to take the path of least resistance when implementing a change.

This happens for various reasons, such as time constraints, unfamiliarity with the system, lack of domain knowledge, poor overall architecture & design, etc.

For example, let’s say we have a typical web application that is using some underlying web framework that invokes some code into our application logic, through to our domain, and then some interaction with a database.

Application Request

When a new feature is implemented, it’s common to look at other features as templates for developing a new feature. Or, worse, it can be using an existing feature and adding the relevant code needed for the new feature throughout the stack. I say worse because this can often confuse two concepts that seem similar but are very distinct. Merging the two concepts within the same code path can add complexity.

Layers

This means we may change existing code through the entire stack, from the client, web API, application code, domain, and our database.

You may decide to piggyback off another feature because of time constraints. It’s not because the feature is difficult to implement. It’s time-consuming or will take more time than you have to implement. Or if you’re new to the codebase or it’s brittle, you might be afraid to make changes because you know it it can cause you to break other parts of the system and don’t want to cause any regressions.

The path of least resistance is making a change that you know isn’t going to break anything that isn’t overly time-consuming, but it’s not necessarily the ideal. It’s likely good for the right now but not good for the long run.

Technical Debt

Technical debt isn’t inherently bad. For me, technical debt comes in two forms. The first is when you’re aware and choosing to take on technical debt at a very moment, knowing it adds value now but will cause issues in the future. This awareness of choosing to make this explicit decision isn’t bad.

However, when you’re unaware that you’re making these types of decisions is when you’re headed in the wrong direction.

If you’re making explicit decisions about the tradeoffs of technical debt, you’re aware of the debt being incurred. You can then explicitly choose when to pay off (refactor) that debt. For example, with a startup, you might incur debt right now so that you have a future.

On the other side, if you’re unaware that you’re incurring technical debt, then when would you realize all the debt that’s been incurred and needs to be addressed? Taking the path of least resistance, without realizing it, is one form of this happening. While it seems like it’s helping you now, it could be hindering you now and even more so in the future.

Coupling & Cohesion

Software Architecture is about making key decisions at a low cost that give you options in the future. Having a good architecture allows you to evolve your system over time. As a codebase and system grow, it should not hinder future development. I’ve talked about this more in my post What is Software Architecture?

Why is a system brittle and hard to change? Generally, it has a high degree of coupling from higher and lower levels within a system. I find this is often because of the focus on data and informational cohesion rather than functional cohesion.

For example, let’s say we are in an e-commerce and warehouse system. There is the concept of a product. When we primarily think about data first, we think of a singular product. It holds all information for everything related to an individual product. The name, price, location in the warehouse, the quantity on hand, it is available for sale, etc.

Product Entity

In reality, a system for e-commerce and a warehouse would be huge. A large codebase that multiple departments would use in an organization. Sales, Purchasing, Warehouse (shipping & receiving), Accounting, and more.

In other words, I’m simplifying this example only to show a few different pieces of data related to a product, but in reality, there would be a lot.

When focusing on data primarily, we lose sight of the behaviors that relate to this data. What does the QuantityOnHand have to do with the Price? What does the Location have to do with the Description?

Nothing.

We’ve lumped all aspects into one concept of a product. However, in a large system like this, the concept of a product would exist in many different forms depending on the behaviors provided.

Product Entities

Sales have the concept of the product that cares about the Selling Price and if we’re selling. It’s customer focused.

Purchasing cares about the price from the vendor or manufacturer, which is our cost. It’s vendor-centric.

The warehouse cares about the location of the product in the warehouse and the assumed quantity on hand.

Each logical boundary has a concept of a product but has different concerns in each of its own contexts.

This means instead of mixing all these different concerns up together, instead be driven by the capabilities of each boundary and then the data ownership for those capabilities.

Low functional cohesion will lead to a high degree of coupling.

Coupling

Defining logical boundaries by grouping related behaviors will lead to higher cohesion, which can then lead to loose coupling.

Logical Boundaries

Awareness

Some of the trade-offs of taking the path of least resistance is being aware of the trade-offs you are making between coupling and cohesion. Earlier I mentioned piggybacking off an existing feature to implement a new feature. You’re coupling. Again, not a bad thing if that decision is explicit.

Over time, left unchecked, if you’re unaware of the technical debt you’re creating, you’ll end up with a large turd pile that’s brittle and hard to change.

Incurring Technical Debt

If you are aware you can choose when to pay down debt (refactor) and keep making those decisions over time, you can manage the amount of debt incurred, never letting it get out of reach.

Paying Back Technical Debt

I often say a system is a turd pile because nothing is perfect. It’s a constant battle to pay down debt, whether you choose it explicitly or not.

Join!

Developer-level members of my YouTube channel or Patreon get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Follow @CodeOpinion on Twitter

Software Architecture & Design

Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design

Data Partitioning! Don’t let growth SLOW you down!

Why would you want data partitioning? Data keeps growing. Hopefully, you’re working on a system with active customers with a good lifespan! If so, the amount of stored data will keep growing over time. The challenge can be managing data growth, which can start to impact performance. I will discuss ways to use this data and strategies to avoid it affecting performance.

YouTube

Check out my YouTube channel, where I post all kinds of content accompanying my posts, including this video showing everything in this post.

Multi-Tenant

In a multi-tenant application, there are multiple ways to silo data. First, you can use the same database instance, but a partition key indicates which data belongs to which tenant.

Multi-Tenant

For example, tables in a relational database would have a TenantId, allowing you to query and retrieve data for a specific tenant.

Partition Key

For various reasons, data growth is one of them; you may choose to instead silo data in its database instance per tenant.

Silo Data

There are many reasons to silo data; check out my post on Multi-tenant Architecture for SaaS for more on various aspects of sharing compute, identity, and more within a Multi-Tenant SaaS architecture.

Regardless of if you use a partition key or silo data, hopefully, you’re working in a system where data is growing! Ultimately this would mean you’re building a system with a lot of activity and users! Congrats!

The issue is over time; more users means more data.

Data Growth

How you initially wrote your system to query specific data may not be ideal now that you have a vastly different volume of data. Queries may have performed fast a couple of years ago, but the volume of data now those queries could be much slower, impacting overall system performance.

Lifecycle

In many line-of-business and enterprise systems, not all data is actively relevant. Many business processes and workflows related to data have a finite lifecycle.

There is generally some initial beginning creation, some work in process, and finally, some completion.

Lifecycle

As an example, let’s use a support ticketing system. When a user opens a support ticket, it goes through a lifecycle till the support ticket is finally closed.

Support Ticket

At this point, this support ticket is not likely very active in terms of interactions. A support ticket that was closed 2 years ago is not likely active at all, and users aren’t viewing them.

A small portion of the total data we store in our database is likely active/hot. Older data that its lifecycle has completed is still available to be queried or changed however; it’s more considered warm data that isn’t active.

Warm & Hot Data

Without any type of partitioning, we have a mix of hot and warm data in our database.

Mixed Hot and Warm Data

Instead, we can organize our data by whether it is warm or hot and increase system performance because we deal with a much smaller dataset when performing actions against hot data.

Partitioned Hot & Warm Data

As for implementation, this could mean that when a ticket is closed, we move it to a separate table/collection or even a separate database. The strategy of when is up to you, but the gist is to segregate between hot and warm data.

Time

Another strategy which is often also hot/warm, is to partition by time. It is very typical in many different domains to have a time-bound partition. An excellent example is fiscal or calendar years for finance or employee pay periods, often weekly or bi-weekly.

Data Partitioned by Year

Summary

Often transactional data is only needed to summarize it, at which point the summary becomes the truth. I worked in a system with a high volume of transactional data partitioned by the hour of the day it was received. After each hour, the data were summarized and removed from the hourly table.

Data Partitioned by Time

Often these are actual business concepts that you can capture. A warehouse with physical goods performs stock counts to confirm the actual quantity on hand that is recorded within the system. The database/system is not the point of truth; the warehouse is. Just because the database shows a quantity of 10 for a product does not mean there are 10 in the warehouse. A product could be damaged, stolen, or can’t be located. The transactional data of products received & shipped “should” be the quantity on hand. But it’s not. That’s why stock counts exist to reconcile what is in the warehouse.

Summary Business Concepts

We can use these business events as a summary. The stock count is the summary event we can use. The other transactional events could be archived.

Cold Data

Speaking of archived, you do not need to delete data if you’ve summarized it. You can archive it as cold data. Data that is archived can be restored for compliance or regulatory reasons.

Archived Cold Data

This data isn’t immediately available as it’s archived, and it is removed from warm and hot data storage.

Data Partitioning

Hopefully, this gave you some ideas about how can partition data over time as it grows. Time is a key aspect, as most business data is time-bound or goes through a finite lifecycle or lifespan. You can also leverage business concepts that inherently are summaries of a point in time where previous transactional data can be archived.

Join!

Developer-level members of my YouTube channel or Patreon get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out the YouTube Membership or Patreon for more info.

You also might like

Follow @CodeOpinion on Twitter

Software Architecture & Design

Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design

What is Software Architecture?

Software architecture is about making critical decisions that will impact how you can make decisions in the future. It’s about giving yourself options at a relatively low cost early on so your system can evolve without a high cost. Software architecture is about options.

YouTube

Check out my YouTube channel, where I post all kinds of content accompanying my posts, including this video showing everything in this post.

Cost of Options

Software should be malleable. It shouldn’t be so rigid that you can’t change it because of new requirements or insights about the domain or the model you’ve built. You want to be able to evolve your system over time as all these emerge. You don’t want to have to pay a high price (complete rewrite) because you’re system is hard to change. Giving yourself low-cost options means making decisions that allow you to evolve your system over time but don’t come at a high cost (time/effort/etc.)

This means you have to pay the price initially to give yourself these options, usually early on in a project/product. These are critical decisions in choosing which options are low-cost and high-value.

I’m not talking about “what if” scenarios. Developers tend to make assumptions, often related to technical and business requirements. I’m not referring to all kinds of edge cases or technical concerns like scaling, which developers love to focus on.

The options I’m talking about are fundamental to your architecture. How you develop the system allows you to evolve it over time.

Coupling & Cohesion

A lot of software design comes down to understanding and making decisions based on coupling & cohesion.

To me, coupling & cohesion are the yin-yangs of software design. They are a push & pull against each other. You’re trying to increase functional cohesion and lower coupling.

There are different forms of coupling, but to give a definition:

“degree of interdependence between software modules”

ISO/IEC/IEEE 24765:2010 Systems and software engineering — Vocabulary

And for cohesion:

“degree to which the elements inside a module belong together”

Structured Design: Fundamentals of a Discipline of Computer Program and Systems Design

Why do coupling & cohesion matter? Because ultimately, a lot of the decisions made are rooted in one or both of them, even if you don’t realize it.

If you do understand coupling and cohesion, it can better help you make decisions that provide options.

The 3 concrete examples I will provide in this post are all rooted in coupling & cohesion.

For more on coupling & cohesion, check out my post: SOLID? Nope, just Coupling and Cohesion

Logical Boundaries

The first way to give yourself options within your architecture is to define logical boundaries. Grouping related behaviors and functionality (capabilities) of what your system provides. Having groupings of capabilities that are functionality cohesive.

Not having a system that is free for all of functionality without any boundaries.

A piece of functionality shouldn’t be intertwined with other unrelated functionality. In other words, the dependencies for one piece of functionality shouldn’t affect another. An example of this is a database. A set/grouping of features should own and be responsible for the underlying data for that feature set.

Define logical boundaries where you’re grouping functionality that works together on a set of underlying data. Focus on the capabilities and behaviors of your system. Group those capabilities into logical boundaries.

Within a logical boundary, you can make decisions that are isolated within it. How should you perform data access? Which type of database is best for the data set of that logical boundary? How should we model within the given context? Different boundaries will have different models. By defining logical boundaries (cohesion), you can make all kinds of decisions that are best for the feature set within that boundary. This gives you options.

Boundaries are one of the hardest things to define correctly, yet the most important things to do. Check out a whole series and talk: Context is King: Finding Service Boundaries

Loose Coupling

If you’re defining boundaries, how do you communicate between them?

In a free for all, there is coupling everywhere. You have different parts of the system that are directly coupled to other parts of the system. This could be coupling between classes/modules or, generally, at it’s worse, via the database.

I often refer to this as a turd pile. But it’s just a system that’s lost control of coupling.

If you’ve defined logical boundaries, as explained earlier, you’ll likely need to communicate between them. Any system will have long-running business processes and workflows that span many logical boundaries.

To remove tight coupling, we can leverage asynchronous messaging. Removing direct communication between boundaries means we are also removing temporal coupling. In other words, you’re not bound by time.

This means that one boundary can send a message to a queue for another and can be processed independently.

Because you have logical boundaries (cohesion), this works best with long-running business processes or workflows. So often, we model our workflows as being synchronous requests/responses when in reality, we could be building a much more resilient system by making them asynchronous.

Asynchronous messaging and event-driven architectures give you options by loose coupling! Check out my Real-World Event Driven Architecture! 4 Practical Examples

CQRS

Unfortunately, CQRS is a buzzword (acronym) that is widely misunderstood.

Command Query Responsibility Segregation is often conflated with Event Sourcing, Asynchronous Communication, Domain Driven Design, Multiple Databases, and more. If you search and read enough posts, you’re bound to find a similar diagram.

CQRS Confusion

Sadly, while this is CQRS, as mentioned, it’s also conflating a bunch of other patterns or concepts. CQRS is nothing more than separating reads and writes even from a service layer.

Yes, really. It’s that simple. Still don’t believe me? Check out my post CQRS Myths: 3 Most Common Misconceptions, where I reference many of the early blog posts from Greg Young.

So why is it important? Because it gives you options. Defining separate paths for reads and writes allows you to make decisions for each path and each occurrence.

If you look at the first diagram illustrates a Command Bus, a Domain model, Event Sourcing, and a projection (multiple databases). All of that is facilitated by the decision to separate commands and queries.

CQRS is a gateway to other patterns and concepts, but at its core, it’s pretty trivial but it gives you options!

What is Software Architecture?

To me, software architecture is about critical decisions, usually early on within a product/project, that give you future options. Options that allow you to evolve your system over time. The cost is usually relatively low when making decisions and giving yourself options at the very beginning. Defining logical boundaries, loose coupling between boundaries with a message/event-driven architecture, and CQRS.

As always, it’s rooted in coupling and cohesion.

Join!

Developer-level members of my YouTube channel or Patreon get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out the YouTube Membership or Patreon for more info.

You also might like

Follow @CodeOpinion on Twitter

Software Architecture & Design

Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design