Event-Driven Architecture on AWS, Part III: The Hard Basics

My previous post in this series discussed the reliability issues many messaging-based systems suffer from and how to address them by implementing the outbox pattern. This post switches the focus from publishing to processing messages published by other components of the system.

As before, I will start by defining the problem we have to tackle.

The Problem

Let’s revisit the outbox pattern once more. It addresses the common problem of publishing messages after the underlying business transaction has already been committed: the events might not get published at all. By implementing the pattern, you ensure that all the events will be published—at least once. Let me explain the last part.

The outbox pattern Figure 1: The outbox pattern

[Read More]

Event-Driven Architecture on AWS, Part II: The Advanced Basics

In my previous post in the series, I discussed the basic building blocks for implementing event-driven architecture (EDA) using AWS managed services. This post is about the advanced basics. It’s advanced because, based on my consulting experience, very few companies apply the practices I want to discuss. However, I still consider this as basics because these practices are not optional but essential for implementing reliable messaging-based systems.

Although the examples in this post use AWS services, the material applies to any system, regardless of the infrastructure it runs on. The implementation might differ, but the underlying principles remain the same.

Since this post is about a pattern—and patterns, by definition, are repeatable solutions to commonly occurring problems—I want to begin by discussing a commonly overlooked problem in EDA-based systems.

The Problem

In the previous post, I suggested using AWS SNS for publishing messages in an event-driven system. Here’s a common example of how such publishing is carried out:

...

sns.publish(
    TopicArn=users_topic_arn,
    Message=json.dumps({
        'event_type': 'user_registered',
        'event_id': str(uuid.uuid4()),
        'user_id': 'USER12345',
        'name': 'John Doe',
        'email': 'john.doe@example.com',
        'source': 'mobile_app',
        'registration_date': '2024-10-11T20:01:00Z'
    })
)

...

In the above example, an event of type user_registered is published to an SNS topic. But did that event come from thin air? Is that all services do, just publish messages? Of course not. The event is part of a larger business process that typically involves updating some state in an operational database before notifying external components about it. A more accurate representation of this process would look like this:

[Read More]

The Essence of Coupling and Cohesion in Under 500 Words

Integration Strength and Distance

When you couple two components, they need to share some form of knowledge: public interfaces, functional behavior, implementation details, or other types of information. Let’s call it the integration strength. The more knowledge is shared, the stronger the integration. The stronger the integration, the more both components will have to change together. Therefore, you should always look to minimize the integration strength. That said, minimizing integration strength isn’t always possible—sometimes, extensive shared knowledge is necessary. For example, when the components implement closely related business functionalities.

Another important property of integrated components is the distance between them. For example, objects in a module are located closer to each other than the source code of two microservices. The greater the distance, the higher the induced cognitive load and coordination challenges, and the more effort is needed to change the integrated components simultaneously.

Now let’s examine four extreme combinations of the two dimensions:

[Read More]

Balancing Coupling in Software Design🎉

Frist of all, today is the 🐒 book’s birthday! 🎉

Exactly three years ago, Learning Domain-Driven Design was published! And what a birthday present the monkey received: today marks the official release of Balancing Coupling in Software Design! The new book is available in print and electronic formats on Amazon, InformIT, and other major book stores 🎉

Learning Domain-Driven Design's third anniversary

[Read More]

Event-Driven Architecture on AWS, Part I: The Basics

The abundance of services provided by AWS often makes it possible to implement the same functionality in different ways. In the case of messaging systems, AWS offers services such as Simple Notification Service (SNS), Simple Queue Service (SQS), EventBridge, Kinesis, and Managed Streaming for Apache Kafka (MSK). It may seem that at least a subset of these services duplicates the same functionality. In this post, I want to describe my go-to architecture and explain why, in my opinion, it is the simplest, most cost-effective, and robust solution for the majority of cases.

Event-Driven Architecture

Components of an event-driven system communicate by publishing and subscribing to events. The asynchronous integration offers significant non-functional advantages. For example, it decouples the integrated components’ lifecycles. To a certain degree, the system can keep functioning even in the face of unavailability of some of its services, thus also reducing the coordination overhead needed to deploy updated components, and to evolve the system.

A basic event-driven integration is comprised of two parts: a component of the system can publish events describing important occurrences in its lifecycle, and it can react (subscribe) to events published by other parts of the system. Let’s see what managed services we can leverage for implementing this.

[Read More]

Untangling Microservices, or Balancing Complexity in Distributed Systems

Untangling Microservices, or Balancing Complexity in Distributed Systems

The microservices honeymoon period is over. Uber is refactoring thousands of microservices into a more manageable solution [1]; Kelsey Hightower is predicting monoliths are the future [2]; and even Sam Newman is declaring that microservices should never be the default choice, but rather a last resort [3].

What’s going on here? Why have so many projects become unmaintainable, despite microservices’ promise of simplicity and flexibility? Or are monoliths better, after all?

In this post, I want to address these questions. You will learn about common design issues that turn microservices into distributed big balls of mud – and, of course, how you can avoid them.

[Read More]

Anti-Pattern: Optimistic Consistency

Ladies and Gentlemen, lo and behold a new consistency model - “Optimistic Consistency”. Implementation of this pattern is quite simple: commit as many transactions as you want, against as many storage mechanisms as you need. That’s all. Let’s see an example. Example Say as the result of some user’s operation the system has to: Modify data in MongoDB Publish some events to Kafka Update search model in Elasticsearch; and Execute an operation on some remote system You implement this logic by executing those operations one after another. [Read More]

Zen of Software Engineering

There is a pet peeve of mine, that I’m encountering way too often: the condition I call Silverbulletitis. This virus spreads among developers on many levels — juniors, seniors, and even architects. In this post I’d like talk about why this condition is dangerous, and how it can be treated. Symptoms All too often you can hear statements that some tool or technique is bad, but another one is the greatest thing since sliced bread. [Read More]

IOS 11: Fixing APN Configuration Issues

So a few days ago I’ve decided to move to another cellular carrier. I pushed the new sim-card in, and got greeted with “Could not activate cellular data network”. First instinct was to install the new carrier’s APN configuration profile, but this resulted in another bummer: “Only one APN configuration can be installed at a time”. Ok, so I have to delete the previous carrier’s profile. Settings -> General -> Profiles …and there weren’t any profiles: “No profiles are currently installed”. [Read More]

Tackling Complexity in Microservices

Tackling Complexity in Microservices

Microservices have taken our industry by storm. Teams around the world have ditched their clumsy monoliths in favor of chasing the dream of loosely coupled, manageable, and independently deployable microservices. Unfortunately, many of them ended up worse off than they were before - in the realm of distributed monoliths. Why?

“If I had an hour to solve a problem I’d spend 55 minutes thinking about the problem and 5 minutes thinking about solutions” - Albert Einstein

In my opinion, this quote is the key to solving most of the issues we have when designing microservices-based systems. We rush into solutions without ensuring we know what exactly a microservice is. In this post, I’d like to take a stab at answering this question.

[Read More]