CQRS and Event Sourcing Intro For Developers

We live in a world of dynamically changing technologies. New ways of architecturing our solutions, new frameworks and libraries seem to appear on almost daily basis.
But good software engineering is not about fancy frameworks and solutions aggressively promoted by their vendors. It is not about doing something because Netflix or Google did it. It is about taking well-thought-out decisions based on facts and knowledge. That’s why it is important to be familiar basic architectural concepts like CQRS. It is one of the tools we use in our software house every day. We mentioned CQRS in the article which is part of the series about Microservices on .NET Core, but it was presented from technical perspective and here we want to focus on basics concepts explanation with visualisation and examples.

What is CQRS?

Command-Query Responsibility Segregation is a pattern that tells us to separate operations that mutate data from the ones that query it. It is derived from Command Query Separation (CQS) principle devised by Bertrand Mayer – author of Eiffel.
CQS states that there can be only two kind of methods on a class: the ones that mutate state and return void and the ones that return state but do not change it.
Greg Young is the person responsible for naming this pattern CQRS and popularizing it. If you search internet for CQRS you will find many excellent posts and videos made by Greg. For example you can find excellent and very simple explanation of CQRS pattern here – http://codebetter.com/gregyoung/2010/02/16/cqrs-task-based-uis-event-sourcing-agh/.

We want to show example from insurance domain – PolicyService. A service responsible for managing insurance policies. Below is a snippet with interface before applying CQRS. All methods (write and read) are in one class.

interface PolicyService {
    void ConvertOfferToPolicy(ConvertOfferRequest convertReq);
    PolicyDetailsDto GetPolicy(long id);
    void AnnexPolicy(AnnexRequestDto annexReq);
    List SearchPolicies(PolicySearchFilter filter);
    void TerminatePolicy(TerminatePolicyRequest terminateReq);
    void ChangePayer(ChangePayerRequest req);
    List FindPoliciesToRenew(RenewFilter filter);
}

Before apply CQRS

If we apply CQRS pattern to this case, we get two separated classes, better fulfilling the SRP principle.

interface PolicyComandService {
    void ConvertOfferToPolicy(ConvertOfferRequest convertReq);
    void AnnexPolicy(AnnexRequestDto annexReq);
    void TerminatePolicy(TerminatePolicyRequest terminateReq);
    void ChangePayer(ChangePayerRequest req);
}

interface PolicyQueryService {
    PolicyDetailsDto GetPolicy(long id);
    List SearchPolicies(PolicySearchFilter filter);
    List FindPoliciesToRenew(RenewFilter filter);  
}

After applied CQRS

That’s all. If you do this, you can show off in front of your colleagues over beer that you’re using CQRS 🙂 But it is just a mechanical transformation, which is first step in applying CQRS. What is a simple transformation has great consequences and opens new possibilities, which we will explore it in further part of this post.

What CQRS enables?

Most of the time data needed to change state are different in form or quantity than data needed by the user to make a decision. Having the same model to handle both queries and commands result in a model that is bloated with things needed only by one type of operations, also model complexity is increased and aggregate size is usually bigger.

CQRS enables us to have different models for mutating state and different models to support queries. Usually write operation are less frequent than reads. Having separate models and separated database engines allows us to independently scale the query side and to better handle concurrent access as reads no longer block writes or the command side (in opposite case). Having separate models for commands and queries let us assign these responsibilities to different teams with different skills.
For example, you can assign command side to highly skilled OOP developers while query side can be implemented by SQL developers. CQRS let you scale your team and let you best devs focus on the core stuff.

Is CQRS an architecture?

People often get it wrong. CQRS is not a top level/system level architecture. Example of architecture styles are: layered, ports & adapters.
CQRS is a pattern you apply on “the insides” of your service/application and you may apply it only to portion of your service.

Example of implementation – evolutionary approach

There are many ways to implement CQRS. They have different consequences, complexity and applicability of such solution depends on your system context. You need different approach if you are Netflix with 150 millions of users and different solution will work for your typical enterprise app with just hundreds of users.
In our opinion, especially when dealing with existing (legacy) project,  the best approach is tackling CQRS evolutionary. Below we will try to describe the some of “levels of advancement” of CQRS implementation in the application. But remember, often you do not need to use advanced patterns and solutions. Venkat Subramaniam wrote great tweet about this: No good programmer retires thinking: I wish I had implemented more design patterns. Let’s solve real problems using simple pragmatic solutions.
Not always (in fact quite rarely) your application needs to reach the highest level, because this level may results in a very complex technical solution without satisfying benefits.

No CQRS

We begin with solution that is not using CQRS.

UI (through controllers layer) uses service facade layer, which coordinates business operations performed by domain model. Model is stored in relational database. In our example there is the PolicyService class (Java, C#), which is responsible for handling all business methods related to the policy.

We use one model for reads and writes. When performing business operations, we use the same classes that we use when searching. This may result in your domain model having attributes needed only for search or, in a worse case, you may be force to design your domain model to be able to query it easier.

If you want to learn more, check our examples, in Java or in C#.

Separate Models Commands and Queries

In this example we want to show code in which developers use separated models for write side and for read side.

In this example a mediator pattern is used. The role of the mediator is to ensure the delivery of a Command or Query to its Handler. The mediator receives a Command/Query, which is nothing more than a message describing intent, and passes this to a Handler which is then responsible for using domain model to perform the expected behavior. This process can therefore be looked upon as a call to the Service Layer – where the Bus does the plumbing of the message in-between. In Java example, we created Bus class, which is implementation of this pattern. Registry is responsible for linking handlers with commands/queries. In C# example, we use MediatR library that does all of this for us. You can read more about MediatR in our other article.

Now that we have separate entry points for commands and queries we can introduce different models to handle it. NoCQRS solution used RDBMS and ORM – typical stack in enterprise app. Without changes to our stack we can start using CQRS. With this setup we can use our domain model as command model. This model gets simplified: some associations used only for reads no longer necessary, some fields no longer necessary. On query model, we can define views in a database and map it with ORM. Alternatively, for query model, we can stop using heavy-weight ORM and replace it with plain old JDBC Templates or JOOQ in Java, or Dapper in .NET. If you want to avoid complex queries that define views in database you can take next step and replace views with tables that will be designed to handle queries. These tables will have simple structure with data mapping 1:1 to what user sees on the screen and what user needs to search for. Adding such tables instead of database views removes the burden of writing complex queries and opens new possibilities for scaling your solution, but it requires that you somehow keep your domain command model “in-sync” with your query model tables.

This can be done by:

• synchronously in the same transaction using something like Application Events in Spring (example) or using Domain Events,
• synchronously in the same transaction in command handler,
• asynchronously using some kind of in-memory event bus, resulting in eventual consistency,
• asynchronously using some kind of queue middleware like RabbitMQ, resulting in eventual consistency.

What are best practices for query model building like this?

• You should build one table/view per screen/widget (screen fragment).
• Relation between tables should be model like relations between screen elements.
• “View tables” contain columns per each field displayed on the screen.
• Read model should not do any computation, calculate data in command model and update read model instead.
• Read model should store precomputed data.
• Last but not least: don’t be afraid of duplicates.

Choice of approach for synchronization between command model and query model depends on many criteria. Even with database views you can achieve great results as you can scale your database with read-only replicas used only for querying with the views you have created. Having separate tables simplifies reads as you do not have to write complex SQL anymore, but you have to write code for updating query models yourself. There is no magical framework that will do this for you. Number of read models related to given command model part is also a decision factor. If you have 2-3 query models for an aggregate you can safely call all updaters in your command handler. It won’t affect performance, but if you have 10s of it, then you may consider running it asynchronously outside of the transaction that updates your aggregate. In that case you must check if eventual consistency is allowed. This is more business decision than technical one and must be discussed with business users.
Having separate query tables is a good step to taking our CQRS solution to next level.

If you want to learn more, check our examples, in Java or in C#.

Separate Storage Engines

In this approach we use different storage engines for queries models and for command models, for example:

• ElasticSearch for queries side, JPA for commands side,
• ElasticSearch for queries side, DocumentDb for commands side,
• DocumentDb for queries side, storing aggregates as JSON in RDBMS for commands side.

Each command handler should emit event with information what happened. Event is a named object that represents some change which happened in a given object. Event should provide information about the data that changed during the business operation. Events are part of domain. In our example we have some events regarding the insurance policy – PolicyCreated, PolicyAnnexed, PolicyTerminated, PolicyAnnexCancelled (Java example, C# example).

On read side we created event handlers (methods execute whenever a specific type of event comes in), which are responsible for projection creation. These event handlers perform CRUD operation upon the persistent read model (Java example, C# example).

What is projection? Projecting is a process of converting (or aggregating) a stream of events into a structural representation. Projection is a converted (or aggregated) stream of events into structural representation. This can be called many names: persistent read model, query model or view.

With this approach we can apply different tools to do queries and different to do state updates. This way we can achieve better performance and scalability, but at the expense of complexity. In typical business systems, the vast majority of operations performed in the system will use read side/query model. This element should be prepared for a much higher load, it should be scalable and allow to build complex queries allowing advanced search.
With this approach we will have to deal with eventual consistency as distributed transactions between various data sources are performance killers and most of NoSQL database do not support it.

You can take this example one step further and make query handlers in a separate microservice and command handlers in another separate services. Now you can scale independently your query and state change sides.

CQRS with Event Sourcing (CQRS-ES)

Next step is changing command side to use Event Sourcing. Architecture of this version is very similar to above (when we use separate storage engines).

The key difference is in command model. Instead of RDBMS and ORM, we use Event Store as a persistent storage. Instead of saving only actual state of object, we save stream of events. This pattern for managing state is named Event Sourcing.
Instead of keeping only current state of the system by altering previous state, we append events (changes) to sequential list of past events (changes). This way we not only know the current state of the system but we can easily track how did we reach this state.

Below example shows different state management method based on football match domain.

Traditional state management

An above diagram shows the traditional state management for Game object. We have information about the result of the match and when the game started/ended. Of course, we could model additional information here, such as a list of goals scored, a list of committed fouls, a list of corners taken. However, you must admit – the domain of a football match is ideally described by a series of events that have occurred over time.

Event Sourcing state management

When using Event Sourcing to manage the state of the Game object, we can accurately reproduce the entire game. We have information about what events have affected the current state of object. The above picture shows that each event is reflected in a specific class. That’s the magic of Event Sourcing 🙂

One of the main Event Sourcing advantages that are mentioned in most articles is that you do not lose any information. In traditional model every update erases previous state. The previous state is lost. You can say, that there are logs, backups and libraries like Envers but they do not provide you with explicit information about reason of changes. They only show you what data has changed, not why. In event sourced approach you model your events after business events in your domain, therefore it not only shows data changes but reason for change.

Next advantage is that saving your domain aggregates as series of events greatly simplifies your persistence model. You no longer have to design tables and relations between it. You are no longer constrained by what your ORM can and cannot map. During working with extremely advanced solutions like Hibernate we found cases, when we had to resign from some design concepts in our domain because it was hard or impossible to map to database.

There are more and more solutions supporting the creation of applications that use Event Sourcing (EventStore, Streamstone, Marten, Axon, Eventuate). In our examples we use own implementation of in-memory event store (Java example, C# example) that is derived from Greg Young’s example. This is not production-ready implementation. For production grade solution, you should apply more complex solution, like EventStore or Axon.

In which systems is it worth using Event Sourcing?

• your system has many behaviors that are not an ordinary CRUD,
• recreating historical states of objects is very important,
• business users see advantages of having full history for statistics, machine learning or other purposes,
• your domain is best described by events (for example, an application to track a assistance vehicle activities).

Should I use CQRS/ES frameworks?

If you are not experienced with CQRS/ES, you should not start with any framework. Start with your core domain stuff and implement some business functionality. When your business stuff starts to work, focus on technical stuff. Evaluate available choices like Event Store or Axon before going to implement your own event store or command bus. There many things to consider and many traps along the way (concurrency, error handling, versioning, schema migration).

Summary

There two camps: one says you should always use CQRS/ES the other says you should use only for parts of your solution and only when you need in highly concurrent systems with high performance/availability/scalability systems.
You should always evaluate your choices in the context of your requirements.

Even the simplest form of CQRS gives you good results without introducing much complexity. For example using views for searches instead of using domain model for it simplifies things a lot. In our systems we also found lot of places where adding specialized read model tables and updating it synchronously gave us very good results (like getting rid of 20+ tables join view definition with 4 unions and replacing it with one table).  
Using a dedicated search engine like ElasticSearch is also a safe bet, as long as eventual consistency is allowed.

CQRS may results in a very complex technical solution if you choose to use different storage engines, event buses and other technical components.
Only some complex scenarios and scalability requirements justify this complexity (if you run on Netflix scale). At the same time you can also apply CQRS using simple technical solution and benefit from this pattern – you don’t need Kafka to do CQRS.

Here, at Altkom Software & Consulting, we are fascinated with new technologies and frameworks which allow us to improve our solutions. However, we are aware that every change costs and is an investment that should gives our customers business benefits. We don’t use Hype Driven Development or Resume Driven Development in everyday work. We solve real-life problems using well-crafted pragmatic solutions.

We prepared two version of demo for this blog post, one is for Java developers and second is for .NET developers. Links below:

• .NET CQRS Intro: https://github.com/asc-lab/dotnet-cqrs-intro
• Java CQRS Intro: https://github.com/asc-lab/java-cqrs-intro

At the end, for the most tenacious, a brief summary of the pros and cons of CQRS & Event Sourcing.

CQRS pros & cons

Pros:

• better performance and scalability of your system,
• better concurrent access handling,
• better team scalability,
• less complex domain model and simple query model.

Cons:

• read and write models must be kept in sync,
• maintenance and administration costs if you choose two different engines for read and for write side,
• eventual consistency is not always allowed.

Event Sourcing pros & cons

Pros:

• append only model is great for performance, scalability
• no deadlocks
• events (facts) are well understood by the business expert, some domains are inherently event sourced: accounting, healthcare, trading
• audit trail for free
• we can get object state at any point in time
• easy to test and debug
• data model decoupled from domain model
• no impedance mismatch (object model vs data model)
• flexibility – many different domain models can be constructed from the same stream of events
• we can use this model with reversing events, retroactive events
• no more ORMs – thanks to the fact that our object is built from events, we do not have to reflect it in a relational database

Cons:

• not natural way for developers to manage state and construct aggregates, takes time to get used to
• querying beyond one aggregate is a bit harder (you have to construct projections for each type of query you want to add to the system),
• event schema changes is much harder than in case of relational model (lack of standard schema migration tools)
• you must consider versioning handling from the beginning.

Are you interested? Check this!

CQRS by Martin Fowler
Simple CQRS by Vaughn Vernon
CQRS, Task Based UIs, Event Sourcing agh! by Greg Young
Types of CQRS by Vladimir Khorikov
When to Avoid CQRS by Udi Dahan

Event Sourcing by Martin Fowler
Event Sourcing Pattern by Microsoft
Why Use Event Sourcing by Mirosław Pragłowski
Why Use Event Sourcing by Greg Young