Besides some changes to existing building blocks, such as the event bus, which is now much more powerful, the 0.5 version also includes some new features.

Read on to find out more.

New features

• Code restructuring. The package structure has been changed to better reflect the different architectural components Axon Framework provides. It should be easier to find the one you’re looking for.
• Command Bus. The command bus is added to Axon. It provides you the ability to explicitly define commands and dispatch them to your command handlers. Furthermore, the command bus provides you the ability to process commands regardless of their type using interceptors. This is useful for, for example, logging, authorization and correlation of incoming commands.
• JPA Event Store. The easiest CQRS configuration is on using full-consistency. That means everything should run within a single transaction. Since transactions over multiple data sources involve a huge performance penalty, Axon provides a JPA Event Store. Its performance is not as good as the FileSystem version, but is does provide transaction support.
• Easy switching between full-consistency and eventual consistency. You can easily choose to process all commands and related events inside a single transaction, or to handle events asynchronously. Choosing for consistency or high-performance is just a matter of configuration. No coding required.
• Per-event listener configuration of asynchronous processing. It is now possible to decide on synchronous vs asynchronous event processing for each event handler individually, just by adding an annotation. If you configure a transaction manager for your event listeners, Axon will process the events in batches and manage the transactions around them.
• Support for rolling snapshots. All event stores will automatically pick up snapshot events. Snapshot events are an important performance booster when aggregates generate a lot of events. Instead of reading all passed events, the event store just needs to read the last snapshot event and the regular events created since the snapshot.
• Transactional Event Processing. Configuring transactions in asynchronous event processing is now a lot easier. 0.5 includes a SpringTransactionManager you can use in combination with Spring’s PlatformTransactionManager.
• Major documentation update. The documentation has been restructured to make it easier to find what you’re looking for.

Maven Central

Where the 0.4 version required configuration of a repository in your project’s pom.xml, the 0.5 version doesn’t. All required artifacts are available in the maven central repository.

Workshop and professional support

We believe that the 0.5 version of Axon Framework is a major step towards production readiness. Therefore, JTeam has decided to provide professional support for the Axon Framework and organize workshops to get you acquainted with the numerous features and choices involved with CQRS.

The first workshop is planned for Friday May 21st in our office in Amsterdam. For more information, visit http://www.jteam.nl/training/workshop/cqrs-axon-framework-training-workshop.html.

Getting started

Want to get started? Visit www.axonframework.org and download the reference guide. That should contain enough information to get you started. If you still have questions, drop me a message.

Measuring code quality is not a matter of a single metric. Instead, software quality has many aspects, some of which can be captured in metrics. Those metrics can be nicely assembled within a single application, which gives a nice overview of the state of an application: Sonar.

Sonar (version 1.12 at the time of writing) is a web application that can be installed standalone or inside an existing Java Web Application Container, such as Tomcat. The standalone version is quite easy to install. Just download, start and a few moments later it’s ready on port 9000.

You can capture metrics using maven by running mvn sonar:sonar on your project. In some cases, your pom.xml file will need an explicit reference to this plugin, since it is not a default apache-maven plugin.

<build>
  <plugins>
    <plugin>
      <groupid>org.codehaus.mojo</groupid>
      <artifactid>sonar-maven-plugin</artifactid>
    </plugin>
  </plugins>
</build>

After running mvn sonar:sonar on your project, Sonar will contain metrics information of your application. Browsing to “localhost:9000″ will show this.

Sonar has a nice overview screen that shows certain metrics (you can configure which ones you want to see) for all project. In a software development organization, this means that you have a single point where you can measure each project’s “technical health”. You could even add custom metrics, such as budget, number of people on the project, number of story points left in the backlog, etc.

Each project has a dashboard. The dashboard is an aggregation of more detailed metrics on a single page. This page gives a pretty good overview of the technical state of a project. When doing project reviews, this page is a good starting point to find items to investigate.

Almost every metric on the dashboard is clickable. When you click on a metric, you get more information about violating classes.

Configuring Metrics

Sonar uses findbugs, checkstyle and PMD to measure your code for bugs, ugly code and possible violation of code style policies. Sonar comes with a pretty good basic configuration, but the chance is big that you want to fiddle with the settings.

The easiest way is to take a basic configuration and make a copy of it. You can then change the configuration as you like. Note that you have to tell Sonar which projects should use this configuration.

Note that you must be logged in to be able to change the configuration. By default, there is one user, with username and password “admin”.

Another way to create a configuration is by specifically creating a new one. This is useful if you already have checkstyle, PMD and findbug configurations in your organization. You can then upload these files when creating your new configuration.

Sonar Database Configuration

The default configuration works nicely if you have Sonar installed locally and don’t mind to use the default Derby Database. For larger development environments, however, this default configuration is not enough.

You can configure Sonar to use a database quite easily. The “sonar.properties” file contains the basic Sonar configuration. Configuration examples are available for most databases. All you need to do is uncomment a block of configuration and make sure the URL, username and password are correct.

In maven, you have to configure the same URL’s. Personally, I find that a little awkward, since you need to share database details over multiple applications.

You can configure Sonar in your pom.xml file, which makes the configuration accessible to the projects using that pom. The following example configures a MySQL database.

<project>
  <properties>
    <sonar.jdbc.url>jdbc:mysql://my_sonar_db:3306/sonar?useUnicode=true&amp;characterEncoding=utf8</sonar.jdbc.url>
    <sonar.jdbc.driver>com.mysql.jdbc.Driver</sonar.jdbc.driver>
    <sonar.jdbc.username>sonar_user</sonar.jdbc.username>
    <sonar.jdbc.password>sonar_pass</sonar.jdbc.password>
    <sonar.host.url>http://my_sonar_url:9000</sonar.host.url>
  </properties>
</project>

Alternatively, you can pass all these parameters in the maven command line, or configure your maven settings.xml (on your build server, for example) to set these parameters.

Metrics don’t say it all

Although I believe that Sonar is a very nice tool to create code quality awareness, metrics don’t say it all. Some quality aspects simply cannot be caught in a metric. Just solving the issues that Sonar arises is not a good way to improve quality. Sonar can show problem area’s. It is up to the development team to find out where these problems come from, and solve them.

No matter how good the metrics are, each and every software project still needs a good process, a good project management, a hands-on architect and a smart development team to become a success.

Attendance is free, but registration is required. Read on for details.

About the meetup

The Domain Driven Design Netherlands User Group (DDDnl) organizes bi-monthly meetups where software developers, architects, product owners and managers join to discuss DDD related issues. During each meetup, one of our members gives a presentation. The upcoming meetup will cover the concept of CQRS.

Command-Query Responsibility Segregation (CQRS) is a technique developed by Greg Young that combines the Event Sourcing pattern with Domain-Driven Design. This provides various benefits, amongst others: a fully encapsulated domain model without the need for any ORM tools, separate query and reporting database, 100% correct historical information, and easy integration with other systems.

In this presentation we’ll cover the core concepts of Command-Query Responsibility Segregation, its impact on the application’s architecture, and the various advantages and disadvantages. Some (Java) code will be shown to highlight how such a system can be implemented.

Time and location

The meetup will be held at the JTeam Head Office at Frederiksplein 1, Amsterdam. Check http://www.jteam.nl/contact for directions.

You are welcome from 17:30. We will start with some food, which is expected at 18:00. The presentation starts after dinner.

Registration

You can confirm your presence by sending an email to info@dddnl.org. Please include your name and a phone number where we can reach you, just in case.

If you want to keep updated about other DDDnl meetings, make sure you register on our website: www.dddnl.org as well.

Read on to find out more.

About the Axon Framework

Axon Framework helps build scalable, extensible and maintainable applications by supporting developers apply the Command Query Responsibility Segregation (CQRS) architectural pattern. It does so by providing implementations, sometimes complete, sometimes abstract, of the most important building blocks, such as aggregates, repositories and event the bus – the dispatching mechanism for events. Furthermore, Axon provides annotation support, which allows you to build aggregates and event listeners without tying your code to Axon specific logic. This allows you to focus on your business logic, instead of the plumbing, and helps you makes your code easier to test in isolation.

Features available in 0.4

We have made quite a few additions and changes since the 0.3 release. Unfortunately, this also means I had to make some minor API changes. There is now a bigger choice of abstract classes for most of the building blocks. This allows you to choose whether you want to use the logic provided by Axon, or prefer to develop more of your own. Whatever your wish is, there should be a class to help you on your way.

The biggest change since 0.3 is the addition of an Asynchronous Event Bus that supports transactional processing of events. Since it is asynchronous, it means that your command processing doesn’t have to wait for events to be processed. However, this Event Buss can also guarantee the order in which events are delivered to the event handlers. You can choose full concurrent processing (without any guarantees about ordering), full sequential processing (FiFo guarantee) or anything in between. You could, for example, guarantee the sequential processing of Events that come from the same aggregate.

The Event Bus also provides transaction support. That means you can tell the event bus to retry an event after a certain amount of time when processing fails, for example due to an error in the underlying mechanism, such as a database. For performance reasons, you can configure the event handler to process multiple events within a single transaction. Database updates are a lot faster if you process several events within a single transaction before committing it. Within your event handler, you can configure transactions.

Besides Domain Events, Axon now also allows you to explicitly define two other types of events: Application Events and System Events. Application Event can notify your event listeners that something interesting happened in your application. System Events are a special type of application events and can be used to indicate that a part of your application has changed state, for example when the database is no longer available. These System Events could provide interesting operational information about your application.

Another major change since 0.3 is the documentation. We have brought the documentation up to par with the code-base. All features are now extensively described in the reference guide. You can download the manual from the Axon Framework website: www.axonframework.org.

Last, but definitely not least, we have created a demo application that you can easily download and deploy to have a look at how you can use events to your advantage. This demo application is built using a Flex Client and an Axon-based server side component. The Flex Client registers itself as an event listener and will automatically process incoming events, at real time! That means that you can have 2 windows open, change something in one window, and see the change come in immediately on the other window, too. Check our Sample Installation page for more information.

What to expect in upcoming releases

The major components of the CQRS pattern are already available in the 0.4 release of Axon Framework. However, there is still a number of features we want to include in the 1.0 release. Although one of the advantages of CQRS is that it makes an application scalable and extensible, the 1.0 release of Axon Framework focuses on what happens inside a single JVM. Streaming events between JVM’s is not made impossible though, the Spring Integration connectors can be used to publish events to JMS, MQ, mail, HTTP and any other protocol supported by Spring Integration.

Some features to expect before the 1.0 release are rolling snapshots. When aggregates live for a long time, the number of events to read in each time the aggregate is loaded could easily run in the thousands. Reading in a thousand events takes a long time. Instead of reading in a thousand events, you can summarize all these events into a single snapshot event. Axon Framework will provide the plumbing necessary for you to build snapshot events.

Another feature that we’ll be spending some attention on is failure recovery. Unfortunately, applications sometimes behave unexpectedly and sometimes stop entirely. The reason of a crash won’t be the Axon Framework itself, of course. But since we have a reliable source of events in the events store, we can use this to recover from failure. Axon Framework will provide some building blocks that allow you to republish events to event handlers that did not get the chance to process them. This same mechanism will allow you to restore your application state from backups.

The last feature currently on the roadmap is JMX support. Since Axon Framework deals with the dispatching and invocation of event handlers, it has interesting information about an application’s performance and technical state. We are planning to expose this information using JMX MBeans.

If you have any other ideas, you can submit a feature request on www.axonframework.org/issues.

Getting started with Axon

If you’re eager to get started using Axon Framework for your own application, check out the Axon Framework website. There, you can download the binaries, sources documentation and a sample application. If you use Maven, the reference manual will explain how you can configure the dependencies on the Axon binaries.

If the information on the website is not enough to get you started, don’t hesitate to send me a message or post a comment at the bottom of this blog entry.

A special thanks to…

Before I wrap up, I would like to thank Jettro Coenradie for his help on the sample project and the reference documentation. I think the sample turned out into a pretty cool demo environment for some of the advantages that Axon can provide.

Also many thanks to JTeam for the time they allowed me to spend on the project. Without it, I am sure I couldn’t have delivered this release. Thanks!

In this post, I describe some CQRS event basics and design considerations that help keep your application extensible.

The role of the domain event

In CQRS, domain events are the source of all application state changes. The diagram below shows a high-level overview of a CQRS architecture. The separation of the components that deal with commands and those that deal with queries is clearly visible.

When a command is executed, it will (most likely) change state of one or more aggregates (see DDD definition of aggregate here). As a result of these state changes, one or more events are dispatched.

These events are picked up by Event Handling Components that do whatever they were built to do. Common Event Handling components are those that update database tables in the query database, or those that send an email notification to a user. You can also use Event Handlers to execute commands on related aggregates.

The command handling component is not aware of the listeners that will act on the generated events. This is an important feature to make an application extensible. The downside is that you cannot foresee how events will be used and what information these event handlers will need.

Information contained in an event

Since events are broadcasted throughout the entire application, events should contain the information needed by the variety of possible event listeners that exist or might exist in the future. It is impossible to exactly predict what information is needed by event handling components. However, there are a few guidelines that will cover most of the cases.

Recently, someone asked me for advice on event design. He asked me if it was a good idea to use an “CustomerDetailsModifiedEvent”, which would contain the full customer details of the customer, after the change was applied. Although this sounds safe to do, it will get you in serious trouble at a later stage. I’ll go into that a little later.

Domain events should be as specific as possible. If only the address changed on a Customer, consider using an AddressChangedEvent. That event would then contain the new address. This allows components acting on these events to listen to more specific types of events. If you want to send a (paper) letter to someone if his address changed, the “CustomerDetailsModifiedEvent” won’t get you very far, as you don’t have a way to find out what exactly changed.

In some cases, it might be important to know the state as it was before the command was executed. For example, if it is vital for your application to know whether the person moved to another country (because the letter template is different in these cases, for example), then you should also provide the old address in the event. Since the event is generated by the component that also makes the actual change, this shouldn’t be too hard to implement.

public class AddressChangedEvent extends DomainEvent {

    private final Address newAddress;
    private final Address oldAddress;

    public AddressChangedEvent(Address oldAddress, Address newAddress) {
        this.oldAddress = oldAddress;
        this.newAddress = newAddress;
    }

    public Address getNewAddress() {
        return newAddress;
    }

    public Address getOldAddress() {
        return oldAddress;
    }
}

The code sample above shows the basic information contained in an AddressChangedEvent in case it is important to know the address of a customer before and after the modification. In this sample, the DomainEvent will contain information about the customer on which the address change was applied. Typically, this information is limited to the unique identifier of the aggregate.

Include the intent of the change

Let’s assume that our business case requires us to send a letter to each customer that changes address. This letter serves as some sort of validation that the address exists and gives the customer feedback that the address change was successfully handled.

In CQRS, we would just create a Event Handling Component that listens to events of type AddressChangedEvent and automatically send a letter to this customer. The information needed for the letter, such as customer name and gender are queried in the query database, since they are not contained in the event itself.

Consider an event with the following information:

AddressChangedEvent {

- aggregateId 123

- newAddress (“someStreat 2”, "13432”, “SomeCIty”) <- notice the typo in the street name

- oldAdress (…)

As a result of this event, a letter is sent. It is very unlikely that the typo in the street name will cause the letter not to arrive at the destination.

A little later, a sales representative notices the typo in the street name and want to change the address. This change will cause another letter to be sent, most likely to the same address. That’s not really the signal the business likes to send out.

In this scenario, we have captured two different intents of address changes: one is to report that a customer moved to another address, the other is to make a correction in an existing address. The way we deal with these different scenario’s is different.

Some components, like a database updaters, typically don’t care about intent. They will change the address to the new one, regardless of the reason. Furthermore, the information needed by listeners in both scenario’s is identical.

When designing events, consider making an abstract event type that defines “what” has changed. For example, an abstract AddressChangedEvent. We make it abstract, because we never want an instance of this type. This abstract then has a subclass for each intent of the change, representing the “why” of the state change. For example, AddressCorrectionEvent and CustomerMovedEvent. Since they both extend the abstract AddressChangedEvent, the both contain information about the old and new address.

Our event handling components now have a choice. If they don’t care about intent, such as the database updater, they will handle the abstract AddressChangedEvent. If they do, like our letter sender, they can listen to intent specific subclasses. In this case, the letter sender would only act on events of type CustomerMovedEvent.

Event immutability

Domain events represent a notification of a state change of an aggregate. It is a representation of something that happened. Whatever you do, what happened has happened. That said, it doesn’t make much sense to change any information in the event while it is being processed.

In CQRS, events are ubiquitous. This means that any component anywhere in the application could listen to any event. Once properties on these events start to change, it is very hard (if not impossible) to predict which component saw which information on the event.

Therefore, make sure all information contained in the event is immutable. At least from the time the event is dispatched. In java, and likely also in other programming languages, making fields immutable provide valuable multi-threading guarantees.

Event design in the Axon Framework

Recently, we released version 0.3 of the Axon Framework, an open source framework that helps you with the implementation of applications with a CQRS architecture. Axon Framework provides base implementations for most of the components that make up a CQRS architecture, such as some base classes for events.

All domain events should extend the abstract DomainEvent class. Axon will then ensure that the correct aggregate identifier is registered on the event and will also set a sequence number on the event that allows you to see in which order events were generated for an aggregate. However, the rules and guidelines mentioned above still apply to all domain events.

Starting at version 0.4 (planned for release mid February 2010), Axon Framework also provides support for Application Events and System Events. The former are events that do not represent a state change in an aggregate, but might be of importance to event handling components. The latter provide information about state changes of application components. They could, for example, report that a database is down or a mail server cannot be reached.

For more information about the Axon Framework, visit the project website: www.axonframework.org.

Command Query Responsibility Segregation (CQRS) is an architectural style that makes a clear distinction between commands that change the application state and queries that expose the application state.

Background

My interest in CQRS was triggered when I saw Greg Young explain “State Transitions in Domain-Driven Design” on InfoQ. In this interview, Greg explains how he sees that application would benefit from using separate models for state validation and transition on one side and maintaining a view on the current state on the other.

One of the problems that Greg describes is the fact that a single model is often used for different purposes. It is nicely illustrated by the SQL query below. It shows how the model chosen in the application is not suited for the purpose of providing certain information (messages between users, in this case).

queryBuilder.append(
  "m.*, " +
  "m.origin_participant_id as message_origin_participant_id, " +
  "po.first_name as message_origin_participant_first_name, " +
  "po.avatar as message_origin_participant_avatar, " +
  "po_ua.username as message_origin_participant_username, " +
  "CASE WHEN !isnull(po.fieldworker_project_id) THEN 'fieldworker' WHEN !isnull(po_ap.project_id) THEN 'fundraiser' ELSE 'player' END AS message_origin_participant_type, " +
  "po.city as message_origin_participant_city, " +
  "c.name as message_origin_participant_country, " +
  "pd.first_name as message_destination_participant_first_name, " +
  "pd.avatar as message_destination_participant_avatar, " +
  "pd_ua.username as message_destination_participant_username, " +
  "CASE WHEN !isnull(pd.fieldworker_project_id) THEN 'fieldworker' WHEN !isnull(pd_ap.project_id) THEN 'fundraiser' ELSE 'player' END AS message_destination_participant_type, " +
  "m.destination_participant_id as message_destination_participant_id " +
  "from internal_message m  " +
  "left join player po on m.origin_participant_id = po.id " +
  "left join (select player_id, project_id from ambassador_project where enabled = true group by player_id) po_ap on po_ap.player_id =  po.id " +
  "left join user_account po_ua on po.user_account_id = po_ua.id " +
  "left join country c on po.country_id = c.id " +
  "left join player pd on m.destination_participant_id = pd.id " +
  "left join (select player_id, project_id from ambassador_project where enabled = true group by player_id) pd_ap on pd_ap.player_id =  pd.id " +
  "inner join user_account pd_ua on pd.user_account_id = pd_ua.id " +
  "where m.destination_participant_id = ? "
);

Wouldn’t it be a lot nice to have a query such as the one below instead?

</pre>
SELECT * FROM messages WHERE receiving_participant = ?
<pre>

Achieving such queries is very easy, but doing so without making your model impossible to use for maintaining state integrity and guarding invariants is a lot harder. Well, unless you apply CQRS.

Architecture

The diagram below shows an overview of a typical CQRS architecture.

When a command comes in, it will load an aggregate from the repository and execute certain operations on it. As a result of these operations, the aggregate produces events, which are picked up for storage by the repository and for dispatching by the event bus. The event bus will dispatch each event to all (interested) event handlers. Some of these event handlers will perform actions on other (related) aggregates, some others will update the database tables they manage.

Having handlers update the data in the database means that your tables do not have to be normalized anymore. Instead, CQRS allows you to optimize your tables for the way you want to query them. This makes the data layer processing your queries really simple, and maintainable.

Furthermore, since all state changes are initiated by events, these events become a reliable source for an audit trail. By storing these events, you have an audit trail that you can use to replay the entire application history. Instead of just seeing “current application state” only, you can see all the changes that have led to the current state, providing valuable information trying to find bugs or deal with customer complaints.

Benefits

Matches the natural language used

At first glance, such an architecture might look very complex and over-engineered. However, after taking the first implementation steps, it felt pretty natural. In fact, when you explain the behavior of an application, you use a style that fits CQRS quite nicely: “when an order is approved, we send a confirmation email to the customer”.

Extensibility

Imagine an order management application. With CQRS, you could have an “OrderApproved” event, which is caught by a database updating event handler as well as one that sends an email to the customer with order information. If you later on decide to store the order information in an accounting tool as well, all it takes is adding an event handler that does the accounting integration. And since you can reload historic events as well, you can even reconstruct historic information to put in the accounting table.

Reliable audit trail

As I said above [see section Architecture], the model that is used to process command will generate events. These events are the sole source of state changes for the domain classes. If you want to load an aggregate from persistent storage, it will actually load all the past events of that aggregate. This process is called Event Sourcing. When events become too numerous, you can combine a number of historic events into a single snapshot event. The events that have been combined can then be archived for auditing purposes.

Event Sourcing turns your event storage into an extremely reliable audit trail. The events do no only show what happened and when, but they will also reveal the intention a user had. Not only is it an audit trail that will never move out-of-sync with your application, you can use the audit trail to actually replay all actions in the application. Events are not just a notification of state change, they are also the source of state change.

Transparent distributed processing

When using events as the trigger for state changes, you can easily distribute your application over multiple processing units (servers, JVM’s, whatever). Events can easily be serialized and sent from one server to another over JMS, via the file system or even email. The Spring Integration framework –a messaging framework – fits nicely with the event processing concept.

You could even let certain types of aggregates live on dedicated machines. This allows you to choose different SLA’s for different parts of you application, such as giving order creation a higher priority than sales reporting.

Performance and Scalability

Since event handling is done asynchronously, a user does not have to wait for all changes to be applied. Especially when integrating with external systems, this can improve liveness of an application significantly.

Another aspect of CQRS is that it heavily builds upon BASE (Basic Availability, Soft-state, Eventual consistency) transactions. Although the “eventual” part scares a lot of customers and developers, the price of distributed ACID transactions is one that customer typically resent. In most cases “eventual” is just a matter of several milliseconds. But you’re free to make that seconds, minutes or even hours if your SLA permits it (think of management reports that are only read once a week or month).

Asynchronous analysis

When you build a CQRS style application, all activity in your application is processed using events. This makes it easy to catch these events and monitor them for inconsistent behavior. Event analysis tools like Esper allow you to monitor event streams for patterns that indicate possible fraud.

For example, when you detect that a sudden large number of users has an increased amount of failed login attempts, you could decide to block these accounts for a small period of time. In one of our projects, we use RSA Adaptive Authentication to increase the level of authentication required when someone tries to login on an account that might be the subject of dictionary attacks.

cqrs4j – an open source CQRS Framework

A few days ago, I published the source code of a CQRS framework – cqrs4j – on Google code. This framework aims at reducing the amount of “plumbing” and boilerplate code to a minimum. It also provides integration support for Spring Integration and transactional event processing.

You can find project information, source code and documentation at http://code.google.com/p/cqrs4j/.

[Update: cqrs4j has been renamed to Axon Framework. You can find more information on www.axonframework.org].

Upcoming presentations and events

JTeam organizes monthly Tech Meetings, where interesting technologies are discussed. The January 2010 edition will contain a presentation about CQRS. Registration is required, but free of charge. The tech meeting is held on January 7th in our office in Amsterdam [directions] and starts at 16:00. It includes dinner.  Send us an email if you like to attend.

The DDDnl user group organizes a presentation and discussion session on March 9th, starting at 18:00. It is also held at the JTeam office in Amsterdam [directions]. Attendance requires registration with the DDDnl user group, which is completely free of charge. Visit dddnl.org for more information.

The increasing demands from the business have a direct influence on the applications complexity. Each time, especially when joining external projects, I am amazed by the way complexity is dealt with in most applications. “Building software isn’t rocket science”, but sometimes we do seem to make it just as complex.

In this article, I provide some insight in different forms of complexity, and how software development teams can design their application in order to deal with the ever growing demands from the business.

At the dawn of each software development project, the sky is blue and the grass is green. That’s the type of project most developers like to do. However, as time progresses and features are added and changed (they are hardly ever removed, for some reason), the developers become entangled by their own code. Concepts –in which lots of time was invested to get them defined- become muddled and intentions of methods and services become vague. Eventually, the application will evolve into a Big Ball of Mud.

Causes of complexity

The degree in which the project team deal with complexity significantly influences the speed at which a codebase evolves into a ball of mud. There are numerous sources for complexity, that can be separated into three categories.

It is important to distinguish the different causes of complexity

Project Environment

The first source is the project environment itself. This mainly includes the availability of resources (e.g. people, money, knowledge) and the management style of the project manager and customer. Some project managers truly believe that when a developers estimates a piece of work at 1 week, that the sentence “I’ll give you 3 days” will actually do good to the project on the long term. Although an interesting topic, it is not the type of complexity I want to deal with in this post.

Technical Complexity

Another aspect influencing complexity is the technical environment of the project. What are the frameworks and platforms used in the development and deployment of the software. As with any technology, it is important to know the powers and limitations of each framework. If a framework requires expert knowledge to interact with it, then try to abstract that knowledge away into separate modules.

Every development team contains a couple of developers that is very well capable of solving these technical solutions. However, once solved and abstracted away, this technical complexity is mitigated and hardly a risk to the project. Of course there are extremes, where project heavily rely on bleeding edge technology. On those projects, you’re likely to keep a high level of technical complexity in the project. But no matter what, it should be clearly separated from the other type, discussed in the next paragraph.

Business Complexity

The third aspect is the one I will be focusing on in this post: the business domain. It is the sum of the knowledge, rules and activities of your customer that makes them survive. Solving the problems in this domain is often highly complex, but not unsolvable.

The majority of the business complexity is composed of the business rules. In contrast to the technological environment, the business environment is continuously in motion. And for some reason, this always means the addition of new rules and the changing of old ones, hardly ever removing them. The result of this process is an ever-growing application with increasing complexity.

The next sections cover a few guidelines that help you manage complexity and maintain it in acceptable ranges. In the end, people have to be able to understand the code.

Application Layering

The use of layers in software engineering is a concept I encounter in almost any project I join. The boundaries and definitions of these layers, though, are often fuzzy or undefined. The result is that, as the project progresses, different types of concerns leak into layers of the application where those concerns do not belong.

When correctly defined, layers can help developers make a clear separation between different types of complexity. Solve technical issues in another layer (e.g. the infrastructure layer) than the business complexity (e.g. application or service layer). Define clear boundaries between these layers and make sure each developer understands and respects them.

Use a model

The real world as a whole is useless when it comes to solving problems in software engineering. If you’re in the theatre ticket selling business, you don’t care that seats come in different styles, different colors and that some types of seats have legs. In other words, you don’t care what a seat is in the real world. I don’t expect to be telling new stuff here, but for some reason, I see it go wrong many times.

A common mistake I encounter in programs is that there is no real model at all. Well, there is a modules in the project called model, and it contains classes with getters and setters. Technically, it is a model, but is has been simplified to the extent that it is pretty much useless in solving the business problems. These problems are then solved in the service layer. But with the additions of new rules, this layer grows out of proportions with in some cases hundreds of methods, some of which are quite-the-same-but-not-really.

Another pitfall is the desire to make their model similar to the real world. The real world is not relevant to the application. An application needs to solve problems. The real world might just not be the ideal place to do that. Instead, the business itself should drive the modeling choices. And if two parts of a business (e.g. divisions) have different views on the same actual thing, consider creating a different model for each one of them.

A third consideration often forgotten in the development process is that not every part of the application should have to use the same model. A model is used to solve (complex) problems. Each part of an application solves different problems in the domain. A logical conclusion is that these parts do not necessarily ask for the same model. A good indicator for the possibility of splitting the model is that a single word is used in different contexts, and has a different meaning in each context. The word “Flight” will mean different things to the ground crew and the cabin crew. They will all have different intents when using the concept of a Flight. Combining all these intents into a single (shared) domain concept will most likely clutter your domain and make it harder to understand.

Watch your language

Ideally, the names of the concepts in the model reflect the actual words used in the business. If a new concept is added to the model, that name should be introduced in the business as well. Eric Evans calls this the “Ubiquitous Language”, the language used by all people and software involved in the domain the software runs in.

Why is this important? It helps new developers on a project. If they are familiar with the terminology, they will most likely understand the application right away. If they don’t, they will pick up the jargon faster and be able to talk to domain experts earlier.

It will also help in finding the locations where changes need to be made. If the model is correct (i.e. fits the business nicely), the language the domain expert uses will pinpoint you to the location where changes need to be made.

Modularity

Where simple applications will contain just a few concepts (where each concept is implemented as one or few classes), large applications may contain literally hundreds or thousands of concepts. A human being is simply not able to maintain a good overview of that amount of concepts.

Therefore, separate related concepts into modules. Within a module, these concepts may have tight relationships. In other words, the classes representing these concepts will have dependencies towards each other. Across modules, dependencies should be minimized. I often encounter dependencies between modules that don’t really need to be there. Removing them decreases application complexity and increases its maintainability. Especially “Entity”  to “Entity” relationships should be avoided as much as possible.

For example, let’s say we have an Order that references a Product, both in different modules as different departments manage the lifecycle of these objects. What happens if the Product price changes? All orders containing that product (and most of them have probably been handled already) will change as well. The solutions is simple, add a property to the order that states the price at which the product was sold. What if other characteristics change? The name for example? Remember, we don’t care about the real world , where products don’t just change name, we care about the world the way the business wants it. Instead of referencing the Product entity as a whole, perhaps it is sufficient to copy the relevant data from the Product into the Order at the time the Order is created. Or perhaps the entity reference is only maintained for as long as the Order is being processed. This allows you to delete Product instances without losing data in the Orders module.

The choices you can make in keeping nicely separated modules heavily depend on the way the model has been built. The better the model is, the more flexibility you have in making choices that help your application.

Conclusion

In this post, I have briefly touched a few types of complexity encountered in software projects. Don’t deal with multiple types of complexity in one location (i.e. layer) in your application. Make a clear separation between these to prevent technical choices from interfering with your business logic and vice versa.

When it comes to business logic, try to make a model that fits the way the business likes to work. Don’t try to make it look like the real world. That world has proven to be less than ideal when it comes to solving complex problems.

Use the Ubiquitous Language to define the terms and concepts in your model. This helps new developers to learn the language faster and get a better understanding of the application.

Finally, when the number of concepts expands beyond human capability of maintaining an overview of them, separate concepts into modules. Choose your modularity carefully. Try to keep dependencies tight inside the module, while limiting the dependencies between modules.

In this post, I will elaborate on how Domain Driven Design has helped us build a clean and maintainable application, mainly focusing on some technical and implementation choices that we have made.

DDD Background

Domain Driven Design is really about the way software is developed. Hence, it will say more about the way to get to an implementation than about the actual implementation itself. However, if you do it correctly, you will find some of the DDD principles back in your implementation.

The most important concept in DDD is the “Ubiquitous Language”: the language that is used within the domain by all people involved, such as domain expert (our customer), developers and stakeholders. Since this language is used by everybody, it is best to reflect that language directly –hence without translation– in your code. Since the language will contain both nouns and verbs, this means that you will end up with a Rich Domain Model.

Another really important principle of DDD is “make it explicit”. I can’t even recall how many times Eric Evans said that during the four day training he gave two weeks ago. This means that, for example, every time you come across some relationship between two values, you should assemble these two values and name the combination. This way, you make the relationship between these values explicit and named. This name should either come from the Ubiquitous Language or should be introduced as part of it. Yes, that means no single developer can make up a name for something without consulting the domain expert first.

How this reflects in the implementation

Most people I talk to are curious about how this reflects in the implementation. Some people even seem to reject DDD because it is not really all that different in the implementation. Of course it isn’t, at least, not on first sight.

In the OSIX domain, people can use wireless access points in Libraries to gain access to the Internet using their laptop. To do this, they need to register themselves in a central application, owned and maintained by OSIX. This application serves as a web shop where users can buy credits, it provides administrators to block or enable users and keeps a log of actual time spent online.

This means our domain model implementation contains classes like Library, User and InternetSession. This last one is DDD-implementation wise quite interesting.

We needed to log the time a user logged on and when he logged off again. Hence, our first name for this class was “LogEntry”. Technically, we are just talking about a giant Log with a lot of entries. However, DDD requires that everything is to be made explicit and named according to the Ubiquitous Language. So, we consulted the domain expert. He said he calls it Internet Sessions. A user starts an internet session, and an internet session is ended when a user logs of.

Does it change the implementation of the class much? Not really, except for some different method names, perhaps, nothing is really different. In the end, we’re still programming Java, and the same things need to be done. And if you’re used to Anemic Domain Models, you’ll probably have to get used to the fact that invariants are guarded inside the domain object itself, instead of a service.

So, why is this naming important? First of all, it allows your developers and domain expert to talk the same language, without the need of continuous translation. Secondly, new developers joining the project will easily pick up the language and recognize the concepts in the code. This really showed when we needed to change some developers. The new developer was quickly able to see which functionality had been implemented and which still needed some work. Finally, our client, who has a fair understanding of the Java language, was able to understand our implementation and give us direct feedback.

How about persistence?

Somehow, many people see Domain Driven Design and Rich Domain Model as the same thing. It’s simply not true. A Rich Domain Model is the logical consequence of a DDD project, since everything is made explicit. And making things explicit means providing names, properties and behavior to concepts.

Another assumption often made is that Rich Domain Model means “put all logic in the domain”. Which is also not true. It’s not called a model for nothing. Using a model means you consciously leave things out of your domain, because they are either not relevant or not practical in your implementation.

In the OSIX project, we assigned the responsibility of retrieving and persisting objects to the application layer. This means the application services call on the Repository implementations to retrieve or save objects.

Furthermore, we decided to keep all modifications to the state of Entities inside the scope of the Application Layer. In the implementation, this meant two things. First of all, the Transactional boundary is located on the service layer, meaning that entities leaving the Application Service are detached from the persistence context, meaning that changes will not be seen by the persistence manager and are ignored. Secondly, the service layer will never accept an entity as a parameter. If an entity is needed, the application service will retrieve it based on its identity, which is passed as a parameter instead.

Some of the Application Service interfaces methods look as follows:

InternetSession startSession(final LocalDateTime startTime,
                             final UserName userName,
                             final BranchCode branchCode,
                             final IpAddress ipAddress);

void endSession(final long id,
                final LocalDateTime endTime,
                final BranchCode branchCode);

All method parameters are value objects, and thus immutable, thread safe and the whole shebang. The startSession method returns en entity: InternetSession. If you wish to change the state of that object in a way that is useful to the application, you need to call another method (e.g. endSession) using the identity of the session, a long in this case. (Yes, in the case of “make it explicit”, SessionId would be a better name for it.)

The implementation of this application service is no rocket science either. It contains only little logic, since it delegates most of it to the domain objects themselves.

@Transactional
public void endSession(final long id,
                       final LocalDateTime endTime,
                       final BranchCode branchCode)
               throws NoSuchSessionException {

    InternetSession session = internetSessionRepository.loadSession(id);
    if (!session.getBranchCode().equals(branchCode)) {
        throw new NoSuchSessionException("This session belongs to another Branch");
    }

    if (session.isOpen()) {
        session.closeSession(endTime);
        internetSessionRepository.saveSession(session);
    }
}

Line 7 shows the actual InternetSession entity being retrieved from the repository. In Hibernate/JPA terms this means the entity is attached to the PersistenceContext, and all changes are observed by the persistence manager.

Line 8-10 show some very simple validation logic to prevent sessions from another Branch being closed accidentally.

Line 12-14 do the calls to the actual business logic involving the closing of a session.

You could state that the Application Service is responsible of the orchestration of business logic execution. It doesn’t say anything about what the business logic actually means (e.g. what does it it mean that a session is open?).

Conclusion

In the end, DDD doesn’t really drastically change the way most applications are built. The most important change is the actual naming of objects and methods and the location of the business logic. However, this seemingly subtle change can have a big effect on the maintainability of your application.

Domain Driven Design is mainly about making things explicit, and doing so in the Ubiquitous Language.

In this post, I will focus on some technical and organizational topics that were addressed during the project.

Search and data import

The initial version of welke.nl contained two major parts: the product database and articles. A Content Management System (CMS) has been implemented to help the authors and publishers manage the content on the website.

The product database, however, required more specific features than the CMS could offer out-of-the-box. To enhance the customer experience, welke.nl visitors have to be able to filter through the available products to limit the selection to only those products that are of interest to them. To enable this, we used something called facetted navigation (or attribute-based search). Facetted navigation provides a very intuitive view on the current data set by showing the values of several, well-picked attributes of the search results including the number of products that match for that attribute value. By choosing one of those facets, the search result are then further limited to only results that match that value. This allows users to easily drill down the entire data set and has the added value that users will never get to an empty search results set. Next to normal values (e.g. choosing a specific brand) some facets allow you to choose a range (e.g. price) or multiple selections (e.g. color).  You can see a working example on the website.

Another major difference between the product database and the articles section is the way data is inserted. MediaMij maintains close contact with their main suppliers in the Netherlands and helps them write interesting articles about new products. The suppliers provide the data to MediaMij in files containing the details of all their products. These files are imported into the CMS using a simple tool and subsequently synchronized with the search engine to allow users to access the information.

Empowerment

Welke.nl is divided into several domains, based on the types of products that are offered. The bathroom domain was chosen as the first domain to be developed. In addition to the JTeam project staff, two MediaMij developers were included in the development of this first domain. As part of the project, so during the development, we coached them and trained them in the use of the frameworks and technologies that we used on the project. The idea behind this empowerment is that MediaMij could continue development of other domains independent of JTeam.

Only several weeks after the successful launch of the bathroom domain, the kitchen domain was developed and launched with only limited support from JTeam. The subsequent development and launch of the fireplace domain was done by MediaMij without any support from JTeam at all. This proved to the customer that our empowerment approach had indeed worked. In fact, JTeam is not involved at all in the development of the remaining domains, allowing us to focus on more advanced features the client would like to see implemented (for all domains).

The result, a happy customer, who is able to quickly move into new domains, without any dependency on JTeam, so no vendor lock-in. And in turn, JTeam can help the customer on the more technically challenging area’s.

Going further…

Development continues on the JTeam side, implementing some additional requirements. Currently, we are in the process of implementing dealer locator functionality, basically a location-based search. This allows users to search for dealers of products that are nearest to them, based on a location they provide. Luckily, JTeam previously implemented similar functionality for iLocal. Of course this location-based search functionality will be fully integrated with the existing product database and CMS.

Conclusion

We can conclude that the empowerment approach has helped the customer in becoming independent in the maintenance of their new website. They no longer rely on the availability of JTeam personnel to make changes or even expand to new domains. Moreover, it also allowed us to focus on the innovative areas and help improve their customer experience even more.

In some discussions I recently had with developers as well as other architects, I noticed that Domain Driven Design and Rich Domain Model  are interpreted as “put everything in the domain model”. This means (according to some of them) that each object should have a “save” method to persist its state, external systems calls are initiated by the domain objects, and so on and so forth. I have to agree with them that such an approach would not be practical at all. All it would lead to is bloated domain objects of which the real purpose is hidden in hundreds of lines of boilerplate code. However, I don’t believe that Domain Driven Design is about any of this at all. And looking at Eric Evans’ (the writer of the book Domain Driven Design) sample application, my thoughts are confirmed.

Domain Driven Design is an approach that describes the approach to designing the business layer. It doesn’t say much about the other layers of an application. The main advice is to base the domain model on the language your customer uses: the ubiquitous language. This means that you should not have a method “setAddress” on a Customer instance if your customer talks about “register a relocation”. In fact, the last application I developed hardly contained any setter on any domain instance at all.

You might wonder what happened to the other layers if DDD is really only about the business layers. Well, the service layer is still what it was, although it tends to contain less code. All that is needed in the service layer is to retrieve the correct domain instance(s) from the repository, call the appropriate methods on them and persist their new state by placing them back in the repository. With this approach, the logic of changing the state is located where the state is kept, in the domain object. The application logic, such as persistence, transactions, calling external application, raising application events, is placed in the service layer.

During one of my discussions, somebody pointed out that according to the Domain Driven Design book one should only create a service if the logic cannot be placed in any of the domain objects. Well, the book does say that, but the service the book talks about is not the type of service that you find in the service layer. The book talks about Domain Services. Whether you decide to use one of these services is completely independent of the use of services in the service layer, which are also referred to as application services.

I can conclude that applications where domain driven design has been applied are not really that different from other applications, with the exception of the domain model itself. However, this richer model tends to make applications more maintainable and easier to understand as the model exposes the language that the customer actually uses. So, in the end, Domain Driven Design does not have to be as scary as many people think. It is just, as one of my colleagues pointed out, a set of best practices for Object Oriented design.