Microservices. The good, the bad and the ugly – 1° Part

Back in 1988, when I was first employed by a company for writing software, the world was fairly simple. The development environment we had was character-based, the database was integrated and traversed with cursors, and we built a whole new administrative system covering everything but the kitchen sink. It took us five years to complete the project, basically because the client kept changing his mind every now and then, and because every change in the code could break code elsewhere in the system. Unit testing hadn’t been invented yet, and testing was done by the end users. In production.

So far for monoliths. Then in 1994 I joined a company that build desktop applications – remember the world wide web was only a couple of years old, and web applications hadn’t been invented yet. We used a great tool called PowerBuilder and now we had two components to worry about; the application on the desktop and the database on the server. The applications we build usually served departments or sometimes even a whole company. Not highly complicated, but not highly scalable either. Ah well, we had fun as long as the client-server paradigm lasted.

Component based development

The world got more complex in the mid-nineties. Companies wanted web applications, basically running on intranets to get rid of desktop deployments. And applications needed to serve multiple departments, and sometimes even break company borders. A new paradigm set in: component based development, also known as CBD. The paradigm promised us re-use, scalability, flexibility and a way to harvest existing code (usually written in COBOL). We started breaking up our systems into big functional chunks, and tried hard to have those components communicate to each other. Java was invented and everybody now wanted to write code in Java (apparently some people still to this nowadays). Components where running in impossible technologies such as application servers and CORBA (to impress your co-workers, look that one up on Wikipedia). The good old days of object request brokers!

Scary stuff, an object request broker architecture

At that time I was working for a large international bank trying to set up an methodology for doing component based development. Even with a heavily armed team of Anderson consultants it took us three years to write the damn thing. In the end both the paradigm and the technology where too complex to write decent and well-performing software. It just never took off.

Service oriented architecture

At that point in time, the early years of this century, I thought we got rid of distributed software development, and happily started building web applications. I guess everyone bravely ignored Martin Fowler’s first law of distributed objects: do not distribute your objects. Gradually we moved into the next paradigm of distributed computing, re-packaging the promises of component based development into a renewed set of technologies. We now started doing business process modeling (BPM), and implemented these processes in enterprise services buses (ESB’s), with components delivering services. We were in the age of service oriented architecture, better known as SOA. 

Coming from CBD, SOA seemed easier. As long as your components – the producers – were hooked into the enterprise service bus, we figured out how to build-up scalable and flexible systems. We now had much smaller components that we could actually extract from our existing systems (now not only written in COBOL, but also in PowerBuilder, .NET and Java). The mandatory design patterns books where published and the world was ready to go. This time we would crack it!

An enterprise service bus

I found myself working for an international transport company, and we happily build around SAP middleware – delivering both ESB and BPM tooling. We now not only needed regular Java and .NET developers, but we employed middleware developers and SAP consultants as well. And although agile was introduced to speed up development (I know, this is not the right argument), projects still suffered from sluggishness and moreover, when we got pieces of the puzzle in place, we started to realize that integration testing and deployment of new releases got more complicated by the day.

At last: microservices!

I do hope you forgive me this long and winding introduction to the actual topic: microservices. You might think: why do we need yet another article on microservices? Isn’t there enough literature already on the topic. Well yes there is. But if you look closely to the flood of articles that you find on the internet, most of them only describe the benefits and possibilities of microservices (sing hallelujah), some of them take a look at the few famous path finding examples (Netflix, Amazon, and Netflix, and Amazon, and Netflix…). Only a few articles actually dig a bit deeper, and they usually consist of summing up the technologies people seem to be using when implementing microservices. It’s still early in the game.

That’s were a little historical perspective won’t hurt. I find it interesting to witness that the benefits and possibilities of the predecessors of microservices are still with us. Microservices seem to promise scalable and flexible systems, based on small components that can easily be deployed independently, and thus promote use of the best choice in technologies per component. Basically the same promises we fell for with CBD and SOA in the past. Nothing new here, but that doesn’t mean that microservices aren’t worthwhile investigating.

Is it different this time around?

So why is it different this time? What will make microservices the winning paradigm, where its predecessors clearly were not? What makes it tick? As a developer, there’s no doubt that I am rather enthusiastic about microservices, but I was enthusiastic as well (more or less) when people came up with CBD and SOA.

I do suppose there are differences. For the first time we seem to have the technology in place to build these type of architectures. All the fancy and complex middleware is gone, and we rely solely on very basic and long-time existing web protocols and technologies. Just compare REST to CORBA. Also we seem to understand deployment much better, due to the fact that we’ve learned how to do continuous integration, unit testing, and even continuous delivery. These differences suggest that we can get it to work this time around.

Still, from my historical viewpoint some skepticism is unavoidable. Ten years ago, we also really believed that service oriented architecture would be technologically possible, it would solve all our issues, and we would be able to build stuff faster, reusable and more reliable. So, to be honest, the fact that we believe that the technology is ready, is not much of an argument. Meanwhile the world also got more complex.

Over the last year I’ve been involved with a company that is moving away from their mainframe (too expensive) and a number of older Java monoliths (too big, and hard to maintain). Also time-to-market plays an important role. IT need to support introducing a new product in months, if not in weeks. So we decided to be hip and go microservices. Here’s my recap of the good, the bad and the ugly of microservice architectures, looking back on our first year on the road.

The good

Let’s start with the good parts. We build small components, each offering about two to six services. Good examples of such small components are a PDF component, that does nothing more than generate PDF’s from a template with some data, or a Customer component that allows users to search for existing customers. These components offer the right size. The right size of code, he right size of understandability, to right size to document, to test an to deploy.

The good

Our team tend to evolve towards small teams designing, implementing and supporting individual components. We didn’t enforce ownership, but over time small teams are picking up the work on a specific component and start feeling responsible for it.

When we outlined the basic architecture for our microservices landscape we set a number of guidelines. We are not only building small components, but are also building small single-purpose web applications. Applications can talk to other applications, and can talk to components. Components handle their own persistence and storage, and can also talk to other components. Applications do not talk directly to storage. Components do not talk to each others storage. For us these guidelines work.

Microservices lives up to some of its promises. You can pick the right technology and persistence mechanisms for each of you components. Some components persist to relational databases (DB2 or SQLServer), others persist to document databases (MongoDB in our case). The hipster term here of course is polyglot persistence.

We also stated that every application and component has it’s own domain model. We employ the principles and patterns of domain driven design straightforward. We have domain objects, value objects, aggregates, repositories and factories. Because our components are small, the domain models are fairly uncomplicated, and thus maintainable.

Although we have had quite a journey towards testing our components and services, from Fitnesse to hand-written tests, we are now moving towards testers specifying tests in SoapUI, which we are running both as separate tests and during builds. We had to learn to understand REST, but we’ve got this one figured out for now. And testers love it.

When we started our microservices journey, the team I’m was in a highly reactive mode – tell us what to program and how to program it, and we’ll do our job. I probably won’t have explain what that does with the motivation of the team members. However, with microservices there is no such thing as a ready-to-use cookbook or a predefined architecture. It simply doesn’t exist. That means that we continuous find ourselves solving new pieces of the microservices puzzle, from discovering how to design microservices (we use smart use case), to how implement REST interfaces, how to work with a whole new set of frameworks, and a brand new way of deploying components. It’s this puzzle that make working on this architecture interesting. We allow ourselves to learn everyday.