Practical AgileSOA - Making the transformation stick

Of course it would be great if the small victories were permanent ones. However, unless there is an obvious benefit to changed behavior, people tend to revert to their old ways.

The business needs to understand how SOA is going to move some of their Key Performance Indicators. That progress needs to be baselined and tracked vigorously. This of course does not mean reams of BS data. On the contrary, a few metrics are selected based on their true meaning to the business. It is pointless to measure throughput rates and latency if those numbers cannot be translated into business value.

The change SOA brings about needs to be made very visible, even if that change is incremental and under the covers. When business and IT alike learn that they can be more efficient, more flexible, less expensive and more aligned by embracing the mindset of Agile and SOA, it is more likely that the changes brought about by these philosophies become permanent.

So, after all this, it is time for you to get started with AgileSOA.
Or if you have already been trying to get it off the ground, revise your current SOA approach and take it more one step at a time.

Next steps for the reader:

1. Find your candidate services
2. Go make them, simple, clean and quickly.
3. Chronicle what you learn
4. Share your experiences
5. Repeat

Good night, good luck and good learning!

Practical AgileSOA - Getting some governance started

As developers get started with building services, they will gather invaluable experience on how to develop SO applications within the context of their organization. This experience, plus best practices and emerging standards, need o be captured and chronicled. An internal group, overtly supported by executive management, needs to be created to become the Service Competency Center (SCC).

The mission of this group is not to dictate, but to facilitate. They are the SOA body of knowledge (the SOA-BOK, if you will) and they are responsible for getting the early trappings of an SOA infrastructure started. They are also responsible for informing development teams about the virtues and drawbacks of SO AD, via workshops, internal seminars and intranet forums.

The SCC must position itself to detect, analyze and propose to resolve organizational constraints for a true, firm wide SOA to flourish. One of these constraints is inter-departmental priority conflict. Others are incorrect technological assumptions and general resistance to change.

The SCC must emit an atmosphere of enablement, of inclusion. To that effect, an SOA "social worker" may be very useful. Such an individual would visit with groups attempting to embrace SOA and work with them to overcome any obstacles, technical or organizational.

This is a departure from the traditional Technology evangelist concept, which has historically been met with skepticism and suspicion.

The SCC needs to minimally advocate an incremental approach to SOA, and optimally promote Agile practices to the full extent possible. Individuals and teams are generally overwhelmed by large enterprise IT projects and feel lost, out of touch or simply left in the dark. The continuous feedback channels promoted by Agile, plus the transparency of progress and resource use will greatly improve developers' sense of accomplishment.
This in turn will create buy-in and eventually commitment.

Finally, the SCC must keep executive management with their eye on the prize. There must be unwavering executive support for the SOA endeavor. The SCC can accomplish this similarly by setting small, short-term goals, providing transparency, measure visible progress and hitting these goals consistently.

That will instill the trust of the execs and keep their attention.

Next post: Making the transformation stick

Practical AgileSOA - How to promote and incentivise reuse

Question: why would anyone use other people's products?
Answer: when they are better than they can make themselves.

Traditionally, software development was not considered proper unless an application was built from scratch. Over the years, more and more components have become available and these products have become better. Today many of these products are so good that it doesn't pay to rebuild them. So software development is now done using a multitude of other people's products

So it must be with services.

Services must be so solid and reliable that it doesn't pay for others to re-implement their functionality. But how do you know when a service is solid and reliable enough?
While there is no absolute formula or metric that makes a service pass a standardized quality test, it serves to understand what the critical aspects are of designing and building services that people will want to use.

First and foremost, the service functionality needs to be designed so that it actually does something useful. While that is different from business to business, the concept should be obvious. A service that just serves up database records provides no value. A service that locates the correct database records, processes them and serves them up as a useable bundle provides something useful. A service that simply collects performance data may be a good building block, but it truly comes to life when it starts to interpret that data to provide intelligent monitoring.

Secondly, a service needs to be built using TDD. All functionality needs to be unambiguously testable. These tests need to be run every time a change is made, preferably using continuous integration tools. A service without bugs trumps just about any application. In this case, statistics are your friend.

Thirdly, a service should behave like a good building block. That means that a client has to have just knowledge of how to invoke it or subscribe to it, and what to expect back.
It should not have to be aware of the state of a service, unless that state is what the service provides.

Next post: Getting some governance started

Practical AgileSOA - The emerging infrastructure

Vendors would like us to believe that there will be no SOA without their products.

Nothing is further from the truth.

However, an emerging SOA is supported by an emerging SO infrastructure.

Here we talk about which pieces may be needed at which level of SOA maturity.

If you are starting with synchronous request-reply services using XML to exchange data, you do not need anything but standard development environments. Java and .Net come packed with parts to create and parse XML documents and send and respond to requests.

If you want to start with Web services (SOAP over HTTP) then you can decide to use the basic parts of your development platform or use what's called a SOAP stack. This is a collection of libraries and code generators that take care of the transformation of your objects to SOAP XML and back, and take care of the request and reply formatting.

The beauty of these SOAP stacks is that they make building Web Services a snap. The main drawback is that they rely on generated code, which is notorious for quality and performance problems. (A friend of mine was even working around a bug in his code generator without blinking)

Using SOAP has many advantages. For one, there are some great products out there that support the testing, running and management of SOAP-based Web services.
The disadvantage of SOAP is that it has an unwieldy XML format, which makes its application limited to situations that do not have sub-second response time requirements.

The next step in your SOA infrastructure must be monitoring. After all, with measuring request volume, response times and latencies, you can prove that:

1. your services are used
2. who uses them
3. what your average performance is

This is a critical step in your emerging SOA, since at this point you can demonstrate that your existing services are indeed being used and to what extent.

Depending on the level and type of security required by your organization, you may have to think about that from the get-go. But this is not the case in every organization. Make sure that you define the authentication and authorization requirements for each service that is being launched. Sometimes, an access control list will do. In other cases, you will want to have full authorization, authentication and audit trail.

Whatever the detailed security requirements are, you will want to have a central security management facility, such as a federated identity management service that hands out certificates upon successful authentication. That way, new services can simply reuse (there's that word again) what has already been fleshed out and capitalize on security improvements as they become available. Making services use certificates to request from or subscribe to services works like a daypass to an amusement park: whatever type of pass you bought determines to which rides you can get in.

A whole string of posts could be dedicated to BPM, BPEL, orchestration and choreography. Let me just note that some type of business process monitoring should be baked in to your services from the start. How you use that will become apparent as your SOA matures. Hold off on buying even more SOA products, now those that purport to do modeling and monitoring out of the box for you. Yes, they may have great demos and user interfaces. But think about what you are trying to accomplish. Evaluate later on whether the number of BPM-enabled services are becoming hard to manage and if a commercial product will help you grow.

Next post: How to promote and incentivise reuse

Corollary comment on the comment

AND ANOTHER THING!!!

Seriously, the most important thing to realize when learning to apply service-oriented techniques is that, as a provider of a service, you are owning a product.

The moment you bring your service in the open and people not under your control (like colleagues) start using it, you are operating a mini-software company, like it or not.

Once you have embraced this notion, you need to look at how successful software companies interact with their customers. You will then realize SOA is not such a bad model after all, and the patterns for how to resolve the new and unfamiliar issues are already out there.

Comment to "Versioning Services"

Since spammers have discovered blog comments as a way to get to you, I disabled them.

Honestly, spammers should be summarily and ceremoniously executed.

Anyway, my colleague Ade Oshineye sent me this comment:

Have you seen this article: http://martinfowler.com/articles/consumerDrivenContracts.html where Ian Robinson talks about the gritty issues involved in versioning services?

Sure. I am not dismissing the complexities of versioning mature services. The point of this series of posts is to get people to stop worrying about details like these until they have obtained a firm grasp of what it is like (and what it means) to build applications with services.

In true Agile sense, plan and build what you understand. Analyze and design what you can't get your head around yet later.


It is this engineering habit of wanting to have ALL the answers to ALL the questions that is killing efforts to change the way we do things in software development.
That's why Agile/XP is such a beautiful thing, because it does away with that notion and accepts that you CAN'T know all the questions up front, let alone have all the answers.

Many times have I come across budding SOA efforts that were mired in pointless analysis and design. "But we have to have our message headers defined!".

NO, NO, NO! What does a full definition of message headers teach you about making SO applications? Nothing! Where's the learning? Where's the difference? Where's the business value?

The point is to learn how to do things differently and discover how you and your team can benefit from it. The many issues that undoubtedly will arise are there to be SOLVED, not avoided. If early man had this attitude of wanting to have all the problems solved before embarking on something new, we'd still be living in caves, without fire, mind you.

What does the sentence: "With Web services, commercial registries can handle much of the versioning" actually mean? Are you suggesting that using a commercial registry to locate business-critical services is a viable approach? Even though the first I'll know about a (possibly incompatible) service upgrade is when the registry notifies me? At this point the upgrade has already happened which is far too late to do anything about it.

Commercial registries like Systinet have the concept of service release stages. For this you need to install more than one registry and add the newly versioned service to the non-production registry first. This very action will notify interested parties that a new version of a service is pending, so they can take action if required.

There are several sound strategies for introducing a new version of a service. One is to leave the previous version intact while the new one comes on-line and set a phase-out schedule.

Of course, this sort of approach requires that all parties have a means of communicating. And here now is the crux of a successful SOA implementation: it's the PEOPLE:

As per the Agile Alliance: (http://www.agilealliance.org/intro)

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

As you can see from Ian's article it's not enough to just append extra content on the end given that most of the tools people use to interact with web services insist on validating against the entire schema.

Sure, if you validate against an entire schema, you are shackling your consumer to a service, i.e. coupling it tightly. While schema validation has its place, if you are in an environment where message formats change frequently (e.g. in options trading) then that coupling is undesirable.

I'm not pro or con validation, I'm just suggesting to use common sense.

Moreover putting version numbers on each message sent is not in itself an acceptable versioning strategy unless we know that our consumers can actually handle the new version. For instance in an investment banking context if there were a service that published newly booked trades it would be unacceptable to silently upgrade that service. If business critical applications suddenly stopped working because of a silent upgrade it would not be enough for the service provider to point to the error queues full of messages in the new format. Each upgrade has to be verified against each consumer in some way because in practice the levels of coupling of between service provider and consumer are still just as high in an SOA. The difference is that the coupling is often implicit and only exists at runtime. This creates situations where there appears to be low-coupling at the architectural level but support staff have to manually resolve the high levels of coupling that exist at runtime in the production environment. This usually ends up taking the form of increased communication between the humans who support the service providers and consumers e.g. co-ordinated releases, staging areas, sandboxes for integration testing by replaying messages from the live system.

Again, use common sense.

• Introduce a new version of a service in a staging environment
  
• Release it to production on a different URL or topic, if you can
  
• If new data is added to a message, put it somewhere so that it doesn't interfere with existing consumers
• ...and version numbers in messages DO work. As a consumer of somebody else's service, I know that I'm beholden to the format it sends its data. So if version numbers are available, it would behoove me to build my consumer to check for that and do something useful with it
  

Indeed, until we can write software that can figure out what to do with new data all by itself, we have to have proper human communication. This need not be heavy-weight and procedure-laden. But every time one human provides something to another human, the provider has to choose between either improving it and coordinating with the consumer, or leave it stagnant and risk the consumer walking away.

If you feel I'm being too light-hearted and cavalier about this, maybe you oughtta try that yourself. It's quite liberating, and very effective.

Practical AgileSOA - Versioning services

In the previous post, we saw how by simply doing, we can get those first services built. This activity will answer a lot of questions, but also will show us which questions to ask next.

One of these is how to deal with service versions.

Undoubtedly, your services will undergo revisions and change.
That in and of its own is not a problem. The trouble happens when you change the interface definition and the functionality.
If the service starts behaving differently because


With push services (asynchronous) it is a good idea to include a version number in the messages it sends.
That way, a subscriber can be made aware that things have changed.
One can think of a subscriber routing incoming messages through a validation component if the expected message version does not match the incoming message version.

With Web services, commercial registries can handle much of the versioning.
Consumers can subscribe to registry updates and be rigged to take action when a service changes.

As a rule, you should be wary of deleting information from service outputs.
Not that you should keep everything around forever; just be aware that consumers are notified in some form.

In general, new content should be added in such a way that a consumer can continue to use the incoming data without knowing how to handle the new stuff. For instance, if you use a byte-encoded message, add new content at the end.

With self-describing message and XML messages, this restriction does not really apply, since you can just pull out what you're looking for, regardless of the extra content.

(See, even with standardized services, you are still not entirely decoupled, just loosely coupled.)

Next post: The emerging infrastructure

Practical AgileSOA - Making the first services

The question that always comes up is:
"How do I know where to draw a service boundary?"

Answering that question requires some more investigation.

Begin by looking at the piece of functionality that the client wants to cast into a service.
Let's take a universal example: getCustomer. All companies have customers, and many companies have a multitude of customer data sources, be it through isolated IT projects or through Mergers and acquisitions.

Some questions that come up at this point are:

"What do I use as inputs?"
"Do I return the intersection or the union of all the data from the various customer systems?"
"What if some data exists in system A and not in B?"

We need to step back here and ask ourselves:" What is it that you want this service to DO?"
Well, I want it to return customer information.
OK, and what do you want to use as search criteria on our first attempt to make this service?
I want to use, uh, first, last and company name.
Very well. We now have something to start with.

"...but what about returning duplicate records, and mismatches, and... and ... and..."

Relax, it's Agile! We'll fold exception scenarios in later.

Let's now make this service.

First, we make a client that invokes the projected service. Do we want to use a SOAP stack? If you must, sure. If not, that's fine too. Suffice to say that this client will act as our service "test".

Now we make a simple service that just returns data from a preloaded file.
Then we expand it to invoke one of the legacy customer data sources.
We continue to expand this service by adding another legacy data source and work out how to consolidate duplicate information.

At this point we may find that submitting first, last and company name is not detailed enough, so we need to add a parameter to the request. In essence, our service has now changed its interface and therefore is a new version.

Next post: Versioning services

Practical AgileSOA - Intro

Clients invariably ask me;

"How do I actually DO this SOA stuff? Concretely? What is my Step 2?"

In this series of posts I will detail how we at ThoughtWorks typically execute the various aspects of Agile SOA.
Let me start by saying that we are Agilists, which means that we do just about anything in continuous cycles of Design-Make-Test-Reflect.

Similarly, we get SOA efforts off the ground by simply doing, not by talking and designing ad infinitum (or ad nauseam, rather)

These are the efforts we want to cover:

• Making those first services
• Versioning services
• How an infrastructure emerges
• How to motivate reuse
• Getting some governance started
• Making it all stick

After you have read this series, you will be able to get off the mark and get some traction on SOA. Incidentally, you also may pick up some Agile Best Practices. After all, we think Agile and SOA are closely related.

Next post: Making the first services