Carl's Consulting Adventures

Insane or Enlightened Leader?

2011-09-11T09:25:00.000-07:00

A friend of mine alerted me of this book, A First-Rate Madness by Nassir Ghaemi.

In it, the author makes a case for insanity as a positive indicator for effective leadership and sound mental health as an indicator for effective status-quo management.

I take this to mean that in order to keep things running, we need the sane to keep the lights on and the insane to make things better.

Loosely translated, the Managers who see as their mission to keep things the same are best suited for a status quo situation. They tend to resist change and see it as a direct attack on their whole being. They experience deviations from protocol and procedure as failure. These types are the most prevalent in top management today because they gives us comfort that they can be controlled, or at least won't do something unexpected. As such they are the main impediment to progress.

The Leaders who see as their mission to change things are best suited for an undesirable situation (like several around us now). Their appetite for risk is much greater than the ordinary person. Rather, they experience a low-risk situation as uncomfortable and even frightening. These people are often attributed with mental illnesses ranging from ADHD to paranoid schizophrenia. These individuals scare common folk with their outlandish ideas and frighten managers with their roguish behavior. What makes things worse is that these kinds of characters have a tendency to achieve actual results, which creates a power struggle between the Managers and the Leaders.

In order to achieve true results, Managers and Leaders must collaborate and learn to appreciate each other's points of view, giving each other the space to apply their talents while managing each other's perceived neuroses.

Introduction to Patterns of Corporate Behavior

2011-04-10T15:04:00.000-07:00

After observing people and organizations at all levels for a long time, a set of standard behavior patterns emerge. Some are helpful, but a surprising number are quite harmful. In this series, I’ll discuss the harmful ones, how to spot them and how to replace them with helpful patterns.

Mind you, this is no corporate self-help series. Just like behavior therapy, there is no substitute for someone coaching an individual through improvement. But it will allow the reader to start to identify whether something’s wrong and realize that there’s something that can be done about it.

For example, imagine a person with so-called Borderline Personality Disorder. These individuals come across as perfectly normal, if slightly quirky. But when interacting with them for a while, you’ll find that their reactions to ordinary events are somewhat exaggerated. When they turn their fury on you, for no apparent reason, you’re left to wonder what it is that you did that was so heinous. If you don’t know the pattern, you’ll be challenged to see the symptoms, let alone the underlying issues. What’s worse, you will react in a counter-productive way, exacerbating the problem.

Recognizing patterns is only the beginning. A critical aspect of improving behavior is the driving motivation. Incentives often are long-term: if you stop smoking NOW, you’ll live longer. If you start exercising NOW, you’ll be healthier at advanced age. If you try Agile NOW, you’ll have a better grasp on your project portfolio a year from now.

The motivators that keep people locked in their behavior patterns are immediate. If those are not diminished or neutralized, there is nary a chance for any improvement. And without changed behavior, any Program Of Change is bound to fail.

Notwithstanding the pathological aspects of addiction, if a smoker were to get nauseous soon after taking her first drag, if an alcoholic were to get a splitting headache soon after one drink, it would be hard for them to keep up the habit. So if bad behavior were to have immediate negative repercussions, whereas good behavior would have immediate positive effects, the person in question would be more bound to stick with the good behavior.

As with Cognitive Behavioral Therapy, one cannot be expected to figure this out on their own. You have to be coached through it. Instead of taking a pill to numb the symptoms (e.g. push your teams to be more Agile), you change the behavior that creates the symptoms in the first place. Therefore, there has to be an incentive to change that is almost as immediate as medication.

Let’s look at a few textbook examples of seemingly laudable, but ultimately destructive behavior. The Over-promising Sales Guy, the Blame-Storming Executive and the Bleeding-Edge technologist

The sales guy is afraid to disappoint the customer, yet by over-promising beyond his company’s capacity, he’s doing just that. But he doesn’t know, he thinks he’s being aggressive and pushy. If every time he over-promises, he is confronted with this early on and has to go talk to the customer and apologize, he will experience exactly what he’s trying to avoid, immediately.

The newly minted executive has learned that holding your staff accountable is the way to stature and promotion. After all, as project manager she got her projects delivered come hell or high water. And now that she manages other project managers, getting them to get things done should work the same. Especially if she publicly berates under-performers her staff will get to know her as someone to be reckoned with. When there’s trouble, she holds lengthy blame-storming sessions, not so much to figure out what collectively went wrong but on who to pin the blame. Her expectation is that her staff will fall in line and execute her assignments to the best of their ability if she holds them personally accountable.

But instead, her project managers and other staff spend more time covering their backs than getting actual work done. Now the whole project portfolio is making hardly any progress. Unfortunately for our newly promoted program manager, her PMs are Agile and the waste from cumbersome process becomes apparent within a few iterations, indicating her as the main impediment.

For the chief architect, playing with the latest gadgets and tools has become a way of life. He became chief because he helped his firm enter a new age of computing by introducing SOA. He continues to introduce new technologies, well beyond the needs of the firm and the capacity for adoption of the IT staff. Implementations are not thought through and as such, the support burden for developers continues to grow. At some point, the company is hamstrung by the very technology that was supposed to give them a competitive advantage. The recent close collaboration between Operations and Development make it abundantly clear what behavior has gotten them in this bind.

Patterns persist because people have seen their superiors be successful with them. It’s learned behavior. Conversely, people promote others who are like them, for better or for worse. So people are filtered based on their behavior, not so much on their results, ensuring the perpetuation of these patterns.

Much like certain forms of abuse, these destructive corporate behavior patterns are hard to break and are passed on from senior to junior workers. it is imperative that they are recognized and treated with short-term incentive programs in a low-risk setting.

We will discuss each pattern and outline possible approaches to disrupt them or at least diminish their impact.

Agile and the Trough of Disillusionment

2011-01-10T18:09:00.000-08:00

Everywhere you look, companies are turning to Agile. Even my sister, who works for a huge European city’s government, said that she’s introducing Agile practices. Professional Services organizations say they need to “go Agile” because their clients demand it.

Yet, when examining what that means in practice, not much is really changing. People are still starting multi-year projects, with massive scope, budgets and resources. They still fail to establish proper project context within which costs and risks can be contained. And they still plan the execution in excruciating detail, to then execute the plan to the letter. Wavering from this plan equals failure. And as such, people expecting Agile to finally solve their organizational challenges are very disappointed: Agile seems like a lot of smoke, mirrors and snake oil.

The rub lies in how organizations measure success.

The only reliable measure of success is the value of capitalizable assets. It doesn’t matter how well you executed the project to plan, if whatever your team delivered does not generate revenue, make you more competitive or reduce costs, it was wasted effort.

The concept of putting a clear business value on each project already sets them up for success. Success as measured in low-risk, high-value execution. Stopping a project because it has become too risky or net negative value is only sound business. Better to use the resources on a different initiative that remains viable rather than have them go to waste on a deathmarch project.

Agile, unlike Scrum and XP, is not a set of activities and processes: it’s a risk-mitigation philosophy. The activities employed to implement this philosophy tend to be industry-specific. Furthermore, the way these activities are executed differs for each organization. If the results are supporting the philosophy, there is no wrong way to do Agile. If an organization continuously evaluates its effectiveness and output, and seeks to diminish any factors that negatively impact it, it could be called Agile.

From this follows that ‘going Agile” is not something that can be relegated to the troops while the generals continue business as usual. On the contrary: there has to be very strong leadership, by example and by changed behavior, to support an organization’s move to Agile.

So in conclusion, the disillusionment with Agile is understandable. Now that practitioners are waking up to the root causes, we’re entering a new, more mature stage of Agile-ness and move into the era of Executive Agile Transformation.

Corporate Behavior Therapy

2010-12-16T19:02:00.000-08:00

A while back I started to write about how organizational transformation is not rooted in activities and tools, but in people’s behaviors.

After several more years in the field, I have noticed that the most effective approach to OT is akin to Cognitive Therapy. In a nutshell, CT aims to change how people think about and react to inputs and as such change how they view themselves and the world around them.

OT has a very similar goal. You aim to change the way a company or division views themselves and their ecosystem. And that is done most effectively by changing people’s behaviors.

The most critical part of that behavior is how people measure each others performance. Do political apparatchiks make it to the top, despite their lack of business contribution or are people measured on their impact on the firm’s revenues, reputation and talent retention? From a simple survival perspective, people will behave according to how they can be judged most successful.

On a more tactical level, how project portfolios are handled and how the projects therein are valued is another critical component of how a company executes. Is the concept of a project portfolio a list of projects with people’s allocations? Or is it an asset portfolio, with constantly changing valuations and real-time feedback of the overall health? Are projects judged by how well they follow some process or by how rapidly and accurately business needs are met?

Then finally, how are innovators treated? Does every new idea need to be vetted with one person or a committee or are people given a forum where they can display their ideas and stir enthusiasm? Do a few people have the power to squash new initiatives unchecked or are these evaluated on their objective merits?

Most corporate behaviors are rooted in its culture. So to change those if needed, a culture shift will be necessary. That is a huge challenge, often avoided by OT practitioners. It’s a lot easier to talk about specific activities and tools than about people themselves. However, this topic cannot be avoided if OT is to have a chance to stick.

The Real Adventure Begins

2010-12-10T19:10:00.000-08:00

Well, I finally did it!

I started my own management consulting practice. It sounds like a bigger deal than it is, really. But being a free agent surely has its advantages. And I just LOVE that title, president…

Starting an S-corp is not the hassle it used to be. You can incorporate, get a domain with a website, get your EIN, open a corporate account, and pay your taxes, all without getting dressed. You still have to mail in your articles of incorporation and fax in your form 2553, though. Poo…

So the next challenge is creating a pipeline of work. As I have written before, The Network Is Your Career, and no more so than when you are your own product.

Here is where your reputation and the connections you have nurtured over the years make all the difference. In keeping with the season, if you have been naughty and driven a hard bargain, you may end up with lumps of coal for projects. If you have been nice however, and played fair, were reliable and kept your clients’ interest at heart, you are welcomed back wherever you go. In today’s hyper-connected world, it’s easy to spread the word. But there still is great value in contacting old friends and colleagues and explaining to them what your focus is. They may not be prospects, now or ever. But they talk to people that may need your help. And the better your network knows what value you can bring, the more leads you get.

Coming out of a nasty recession, it appears many companies are realizing that business as usual no longer cuts it. But they are grappling with formulating a vision of what “better” would look like. What makes improving even harder is that executives and directors have not woken up to the notion that nothing will change unless they start measuring their reports differently. A Program of Change is for staff, not the corner office.

Per the Gartner Group Hype Cycle, Agile is stuck in the Trough Of Disillusionment, and rightfully so. Agile was hijacked by faddists and me-too companies, just like SOA was, and subverted into another prescriptive, waste-prone process. Many people have not had a chance to witness the power of simplicity and transparency agile approaches bring. Pilot projects were started, agile and scrum coaches were brought in and things started feeling differently. But when the coaches left and the pilot projects were delivered, somehow everything went back to the way it was. The root cause is that people went through the motions of agile practices, but did not internalize its philosophy nor change their behavior. They can’t because the way their performance is measured hasn’t changed.

That’s why I decided to focus on Executive Agile Transformation, or EAT. EAT your own dog-food. You are what you EAT. Whoa, I better go trademark that…

All kidding aside, programs of change do not stand a chance unless a company changes the way it measures business contribution, from the top down. Yes, executives are a tough bunch to change. But they can do it and when they do, miracles happen. And not only at Christmas.

My biggest challenge will be to get executives to admit they have a problem. Here is where we will take guidance from psychology and apply practices from Cognitive Therapy.

And so the adventure begins…

Gradual versus sudden shift from waterfall to agile

2010-03-02T07:47:00.000-08:00

Hello readers!

Well, it's time to pick up the thread on my consulting adventures. Too many thoughts to share.

A former client of mine asked me:

"Do you have a quick summary of the plusses and minuses of a gradual versus sudden shift from waterfall to agile for an org? In other words what are the implications of transitioning over 2 months versus 2 years?

And if you have a portfolio of programs, with a common pool of resources, do you let the majority of programs stay on waterfall and move one or two projects to agile, or is it better to move everything to agile?

Do you have any papers, books, case studies to shed light on this topic?"

My experience is this.

Gradual change has the following advantages:

People have more time to adjust if changes are introduced piece-meal
People will be less likely to resist change if it's something they can get their heads around
Each incremental change can be evaluated on its impact and rolled back if it's net negative Incremental changes will have the time to engrain themselves in the daily routine, creating a ratchet effect: once the change is part of daily life, it will be hard to roll back.

The drawbacks are:

It takes a long time to see a payout from the changes
When changes are split up too much, they may fail to have a positive effect, causing the organization to falsely reject the change
The longer the rollout of changes, the more chance detractors have to discredit the change program
A long-running program of change is at risk of falling victim to business cycle volatility. An economic downturn will surely kill such a program

At a former employer, I introduced changes in packets. When I slowed the rate of change down too much,giving people a chance to catch up, the program stalled under constant criticism of detractors. When the firm had to struggle for its survival, (which it lost), they all but killed the program.

You move one project, or a cluster of dependent projects at a time.

The client responded with:
"Hmmm, hard to imagine an independent project in a portfolio. Shared resources, shared outcomes, shared technology infrastructures, shared release dates. If my assertion is true (there's no such thing as an independent project), would this not say you should go all-or-nothing with agile?

Assume you go "all-in" with agile, then it becomes a question of adoption speed, no?"

Ah, when I say independent, I mean independent delivery assets.

Another way to divide it is to decide a point in time after which new
projects are run in an Agile fashion. From that point on new projects
are properly incepted, storied and tracked.

Going all-in with Agile, or any program of change, is a recipe for disaster. You should approach the introduction of Agile as an Agile project in and of its own.

Agile, the SOA hangover cure on InfoQ

2006-08-01T20:15:00.000-07:00

Here's a fun little story about making SOA happen by taking simple yet strategic steps, using Agile practices.

http://www.infoq.com/articles/Agile--The-SOA-Hangover-Cure;jsessionid=EFD8B5DED840DF9502597D7A080B1B8B

BTW, this will be my last Agile/SOA/Consulting Adventure post, since I have given up being a consultant for a more earth-bound existence as a Head of Project Management.

The adventure continues however with a sassy blog about Agile Organizational Transformation.

Practical AgileSOA - Making the transformation stick

2006-07-21T10:03:00.000-07:00

Of course it would be great if the small victories were permanent ones. However, unless there is an obvious benefit to changed behavior, people tend to revert to their old ways.

The business needs to understand how SOA is going to move some of their Key Performance Indicators. That progress needs to be baselined and tracked vigorously. This of course does not mean reams of BS data. On the contrary, a few metrics are selected based on their true meaning to the business. It is pointless to measure throughput rates and latency if those numbers cannot be translated into business value.

The change SOA brings about needs to be made very visible, even if that change is incremental and under the covers. When business and IT alike learn that they can be more efficient, more flexible, less expensive and more aligned by embracing the mindset of Agile and SOA, it is more likely that the changes brought about by these philosophies become permanent.

So, after all this, it is time for you to get started with AgileSOA.
Or if you have already been trying to get it off the ground, revise your current SOA approach and take it more one step at a time.

Next steps for the reader:

Find your candidate services
Go make them, simple, clean and quickly.
Chronicle what you learn
Share your experiences
Repeat

Good night, good luck and good learning!

Practical AgileSOA - Getting some governance started

2006-07-20T09:59:00.000-07:00

As developers get started with building services, they will gather invaluable experience on how to develop SO applications within the context of their organization. This experience, plus best practices and emerging standards, need o be captured and chronicled. An internal group, overtly supported by executive management, needs to be created to become the Service Competency Center (SCC).

The mission of this group is not to dictate, but to facilitate. They are the SOA body of knowledge (the SOA-BOK, if you will) and they are responsible for getting the early trappings of an SOA infrastructure started. They are also responsible for informing development teams about the virtues and drawbacks of SO AD, via workshops, internal seminars and intranet forums.

The SCC must position itself to detect, analyze and propose to resolve organizational constraints for a true, firm wide SOA to flourish. One of these constraints is inter-departmental priority conflict. Others are incorrect technological assumptions and general resistance to change.

The SCC must emit an atmosphere of enablement, of inclusion. To that effect, an SOA "social worker" may be very useful. Such an individual would visit with groups attempting to embrace SOA and work with them to overcome any obstacles, technical or organizational.

This is a departure from the traditional Technology evangelist concept, which has historically been met with skepticism and suspicion.

The SCC needs to minimally advocate an incremental approach to SOA, and optimally promote Agile practices to the full extent possible. Individuals and teams are generally overwhelmed by large enterprise IT projects and feel lost, out of touch or simply left in the dark. The continuous feedback channels promoted by Agile, plus the transparency of progress and resource use will greatly improve developers' sense of accomplishment.
This in turn will create buy-in and eventually commitment.

Finally, the SCC must keep executive management with their eye on the prize. There must be unwavering executive support for the SOA endeavor. The SCC can accomplish this similarly by setting small, short-term goals, providing transparency, measure visible progress and hitting these goals consistently.

That will instill the trust of the execs and keep their attention.

Next post: Making the transformation stick

Practical AgileSOA - How to promote and incentivise reuse

2006-07-19T16:57:00.000-07:00

Question: why would anyone use other people's products?
Answer: when they are better than they can make themselves.

Traditionally, software development was not considered proper unless an application was built from scratch. Over the years, more and more components have become available and these products have become better. Today many of these products are so good that it doesn't pay to rebuild them. So software development is now done using a multitude of other people's products

So it must be with services.

Services must be so solid and reliable that it doesn't pay for others to re-implement their functionality. But how do you know when a service is solid and reliable enough?
While there is no absolute formula or metric that makes a service pass a standardized quality test, it serves to understand what the critical aspects are of designing and building services that people will want to use.

First and foremost, the service functionality needs to be designed so that it actually does something useful. While that is different from business to business, the concept should be obvious. A service that just serves up database records provides no value. A service that locates the correct database records, processes them and serves them up as a useable bundle provides something useful. A service that simply collects performance data may be a good building block, but it truly comes to life when it starts to interpret that data to provide intelligent monitoring.

Secondly, a service needs to be built using TDD. All functionality needs to be unambiguously testable. These tests need to be run every time a change is made, preferably using continuous integration tools. A service without bugs trumps just about any application. In this case, statistics are your friend.

Thirdly, a service should behave like a good building block. That means that a client has to have just knowledge of how to invoke it or subscribe to it, and what to expect back.
It should not have to be aware of the state of a service, unless that state is what the service provides.

Next post: Getting some governance started

Practical AgileSOA - The emerging infrastructure

2006-07-19T09:14:00.000-07:00

Vendors would like us to believe that there will be no SOA without their products.

Nothing is further from the truth.

However, an emerging SOA is supported by an emerging SO infrastructure.

Here we talk about which pieces may be needed at which level of SOA maturity.

If you are starting with synchronous request-reply services using XML to exchange data, you do not need anything but standard development environments. Java and .Net come packed with parts to create and parse XML documents and send and respond to requests.

If you want to start with Web services (SOAP over HTTP) then you can decide to use the basic parts of your development platform or use what's called a SOAP stack. This is a collection of libraries and code generators that take care of the transformation of your objects to SOAP XML and back, and take care of the request and reply formatting.

The beauty of these SOAP stacks is that they make building Web Services a snap. The main drawback is that they rely on generated code, which is notorious for quality and performance problems. (A friend of mine was even working around a bug in his code generator without blinking)

Using SOAP has many advantages. For one, there are some great products out there that support the testing, running and management of SOAP-based Web services.
The disadvantage of SOAP is that it has an unwieldy XML format, which makes its application limited to situations that do not have sub-second response time requirements.

The next step in your SOA infrastructure must be monitoring. After all, with measuring request volume, response times and latencies, you can prove that:

your services are used
who uses them
what your average performance is

This is a critical step in your emerging SOA, since at this point you can demonstrate that your existing services are indeed being used and to what extent.

Depending on the level and type of security required by your organization, you may have to think about that from the get-go. But this is not the case in every organization. Make sure that you define the authentication and authorization requirements for each service that is being launched. Sometimes, an access control list will do. In other cases, you will want to have full authorization, authentication and audit trail.

Whatever the detailed security requirements are, you will want to have a central security management facility, such as a federated identity management service that hands out certificates upon successful authentication. That way, new services can simply reuse (there's that word again) what has already been fleshed out and capitalize on security improvements as they become available. Making services use certificates to request from or subscribe to services works like a daypass to an amusement park: whatever type of pass you bought determines to which rides you can get in.

A whole string of posts could be dedicated to BPM, BPEL, orchestration and choreography. Let me just note that some type of business process monitoring should be baked in to your services from the start. How you use that will become apparent as your SOA matures. Hold off on buying even more SOA products, now those that purport to do modeling and monitoring out of the box for you. Yes, they may have great demos and user interfaces. But think about what you are trying to accomplish. Evaluate later on whether the number of BPM-enabled services are becoming hard to manage and if a commercial product will help you grow.

Next post: How to promote and incentivise reuse

Corollary comment on the comment

2006-07-18T08:31:00.000-07:00

AND ANOTHER THING!!!

Seriously, the most important thing to realize when learning to apply service-oriented techniques is that, as a provider of a service, you are owning a product.

The moment you bring your service in the open and people not under your control (like colleagues) start using it, you are operating a mini-software company, like it or not.

Once you have embraced this notion, you need to look at how successful software companies interact with their customers. You will then realize SOA is not such a bad model after all, and the patterns for how to resolve the new and unfamiliar issues are already out there.

Comment to "Versioning Services"

2006-07-18T07:10:00.000-07:00

Since spammers have discovered blog comments as a way to get to you, I disabled them.

Honestly, spammers should be summarily and ceremoniously executed.

Anyway, my colleague Ade Oshineye sent me this comment:

Have you seen this article: http://martinfowler.com/articles/consumerDrivenContracts.html where Ian Robinson talks about the gritty issues involved in versioning services?

Sure. I am not dismissing the complexities of versioning mature services. The point of this series of posts is to get people to stop worrying about details like these until they have obtained a firm grasp of what it is like (and what it means) to build applications with services.

In true Agile sense, plan and build what you understand. Analyze and design what you can't get your head around yet later.

It is this engineering habit of wanting to have ALL the answers to ALL the questions that is killing efforts to change the way we do things in software development.
That's why Agile/XP is such a beautiful thing, because it does away with that notion and accepts that you CAN'T know all the questions up front, let alone have all the answers.

Many times have I come across budding SOA efforts that were mired in pointless analysis and design. "But we have to have our message headers defined!".

NO, NO, NO! What does a full definition of message headers teach you about making SO applications? Nothing! Where's the learning? Where's the difference? Where's the business value?

The point is to learn how to do things differently and discover how you and your team can benefit from it. The many issues that undoubtedly will arise are there to be SOLVED, not avoided. If early man had this attitude of wanting to have all the problems solved before embarking on something new, we'd still be living in caves, without fire, mind you.

What does the sentence: "With Web services, commercial registries can handle much of the versioning" actually mean? Are you suggesting that using a commercial registry to locate business-critical services is a viable approach? Even though the first I'll know about a (possibly incompatible) service upgrade is when the registry notifies me? At this point the upgrade has already happened which is far too late to do anything about it.

Commercial registries like Systinet have the concept of service release stages. For this you need to install more than one registry and add the newly versioned service to the non-production registry first. This very action will notify interested parties that a new version of a service is pending, so they can take action if required.

There are several sound strategies for introducing a new version of a service. One is to leave the previous version intact while the new one comes on-line and set a phase-out schedule.

Of course, this sort of approach requires that all parties have a means of communicating. And here now is the crux of a successful SOA implementation: it's the PEOPLE:

As per the Agile Alliance: (http://www.agilealliance.org/intro)

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

As you can see from Ian's article it's not enough to just append extra content on the end given that most of the tools people use to interact with web services insist on validating against the entire schema.

Sure, if you validate against an entire schema, you are shackling your consumer to a service, i.e. coupling it tightly. While schema validation has its place, if you are in an environment where message formats change frequently (e.g. in options trading) then that coupling is undesirable.

I'm not pro or con validation, I'm just suggesting to use common sense.

Moreover putting version numbers on each message sent is not in itself an acceptable versioning strategy unless we know that our consumers can actually handle the new version. For instance in an investment banking context if there were a service that published newly booked trades it would be unacceptable to silently upgrade that service. If business critical applications suddenly stopped working because of a silent upgrade it would not be enough for the service provider to point to the error queues full of messages in the new format. Each upgrade has to be verified against each consumer in some way because in practice the levels of coupling of between service provider and consumer are still just as high in an SOA. The difference is that the coupling is often implicit and only exists at runtime. This creates situations where there appears to be low-coupling at the architectural level but support staff have to manually resolve the high levels of coupling that exist at runtime in the production environment. This usually ends up taking the form of increased communication between the humans who support the service providers and consumers e.g. co-ordinated releases, staging areas, sandboxes for integration testing by replaying messages from the live system.

Again, use common sense.

Introduce a new version of a service in a staging environment
Release it to production on a different URL or topic, if you can
If new data is added to a message, put it somewhere so that it doesn't interfere with existing consumers
...and version numbers in messages DO work. As a consumer of somebody else's service, I know that I'm beholden to the format it sends its data. So if version numbers are available, it would behoove me to build my consumer to check for that and do something useful with it

Indeed, until we can write software that can figure out what to do with new data all by itself, we have to have proper human communication. This need not be heavy-weight and procedure-laden. But every time one human provides something to another human, the provider has to choose between either improving it and coordinating with the consumer, or leave it stagnant and risk the consumer walking away.

If you feel I'm being too light-hearted and cavalier about this, maybe you oughtta try that yourself. It's quite liberating, and very effective.

Practical AgileSOA - Versioning services

2006-07-13T09:37:00.000-07:00

In the previous post, we saw how by simply doing, we can get those first services built. This activity will answer a lot of questions, but also will show us which questions to ask next.

One of these is how to deal with service versions.

Undoubtedly, your services will undergo revisions and change.
That in and of its own is not a problem. The trouble happens when you change the interface definition and the functionality.
If the service starts behaving differently because

With push services (asynchronous) it is a good idea to include a version number in the messages it sends.
That way, a subscriber can be made aware that things have changed.
One can think of a subscriber routing incoming messages through a validation component if the expected message version does not match the incoming message version.

With Web services, commercial registries can handle much of the versioning.
Consumers can subscribe to registry updates and be rigged to take action when a service changes.

As a rule, you should be wary of deleting information from service outputs.
Not that you should keep everything around forever; just be aware that consumers are notified in some form.

In general, new content should be added in such a way that a consumer can continue to use the incoming data without knowing how to handle the new stuff. For instance, if you use a byte-encoded message, add new content at the end.

With self-describing message and XML messages, this restriction does not really apply, since you can just pull out what you're looking for, regardless of the extra content.

(See, even with standardized services, you are still not entirely decoupled, just loosely coupled.)

Next post: The emerging infrastructure

Practical AgileSOA - Making the first services

2006-07-13T09:35:00.000-07:00

The question that always comes up is:
"How do I know where to draw a service boundary?"

Answering that question requires some more investigation.

Begin by looking at the piece of functionality that the client wants to cast into a service.
Let's take a universal example: getCustomer. All companies have customers, and many companies have a multitude of customer data sources, be it through isolated IT projects or through Mergers and acquisitions.

Some questions that come up at this point are:

"What do I use as inputs?"
"Do I return the intersection or the union of all the data from the various customer systems?"
"What if some data exists in system A and not in B?"

We need to step back here and ask ourselves:" What is it that you want this service to DO?"
Well, I want it to return customer information.
OK, and what do you want to use as search criteria on our first attempt to make this service?
I want to use, uh, first, last and company name.
Very well. We now have something to start with.

"...but what about returning duplicate records, and mismatches, and... and ... and..."

Relax, it's Agile! We'll fold exception scenarios in later.

Let's now make this service.

First, we make a client that invokes the projected service. Do we want to use a SOAP stack? If you must, sure. If not, that's fine too. Suffice to say that this client will act as our service "test".

Now we make a simple service that just returns data from a preloaded file.
Then we expand it to invoke one of the legacy customer data sources.
We continue to expand this service by adding another legacy data source and work out how to consolidate duplicate information.

At this point we may find that submitting first, last and company name is not detailed enough, so we need to add a parameter to the request. In essence, our service has now changed its interface and therefore is a new version.

Next post: Versioning services

Practical AgileSOA - Intro

2006-07-13T09:30:00.000-07:00

Clients invariably ask me;

"How do I actually DO this SOA stuff? Concretely? What is my Step 2?"

In this series of posts I will detail how we at ThoughtWorks typically execute the various aspects of Agile SOA.
Let me start by saying that we are Agilists, which means that we do just about anything in continuous cycles of Design-Make-Test-Reflect.

Similarly, we get SOA efforts off the ground by simply doing, not by talking and designing ad infinitum (or ad nauseam, rather)

These are the efforts we want to cover:

Making those first services
Versioning services
How an infrastructure emerges
How to motivate reuse
Getting some governance started
Making it all stick

After you have read this series, you will be able to get off the mark and get some traction on SOA. Incidentally, you also may pick up some Agile Best Practices. After all, we think Agile and SOA are closely related.

Next post: Making the first services

Some Q&A on SOA

2006-04-05T09:46:00.000-07:00

Q: What are the differences in managing SOA applications compared to traditional applications?

A: SOA applications are comprised of component services that do not reside in one area of responsibility. So there have to be clear Service Level Agreements in place to make such an application manageable. Also, clear and unambiguous monitoring and error detection needs to be in place, lest service support groups pass the buck on where problems truly reside.

Q: What new technical challenges do these applications bring?

A: Many developers are not familiar with asychnronicity. Even though multithreading has been around for decades, not many truly have a multithreaded mind. Such asynchronous thinking is fundamental in understanding and managing SOA applications.

Also, the design of, and designing with services is quite different from building a traditional, monolithic application. In the latter, events happen in a predefined order: the program flow. With SOA applications, this order is not fixed. Therefore, a service must not rely on events to happen in a certain order for it to function properly, it must be idempotent.

Another challenge is the concept of marshalling and unmarshalling. Too easily is precious processing time wasted on inefficient serialization and deserialization schemes. Developers must now learn to think about how services can communicate most efficiently. And, unlike using APIs, that may not be by making frequent, granular calls, lugging large objects around.

Q: What are the challenges in troubleshooting performance problems with these applications?

A: Actually, diagnosing and remediating performance problems is much easier with SOA applications. Let’s assume that such an application is entirely made up of Web services (SOAP over HTTP). Each of these services can be invoked independently, in their deployment environment. Using simple test clients, performance problems can rapidly be narrowed down to specific services. The underperforming services can then be improved either through coding, faster hardware or load-balancing.

The real challenge with SOA apps is solving timing problems. This is especially true in asynchronous high-speed environments (thousands of messages per second). If services are not written to cope with differing sequences of events, they will yield different results with different sequences.

Q: How much work is involved in integrating management of these applications with traditional applications?

A: Technically, the amount of work depends on how application-management-savvy an organization is. If they already have outfitted their current applications with management agents, then doing so with services will feel quite natural. If not, then managing services will be a learning curve.

Organizationally, however, a transformation will be needed to build, support and use business-oriented services. The traditional silos of responsibility and accountability will no longer suffice and a much more collaborative environment will have to emerge. This requires unwavering attention and support for SOA from IT management and a modified

Q: What are some of the areas that companies tend to overlook?

A: Popular misconceptions about SOA are:

SOA means installing some web services
SOA is synchronous, request-reply
SOA is expensive to undertake
It takes a long time to see any ROI from an SOA effort
It’s a or a suite of products
It’s a turnkey solution
It’s just EAI
It’s a technology solution to a technology problem
Our organization doesn’t have to change to embrace SOA

From blob to services

2006-02-22T11:30:00.000-08:00

Many clients ask us what to do about their legacy applications.
Should they be converted to servicees ?
Where do you draw the boundaries ?
Do we need to rewrite our whole application?

In a nutshell, the boundary of a service lies where external knowledge of the inner workings ceases.
This means that a piece of functionality can become a service if a consumer can just ask it to do something or can simply consume its messages, without having to tell it how.

The best way to transform an application into services is to start teasing sections of it apart. The right boundaries will become apparent as you do this.
Take one section at a time, learn and apply that knowledge to the next section.

In the category " a picture speaks a thousand words"...

The Interplay of Agile and SOA

2006-02-10T09:23:00.000-08:00

Here you can see more concretely how Agile and SOA enable one another.

Start with a bit of Agile to get the first services out there.
Your first services will make you more Agile, which will enable you to take services to the next level, et cetera, et cetera.

Even the Army is getting it

2006-02-09T10:12:00.000-08:00

So you thought that the government uniformly was an old ddog that couldn't learn new tricks.

They seem to be getting ready to become more Agile.

Read this :

http://www.defensesystems.com/issues/1_1/beyond_the_dog_tag/25-1.html

The Symbiosis of Agile and SOA

2006-02-09T08:12:00.000-08:00

This is the symbiosis of Agile and SOA.

Start small.
Get solid, test-driven functionality out there and prove the skeptics wrong.
This way your will get your first services deployed at very low risk.

It's hard to argue with success.

A spark of Agile can bring an SOA to life

2006-02-09T08:03:00.000-08:00

Some parts of Agile practices are needed to get the first bits of an SOA (a service) off the ground. This manifestation will prove to skeptics that deploying a small piece of business functionality that actually delivers business value (Agile) is possible in the most adverse of circumstances. It proves that the oft-heard cop-out “that will never work here” is bunk.

That first service can then motivate others to make their own services. Providing a registry where the uninitiated can find existing services will then spark a trend where people not on the bleeding edge can adopt services. Note that such a registry initialy can be just web pages.

At this time, introducing a little bit of governance, i.e. some approval process as to which services are listed in the registry, will provide some level of standards.

Adding a services monitoring tool to the infrastructure will validate the use of certain services. If these metrics are available in the registry, then prospective users can see whether a service is actually used and is actually performing within desired specs. This will then spur reuse, which will allow for iterative improvements and additions.

SOA and Agile are locked in a virtuous circle that creates a helix of progress. each stage of SOA maturity will allow for a new phase of Agile maturity.

SOA public sightings

2005-11-05T19:46:00.000-08:00

It pays to be an SOA Practice Leader at ThoughtWorks.
We're getting around these days !

Presenting at SOA Leaders in New York October 27th.
http://events.soaleaders.org/events/register.cfm?location=4

And as a panelist at InfoWorld's SOA Executive Forum.
http://www.infoworld.com/event/soa/schedule.html

Killer SOA definition

2005-09-03T13:37:00.000-07:00

By popular request, here is my killer SOA definition.

"A service-oriented architecture (SOA) is the organizational and technical framework that enables an enterprise to deliver self-describing, platform-independent business functionality and make it available as building blocks of current and future applications."

Put this in front of your execs and they will drool. (Guaranteed)

In the New York Times !

2004-10-04T14:18:00.000-07:00

Was interviewed for the New York Times about blogs. Is this one of those self-references, like rock songs about rock music ?

See here:

http://www.nytimes.com/2004/10/03/jobs/03BLOG.html?oref=login

Week 13 - After the refocus

2004-09-21T18:51:00.000-07:00

This is a bit out of sync, but Iwill fill in the blanks later.

Basically, we have refocused ou refforts to rebuild the core of the customer's code.

So now we get to do real Agile:

Write stories
Spec them out
Write tests
Write code
Next

It's beautiful !

We developed some nitfy Excel sheets to do the story tracking. It recalibrates itself and predicts how long related stories take as actual durations are being entered.

Haven't forgotten about EAI, tho.

Seems like there are some open source projects going on that may be useful:

How about OpenAdapter ? Dresdner Bank spun this off from their in-house EAI platform.
http://www.openadaptor.org/adopters.html

And then there's MuleUmo. It adopted the pattern language in Gregor Hohpe's book (http://www.eaipatterns.com/)to describe their solution.
http://wiki.muleumo.org/display/MULE/Architecture+Guide

Week 8 - Back at the ranch

2004-08-20T15:02:00.000-07:00

This week, we're slummin' at the home office.
Doing more code investigation and prepping for the project kick-off.

Looks like we have to dramatically cut scope, since what looked like a 3 month project likely will take more than a year. Incidentally, the client seemed not at all surprised !

We can automate a lot of the gruntwork, but that sort of defies the secondary aim of the project, which is teaching the client how to start developing "agily".

Reading Jim Highsmith's book Agile "Project Management"
(see http://www.jimhighsmith.com/pubs.html)
Salient detail #1: my colleague Robin Gibson collaborated on that, you know.
Salient detail #2: our CEO, Roy Singham, wrote a comment that's printed in and don the book.

And who joins us for lunch at the office ?
By Toutatis, it's mr Highsmith himself !

I better get to finishing the book before Monday...

Week 7 - Finishing the preparations

2004-08-09T10:14:00.000-07:00

The client has a varied codebase, with servlets and JSPs.

Some of these are using struts.

What are struts ?

http://struts.apache.org/

Apparently, this is a way to abstract model from view better than ordinary JSPs do.

A note about a cool MS Excel trick:

If you want to chart a bunch of data points that do not form a smooth line, don't knock yourself out trying to deduce the 3rd order polynomial !

Select the data range and make an XY scatter chart, maybe even select the subtype with smoothed lines. Now select one of the datapoints in the chart and right-click.

Choose "Add Trendline..." and select one of the types that looks closest to your data.

Select the Options tab and switch on "Display equation on chart".

Now you have your polynomial and can eat it too. Select the text of the equation and paste it into the cells where you want to get missing data. Modify it to include your source cells and there you go.

Week 6 - The knowledge grows

2004-08-02T22:07:00.000-07:00

My colleagues are launching new stuff at me at such velocity, I can hardly keep up.
Wow, I though I knew a thing or two about software, but I guess I've been asleep for the last two years.

A few notes on some very useful tools to better estimate project duration.

Learn about Wideband Delphi
http://www.processimpact.com/articles/delphi.html

Tom DeMarco and Timothy Lister wrote a book about project management from a
risk management perspective, "Waltzing with bears".
http://www.dorsethouse.com/books/waltz.html

If you want to jump into the math, look here:
http://mathworld.wolfram.com/RayleighDistribution.html

Test code coverage can be done using Clover or JCoverage or EMMA.
Here's a user's testimonial on all three.

Week 5 - Goin' Deep

2004-07-26T22:01:00.000-07:00

I'm learning all about Agile Projeect Management.

There is a good body of work out there on agile PM.

"Crossing the Chasm” by Geoffrey Moore

Kepner & Tregoe’s analysis methodology - see
http://www.kepner-tregoe.com

“Radical Project Management” by Rob Thomsett

Jim Highsmith is one of the authorities on the topic.

Also learning about test-driven development.

When writing a JUnit test, sometimes you have eto stub out objects. This can be done using the Mock object found in jmock:

See www.jmock.org

Interlude

2004-07-25T21:54:00.000-07:00

Interlude

Enjoying the weekend in northern California with a visit to Santa Cruz and Bonny Doon
Vineyards.

Just the drive is worth it, through beautiful redwood forests, over hills and
through valleys.

If you get on Highway 17 to Santa Cruz, you get off at Mt Hermon Road. Turn
on Graham Hill, which becomes Felton Empire, which becomes, believe it or not,
Ice Cream Grade. Turn left in Pine and keep going for a while.

The Bonny Doon Tasting house is an number 2.

When you're nicely liquored up, you turn left out of the BD parking and
continue on to Highway 1.

Right there is a small parking lot. Stop and cross the traintracks along the
highway and look down on a fantastic vista.

Continue down Highway 1 into Santa Cruz.

Follow the signs for Beach and Wharf.

http://www.santacruz.com/

http://www.bonnydoonvineyard.com/

Week 4 - First real gig

2004-07-23T20:09:00.000-07:00

This is the first real gig at a client site.
We're flying to California on an early evening flight.
It turns into a late-evening flight as the plane arrives late and the crew even later.

The cute girl sitting next to me asks me what I'm going to do in California.
I say: "Agile enablement".
She: "Oh, like ThoughtWorks ?"
Me: "No, FOR ThoughtWorks."

And now I'm a hero. She's wearing a shirt that says: "I Nerds"/

By the time I pick my car up, drive to the hotel (miss the @$#@%! exit to the highway) and check in, it's 3:00 AM.
Needless to say, I'm completely fried and hit the hay.

The next morning, I'm having breakfast with my two team mates, one of which turns out to be major ace in the Agile PM field.See here:

http://www.softed.com/sdc2004/robin.html

The client has an organically grown people tracking system that is so complex that nobody dares to make any changes to it.

Enter the knights of Agile. (Pa-Pada-Paaa !)

We're teaching the client that you can indeed bite off little bits at a time, focus on just that delivery and acctually complete something.
So here we're introducing things like test driven development and continuous integration.

Look for books by Jim Highsmith and Alistair Cockburn on Agile Project management.There is a whole series of them by various authors.

Beware of the hucksters and carpet-baggers out there that picked up on the Agile craze, but failed to do their homework.

For most people that have been around for a while, Agile is realy hard to grasp: they have to learn how to talk about projects in a whole new fashion.
What's insane about this, is that an Agile approach to projects comes naturally to people, but because of it's simplicity and "easy fit', people think that it's too "organic".
Many an IT person feels uncconfortable dealing with projects is such an ordinary fashion: there need to be big words, thick documents and much ado about nothing.

Some fun little nuggets that came my way as I'm analyzing the client's codebase.

The obvious Unix tools find/sed/awk/grep can give you a very quick insight into the complexity and organization of a codebase, but these tools up the ante quite a bit by doing a lot more in-depth analysis on class and pacckage interrelationships.

One of these tools even detects whether code was copied and pasted...

Simian (http://www.redhillconsulting.com.au/products/simian) Code duplication detector that finds copy-and-paste reuse in code.
Checkstyle (http://www.sf.net/checkstyle). Does a lot more than simple tab-and-braces checking
Pasta - the Package Structure Analysis Tool (http://javacentral.compuware.com/pasta/) Finds cyclinc dependencies and much more nasty stuff. Not free :,v(
yDoc (http://www.yworks.com/) javadoc plugin that generates UML into your javadoc files. Not free, but cheap.

Oh yeah, we also coined a new term: the Onion diagram.

This thing shows how phases of a project build on each other. E.g. the core is the very first thing you need to do.This sets the stage for the next layer to be started and completed. Each layer needs to have a significant part of the previous layer completed in order to start.

We did a presentation on Agile for the client department today.

Much scepticism. How can something so simple solve our orgganizational problems ?

It's called Agile project management and proper governance, baby...

Week 3 - It's getting interesting

2004-07-16T19:36:00.000-07:00

We're working on a very interesting proposal.
A client is concerned that the corporate IT approach to EAI is flawed and wants us, as an independent, objective, expert party, to prove him right.
So we have to prove a negative.
After several iterations we finally have a presentation ready for him that's just rife with a salesman's version of Agile.

What I take away from it though is the following:

There are three different Integration Models:

Multi-step processWorkflow - Transactions or activities need to be orchestrated between various applications
Composite applicationPortal - Provide single application into a range of back-end systems
Data integrationCommon Data Layer – Data from incompatible systems are transformed into a common format

And there are four Integration Styles

Batch Data ExchangeExport/import data via flat files, aka ETL
Central DatabaseMake all applications access a common database
Remote Procedure CallsApplications call another directly or indirectly
MessagingPropagate documents and events via a common bus

Styles and models can be mixed to suit specific requirements.
Just to give you another push for Agile development (as if you needed it...)

Agile practices and benefits

Manageability of project investment

Visibility of project progress
Adaptive planning
Incremental delivery

Test-driven development

Regression basis
Design evolution

Close Customer Collaboration

Create what customer really needs
Be able to respond to shifting priorities

This wealth of knowledge is gleaned from presentations that our main man Gregor Hohpe and others have given at the various conferences.

At another client, there is an issue with the use of MS BizTalk.

A lot of manual work is going into mapping data from legacy systems to a generic API. Strangely enough, they do this ad-hoc and have a cluster of people field data mapping calls from users.
Sounds like they need themselves some SOA...

Remember the client with the TIBCO InConcert/Integration Manager throughput problems ?We were right in our assumptions that the root cause of their workflow pileups lies in SQLServer locking problems.

But it is more pervasive and conniving than we expected. Get this:

The TIBCO tools create global temporary tables in SQL Server to store workflow data.These tables are not session-related or carry some kind of session ID. So when there are a lot of workflows being created and managed, they deadlock on the temp tables !
Using DBCC TRACE, you can quickly determine who deadlocks what.You can also use the Enterprise Manager tool to configure tracing options for rthe entire server instance.

Guess TIBCO has some "esplainin" to do...

See here for some info on MS SQL Server trace flags:

http://www.databasejournal.com/features/mssql/article.php/1459091
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/tsqlref/ts_ta-tz_646r.asp

Rumour has it that I need to go and do some Agile enablement in California...

Week 2

2004-07-09T12:51:00.000-07:00

Ah, we survived week 1 and 2.
Already been talking to several prospects and catching up on current technology.

I have learned more new stuff in the last week than in the two years preceding. NASTY !

Already poaching ex-colleagues to come and work here.

I mean, what's more beautiful than consulting, right ?
(voice-over mutters: "...that's what I thought then...")

It appears that I have entered a very buzzword-heavy part of fthe industry. Not since the advent of GUIs has any IT branch been more rife with TLA's than EAI.

Found a possibly useful EAI forum site:

http://groups.ittoolbox.com/

They have newsletters with news categorized for talkers or doers.
Are they trying to imply that architects are just EAI bullshitters ???

Anyway, learned that IBM's fantastic WebSphere Business Integrator bundle is going to be replaced with something more integrated.

In the mean time, seems that there is a lot more TIBCO stuff to go around than just financial services. Pretty neat...

A final note: there is a rumbling in the Agile community about using XP for GUI design. Is there any other way ?
A new book on the topic is due out soon, so more on that later.

Learning all about AppServers, servlets, EARs, WARs and SOAPing it up in general.

For good measure, some more links:
http://www.testdriven.com
http://www.xprogramming.com/

The Beginning - week 1

2004-07-02T13:42:00.000-07:00

Well, we're off here at ThoughtWorks and they're throwing me in at the deep end.

After a week of orientation ("dude, you're, like, SO overdressed, dude !") we're starting on our first project.

Quite a change from my previous engagement, where learning external products was highly frowned upon.

The first project involves me learning about WebSphere Business Integrator.
Maybe I should first learn WebSphere...

I'm in the Enterprise Application Integration business, so it would behoove me to post a couple of useful links.

Our main man Gregor Hohpe's site:
http://eaipatterns.com/

Greg Wdowiak's collections and observations:
http://eaiblueprint.com

A PDF on IBM's WebSphere Business Integrator
PDF