Introduction

• High complexity and short time to market, leading to an increased risk of failure
• Low tolerance for failure because of high costs and difficulty quantifying return on investment (ROI) associated with the SOA project
• Dynamic requirements and business needs that need proper management

So, what do you do to overcome these problems? You need an adaptable software methodology -- a systematic approach to the way software is developed -- that can provide rigid controls and processes during the initial SOA rollout but also be adaptable to change and streamlining as you move toward an optimized SOA. Before you consider the software methodology outlined here, you should read the Service-oriented Architecture Maturity Model. Organizations at Level 3 or 4 of this model should use formal software methodologies such as RUP for development; but after they get to Level 5, they should use an agile methodology such as XP.

But how can your enterprise make the transition from one level to another while still maintaining a coherent development process? In this metholodologyI propose a best-of-breed hybrid methodology for building and then continually optimizing an SOA. I've nicknamed this methodology the service-oriented unified process, or SOUP. Before you proceed, you should have a basic understanding of how both RUP and XP work.

What is SOUP?

The major driving factors of a typical software development project are the application development process, project management, and the technologies used. These are all driven by the specific consultants assigned to the project. A generic software development project has four variables: time, budget, scope, and quality. Compromising on any one variable has an impact on the overall project. Because of changing business needs, scope and quality are the two most difficult factors to manage. Technological complexities can lead to problems with time and budget management, as well.

SOA projects are significantly more complex than typical software projects, because they require a larger, cross-functional team along with correspondingly more complex inter-team communication and logistics. Typically, an SOA is not well defined, and the vision for the final result is thus often not clear at the project's inception. An SOA can have a very positive impact on an organization, but it comes with a potentially high price tag. Moreover, an SOA project has a higher risk of failure than other projects. The key factors that can help an SOA rollout succeed are a clear definition of the development process, enhanced lines of communication across project teams that know the business, and clear support and governance policies.

This is a lot of baggage, and during the initial development process there's a need for a formal software methodology that addresses the risks identified above. However, after your SOA project is successfully established, continuing the use of this formal methodology for maintenance and incremental development will reduce the true ROI that you can get on your projects. Nevertheless, an agile methodology like XP might not be formal enough. The SOUP methodology helps address the risks of your SOA rollout but also sets you on a path for continuous SOA optimization.

SOUP is a six-phase approach to software development. Each of these phases represents a distinct set of activities critical to the success of an SOA rollout. Of course, as in any project, you need to tailor the process to your own organization.

Figure 1 shows the SOUP process. There is nothing particularly radical about it: All software or SOA projects are defined in a similar way.

Figure 1. The SOUP Process model

I break the use of the SOUP process into two categories.

• SOUP for initial SOA deployment
• SOUP for ongoing SOA management

SOUP for initial SOA deployment

Figure 2. SOUP and RUP model

Now let's look at each phase in more detail.

Incept
In this phase, you establish the need for an SOA project in your organization. You can apply the SOA maturity model to determine your organization's level of architectural maturity and determine the basic driving factors towards an SOA. In addition, at this point you would explain the basics of the SOA concept to all project teams serving the business and plan a strategy for a feedback and suggestion process.

The key deliverables of this phase are:

• SOA vision and scope: This is a document that outlines the overall vision of the project. It also provides some boundaries that establish the project's scope, which should include at least a couple of lines of business or projects.
• SOA strategy: This is a high-level plan explaining how the project will be executed.
• ROI analysis: This document outlines all the costs and savings that this project will bring.
• Communications plan: This document explains how the SOA team will communicate with other project teams and business users.

Clients (the business stakeholders) do not always fully understand the benefits of new software products in their business. The enterprise teams that define the SOA strategy need to leverage the domain experience of the project teams to help identify business problems and ways and means to streamline operations.

Cross-functional analysts and project managers analyze the business of the clients to determine the advantages of an SOA-based solution. The key factors analysts examine are the clients' internal operations; interactions with their partners, suppliers, and consumers; and their overall business model. These help the analysts develop and recommend an SOA strategy.

At this stage, you also need to do a full ROI analysis of the SOA strategy that you would recommend that the client buy or have built. This analysis would clearly show the cost benefits to the organization -- in the short, medium, and long term.

In my mind, the single most important deliverable at this stage is the communications plan. Ultimately, project IT teams and the business stakeholders know the business better than the architecture teams and analysts. The communications plan ensures that these internal stakeholders understand and are involved in the process.

Define
This phase is by far the most critical of the SOA project. Business and project team involvement here ultimately determines the project's success. Team members need to be committed to participating in the definition the requirements and use cases developed as part of the initial SOA rollout.

This phase of a project lifecycle involves such activities as:

• Requirements gathering
• Requirements analysis
• Nonfunctional requirements definition
• Creating a project plan with timelines and estimates
• Technical infrastructure definition
• Use case definition and realization
• Overall architecture definition and documentation

The key deliverables of this phase are:

• Functional requirements document: This is a detailed description of all the different business processes that the SOA will surface as business services.
• Nonfunctional requirements document: The requirements described here would include performance considerations, service-level agreements (SLAs), infrastructure requirements, and so on.
• Use cases and use case realizations: These include detailed use cases for all the business services that are being built.
• SOA architecture document: This is a description of the overall architecture of the project, including hardware and software components.
• SOA applicability document: This document explains what projects fall within the scope of the SOA project and how ongoing projects can build on top of the SOA.
• Infrastructure definition document: This document includes detailed infrastructure deployment diagrams, outlining the servers, and the locations of and connections between the servers necessary to implement the SOA.
• Project plan: This detailed plan for the whole project includes milestones and dependencies.
• Support and governance model: This is a description of how the SOA will be supported and governed. It includes considerations such as SLA monitoring and management.

Various studies (see Resources for links) have shown that requirements-related issues are the number one reason for the failure of software projects; SOA projects are no different. Software quality is sometimes defined as software's conformance to its requirements. However, the quality of software is probably better measured by the quality of the requirements it meets, not just the number of requirements it meets.

As the descriptions of the deliverables should make clear, the SOUP methodology recommends a formal requirements-gathering and management process. XP does not really define a formal process to gather and document requirements; instead, XP developers create user stories (see Resources for more on these) and try to put these stories in a logical order to provide a sequence to requirements. For this stage of development, such a process is inadequate. In RUP, the process is more detailed and uses more formal techniques to ensure that requirements sessions are complete, documented, and validated. I recommend the use of a requirements repository that will ultimately tie into use cases, design, and code, providing traceability through the project. This repository could be the IBM Rational Requisite Pro tool, or any other Requirements Management tool that you organization chooses to use. A change management process is also important.

In my opinion, the most important deliverable in this stage is the support and governance model. How will the organization support the SOA? What are the governance guidelines? At the end of the day, if you roll out an SOA and no one uses it, the project is a failure, but this document can prevent that from happening.

It is important that the business start the project with a realistic set of plans and objectives. Extra or superfluous requirements should be left for later. An effective business-to-IT (B2IT) collaboration on requirements that are achievable for the expected time and budget can set up the project for success from the beginning.

You define the technical infrastructure required to support the entire SOA early in the life of the project. This includes detailed estimates on servers, network infrastructure, training requirements, and so on. A realistic and pragmatic approach to planning will always be successful.

Design
In this phase, you elaborate on the design artifacts from the Define phase. The use case realizations and the software architecture document translate into detailed design documents.

At this time, SOA architects (typically a subset of the enterprise architects) need to involve project architects to make sure the design presented by the SOA team will work for specific projects. The SOA architects might even choose to do a couple small proof-of-concept projects to prove the SOA vision. The key deliverables of this phase are:

• Detailed design document: This document highlights how the services are designed and built.
• Application programming model: This document provides guidelines on how the development will be structured. The topics covered include the process and technologies being used, coding standards, deployment procedures, and so on.
• Database model: This includes an entity relationship diagram for the databases the SOA uses.
• Testing and QA plan: This plan includes details of the testing and QA plans and will include test cases.

Incept
During the Construct phase, you can basically use any iterative build methodology you like to build your SOA. This phase involves new development as well as integration activities. Your activities are not limited to software but can also involve infrastructure-related subprojects, such as hardware consolidation projects or efforts to centralize hosting of servers. The overall effort is managed in a single SOA project plan, but in all probability various small sub-teams will be working on different construction activities.

This phase of a project lifecycle involves such activities as:

• Iterative development
• Iterative QA and testing
• User documentation

The key deliverables of this phase are:

• Code base: The code should be stored in some kind of source control repository.
• Test results: The results from the execution of the test cases and QA plans should be available for examination.
• Documentation: Documentation should include detailed documentation on the code and any updates to the design documents.

Deploy
In this phase, you actually deploy your SOA. You roll it out to individual project teams, who start using the SOA on their projects in a production setting. Perhaps the most obvious key deliverable for this phase is your application in production. However, even more important is the plan that establishes the projects that will pilot the SOA. This leads into the next step of the SOUP methodology, which determines how ongoing projects will use the new architecture. The other key deliverables of this phase are:

• Deployment model: This outlines the overall deployment structure for the SOA.
• Usage model: This provides guidelines on how to use the SOA. This model becomes important as various project teams and lines of business begin to use the new architecture.
• Ongoing support levels model: This model codifies any updates to the support and governance model that was developed in the Define phase.

Support
Although this is the final step of the software development cycle, it is nevertheless a very important phase. In this phase, you ensure ongoing SOA support, bug fixes, and new functionality development. This phase of a project lifecycle involves such activities as:

• Maintenance
• Bug fixes
• Training
• Continuous project buy-in

At this point, your SOA is in production. But how do your project teams benefit from the architecture? Do they now need to follow all the steps described above in detail when building applications that consume existing services or when designing new services that make use of an existing SOA? As you'll see in the next section, they can in fact benefit from a more lightweight methodology.

SOUP for ongoing SOA projects

Figure 3. Overlaid SOUP and XP processes

What is the key premise here? The value of the SOA project is that it provides a well-defined architecture and firm technology guidelines. You have a framework on which to build your applications as services. The project teams' main tasks are building and consuming services. They are most fit to perform these tasks because they have the necessary business domain knowledge. However, they do not need to worry about the technical architecture or invention of technology, as that is left to the SOA teams.

Projects of this type follow the same SOUP guidelines outlined in the last section; however, as you'll see, each phase is significantly reduced. Many of the deliverables I describe in this section are the same as those described previously; when necessary, I explain how they differ from their counterparts in a project where an SOA is built from scratch.

Incept
When you've got an SOA up and running, there's still development work to be done. You'll no doubt want to build projects that consume existing services or expose new ones. In the Incept phase of such a project, you establish the project's overall scope and vision. The key deliverables of this phase are:

• Project vision and scope: This would encompass the scope of an individual project, not the larger SOA initiative.
• Project strategy: This document describes the strategy and process for this specific project and explains how it leverages the SOA.
• ROI analysis: A detailed cost-benefit analysis for the project is crucial to help make decisions about implementation.

Define
In this phase, you build the tie-ins to the SOA project and understand how to leverage the SOA. You need to identify the services that the SOA already makes available and those that still need to be built to meet the goals of your project.

This phase of a project's lifecycle involves such activities as:

• Requirements gathering
• Services identification
• Establishing a project plan with timelines and estimates
• Test case development

The key deliverables of this phase are:

• Functional requirements document: This is a typical requirements document for a software project.
• Use cases and use case realizations: These cover the new business processes being built.
• SOA applicability document: This outlines how the existing SOA framework applies to this project.
• Project plan: This detailed plan for the whole project includes milestones and dependencies.
• Services strategy: This document identifies services that already exist and can be consumed along with new services that can be exposed.
• Test plan: This plan includes test cases for the new project.

The requirements gathering in this phase does not need to follow the formal standards that were followed during SOA rollout. Instead, you can make use of agile requirements gathering involving user stories or Class, Responsibilities, Collaborator (CRC) cards (see Resources for more on these). In line with XP thinking, test cases are built early on in this type of project.

Design
This phase is really quick, because most of the design was completed in the initial SOA rollout. In this phase, you only have to worry about how to use the services, what more needs to be built on top of the existing services, and what new services need to be built.

The key deliverables of this phase are:

• Detailed design document. This document explains how your design leverages the overall SOA and establishes a design and application programming model.
• Database model. This document encapsulates a logical and physical database design for this specific project.

Construct
This phase involves more assembly than new development. As more services become available, each project will have more to reuse and less to build. XP's iterative development techniques are ideal at this stage of development. In XP, iterative cycles are theoretically set at two weeks, which should be more than ample time for one development-QA cycle when building a small service or consuming a set of services in an SOA environment. With this iterative approach, an SOA can provide valuable business agility via quick software releases.

This phase of a project lifecycle involves such activities as:

• Iterative development
• Iterative QA and testing
• User documentation

The key deliverables of this phase are:

• New services: Any new services that are being exposed will be ready by the end of this phase.
• Test results: Results from the execution of the test cases should be available to all interested parties.
• Documentation: This phase of development should produce detailed code documentation along with any necessary updates to other documents created previously.

Deploy
In this phase, you either launch an application that consumes several services, or you launch new services. The infrastructure is managed by the SOA team; thus, this process is relatively straightforward.

Support
In this phase, you are only providing support for your new services. In doing so, you will follow the support guidelines laid down during the original SOA project; thus, individual projects don't need to spend much time developing new support guidelines.

Managing the SOA lifecycle