Enterprise Initiatives

The promise of SOA is flexibility and agility. I define agility as the ability to adapt to change at the speed of business. In today's global economy which has been fueled by collaboration and Internet technologies, businesses change at a much faster rate than ever before. So how does SOA help companies become agile? Let's look at a logical view of a typical SOA.



You can see in this diagram how each layer of the architecture has been abstracted and is mutually exclusive or "loosely coupled" from the other layers of the architecture. Why is this important? The answer is simple.....ease of change! Here are some advantages of this approach.

• Share services, components, rules, etc. across the enterprise (reuse)


• Isolate changes and reduce dependencies (speed to market)


• Minimize impact of business changes (speed to market)


• Easier to maintain (maintainability)

Let me give an example of how this approach helps companies become agile.

Use Case: New customer data from a new client


Let's say your company provides a service for customers in the retail industry. You have a website that offers online services for consumers and your white label solution is tailored to look like it is hosted by the individual retailers. The problem is that each retailer has their own customer database. In the past you would have to write a ton of code for each retailer that you signed up to use your services. With SOA, now it is a simple data mapping exercise. By abstracting data services, the other layers of the architecture use the logical representation of customer, not the physical implementation. Behind the scenes, the data services layer is translating the request for customer data from logical view to physical implementation. So when you bring on new clients, you simply use a tool in the data services layer to map the new clients customer data to the standard logical definition as defined by the architecture.



You can see from this example that all three retailers have an entirely different implementation of their customer database including different naming conventions and even different attributes. In the data services layer, you can map all of these physical implementations to one standard customer definition. You can also see how the business processes all use the logical view of customer. This allows us to add and change customer definitions on the back end without changing code on the front end. If two more retailers were to sign up tomorrow, we can map their definitions to the logical customer view and be done. No Code!!! That is agile!

I will give examples of agility within the other layers in future posts. If you would like help establishing a data services strategy, give me a shout!

I have discussed the role of the SOA Evangelist and the importance of the architects. Today I will discuss the different developer roles and the characteristics that make a successful SOA developer.

In a traditional 3-tier architecture it was common that there was a presentation layer, a middle tier or application layer, and a data layer. In some cases, a single developer role was responsible for all three layers. In larger companies, there may have been specialized roles such as a UI developer, an application developer, and a database developer. In an SOA, the concept of an application is no longer relevant except from the standpoint of integrating to existing applications. With SOA, we should be building business services that are application independent. The following chart is one I have used in previous posts which shows a typical breakdown of some of the development roles required for SOA.

From SOA Slides

So let's talk about business services. A business service is the compilation of all of the work done by the collection of developers within each layer. For example, let's discuss a "Shopping Cart" business service like the one we use on Amazon.com. It is most likely made up of services and/or components that live within each of the various layers of the architecture. The presentation layer obviously contains the physical implementation which the user sees and interacts within the browser. The business process layer contains the logical flow of steps that walk the user through the checkout process from start to finish. The business rule layer contains rules about tax codes, discounts, memberships, etc. and the underlying data elements and structures are handled in the data layer. In many instances, companies have multiple data structures that serve similar functions due to mergers, acquisitions, years of legacy systems, 3rd party application purchases, and many other reasons. To abstract these different data structures and present a common data interface that hides the complexity of the physical implementation of the data, the data services layer exists (think master data management).

So to develop this shopping cart business service, all of the people working within the different layers of the architecture must collaborate to deliver the service in a way that it both meets the business requirements and the technical requirements as defined by the SOA Governance model that the company has adopted. Examples of the technical requirements might be...

• Adheres to WS-* security standards
• Data encryption policies
• Platform neutral implementation
• Meets specific performance requirements

Why do I bring all of this up? Because developers that will succeed developing in a service-oriented architecture need to have the following characteristics:

• Flexible, open to change
• Collaborative
• Able to work in a setting with peers reviewing their work
• Able to see the big picture
  
• Not married to a certain technology
• Able to withstand constructive criticism
• Innovative

Developers who are protective of their own work or of their technology of choice will likely struggle working on SOA. One of the goals of SOA is to build platform neutral software that is highly flexible, maintainable, and loosely coupled. To build software like this we must shift away from software developing and move to software engineering. In simple terms, we must move from drag and dropping to planning and modeling. We must embrace collaboration, peer reviews, and governance. If your developers don't like these requirements, they either need to change or leave. Otherwise they will be a cancer and will continously feed the forces of resistance.

Ok, off my soap box now. So let's discuss the roles. But before we do let me note that we are talking about roles and not individuals. In smaller companies, a developer may perform multiple roles. In larger companies we may actually see very specialized developers living within a single layer of the architecture. One last disclaimer: I am assuming that an architecture team exists and some form of standards and governance is in place within each layer.

UI developers
If your company can afford to specialize, this is the perfect layer for it. Developers in this layer do not necessarily have to be extremely technical. It is more important that they understand usability, UI standards, and web interface best practices. These people should be comfortable working in prototype mode where they can start with mockups and work closely with the business or business analyst to reiterate through the various visuals until an agreed upon format is approved. These developers must be able to talk to the business in business terms and understand how typical business users interact with web technology.

Business Process developers
There are two distinct roles in this business process layer. One role deals with modeling business processes, the other role deals with integrating business processes with the underlying services and systems. In some companies this role may be served by one person but more likely it will be two because of the difference in the skills required. The business process modeler may not even be an IT person. Some companies have a department that exists within the business that focuses on business process improvement and business process automation. This is the norm for companies that practice Six Sigma or Total Quality Management (TQM).

The business process integrator is a technical role which requires expert experience working with Web services, REST, JMS queues or several other technical areas of expertise. The integrator is the person who connects business processes to the backend technologies which controls the overall flow of the business services and overall composite business application, often coined orchestration.

Business Rules developer
This one is a little fuzzier. Not all architectures have a distinct business rule layer. In some instances, the business rules are controlled in the data layer. For companies who have very dynamic business rules, especially rules that are customer centric and could be updated by end users and even customers, it makes a great deal of sense to abstract the business rules. If a company has a business rule layer and implemented a tool to manage business rules, it is highly likely that you will need technicians who govern this layer much like DBAs govern the database layer. In some case this might be an additional role for a DBA but it does not have to be. Whoever it is they must understand the concepts of abstraction and be looking for ways to empower the end user to react quicker to changes in the business. For example, let's say that a loan approval process requires state specific business rules about what percentage of capital is required from a loan applicant up front before the loan application can be expected. The frequency which this value changes varies by state and must be updated as soon as possible. The best scenario is to take IT out of the process and enable certain users (or systems) with the appropriate authority to update the values as necessary. Whoever is responsible for the business rules must be able to work with the business and/or business analysts to understand the frequency of change, the rights and permissions, and the impacts of the various business rules. There is an amount of overhead associated with creating and managing rules and the person in charge must understand the trade offs.

Data Services developer
Maybe a better name for this person is an information architect. The person in charge of this function is responsible for abstracting the underlying data layer and exposing it the other layers of the architecture and even externally to other systems. For example, let's say that our shopping cart business service allows multiple forms of payment (American Express, Visa, and Pay Pal). In addition, the different products purchased (books, DVDs, clothing, etc.) are managed from different inventory systems. To hide that complexity and to allow us to add additional payment services and inventories with ease, we leverage data services to present the shopping cart service with a logical view of the data. In other words, we create a standard payment and inventory message that the shopping cart communicates with. As long as the shopping cart communicates with these standard messages, the underlying physical implementation of the different payment and inventory services are irrelevant. The shopping cart service simply communicates these messages in the standard format it knows and the translation from the standard message to the physical layout of the receiving systems or services is performed in the data services layer.

From SOA Slides

(This slide is from BEA)

Developers or architects within this layer must have exceptional technical knowledge in the area of data modeling and database design. If the company has a tool to manage this layer (which is highly recommended) then an administrator is also needed.

Database developers
This data layer is typically an area that already exists today within each enterprise. This is where the DBAs live. With SOA, the DBAs must work extremely close with the developers within the other layers of the architecture. They must understand the performance and security requirements of each layer and ensure that the underlying data model meets the needs. Since legacy integration is common in many SOA implementations, DBAs are often called upon to create views or structures or even establish new ETL processes to provide a common view of data to expose to the other layers of the architecture.

Security developers
The security experts may not actually do the development but there must be a person or group of people who understand the new security challenges that SOA introduces. SOA can introduce the following new threats:

• Expose components of legacy systems that were never designed to live outside the firewall
• Require trusts between multiple internal/external systems without additional sign-ons
• Send a single message to multiple partners each with their own distinct security requirements
• Expose credit card and other privacy related data out on the web

From SOA Slides

(This slide is borrowed from the following presentation from Mamoon Yunus of Forum Systems)

Todays n-tier security models are not enough when dealing with B2B type SOA implementations. The developers in this area must have a deep understanding of communication protocols, security best practices, web architecture, regulation (ie. HIPAA, SOX, PCI) and WS-* or other standards.

Summary
As you can see there are many distinct developer skill sets required to implement SOA properly. Trying to find a person that can master each layer is not realistic. If you have this person, it is likely that he or she is on your architecture team. The beauty of this architecture is the ability to divide and conquer the problem within each layer. A single business service can be worked on by numerous people within each layer. It requires solid governance, strong collaboration skills, and teamwork to work this way which is why it is critical that the right team of people is assembled to tackle SOA. My next post will discuss the testers and their relationship with the development team.

One of the most critical tasks that the champion of your SOA initiative must perform is to quantify the total cost of ownership (TCO) of the SOA initiative. Some executives might think that once they buy the ESB they are done. This is far from the truth. Here is a list of items you should plan for up front and create a road map for how much and when each of these items need to be procured:

• The Stack (BPMS, ESB, Data Services, Portal, etc.)
• Testing tools
• Training, training, and more training
• Professional services from the Stack vendor(s)
• Professional services from the implementation partners
• Conferences, site visits, partner meetings, and other learning opportunities
• Acquiring personnel with new skill sets
• Organizational changes
  
• Annual maintenance costs
• Network infrastructure (if necessary)
• Additional security hw/sw (if necessary)
• Funding a project and tools to implement SOA Governance

I am sure I missed several others. The key here is that to successfully implement SOA, your company must make a long term financial commitment to architecture that never ends. Investing in SOA goes far beyond buying the Stack.

The Stack
Depending on your project's objectives, you may need to purchase a variety of products that make up the Stack. In my case, we already had an enterprise portal. We purchased an ESB, a BPMS solution, and a data services tool. With each tool you need hardware, racks, floorspace, etc.

Testing Tools
Unless you already have a robust testing tool for testing enterprise architectures including web services or JMS messages, you probably need additional tools. We have many modules in the Mercury suite, but nothing for performing testing in a service oriented architecture.

Training
Do not underestimate this one. Not only do you need to train administrators for every piece of the stack and for every additional tool you purchase, you need to train a large number of IT staffers about SOA, Governance, and developing and deploying with the new products. There are five different courses offered just for the BPMS tool we bought. One is for administration, two are for programming, and two are for modeling. Training is also a continuous process and not an event that occurs by sending a few people off site for three days.

Professional Services (Stack vendor)
Take my advice on this one. Use the vendor's professional services to install the products. Do not, I repeat, do not try this on your own or with consultants. The beauty of SOA is how it easily integrates across different technologies. My lessons learned is that this is true once you suffer through the upfront configuration nightmares required to create the initial environments. If anyone other then the vendor sets up the environment, you may not get the support you need from the vendor to fix it. Use the vendor to implement the products and have them come back when you have built something and have them certify the environment with your code running on it. Tweak the system now before you go live!

Professional Services (Partners)
Unless you are fortunate enough to have the necessary skill sets within your current staff, I highly recommend finding a partner with years of experience implementing SOA. Choose your partner wisely and embed your staff with them for knowledge transfer.

Learning opportunities
Don't underestimate continuous learning opportunities. I still attend conferences, webinars, lunch and learns, and various other opportunities to talk to people and learn from their experiences. Get out of the office and see what the rest of the world is doing. Learn from their mistakes and their success stories. Don't forget to budget for travel.

New skill sets
Our SOA/BPM initiative introduced a number of new skill sets. You may need to hire incremental headcount or replace existing roles with new roles. SOA talent is in high demand and is not cheap. Don't forget recruiting costs, relocation expenses, and the overhead of onboarding.

Organizational changes
You may find that your current organizational structure is not optimal in the new world of SOA. Reorganizations sometimes have costs associated with relocations, promotions, merit raises, administrative costs, new reward and incentive programs, etc.

Annual Maintenance
Expect to pay between 18-21% in maintenance costs for everything you buy.

Network infrastructure
Your current network infrastructure may not support the requirements of your SOA initiative. More sophisticated load balancers, additional monitoring tools and agents, and hardware accelerators are just a few things you may need to consider.

Security
If you are exposing business services externally to customers and partners, your security requirements are more critical then ever before. There may be additional purchases required for authentication, auditing, logging, encryption, ID management, etc.

SOA Governance
This might be the most underestimated piece of the entire SOA financial model. Implementing SOA Governance is a project within itself. It requires dedicated staff, training, and tools for enforcement. Don't cut expenses here. Invest in governance because the success of your SOA implementation for years to come is heavily dependent on your ability to enforce policies and best practices. Buy a repository. Don't try to manage this manually or you will drown.


To sum it up, SOA is not a tool you buy. SOA is a new way to develop software and touches every aspect of your enterprise. Invest in SOA or the money you spent on your stack will never reap the benefits that SOA can offer.