Enterprise Initiatives

Back in September I shared with all of you the presentation I gave about SOA and Organization Change Management. Today I am happy to share the podcast from that discussion that took place at the quarterly SOA Consortium meeting in Orlando. Here are the slides.

And here is the podcast that goes with it.

The panel discussion at the tail end of the presentation is fantastic. There were a lot of great questions and many lessons learned shared from experts like Todd Biske, Brenda Michelson, Fill Bowen from IBM, and others. If you have the time, listen to this podcast. Failure to recognize and counteract the resistance to change is the number one cause of failure for all enterprise initiatives, not just SOA.

The Dilemma
I have been spending a lot of time recently learning about enterprise mashups. The new startup that I am working for is building a SaaS solution that will be consumed by various types of customers and partners. These customers and partners may want to consume our data services as a RSS feed, gadget, SMS message, web page, within a portal or portlet, or a number of different ways. I do not want to spend the rest of my life developing new output mediums for our services. Instead, I would rather spend my time adding new business services to enhance our product and service offerings hence contributing to the bottom line.

Enterprise Mashups to the rescue
Enterprise mashups will allow me to offer my partners and customers the ultimate flexibility to access our products and services in ways that are convenient for them without having to wait on my IT shop to decide if (a) we think the request is important enough in our priority list, (b) if we have the time and resources to work on it, and (c) how much we will charge them. On the IT side of the house, with an enterprise mashup strategy in place we can be assured that whatever mashups our customers and partners create, they will be subject to the same security and governance as the services we have developed. The diagram below shows a logical view of how our SOA will be designed.



As you can see, we have clearly abstracted the various layers within the architecture and they all inherit our overall security policies. SOA governance is applied to this architectural approach to enforce our standards and design principles. Overall IT governance provides oversight over the entire enterprise which includes legacy systems (we don't have any legacy yet), third party software, etc.

Now let's add the enterprise mashup layer. We want to hide the complexity of our architecture from the end user and expose data services to them to consume. At the same time we want these mashups to be equally secure as the services we write and adhere to the same governing principles. Enterprise mashup products provide tools to make managing this layer easy and efficient. The diagram below shows the enterprise mashup layer inserted into the architecture as a layer on top of SOA.

From SOA Slides

Enterprise Mashups in simple terms
If the concept of Enterprise Mashups is still not clear, check out this white board discussion with JackBe's CTO John Crupi.



I spent some time talking to JackBe's VP of Engineering Deepak Alur a few week's back. Deepak discussed how enterprises have been focusing more on infrastructure and technology and not on the consumers of data. As he coined it, many shops are "developing horizontally and not addressing the needs of the users". He talked about how users were doing their own brute force mashups by cutting and pasting data from various places into Excel. This creates various issues within the enterprise due to lack of data integrity, security, and governance. It is ironic how corporations spend huge amounts of money on accounting software and ERP systems, yet they still run the business out of user created Excel spreadsheets! The concept of enterprise mashups addresses this by shifting the focus back to the user consumption of data. Here are some of the requirements for mashups that Deepak pointed out to me:

• User driven & user focused
• Both visual & non-visual
• Client & server side (although most are server)
• Plug-n-Play
• Dynamic, Adhoc, Situational
• Secure & Governed
• Sharable & Customizable
• Near zero cost to the consumer

Jackbe's enterprise mashup tool is called Presto. Presto is an Enterprise Mashup Platform that allows consumers to create "mashlets" or virtual services. IT's role is to provide the security and governance for each data service that will be exposed for consumer use.



Presto Wires is a user friendly tool to allow users to create their own mashups by joining, filtering, and merging various data services (as shown in the picture below).



In this example the user is combing multiple data points from many different organizations in an automated fashion. They could then present this data to multiple different user interfaces and devices. All without waiting on IT.

Here is a simple example of a mashup of Google Maps with a Carpool data service.


The folks who create the Carpool data service where able to leverage the Google Maps service without having to write their own mapping software and without having to know the underlying architecture behind Google Maps.

How this solves my Dilemma
Back to my dilemma. By leveraging a tool like Jackbe's Presto or WSO2's Mashup Server, I can now present various data services in a secured and governed fashion to my customers and partners without being concerned on how they want to consume it. Whether they want the mashup on their own intranet, as a desktop gadget, as an application on Facebook, or what ever they dream of, all I need to be concerned with is the SLA of my data services. This also makes my product offering more competitive than my competitors who have proprietary user interfaces that do not provide the flexibility and customization that the customers desire. As I said in the title, this is the Icing on your SOA cake. For those organizations who are disciplined enough to implement SOA and follow the best practices of design and governance, the reward can be an simple addition of an Enterprise Mashup Platform on top of your SOA stack. This is the ultimate flexibility and agility that SOA promises.

My friend and colleague Todd Biske has recently published a must read book about SOA Governance. Most books that discuss process can require a ton of caffeine in order to make it from cover to cover. Not this book! Todd uses a unique style of combining a story of a fictional company along with his well served advice. He walks us through a multi year SOA project with a fictional insurance company called Advasco. Advasco, like many of the companies that we work for, has years of silo legacy applications that are the direct result of acquisitions, mergers, and years of developing in silos. The goal of their initiative was to "provide sales agents and marketing staff for the insurance division with a single view of the customer".

Todd takes us through the evolution which spans five years. Over this time, you can see the company make continual strides towards the overall vision but not without challenges. With each challenge comes another opportunity to mature their governance model. At the end of the book, Advasco's IT staff have made a major impact to the company's bottom line and ability to react to market changes. This is one of the few books that actually shows us what success looks like. This is why the book is so good. Many books talk about processes, structure, and controls but fail provide us with a glimpse into what the fruits of that labor looks like. Todd takes us from project inception, to successes, to set backs, to mitigation strategies, and ultimately to enterprise wide success. By taking us through this journey we can get a better understanding of the impact of the recommendations that he makes for SOA Governance. I won't go into details on what he recommends. I will let you read it yourself. But I can guarantee that you will enjoy the book and will be able to relate your real world experiences with the experiences of the fictional characters who helped move Advasco forward by leveraging SOA.

What made Advasco successful?
If you follow everything that Todd recommends in this book, you still are not guaranteed success. There were some specific events and characteristics that made this journey a success. Here is a short list:

• Initiative was driven top down
• Very little resistance to change
• Great working relationship with business and IT
• Very talented staff that was willing to learn
• Effective EA team
• Project was business focused

Final Thoughts
SOA Governance is a must read for any company preparing for SOA or for companies struggling to make their SOA initiative successful. Follow the advice and examples that Todd provides but also address the bullets I listed above. If you have the great staff that Advasco has you will find success. But in the real world, the effort to promote this type of change in the enterprise will usually require much more focus on change. At the SOA Consortium meeting last month in Orlando, Todd provided us with a great presentation about SOA Governance. I followed that up with this presentation on change.



If you focus on Organizational Change Management and heed Todd's advice on SOA governance, your odds of succeeding with SOA will be greatly enhanced. Buy the book, it is the best book on governance that I have read so far.

I have discussed the SOA Evangelist, the Architects, and the Developers. Today I will discuss the role of the Testers and the characteristics required to contribute to a successful SOA implementation.

One of the most important roles and one that I probably should have included in the Architects post is the Testing Architect. As I have written on CIO.com in my article called Six Questions to Consider Before Building a SOA Testing Team, SOA testing requires a much deeper knowledge of technical skills, including development skills. It might be unrealistic to expect your entire testing team to possess the desired technical skills required to successfully test SOA, but your test architect and your leads should be able to understand concepts like statefullness, distributed computing, and data services, to be able to properly validate the underlying architecture. They also need to be able to take developer test harnesses and update them with their own test scripts.

A successful SOA testing strategy starts with the test architect. This person must have a very in depth knowledge of SOA and should work closely with the EA team. I actually recommend that this person is a member of the EA team, but every business and culture is different. The goal of the test architect should be to set up a framework and a core group of policies and procedures (part of IT Governance) so that the rest of the test team has the tools and the guidance to successfully test SOA. Without an established testing architecture, the company will have to rely heavily on expert knowledge from the entire testing team. I have seen three scenarios through my personal experiences and through my research.


Scenario 1: Underestimating SOA
The first scenario is out right failure caused by not having the tools and knowledge required. This is caused by a company not realizing that their current methodology and internal personnel are not well equipped for testing SOA. These companies do not invest enough in tools, training, and governance and usually can only test the presentation layer and possibly the interfaces. By not understanding the concepts of SOA, they are unable to validate the architecture which leads to poor performing and fragile services.

Scenario 2: Paying through the nose
The second scenario I have seen is relying too heavily on "expert" consulting firms for testing. In this scenario the company bets the farm on an experienced SOA consulting firm and pay rates that far exceed $100/hr. This model cannot be sustained for any length of time unless the company is willing to burn huge amounts of capital (which is not a popular thing to do these days).

Scenario 3: Good balance of internal/external expertise
A more desirable scenario is to train or hire a SOA testing architect, build a solid testing platform tailored for the needs of the organization, and govern the testing process while training the other members of the team. Companies should hire one or more experienced SOA tester, find an experienced consultant or two, or train an experienced and credible internal candidate to take the lead for creating the testing architecture. At the same time the testing experts are charged with transferring knowledge over to the rest of the internal team members. This is critical because a highly experienced SOA tester is in great demand and is a flight risk since they are a rare breed. It is critical to grow the knowledge base internally.

Testing needs
The needs can be broken down into these categories: People, Tools, Governance. So what are the characteristics of a successful SOA tester. The answer is dependent on the architecture that is implemented, which is related to the tools and policies that are put in place. Below is a diagram I often use to discuss the typical layers of an architecture.

From SOA Slides

As I mentioned in the previous discussion about the developers, I see the need and desire to specialize within the layers. The same holds true for testers. Work within the layers happens simultaneously in development. I recommend an iterative development and testing approach which means there should be testers working within the layers simultaneously as well.

Here is what I would strive to put in place, keeping in mind that these are roles and some people may fill multiple roles:

SOA Test Architect
Courageous leader with extensive working knowledge of SOA, distributed computing, integration testing experience, coding/scripting experience, and understands the business. It is likely that you may have to go outside to hire this person or bring in a consultant to assist your top level tester.

SOA testing leads
This person or persons must understand all layers of the architecture (let's not forget security either) and should be able to design test plans that validate both the architecture and the governance model. They should understand how to perform black, white, and gray box tests. Testing abstracted services requires extensive testing in the areas of security, performance, and regression. Throw in the versioning capabilities of services and the fact that service consumers can use services in ways that were not anticipated and the permutations of test cases start skyrocketing. The test leads need to practice risk based testing and balance risks, timelines, and costs. There is just so much more at stake here than in the traditional n-tier architectures.

Business process testers
The business processes should be modeled within some tool and will likely call one or many services. The process flow requires a series of decision points based on variables and constants and can trigger events such as notifications, alerts, other processes, error handling routines, services, and a variety of other possibilities. The testers need to validate the business process as a stand alone entity. For example, if the business process is "validate credit card", the tester needs to ensure that this process handles the inputs correctly, properly runs the validation process, and generates the appropriate output. At this point, the tester need not be concerned with any other processes or services. These testers must work closely with the business and/or business analysts and do not need the breadth of technical knowledge that the leads and architects must have (although it would help). They should be approaching the testing from a business standpoint.

Integration testers
These testers must be much more technical and understand how to work with XML, SOAP, WSDLs, networking & telecommunication concepts, statefullness, and various platforms and technologies (Java vs .Net, Windows vs Unix/Linux vs. mainframe, etc.).

User Interface testers
The company most likely already has an abundance of people who can test the UI. If your company is leveraging mashups, wireless devices, or consumer facing UIs, the complexity of the testing will be greatly increased. These testers may be need to become familiar with AJAX, RIA, portal technology, RSS, and a variety of social software.

Data services testers
These testers must understand concepts of data modeling, database CRUD operations, transformations, security and roles, authentication, and much more. Everything starts with the data and if errors are introduced in this layer, everything else is doomed to fail. You must have a very strong testing lead working in the data services layer since data is the foundation of any successful implementation.

Other areas of focus
The name of the game is speed to market and test automation is a critical component for making that a reality. Performance testing is extremely critical and organizations should practice simulations to try to anticipate future performance of all services. Validating the security model and the governance model should also be part of the overall test strategy. Whoever is responsible for designing the security test plan should read (and fully understand) this book.

Closing comments
I am by no means a testing expert (but I did stay at a Holiday Inn Express last night). I do read a number of testing blogs listed below:

• Randy Rice's Software Testing & Quality Blog
• Frank Cohen's blog
• The ITKO Lisa Soapbox: SOA testing, validation & virtualization
• SOA Testing
• Mike Kelly's testing blog

Make sure your organization funds the necessary training and tools for testing. From my own personal lessons learned, make sure you invest as much effort in testing and security up front as you do in architecture and development. On my project a few years ago, I wore my developer hat too much in the research phase and did not properly budget for what was necessary for testing. Do not make this same mistake. As hard as it is to sell the business on SOA, it is much harder to go back for more money, especially before any business value is realized!

For those of you who have been reading my blog for the last two years, you might know that after 13 years at my previous job I left the company this summer to pursue multiple opportunities that were available to me. I have been doing some consulting and freelance writing in the mean time but have finally found my new gig. I will be the CTO/Chief Architect for a startup (more details forthcoming in the future) and will have the opportunity to work on some of the newer technology initiatives. So here are some of the future blog topics that I will start covering in 2009 once I get rolling:

1. SOA - can't seem to let it go! As I wrote in a post at CIO.com, I believe SOA is not just for legacy systems and can be the foundation that startups can leverage as a competitive advantage (see the reasons why)
2. Cloud Computing - Startups typically have low budgets and leveraging platform (PaaS) and/or Infrastructure as a Service (IaaS) can be a great way to both minimize costs and reduce time to market. As I go through the requirements analysis, risk analysis, vendor analysis, proof of concepts, prototypes, and deal with issues such as stability and privacy, I will share my lessons learned here.
3. Open Source - We will likely leverage many open source solutions and I will discuss those experiences and decision points here.
4. Social Software - The company is based in the North East and I live in Florida. It is highly possible that the team I build will be dispersed all over the country and possibly even the world. There will be great lessons learned discussions as well as evaluations of tools to help enable working remotely (virtual whiteboards, live meetings, SaaS solutions for defect tracking, etc.).
5. Agile - One of our goals is to deliver quickly and at low cost. This involves a lot of the above mentioned topics and will also lead us into the world of Agile Development. I will discuss our challenges and lessons learned in this area as well.
6. C-Level IT topics - In my new role I will be working side by side with other C-Level people within the company as well as with C-Level people of partner and customer companies. I will have to hit the road and sell and/or promote our products/services and will try to shed some light on some of the interesting topics I come across.
7. Misc - Other topics that might be worthy of discussing (examples: impacts of financial crisis, VC funding, selling technology to business people and customers, etc.)

In addition, I still do some freelance writing and blogging from time to time. I have a series of discussions coming up on enterprise mashups and will blog about it in the next week or two. I am attending and speaking at the EDM Summit in Orlando next week and will definitely be sharing my presentation on Business Intelligence as a Service as well as covering topics like business rules management, data warehousing, IT governance, and others. I also have two books from Packt Publishing to review:

• SOA Governance by Todd Biske
• SOA Cookbook by Michael Havey

Once I finish reading these I will post a review on my blog. I am half way through Todd's book so far and really like it.

And finally, I am open to covering any topic that any of my readers would like me to discuss whether it is on this blog, over the phone, or in person. Whenever I travel I send a Tweet on Twitter to let people know where I can be found. I will be in Pittsburgh from Saturday 10/25 through Monday 10/27 if anybody lives there and would like to meet. Don't bug me from 4:15-7:15pm on Sunday because I'll be at Heinz Field pulling for my Giants against a tough Steeler team. The rest of next week I am at the EDM Summit in Orlando.

I look forward to sharing these topics with you all. Over the next two months my focus will be on the business plan and creating a team. We officially launch the company in the December-January time frame. At that point expect to see more discussions on the topics I mentioned above.

I frequently discuss SOA with potential clients, fellow architects, and business partners. Each audience requires a discussion that is tailored to the appropriate level of technical and business expertise. Today I had to present to a CEO and a few of his top staffers about what it takes to launch into a SOA initiative. So I put together a presentation similar to the one below and felt like sharing it with all of you.

Before you look at it let me put the discussion into context. The target audience is hi level management, business and IT, who are already bought into SOA and are trying to understand what is involved before they launch into a full blown SOA initiative. This is not a presentation for architects, rather it is to inform C-Level stakeholders what is required from a business, technology, and culture standpoint.

I also had to sell to them that I know what I am talking about, hence the first few slides about my background. One last note: Some of the slides are mostly visual and only get the point across when backed up with talking points.

I hope this adds value to some of you. Enjoy!

Here is a slide show for those who need to explain to upper management the risks involved trying to implement SOA and the importance of SOA governance. This must be sold early on and the necessary resources must be brought in. Trying to implement governance after implementation is risky, expensive, and hard to do. Like SOA, governance should be implemented one step at a time and will take years to evolve to a high level of maturity. Make sure your SOA governance maturity level matches your SOA maturity level. Don't try to implement all of your governance up front because you will never have any time left to implement any SOA projects, hence you will not deliver value to the business. I hope the slides help. Enjoy!

One of the many websites I visit, SOAGovSource, published a free pocket guide today for all of their registered users. This guide has quotes from numerous articles about SOA Governance taken from a variety of popular blog posts including yours truly. You can see the tables of content on the right hand side of this post.

To download this guide, go here. This PDF file gives you great quotes from various authors and links to articles written by industry experts, analysts, and practitioners.

Enjoy!

I attended Zapthink's Practical SOA event on Tuesday in New Jersey. There were many great presentations from both vendors and practitioners. There were three different case studies presented including mine which was on selling SOA to the Business (the video will be posted soon). What was interesting to me was the three different paths the companies in these case studies took. The first case study was from The Hartford. They have a well established governance model and are very far along into the maturity model. If you read a text book on what SOA governance should look like, it will look like the Hartford. They have their processes so buttoned up that they even tie developers' annual review process to how they adhere to Hartford's best practices in reusability. A key take away is the fact that this took several years and a firm commitment from their leadership team to put this in place.

The second case study was from Lehman Brothers which was implemented in a totally different fashion. In this scenario, the governing body had very little control and power over the distributed development groups. So instead of just giving in and giving up, they took a different approach. They leveraged tools and some custom code to discover web services and usage of services in the repository. They would receive a page anytime a new service was discovered. Then they would study the service to see if it was constructed the proper way. If there were opportunities for improvement, the governance team would engage in conversations with the service owners to coach and mentor them on the proper design and deployment of their services.

In my scenario, we have a totally different story. Because we were so successful in selling the business on BPM and SOA, they funded the initiative and targeted very aggressive delivery dates for key functionality. That did not leave us enough time to plan and implement a governance model. So we are establishing a road map for delivering governance one step at a time. We understand the risks of not having enough governance up front and expect some headaches along the way, but the business was not going to wait a year for us to put those processes in place.

So theory and text books say that you must start with governance. The reality is you evolve governance over time. I highly recommend attending these Practical SOA conferences if you get a chance. I jokingly refer to it as group therapy because we are all sharing our lessons learned.
If you missed the conference, there are two more scheduled. The next one is in London on April 25 and the last one is in Las Vegas on May 16. I also highly recommend the LZA SOA Bootcamp which I recapped a while back. Ron at Zapthink told me that they are targeting May 5-8 in New Jersey for the next boot camp if they get enough people interested in it. Send Ron an email at info@zapthink.com if you want to attend.

I just read an entertaining post from Dave Linthicum called Do you Have a DSG (Dumb SOA Guy) Issue? A DSG refers to a person leading the SOA effort who has the political clout but has no clue what SOA is. If you have one of these, you could be in for a long painful and expensive journey. Here is a key point from the article:

Again, the technology around SOA is simple, and I never worry about how we're going to leverage technology to solve a problem. However, the people issues are more concerning and more difficult to fix.

My head was nodding when I read this. I can tell you from experience over the last 18 months that the hard part is culture change, change management, and establishing governance. If you have a few smart architects and a good implementation partner like I have, the technology part can be easily conquered. The hardest part I have experienced from the technology standpoint is getting all of the tools in the stack completely integrated and stable. But even these issues are a one time hit that you have to deal with on your first implementation.

Back to the DSGs. The person leading the SOA initiative needs to be strong in the following areas:

1. Technology - must understand SOA at a conceptual level or better
2. Political Clout - must have a high level of influence to remove roadblocks
   
3. Business - must align technically with business drivers
   
4. People Skills - must know how to motivate and direct people

If your DSG is lacking in any one of those four areas, your SOA initiative's chance of success will decrease dramatically. If your DSG is lacking in more then one area, cancel the project now or get a new DSG!

We are in the process of defining and implementing our SOA governance processes. We are a medium size company so having armies of process cops is not an option, not to mention totally undesirable. At the same time, we need some level of process to ensure that we architect business services in a consistent manner and that we design for the enterprise as opposed to the old application silo approach. To make governance even more challenging, we want to deploy in cycles of 12 weeks or less. So how can we be agile and enforce SOA governance at the same time?

One way is to shift from text heavy documentation to visual documentation. In other words, stop generating hundred page Word documents and start creating UML models, business process models using BPMN, use case diagrams, and architecture diagrams. These artifacts are like blue prints for a building architect. If you were building your dream house would you type up the specs for your house in a Word document and hand it to your builder or would you give him blue prints?

Another way to stay agile and still enforce governance is to tightly control scope and requirements. Users are used to asking for the sun and the moon because they typically have to wait several months to get the next release. It is similar to the way politicians load up a bill with pet projects because they know it might be their only chance to get their stuff done. We must train the user community that in an agile approach, we will deploy more frequently and in shorter time spans. This means that you don't need to get every last requirement jammed into the project because another release is coming on the heals of the current release.

But the most critical way to stay agile while having a solid governance model in place is to use processes where it makes sense. Focus on artifacts that mean something to those who have to develop software. Don't focus on documents that are never used once they get signed off. One of my favorites is the team structure document. I have seen teams waste days creating a document that nobody other then the PM uses that describes all the roles, who the team members are, what their contact info is, how meetings will be held, when they will be held, and so on. Who cares? The PM needs to know this but why waste the rest of the teams time with reviewing this stuff. I can look up people's information on the portal faster then I can even find this document. Over the years I have seen teams create documents for the sake of satisfying a check list. This is a total waste of time. SOA Governance is all about managing services and the service life cycle. SOA governance can plug into your existing methodology (if you so desire). Focus on the processes that add value and discard everything else.

For small and medium sized businesses, we must be careful not to get bogged down in process. Unlike large enterprises, we don't have the luxury of dedicating several full time employees to enforcing processes and procedures. Instead, we must put a solid framework in place and rely on our people to enforce the necessary processes. Setting up a SOA Center of Excellence and an IT Steering committee under the watchful eye of a strong executive sponsor (CIO, Chief Architect, etc.) is one way to approach it.

There is no silver bullet here. Each company must decide for themselves how much process to put in place. If they put too much process in place, they can get bogged down and never meet expectations. If they don't implement enough process, then they will eventually spiral into mass chaos and won't realize the benefits of SOA like reuse, reduced maintenance costs, and increased flexibility.

One of the main reasons why I blog is to share my experiences with other Enterprise Architects so we can learn from each other. Today, I am not going to share a success story but instead will describe some decisions we made early on that we would probably do differently now that we know more.

The scenario:

After we sold the business on BPM and SOA, we were challenged to implement a pilot project quickly to show the value of SOA. It was critical that we deliver early and often as opposed to going off for 8-12 months without giving the business any relief from its legacy business processes. To meet this objective we established a 12 week "pilot" or "beta" project that started as soon as we purchased our SOA stack. This did not give us time to implement governance. We agreed that during the second wave of projects we would implement governance as part of those projects. As we started the second wave of projects, I mandated that all code should be delivered with test harnesses, the build process should be 100% automated, and testing automation should be part of the project deliverables.

Those all seemed like reasonable objectives, so what could go wrong with that plan?

The Reality:

A few things happened along the way that caused the initial pilot project to extend longer then anticipated. First of all, getting all of the individual pieces of the stack (portal, BPM, ESB, and data services) to work properly together took way longer then any of us ever dreamed of. Luckily the vendor understood the urgency of our issues and dedicated many resources to help resolve our issues but this cost us several weeks. Guess what? Now we are mitigating our risks for getting the next six concurrent SOA projects delivered by the end of the year. What do you think is the only thing left to cut out? That's right, governance, build automation, and testing automation.

Lesson Learned:

Don't take any short cuts when it comes to governance and automation. We will eventually get there but it will be much harder to implement this stuff at a later date then it would have been before we moved to production. I am also fearful that we might continue to prolong or delay governance and automation to hit various dates along the way. We can get by today, but several months from now when we have over a hundred services and a dozen projects in production, we better have this resolved or we will be working hard instead of working smart.

Recommendations:

Here are a couple of things I would do differently next time. Staff the governance and automation initiatives with dedicated individuals who are not tied to other deliverables. I would also define and manage these initiatives as individual projects with published milestones and get executive buy in to ensure that these deliverables get the focus they need.

Hopefully this has been helpful for those of you who are considering SOA or are early on in the process. I would love to hear your experiences and/or your recommendations going forward.

We are in the middle of implementing our SOA Governance model. We have put together a ton of lengthy reference documents like the architect guidebook, SOA roadmap, developer guidebook, administrator handbook, and many others. Our biggest fear has been getting people to read these documents. In order to enforce our SOA best practices and standards, people must be aware of what they are. Giving people 200-300 page documents is not an effective way of sharing information.

Enter open source wiki tool, MediaWiki. With a little help from OpenOffice v.2.3, we took the lengthy Word documents and converted them to Wiki Markup to create easy to use, clickable, and searchable documents. Now our architects and developers can easily maneuver through pages of pertinent documentation without being overwhelmed by large sequential documents. In addition, people can collaborate by using the discussion feature in MediaWiki to help the architect team improve upon the documentation.

Our next step is to implement an open source blogging tool. One of our challenges today is to communicate both technical and project specific information pertaining to our SOA initiative in a timely manner. I have held a number of meetings with several IT teams to walk them though various presentations and white board sessions. Unfortunately I don't have the bandwidth to do this often enough. We will start to leverage blogging technology to share daily information about the initiatives. People who are interested will be able to subscribe to these blogs using an RSS feed reader. This is much more effective then pushing more email on people.

And finally, we will create an Enterprise Architecture community in our enterprise portal. This will allow us to bring all of our content together in a one stop shop approach. The community will display our blogs, our wiki, project status, tips & tricks, announcements, and various other information.

One of my lessons learned so far on our SOA project is that we do not communicate often enough. I believe a combination of monthly presentations and white board sessions, along with our wiki, blogs, and portal community provides multiple opportunities for people to keep up to date with our SOA initiative. I also think it is more effective to publish content and let people "pull" it on demand then to "push" it on them.

One of the most critical tasks that the champion of your SOA initiative must perform is to quantify the total cost of ownership (TCO) of the SOA initiative. Some executives might think that once they buy the ESB they are done. This is far from the truth. Here is a list of items you should plan for up front and create a road map for how much and when each of these items need to be procured:

• The Stack (BPMS, ESB, Data Services, Portal, etc.)
• Testing tools
• Training, training, and more training
• Professional services from the Stack vendor(s)
• Professional services from the implementation partners
• Conferences, site visits, partner meetings, and other learning opportunities
• Acquiring personnel with new skill sets
• Organizational changes
  
• Annual maintenance costs
• Network infrastructure (if necessary)
• Additional security hw/sw (if necessary)
• Funding a project and tools to implement SOA Governance

I am sure I missed several others. The key here is that to successfully implement SOA, your company must make a long term financial commitment to architecture that never ends. Investing in SOA goes far beyond buying the Stack.

The Stack
Depending on your project's objectives, you may need to purchase a variety of products that make up the Stack. In my case, we already had an enterprise portal. We purchased an ESB, a BPMS solution, and a data services tool. With each tool you need hardware, racks, floorspace, etc.

Testing Tools
Unless you already have a robust testing tool for testing enterprise architectures including web services or JMS messages, you probably need additional tools. We have many modules in the Mercury suite, but nothing for performing testing in a service oriented architecture.

Training
Do not underestimate this one. Not only do you need to train administrators for every piece of the stack and for every additional tool you purchase, you need to train a large number of IT staffers about SOA, Governance, and developing and deploying with the new products. There are five different courses offered just for the BPMS tool we bought. One is for administration, two are for programming, and two are for modeling. Training is also a continuous process and not an event that occurs by sending a few people off site for three days.

Professional Services (Stack vendor)
Take my advice on this one. Use the vendor's professional services to install the products. Do not, I repeat, do not try this on your own or with consultants. The beauty of SOA is how it easily integrates across different technologies. My lessons learned is that this is true once you suffer through the upfront configuration nightmares required to create the initial environments. If anyone other then the vendor sets up the environment, you may not get the support you need from the vendor to fix it. Use the vendor to implement the products and have them come back when you have built something and have them certify the environment with your code running on it. Tweak the system now before you go live!

Professional Services (Partners)
Unless you are fortunate enough to have the necessary skill sets within your current staff, I highly recommend finding a partner with years of experience implementing SOA. Choose your partner wisely and embed your staff with them for knowledge transfer.

Learning opportunities
Don't underestimate continuous learning opportunities. I still attend conferences, webinars, lunch and learns, and various other opportunities to talk to people and learn from their experiences. Get out of the office and see what the rest of the world is doing. Learn from their mistakes and their success stories. Don't forget to budget for travel.

New skill sets
Our SOA/BPM initiative introduced a number of new skill sets. You may need to hire incremental headcount or replace existing roles with new roles. SOA talent is in high demand and is not cheap. Don't forget recruiting costs, relocation expenses, and the overhead of onboarding.

Organizational changes
You may find that your current organizational structure is not optimal in the new world of SOA. Reorganizations sometimes have costs associated with relocations, promotions, merit raises, administrative costs, new reward and incentive programs, etc.

Annual Maintenance
Expect to pay between 18-21% in maintenance costs for everything you buy.

Network infrastructure
Your current network infrastructure may not support the requirements of your SOA initiative. More sophisticated load balancers, additional monitoring tools and agents, and hardware accelerators are just a few things you may need to consider.

Security
If you are exposing business services externally to customers and partners, your security requirements are more critical then ever before. There may be additional purchases required for authentication, auditing, logging, encryption, ID management, etc.

SOA Governance
This might be the most underestimated piece of the entire SOA financial model. Implementing SOA Governance is a project within itself. It requires dedicated staff, training, and tools for enforcement. Don't cut expenses here. Invest in governance because the success of your SOA implementation for years to come is heavily dependent on your ability to enforce policies and best practices. Buy a repository. Don't try to manage this manually or you will drown.


To sum it up, SOA is not a tool you buy. SOA is a new way to develop software and touches every aspect of your enterprise. Invest in SOA or the money you spent on your stack will never reap the benefits that SOA can offer.

Everyday as I sift through the articles in my Google Reader, I see countless debates about the relationships between SOA & BPM, IT driven vs. business driven, and top down vs. bottom up. There is so much debate about these topics that I sometimes wonder if anyone is getting any work done. Glance at these headlines and ask yourself how confusing this must be for somebody who is in the research stages of SOA.

* The proper relationship between SOA and BPM
* The awkward dance between BPM and SOA
* BPM Driven SOA
* BPM Without SOA: 'Like One Hand Tied Behind Your Back
* Why BPM Screws up SOA

* Business Driven SOA
* Who's in Charge of your SOA?

* Another view: avoid bottom-up SOA like the plague
* Bottom-up SOA is harmful and should be discouraged
* Should SOA be Top Down or Bottom Up

I have mentioned it in the past and I'll say it again. There is no single answer to these questions. There are many factors that can influence these decisions such as:

• Whether you have strong executive sponsorship or not
• Your IT staff's capacity to change
• Your EA maturity level
• Your staff's talent level
• Budget
• How much time you are given to deliver
• The main driver for the initiative

These are in no particular order. I am not going to tell everyone how I think you should run your SOA projects. I will give you insight into the decisions we made on my project.

In this article I tell the story of how we got the business to support a BPM and SOA initiative. IT had been pushing this for a while but could not get the funding. Once we convinced the business to reengineer their business processes we were able to come up with the justification to buy BPM for operational efficiencies and SOA as the technology to enable BPM.

We then launched into a process reengineering exercise which produced a portfolio of a dozen or so initiatives with an extremely attractive ROI. This determined the bottom up approach for us. We were funded specifically for delivering the projects identified from the process reengineering exercise. We then analyzed the projects and identified the services that would be required to support the new business processes. We did this for each project. Then we recommended a priority order which was based on two factors:

1. Business benefits - ROI, operational efficiencies, customer service, etc.
2. Architecture benefits - Service reuse and speed to market

We performed a three week "SOA roadmap" exercise that took these two factors into consideration. There were huge advantages of moving certain projects to the front of the priority list because of the number of services contained in the project that would be shared by the other projects. To figure this out we mapped out all of the services for each project and identified projects in the portfolio that would give us the biggest bang for the buck from the standpoint of service reuse.

The other constraints we had to deal with was time and money. Each of these projects in the portfolio had to be justified individually. They each had very specific funds and very aggressive timelines. The first project which included implementing the stack and delivering a beta version of a B2B portal in 10 weeks really constrained us from a governance standpoint. For this first project it was critical that we delivered quickly to deal with the perception that SOA initiatives take forever to implement. Funding for the other projects in the portfolio were also dependent on the results of the first deliverable. Due to these constraints, we did not have the luxury to establish a governance model up front. We alerted everyone of the risks of not establishing our governance model and agreed that we would be allowed to implement our governance model for the next projects.

As we sit now, we are wrapping up the first project and getting the funding to tackle several projects concurrently. We are gearing up with our implementation partner to start establishing our governance model that will help us grow our SOA with the release of each new project.

So for us, we are building SOA from the bottom up, the business and IT are working together to drive the initiatives that drive SOA adoption. The business is the owner of this multi year initiative. We are focusing on a specific area of the business that has 20 year old processes. Other business units are seeing the benefit and lining up to launch their own initiatives. The company is changing the way they think because of BPM and SOA. Twelve months from now we will be a different company because of this.

To sum it up, I can't tell you how you should address top down vs. bottom up, BPM driven SOA or SOA driven BPM, business driven or IT driven. I don't think there is a silver bullet. I believe the decision should be based on the environment you work in and the constraints that you are faced with.

I would love to hear from others who have been down this road.

I have read and posted several articles about the many business and IT benefits of SOA. I haven't seen much talk about the benefits it creates for the individuals within IT. This article focuses on the potential career path opportunities that a Service Oriented Architecture can create.

Many IT shops that do not have a well defined architecture have a limited career path for those folks who want to stay technical and not go into management. There are typically a few levels of development (programmer, programmer/analyst, sr. programmer, etc.) followed by an architect role. There is only room for a few architects so most people with over 10 years of experience get stuck in the sr. programmer role for long periods of time. People stuck in this role tend to do one of the following:

1) Go into management soley for the purpose of getting promoted. Many of them return to development, leave, or do not perform at previous levels.

2) Leave due to lack of opportunity.

3) Become miserable and unmotivated

Enter SOA. Companies moving towards SOA may develop a layered architecture like the one in the following picture:

Within each layer you can see specialized roles and job responsibilities. Some of the roles (DBA & Configuration Management) will likely already exist in your organization. Some of the others may be new or the job responsibilities may change dramatically. The point here is that for those organizations with many talented IT professionals who are stuck at a dead end technical career path, SOA might be the answer to your prayers.

Some of these roles (Process Analyst and Enterprise Architect) are best suited for technical folks with great communication skills. These people will interact often with the business. For those who would rather not be exposed to the business, there are many other roles for you (Configuration Management, Object Librarian, Information Architect, etc.). Your more junior developers or people who prefer web development can live in the presentation layer. They will be extremely productive since most of the business rules and services will be available for them to assemble into the UI's. The folks who just love to bang out code can live in the business rule layer where they can develop components and services. The more seasoned developers will build enterprise components and services (i.e. encryption, authentication, error handling, etc.) while the others can focus on application specific components and services (i.e. create order, print invoice, etc.) .

Most IT shops will have pockets of people who don't like change. They may not be interested in these new roles. Those folks can focus on maintaining the legacy systems.

In summary, SOA has many benefits. For IT professionals, many new roles will be created and your company's IT career path will most likely be extended. Over the next few years, more companies will be moving towards SOA and the demand for SOA professionals will be great. So embrace SOA if your company is implementing it. SOA will create many opportunities for those who take advantage of it.

I spent the last two days in Weston, FL. attending various BPM and SOA related lectures. One session I enjoyed was George Paras's "Connecting the Dots: Establishing THE Enterprise Perspective". George is the editor-in-chief for the Architecture and Governance Magazine.

George reminded us that as architects we should think, act, and behave as business people. Everything we do as architects should be geared around value creation by affecting change in the name of product innovation, process transformation, technology transformation, and business transformation. In other words, we need to enable the business to be better, faster, more efficient, better informed, etc.

He then went on to point out that one of the reasons that it is so hard to create business value these days is because our technology and our business processes have become so complex. He showed us one slide that states, "Complexity INHIBITS Change....Complexity consequently INHIBITS Business Value". In my opinion, as we begin to model both our operational and technical architectures, we must make it a goal to minimize the complexity and subscribe to the KISS (Keep It Simple Stupid!) methodology.

The last golden nugget that I took away from George's presentation was about maintaining balance. George talked about how companies struggle to make progress towards establishing an enterprise architecture because the business demands so many tactical projects yielding only short term gains. The chief architect or architecture group must figure out how to get the company to start thinking strategically which yields long term gains. But be careful, you need to strike a balance between tactical and strategic. The business cannot stop while you go off for 6-12 months to create an enterprise architecture.

George's answer to the "Balancing Act" is the Managed Portfolio. The Managed Portfolio combines :

• Enterprise Strategy & Planning
• Project Portfolio Management
• Enterprise Architecture

The leaders of these three groups need to be in sync and must have shared goals geared around value creation.

During a break, I had a chance to speak to George (us Greeks need to stick together). I discussed my project and our approach to implement SOA only for the services required to support our BPM initiative, as opposed to creating a full blown SOA implementation. He agreed that we were on the right path which will make me sleep a little better tonight. I then handed him my last business card which was crumpled and stained with spaghetti sauce. Talk about great first impressions!

There were many other meaningful lectures at the show, but this was the best one to blog about.

I am beginning a long journey to implement both a BPMS and SOA solution at work. What I am finding out is that the technology is the easy part (and it is not all that easy). The real key to success is change management. The reason I say that is when I list out and rank the barriers to success, the top 5 are not related to technology. Number one: Resistance to change. Number 2: Becoming a process centric culture. Number 3: Governance. Number 4: Conflicting project priorities. Number 5: Lack of in-house skills.

If you look at this list you can see that all of these issues are related to people and processes. Change management, by definition, is managing the human aspects of change. The 5 barriers I listed are all issues that need to be addressed by either changing the culture, changing the organization structure or processes, or by training people. If we don't address all of these issues up front, then our chance of success is greatly reduced.

I am confident that we will successfully tackle the technical challenges, but we will need some help with the big 5! Any advice would be greatly appreciated.