Enterprise Initiatives

As I have mentioned in many posts before, SOA built right can provide business agility and flexibility. Yesterday I discussed how to leverage data services and a few days back I discussed enterprise mashups. Today the focus is on business processes.



Separating business processes from code is an ideal way to build an architecture that enables systems to adapt more readily to change. Let's make believe that we are working for a large eCommerce site that sells many different categories of products from multiple suppliers and warehouses each with their own distinct business processes. Abstracting the business processes gives us some very important advantages:

• Creates a set of standard business processes for a consistent user experience
• Allows us to quickly bring on additional suppliers
• Allows us to change a core business process once and apply it to all suppliers
• Allows the business to make modifications to its processes with minimal or even no coding

Here is an example of an order fulfillment process.



This work flow speaks to my first point "Creates a set of standard business processes for a consistent user experience". The complexities of working with multiple suppliers, each with their own processes and systems is hidden from the end user. Regardless of what products the customer is buying and which supplier is fulfilling the order, the user experience is exactly the same. On the other side of the equation, bringing on a new supplier can be as simple as collaborating on a set of services to integrate the supplier's back end system to the core business processes of the eCommerce site. Had we not separated the business processes from the code, we would have to make modifications to our existing code base which is already running in production and would have to go through a complete testing cycle. Instead, we simply tell the business processes which set of new services to integrate with for the new supplier. All of the existing code is not impacted and does not need to be retested. Instead, we can just validate the new services and the new supplier flow.

Here is another benefit. Let's say that we just signed agreements with three new suppliers and have contracted to launch them all in 4 weeks. Without the separation of business processes from code, all three supplier integration efforts would have to be coordinated and most likely bundled into one build. This creates huge project risks because slippage in any one integration effort can delay all three. By separating the business processes from the code, each supplier integration can be implemented whenever it is ready. We can stub out a call for each supplier and assign the appropriate WSDL when the services are completed. For example, the "Fulfill Order" process could be an integration with SAP for supplier 1, PeopleSoft for supplier 2, a custom built fulfillment process written in Java for supplier 3, and a custom built fulfillment system written in .Net for supplier 4. The business and the business processes have no need to be concerned with the technologies behind the fulfillment process. They are only concerned with the inputs, the outputs, and the quality of service (QoS).

As we move up the stack it gets even better! Next post I will discuss the importance of business rules. Stay tuned.

I was invited to speak at a forum for IT executives in Detroit this week sponsored by Information Week. The purpose of the forum as described in the agenda goes like this...

This executive breakfast, specifically designed for senior business-technology executives, will explore why the pressure is on IT to help the business transform, and how it can meet those expectations. More than ever before, companies are demanding their CIOs to be strategic thinkers in helping them innovate and operate at peak performance - especially as businesses are under pressure from the poor state of the economy and the ever-faster pace of change in a global market. In this environment, you can't miss this opportunity to gain insight into the tactics and strategy that will help you be on your best game.

I was specifically asked to talk about why transformational IT initiatives like BPM & SOA fail and what advice I would offer to prevent failures from happening. I put together the following presentation which is a combination of some of my previous presentations, Preparing for SOA and SOA & Change.

I wrote a very popular article on CIO.com a while back about the Top 10 reasons why SOA fails. I speak to each of these points in the presentation and present solutions for each. I also discuss using John Kotter's 8-step process for managing change which I highlight in the presentation. Here are my keys for preventing failures.

1. Plan for and manage organizational change


2. Key drivers should be business focused not IT focused


3. Evaluate internal skills and fill gaps. Do not try it without help!


4. Don't let the vendors drive your architecture. Do your homework.


5. Grow your governance model over time

Speaking of governance, here is an analogy I like to use...

Implementing SOA without a solid governance model is the equivalent to having an airport without a control tower. Sure, there are some very good planes and talented pilots, but without the proper planning and timely information the end results would be disastrous. So make sure you build a control tower and hire some air traffic controllers!

If you would like me to create and present a custom presentation like this one, feel free to contact me

Packt Publishing recently asked me to review a copy of Michael Havey's new book, SOA Cookbook. I finished the book yesterday and really enjoyed reading it. This book is made for technical folks, so non-technical managers and drag-n-drop programmers should not waste their time. Michael Havey takes a really neat approach in this book to assist architects and developers to understand the concepts of SOA and process modeling. In each of the nine chapters, Michael discusses pertinent patterns, use cases, or disputes and then offers very clear and easy to understand examples of what the resulting models would look like. He also goes one step further and shows specific examples within each of the following vendor stacks (IBM, BEA, Oracle, and Tibco). I find this extremely helpful for technicians who are modeling processes in an SOA for the very first time. He even provides a URL
to download the sample code.

One of the most important points in this book and one that I totally agree with is that the typical three layer SOA stack that most vendors offer is "unmistakenly process oriented. There are processes at each layer". This is obvious in the BPM layer. Michael goes on to discuss how ESB processes are single-burst stateless processes and how orchestration processes are both stateless and stateful. So why is this important? Michael states that "it is easier for developers to model a service as a process than to code it explicitly in a third-generation language". Another important point is that since SOA is a business driven architecture and business analysts and business people are more comfortable looking at use cases and process models, technicians need to start thinking in these same terms. After all, "Bad processes mean bad SOA" as Michael concludes.

The remainder of the book dives into topics such as design decisions on whether a process belongs in the BPM layer or within the lower SOA layers, modeling using BPMN & BPEL, design decisions for both short & long-running processes, using the "Flat-Form" method of modeling processes, dealing with dynamic processes, Simulating SOA, and concluding with measuring SOA complexity.

Some key points worthy of mentioning:

• A very worthy discussion of simulating SOA performance takes place in chapter 8. Michael recommends that simulation occurs before a line of code is written. When this does not take place it raises the risk of performance surprises in the production environment which may or may not occur right away. When these issues occur in production, resolving it becomes extremely expensive and can have catastrophic impacts.
  

• The Flat-Form method of modeling processes is a must read. I won't spill the beans here but the methods intent is to flatten the process models to improve maintainability and simplicity. Great stuff!

Overall, I really enjoyed the book. I have read a ton of books on SOA and most of them are high level discussions on what SOA is and how to get started. This book picks up after those discussions and is a great tool for architects and developers who have to actually build and deliver SOA. I highly recommend it! I also recommend that your testing lead reads the book as well. If you don't have a tester who understands the content of the book, you better get one!

Next up is my colleague Todd Biske's SOA Governance book which I will finish reading this weekend!

I have been blogging about our SOA project for over a year now. We are closing in on our first year of development and are ready to deliver eight different projects over the next two months. A few months ago we started our second process reengineering effort in another line of business. In this initiative we were able to access the process from pre-sales through contracts/proposals all the way through delivery. In the pre-sales area there is plenty of data mining and data discovery processes that take place so the sales team can go after the best opportunities. The pre-sales process is a business problem that is perfectly suited for our business intelligence (BI) tools. Historically, our BI tools were delivered as stand alone solutions. Now we have a need to allow the users to drill into data, select one to many result sets, and pass the information onto the next business process. In short, this means that we need to connect our BI results to our BPM tool.

In our current implementation, our SOA stack looks like this...



Our BI architecture (we use Microstrategy) looks like this...



But if treat our BI architecture as just another abstracted presentation layer, we get this...



In the previous picture, you can see that our SOA stack does not need to understand or even interface with the complexities of the BI platform. Instead, the selected result set that the user chooses from the BI user interface can trigger a web service to pass the results as an XML message to the SOA stack where the BPM engine takes over and moves the data into the next step of the process. Below is a example of a BI interface that I "borrowed" from the Microstrategy's web site.



Imagine drilling into the subcategory of computers to find an opportunity to sell a loyalty campaign to a customer based on certain performance metrics and demographics. Once the sales person selects a potential opportunity and hits the submit button, the data can be fed into the proposal process and much of the order entry data can be prepopulated to improve quality and save time.

So why does this matter? The point is that when we leverage architectures that abstract the presentation, data, business processes, and business rules, it becomes really simple to integrate applications whether they are home grown or commercially built. Because of the abstraction layers in both our SOA and Microstrategy's architecture, both solutions can treat the other as a black box and simply communicate via services. What makes this even more attractive is if our customers or partners have their own BI tools that we want to integrate with our systems we can simply provide them with an XSD so they know how to format the XML message (of course this assumes that the proper security is in place).

Another benefit of leveraging our BI tools as an abstracted presentation layer is that we can take advantage of many out-of-the-box features from the BI platform. Features like subscription services, alerts, flash enabled emails, mobile support, scorecards, and dashboards are just a few of the many rich features that you do not have to build from scratch. But the big bang is that your CIO will be happy to see you leverage the company's large investments in both BI and SOA while wowing your customers at the same time.

In response to the recent surge of bad publicity that SOA has received, I have been screaming from my soapbox that SOA is the real deal, it's just the people who keep screwing it up. I offered my recipe for success and my ideas about change management. As I look back at my SOA journey, I identified one new role that I would add if I ever get an opportunity to lead another SOA initiative. That role is an Organization Change Management (OCM) specialist.

What is the OCM Role?
The OCM role is critical to the success of any large scale culture changing initiative. This person is responsible for accessing the readiness of the organization to change and then creating a change management strategy to help the organization transition from its current state to the desired future state. Developing a communication plan is also a key deliverable. People at all levels of the organization need to receive frequent communications of the impact of change, when the change is coming, what it means to them, how their jobs will be impacted, what the deliverables are, what training they will receive, and when will it be delivered. To make matters more challenging, each layer within the organization will need this information in a way that makes sense to them. For example, developers will want the low level detail, the business will want it in business terms, the financial people want it in dollars and cents, and senior management want an executive summary. There is no one communication fits all.

Then there is the skills assessment. What skills do we need? What can we address with training and what do we need to go outside for? Do we need to change our incentives and rewards programs, our recruiting process, our software development life cycle process, etc.? Does our existing job titles and pay scales make sense for the skills we need?

You can see that from this list of questions the scope of this role is more public relations (PR) and human resources (HR) then it is architecture.

Who fills this role?
There are a few options. This can be filled internally by an executive sponsor, an HR generalist, or even a project manager. However, only fill this internally if the person has clout and is near 100% dedicated to the project. If they have another day job then don't bother. External resources are also an option. This has many benefits. First, you can get someone who lives and breaths change management and has been through many of these types of initiatives before. Why reinvent the wheel and learn this from scratch when there are experts in the field. Second, the hours needed each week may fluctuate. You may need 40 hours one week and 20 the next. With an external resource you can pay only for the hours you need. Third, it is advantageous to have a fresh perspective from an outsider who is not tied to existing processes and cultural barriers. Fourth, this person can give candid feedback to the higher ups without worrying about getting fired. For example, let's say a senior executive is not pulling his or her weight in one area of the project. The consultant can confront that person and in extreme cases go above that person to provide feedback without worrying about their next pay check.

What are the benefits?
Investing time and money in change management can make or break a project. In my case, several of us took on various tasks in this area, but collectively we did not have sufficient time to communicate at the level we needed to. Some areas we didn't even get a chance to address. Leveraging an OCM specialist greatly improves communication, addresses project risks, helps people adjust to change, and greatly reduces resistance. You can survive without this role, but it will be much harder, cost more, and take longer due to resistance to change and communication gaps. Take it from someone who has spent the last two years fighting the good fight, change is good, but dedicated change management specialists are better.

Here is my presentation from Zapthink's Practical SOA conference in New Jersey back on March 25. My topic was How to Sell SOA to the Business. My three key take aways:

1. Align SOA w/Business Drivers
2. BPM is the Killer App
3. Speak in Business Terms
   

Click this link and go to the 4th video (you must register on Zapthink...it's free)


Feel free to leave comments or send me your questions at mkavis@yahoo.com.

Reading about SOA is like watching a tennis match. One day it is the answer to everyone's prayers, the next day it is too complex and the business doesn't want it. The arguments go back and forth day in and day out. I can tell you that SOA done right can totally transform a company. There is no one way to approach it but I will share the method that we used.

Step one - Research the living daylights out of SOA. Read blogs, go to conferences, interview companies who have had success, read books, talk to consultants who have been through many implementations.


Step two - Find a business problem to solve. What is a major pain point for the business? What key business drivers or goals can you align with? Can you create revenue opportunities or cut costs by reengineering business processes? If so, sell SOA as the technology to enable BPMS tools.

Step three - Find a strong executive sponsor on the business side to support this initiative. Remember, this must drive business value in business terms. Reusable business services means nothing to the business. Reducing costs, increased sales, and improved customer satisfaction does.

Step four - Create a multi year vision supported by a road map of projects that deliver incremental value to the business in short time frames (2-4 months or less). Produce a business plan with supporting financials for each and every project within the road map. If you try to justify the entire road map as one number, you are putting the whole initiative at risk. We found that justifying each project as a stand alone deliverable allowed us to get most of the projects approved and funded. Don't make it an all or nothing proposition. You will likely lose.

Step five - Present plan to executives, including vision, road map (future state), and financials. Show the ROI, IRR, and your best estimation of when the break even point is. It helps to know your executives and what their thresholds are. If you know this information, keep working the plan until you can produce a road map that meets their needs for an acceptable investment. If you can't produce numbers good enough, either you picked the wrong initiatives or the time is not right for you to pursue SOA.

Step six - If you successfully secured funds, now the fun really starts. Now comes the SOA planning and change management (culture) challenges. Go to websites like Zapthink or take their 4-day bootcamp to figure out the best way to approach SOA. No need for me to publish that here when there are many great resources available on that topic. Here is a link I did on change management a while back.

That is a recipe that has worked for me. As I said before, the problem with SOA is not the technology. Only people can screw up SOA. That doesn't imply that SOA is easy, but all failures point to one or more of the following:

• No business buy in
• No executive support
• Poor project management
• Lack of understanding of SOA concepts
• No governance
• Doing SOA "on the cheap"
  

None of these are problems with technology.

Recently the SOA discussions have turned ugly and everyone is talking about SOA failures. As Eric Roch said yesterday:

It is troublesome to hear SOA architects argue about the arcane details of SOA implementations, especially when there is no grounding in actual requirements.

Eric goes on to say that...

SOA should have a business sponsor and a steering committee like the huge IT initiative that it is. SOA should change the business and for the better! In the age when the vast majority of IT are buying not building applications many SOA pundits talk as if we can throw everything away and start over for the sake of architecture purity. The fact is we are stuck with legacy systems and are constrained by the SOA strategy of the application vendors like SAP and Oracle.

Eric seems to be the only person out there who agrees with me that SOA should be sold to the business. I have constantly been told my numerous industry experts and popular bloggers that this is impossible and is a total waste of time. Maybe I need to clarify my position and finally convince some people that this is the way to get the business to support and fund your SOA initiative.

Selling SOA to the Business
When selling SOA to the business, you must talk in business terms. In fact, you shouldn't even mention the word SOA. My company went through an impressive growth spurt over our first two decades growing at rates of 20-30% annually. Over the past few years our growth has peaked and now we are growing at a more modest rate. Now that things have calmed down from the high growth years, we are looking at ways to control costs. One area that has tons of opportunities is our business processes. We had been moving so fast for so many years that our processes have become largely outdated and inefficient.

A few of us in IT saw a huge opportunity for the business to reengineer their business processes. We convinced the business to hire a consulting firm to perform a business process analysis. The consultants documented the current state processes, worked with us to derive the future state processes, and performed a gap analysis that uncovered a huge opportunity with a very large ROI. Now the business was motivated because they could envision a future state where they would improve the time to market for our products, improve customer satisfaction, reduce the learning curve of their systems, and streamline operations. They were hungry for change.

That opened the window of opportunity to sell SOA. And sell it we did! We did not talk about SOA in technical terms. What we did tell them is that by leveraging a "new" technology, we could automate their entire workflow from contract to delivery without having to rewrite any of our backend systems. This new technology would allow us to connect their new web based systems to the existing systems through an "adapter". Of course, by adapter I mean services, but that term means nothing to the business. So here is what the business heard:

• BPM will make our life easier, eliminate waste, and save us a ton of money
• If we buy IT this "new" thingy, we won't have to wait years for them to replace our current systems.
• IT will be able to deliver these new web based systems this year!
  

The result, the business funded both the BPM and SOA technologies and three years worth of projects as defined in our 3 year SOA road map. We created a steering committee led by the business that is the driving force of our BPM/SOA projects. We delivered our first project within a few months of buying the software, and have the next eight projects in the hopper to be delivered over the next two months. We are now starting another round of projects for a different business unit and discovering that we are already achieving a 30% reuse rate. All of this in less then 10 months.

So who says I can't sell SOA to the business? And who wants to argue that BPM is not the killer app for SOA? I am sure that I will get the same responses from the same people, but my project is living proof that IT can get the business to drive SOA. Without the business on our side, SOA would be nothing more then a fantasy for us.

My company is hosting the current Zapthink LZA SOA Bootcamp this week. We have a mixture of attendees from various government agencies, consulting companies, and from the marketing and advertising industry. The class is being taught by Jason Bloomberg from Zapthink who knows SOA inside and out. I highly recommend this class for any company who is trying to implement SOA.

Day 1 was focused on the fundamentals of SOA and service-oriented analysis and design. I'll highlight some of the key take aways below.

Companies require agility. IT needs to respond faster to change. We must architect to change.

The reason for architecture is to support the problems of the business. In my opinion, many times architecture focuses solely on IT and forgets that we are here to enable the business to meet its objectives.

Service Oriented is a business concept. Many people think SOA is all about connecting things. SOA is really all about enabling flexible business processes and providing agility.

SOA is architecture. You can not buy SOA, you do SOA! It is a set of best practices and the discipline to follow them. SOA is all about abstraction and creating loosely coupled business services. These services should be interoperable, unbreakable, reusable, and composable.

SOA is not Web Services and Web Services are not SOA. SOA is an architectural approach while web services are a standards based interface. You can do SOA without web services.

SOA is distributed computing. SOA shifts us towards decentralization and empowers the business. It simplifies things for the business but greatly increases the complexity for IT. IT can no longer ignore governance if it wants to successfully implement SOA.

SOA doesn't replace anything, it extends it. The beauty that I see in SOA is its tie to BPM. I can deploy a brand new workflow driven, composite user interface that leverages SOA to extend the life and value of years of legacy systems, without having to spend years rearchitecting fully depreciated and working systems. In other words, I can provide the business with more modern tools and optimized workflow without blowing up my existing systems.

SOA is not about portability, it is about interoperability - If your implementation is dictating a certain technology, then you are not doing SOA. You must think in terms of the business. The business doesn't care if the you use Java or .Net, the business only cares that it works. You must architect the business service in a way that the underlying technology does not matter. Changing the underlying technology should not impact the business services. Think of the universal remote control. You can program it to work on any brand and on multiple devices (TV, VCR, DVD, CD, Home Theather, etc.). The remote control has no idea how the technology is physically implemented or even what it looks like. It does know how to send messages back and forth to the devices.

SOA is declarative not programmatic. This means that the business logic is configurable as opposed to being hard coded. SOA provides the ultimate flexibility and agility.

SOA supports business goals. With this mindset, software plays the supporting role as opposed to dictating how the business works (ie. SAP, PeopleSoft, etc.).

BPM is the killer app. You can't do BPM without SOA and visa versa. Integration is a side effect of enabling business processes.

This summary is just a fraction of the wealth of information that was shared today. In addition to the great lecture and corresponding slides, the class had numerous group exercises that allowed the different companies to share their personal experiences from their SOA projects. These experiences were just as valuable as the presentation. All in all it was a very productive first day!

Thanks to Joe McKendrick I found a great post on the Financial Times by Kaushal Mashruwala called SOA strategy and execution is failing in many companies. The article discusses how companies are struggling with the value proposition of SOA and how the business tends to shy away from the next big IT buzzword.

One key reason is that the SOA value proposition is fundamentally unimportant to business people. It’s just another way to implement an application.

The article goes on to recommend combining BPM with SOA to give real value to the business.

Give business people a reason to care about SOA—give them BPM. It gives many businesses an approach that balances the architectural benefits of SOA against the cultural shift that business people are demanding.

I have been singing this tune for quite some time now (Selling SOA to the business). Many fellow EA bloggers called me out on my article saying that it does not make sense to talk to the business about SOA and that Top Down SOA is the only way to go. I disagree. There is no single way or silver bullet for delivering SOA as I mentioned in SOA Theory vs. Reality. I sold the business on SOA by explaining to them in business terms how SOA is the technology that enables BPM. The business understands BPM and how process reengineering and optimization can eliminate waste, improve customer and employee satisfaction, and provide visibility into the the state and health of business processes.

As a matter of fact, we were so successful in selling BPM with SOA that when the CEO announced the top 5 initiatives for the next three years, Business Processing Reengineering made the list. The company is now investing in training, tools, and various key initiatives which will leverage BPM and SOA. Without the link to BPM, none of this would have been possible. Like Kaushal mentions at the end of his post, if we had not combined BPM with SOA, our SOA would be SOL!

I will be sharing our story at Zapthink's Practical SOA conference next month.

One of the challenges of process improvement initiatives is getting the subject matter experts (SMEs) to think outside of the box. Quite often, people defend existing processes even though they know that there is a better way to do it. I often hear statements like "sales insists we do it this way", "we can't expect our customers to change their processes", and my favorite, "we don't have time". We need to get out of our old habits of living under constraints of legacy rules and start challenging why these processes exist and see if they still support today's business.

So when people declare that "sales insist we do it this way" we should find out why. What business reason does this rule apply? What is the cost of this business decision? Is there a better and more cost effective way to do this? What would it take for this rule to change?

When people say "we can't expect our customers to change their processes", do we know that for sure or are we making assumptions? Have we engaged our customers in a discussion about process improvement? Are they happy with the existing processes? Before we write off any process improvements we should first ask the questions.

When people say "we don't have time", do they realize that this excuse is the underlying reason why the processes are ineffective to begin with? Always taking the short cuts leads to ineffective and non-value added processes that waste company time and money. If you don't take the time now, don't expect to find time later to fix it.

In an effort to break the bad habits of old, we must constantly educate people on the value of process improvement and the true costs of waste and low value processes. When people resist changing an obviously inefficient process, you owe it to your company to not give in without at least asking some questions. Here are some questions to ask:

• What percentage of time does this occur? (does the 80-20 rule apply?)
• How frequently does this occur? (are we addressing something that happens infrequently?)
• Does this apply to all customers or a few? (are we addressing our high or low value customers?)
• If we were starting from scratch, would we still do it this way?
• What problem does this solve?
  

In addition to asking questions like these, we should perform an analysis to discover what the cost of an existing process really is. Not everyone understands process improvement, but most people understand money. Computing the cost of low value processes is often an eye opening experience for users. Once they see the expense, the amount of waste in labor, and/or the impact on quality, they become more open to change. Simply telling them to change without any real data is a tough sell.

Over time, the users will experience the power of process improvement as they work on new systems that leverage BPMS tools. The more hands on time they get with the tools the more they will see the benefits in the form of ease of use, visibility into the workflow, assess to information, and improved quality. When you first launch into your BPM initiative, these benefits are not as obvious. The best way to break them of bad habits is to quantify the costs of the waste. You will not always be able to convince people of a better way, but if you simply accept defeat without asking the questions, you may miss a huge opportunity to make a significant impact to the bottom line.

Another lesson learned I encountered recently is to clearly define the intentions of your BPM projects. There is a huge difference between attempting to improve processes versus automating processes. First let's discuss the pros and cons of each:

Process Automating in a BPM world

Pros

• Quick wins (unless processes are extremely complex)
• Less culture shock
• Reduce human error
• Start collecting process metrics baselines to analyze future process improvements
• Reduce paper, fax, emails, etc.

Cons

• Only small efficiency gains are realized
  
• Automating non-value add processes
• Missed opportunity to optimize work flow
• May never get a chance to improve processes

Process Improvement in a BPM world

Pros

• Reduce or eliminate waste
  
• Potential huge return on investment
  
• Improve employee performance and customer experience
• Reduce complexity of systems
• Financial justification of process steps
  

Cons

• Potential culture shock issues
  
• Requires more executive support
  
• More expensive to implement
  
• Requires more attention to change management

Like any other large enterprise initiative, a company must have a vision or a roadmap. Some companies may want to achieve financial benefits as soon as possible. For those companies, they should embark on a process improvement initiative right out of the gates. This means that the company must invest in training the business on process improvements with tools like lean sigma, TQM, and others or bring in talent to assess and recommend new processes.

Other companies may want to ease into the culture transformation and start with process automation. This allows the employees to get familiar with the BPM tools and to start collecting data about the current processes (time, costs, bottlenecks, etc.). This information can lead to process improvements down the road. The downside to this approach is that it will take the company much longer to reach the "promised land" of having streamlined and cost effective processes. The real danger that I see is other priorities may take precedence and the company may never get the opportunity to eliminate waste. In my opinion, process automation by itself may not justify the expense of purchasing a BPMS tool. The real value is in process improvement.

Fellow ITToolbox blogger Vladimir Stojanovski wrote a good post on this topic several months ago called Synergies of Process Improvement and Process Automation. In his post he pointed out some interesting statistics that I will mention here:

• Process automation resulting from technology investments yields 2 percent productivity gains. These are the types of projects where technology is used to make old, tired, and broken processes run faster.

• Process improvement through re-engineering yields 8 percent in productivity gains.

• When done together, productivity improvements from both process automation and improvement yield 20 percent in productivity gains.

Mark Smith wrote a post back in 2005 called Process Automation Can Undermine Performance. The key take away from this post is his closing statement:

To succeed at BPM, assess for success, think beyond automation and make performance improvement your number-one process improvement goal.

My point is, the real value in leveraging BPM is process improvements. Process improvement combined with process automation is a powerful combination that can bring real value to an enterprise. Process automation by itself is nice, but it still allows waste to exist in the organization. But whatever your BPM strategy is, make sure it is clear to all team members what the direction is. If some people have an expectation that the goal is automation, while others think the goal is improvement, you will create a lot of unnecessary conflicts.

Many articles that I have read speak about the complexity of BPM and SOA technologies. In my experience, the most complex part of implementing these enterprise wide initiatives is dealing with change. And the number one way to address change is to have a solid communication plan. But is a good communication plan enough? Managing perceptions, attitudes, fears, and confusion takes more then a communication plan. The execution of the communication plan is where the rubber hits the road.

What I am getting at is the Chief Architect, the business sponsors, the enterprise architects, the CIO, and all of those people who are thrust in the spotlight need to be consistent in their message, their attitude, their goals, and their delivery.

With many culture changing initiatives, you have people who are ready for change, people who fear change, and people who take the "wait and see" approach to change. The wait and see people are like the swing voters in the upcoming elections. They need to be convinced before they commit. What that means is that they scrutinize everything about the candidates from what they say, to how they say it, to how they react to certain questions and circumstances. They same thing applies in your BPM/SOA implementations. If the people driving the change show signs of weakness, frustration, or stress the "swing voters" will pick up on this and may be turned off. The people driving these initiatives must also speak publicly as careful as a politician so not to say things that can cause the swing voters to run away.

Early on in my implementation, we had some challenges with some of the vendor tools that we purchased. I was pushing the vendor hard to resolve the issues but also expressed some frustration with the vendor that many swing voters picked up on. The next month or two I was in damage control mode fighting the perception that our SOA stack was a pile of garbage. The SOA tools in the market place today are early in the maturity phase and tend to have various integration issues across the various components of the stack. That doesn't mean that the products are garbage, it just means that stabilizing the new environments are challenging. Because I am very visible on this project and people saw my frustration with the vendor, I had to spend a lot of energy changing the perception of the tools. This is just one of many examples of how the swing voters' perceptions were swayed by the actions and delivery of both verbal and non verbal communications.

After dealing with and heading off many issues, perceptions, and fears, we have put together a more strategic plan for communicating going forward. The business leaders and the CIO will now send out frequent communications to all employees on the progress of the projects. We are spending more time showing demos and prototypes to anybody who wants to see them. The architecture team started blogging about the vision, the technology, and the project. But the most important part of these communications is the delivery. All of these communications must be delivered with confidence, consistency, and must explain the value of the initiative to the audience. We must think of our audience as swing voters looking to decide if they want to join the campaign or turn and run. After all, it is the swing voters contributions to the cause that will make the difference in the long run.

Eric Roch wrote a great piece today called The Difference between SOA Theory and Reality. I started to comment on it but my comments turned into this post. Eric's key point is that there is no one right way to implement SOA.

The reality of SOA is that every company is unique with widely different systems (as-is architecture) and culture .... The approach to governance and organizational structures will depend on the culture and capabilities of the organization.

This is true for all technologies not just SOA. I have read countless articles about the best approaches to delivering SOA, Enterprise Architecture (EA), ERP solutions, Business Process Management (BPM) and many others. Bloggers get into religious wars about how their approach is the only correct approach. These bloggers might generate a lot of traffic but they are doing the community a disservice. As Eric says, every company is different. In some companies, IT budgets are handed to them at the beginning of the year. These companies have to make due with the dollars they have and running a top-down SOA project is virtually impossible. In companies like mine, the business is funding the SOA and BPM initiative. We have specific projects that justify the expense of the infrastructure and capital investments for SOA. Once again, Top-down does not work in this scenario. This requires a bottom-up approach since the business is expecting new functionality every couple of months. In other companies, IT can be more strategic and budget for a multi-year SOA implementation. In this scenario, a top-down approach can be a reality.

Whether it is EA, SOA, ERP or others, it really comes down to these variables:

• Business driver
• Level of executive support
• Culture
  
• Time
• Money
• Talent

The combination of these variables should lead you to the solution that will work in your world. The thing that people need to understand is that there is a difference between what is the best way of implementing technology from theory and text books versus what makes financial and strategic sense from a business standpoint. At the end of the day, it's not about the technology, it's all about the business.

Here is another lessons learned from my world, but this time the focus is BPM, not SOA. I have mentioned many times how we sold the business on BPM and SOA. After modeling the future state processes and creating a roadmap of projects based on a combination of business priority and service reuse, we took our business case to the finance department and secured funding for a number of high ROI projects. The funds secured were for procurement of BPM and SOA tools (BPMS, ESB, Data Services, etc.) and for capital labor to work on the various projects. We also were able to justify two new positions in a new department in the business who is responsible for business processes. Sounds good so far, right?

Well, we should have funded one more thing and that is an initiative to change our culture to be process centric. The two new resources are consumed in the new projects which means the process of becoming a process-centric culture will take a very long time. It takes focus and dedication to the cause to train staff, hire experienced talent, and move the organization to a place where optimizing business processes is the norm. So what is the down side?

Although we are providing a ton of value by automating processes, connecting legacy systems, providing visibility into the workflow, shortening the order life cycle, and leveraging operational dashboards and reporting, we are not creating processes that are as efficient as they could be. In some instances, users are asking for the same functionality and processes that exist today. There is still too much focus on reports and not enough focus on data and there is still too much emphasis on exceptions and not enough standardization.

Is it a disaster? No. Could we be developing more efficient processes? Yes. I am hoping that with each iteration we will improve the processes and the business folks will become more process-centric over time. I do feel that if we could have justified an initiative to create a process-centric culture, we would achieve a higher ROI and see the benefits of it quicker.

So the lesson learned that I would like to share is don't forget to invest in change management initiatives. Getting funds for a culture change initiative is not easy, but you should strongly consider trying. Some companies already have business process centers of excellence or practice methodologies like TQM, Six Sigma, Lean etc. For those who don't, the earlier the culture becomes process centric, the better the Return on Investment.

Here are a few of my other lessons learned from past posts:

• Boiling the ocean
• Things we did right
• Things I would do differently

I just posted my top 10 posts for 2007 and all of them were related to Linux and open source (thanks Digg!). But my real passion for writing is enterprise initiatives like SOA, BPM, Enterprise Architecture, Project and Portfolio Management, Outsourcing, and any initiative that requires a combination of leadership, technical knowledge, and people skills. So here is my top 10 posts for all non open source topics in 2007:

1. How did we become a Dilbert cartoon?
2. What does it all mean?
3. Selling SOA - a True Story
4. Top 5 Bone Head Moves
5. Outsourcing vs. Offshoring
6. Adobe Flex beats Silverlight every time
   
7. Zap Think's Who's Killing SOA
8. Another Benefit of SOA - Career Path
9. Why are you still generating reports?
10. SOA and BPM Consolidation

I would also like to list a few of my lesson learned posts because I believe these types of posts add the most value to the community.

1. SOA Lessons Learned
2. Lessons Learned from our first SOA implementation - part 1
   
3. Lessons Learned from our first SOA implementation - part 2

Happy Holidays!

I read an interesting article from ITBusinessEdge's Lorraine Lawson today called Coping with Disruptive Technologies. She discusses some of the resistance to "Enterprise 2.0" technologies like wiki's and blogs and how the enterprise users are demanding more empowerment. Those people who resist these technologies and laugh at wikis need to first get out of their boxes before they can think outside the box.

As an architect, my job is to build for the future. To do this, I must not constrain myself with the present state. My philosophy has always been to start with a clean slate, remove all rules and constraints, and then think outside of the box. Once the vision is formed, then you can start looking at constraints and start chipping away at the vision until you form something that is attainable.

I am working on a BPM and SOA initiative which challenges us every day to forget about our years of legacy business processes and system constraints and build a future state that reduces costs, improves speed to market, increases efficiencies, and ultimately improves customer satisfaction. On the business side, there are huge opportunities to change the way we collaborate with our customers. On the IT side, this is a drastic shift in the way we approach software development. On both sides, we need to get out of our boxes to create our future state roadmap. Everybody is buying in to the reasons why we need to do this but many are struggling with the future state vision.

Most of the resistance is due to the inability to let go of how things are done today, the rest is fear of change. Today we were listening to requirements on how to deliver a particular business service. The requirements were extremely dynamic almost to the point where it required artificial intelligence to satisfy it. These requirements are all based on the manual way we meet the current business need with spreadsheets. Automating manual spreadsheet manipulation and archiving is probably not the best strategy. Unfortunately, the current business process is extremely hard to automate due to lack of standards and the emphasis on exceptions and customization. This process has evolved through 20 years of business and probably needs to be totally revamped. Everytime I mention this I am informed of the many constraints that prevent this from happening. I don't disagree with the arguments, but those arguments are based on the box that we live in today. To reengineer these processes we must leave our box and derive a solution without constraints. Once we have done that we can brainstorm the new processes and modify them until they satisfy the business needs. Without leaving our box, we will never be able to overcome the limitations of the existing processes and will struggle to accomplish any automation that can improve on our current manual processes.

On the IT side, there is concern for the new SOA governance processes, agile methodology, short delivery cycles, and the shift from text heavy artifacts to model/visual based artifacts. Most of the feedback is based on today's constraints. I totally agree with the concerns. If we constrain ourselves with these new processes the way we constrain ourselves with the old processes, we will fail. However, if we leave our box of constraints behind and think outside the box, we can be successful with the new processes.

As I look at the emerging technologies like Enterprise 2.0, BPM, SOA, and others, the companies that will take advantage of these will be the ones not living inside their box. It is time we started paying attention to social networking, software as a service (SAAS), mobile workforces, open source, and other technologies. It's time to get out of our boxes.

I have read a lot of articles recently claiming that BPM & SOA are nothing more then hype. I am here to tell you that if you align these technologies with the proper business case, you can transform your company. I am watching this transformation happen right before my very own eyes. We had an all day future state meeting today and the eighteen people from a cross function of the business were "building the new company from scratch". It was a great moment for me. Just over a year ago, IT sold the business on BPM and SOA. Last week we delivered our first implementation of a BPM/SOA enabled B2B portal. When we did our first round of process reengineering, the company was new to the concepts and skeptical of the investment in the technology. Now, one year later, we have a dedicated architecture team, a dedicated business process department residing in operations, BPM and SOA technology proven in a production environment, and BPM is one of the companies top five initiatives in our four year corporate strategy.

In today's future state sessions, there were eighteen highly motivated and deeply engaged people who believe in the vision and the technology. Our company is transforming from being reactive and heavily labor intensive to proactive and process oriented. We have extremely aggressive growth goals that our existing processes cannot sustain. By using BPM and SOA, we will totally reengineer the way we do business.

At the beginning of the session I explained how SOA will allow us to wipe the slate clean and build our new processes from scratch. I explained in business terms how SOA is the bridge between our brand new business processes and our legacy systems. In our current state findings meeting, there was a lot of concern about "blowing up" all of our systems. In order for us to not constrain our thinking, I thought it was important to start today by explaining that we do not need to touch our legacy systems, instead we simply "connect" them to our business processes with SOA. Below is a slide similar to the one I showed them:



The way I explained it was that in the past we had a separate UI for each one of the systems in the blue circles. Each UI looked different, had redundant data across the applications, and were disconnected from each other. In the new world, there will be one composite UI that is driven by our new business processes and "talks" to the legacy systems through adapters (really web services) while hiding the complexity of all the legacy systems. In other words, they no longer need to care about the order entry system, the accounts payable system, the inventory system, etc. Now they can focus on the process which is designed by the business in the BPM tool and IT can magically connect their processes to our existing systems with a minimal amount of effort (once we built the infrastructure).

Once they understood this, they started building our new company from scratch. It was almost like someone freed their minds and took the handcuffs off. At the end of the day, we had a vision of what our new world might look like and it looks good. No paper, portals, dashboards, workflow, alerts, KPIs, self service, real time data, electronic approvals, etc. That sure beats the sticky notes, emails, phone calls, reams of paper, and rows of filing cabinets. This is all possible because of BPM and SOA. So the next time someone who is afraid to change starts calling the hype card, send him/her this post. Better yet, have them email me. These technologies are the real deal!

A couple of days ago I wrote part 1 of this series and highlighted the things we did that made us successful on our first SOA implementation. Today, I will discuss the things that we should have done differently. Before I start, I would like to note that we are working on six different projects concurrently under the parent program. So some of these "what not to do's" apply to all of these projects not just our initial one that we just rolled.

What not to do...

1. Not enough professional services - Our budgeting process only included enough hours to install the stack. We should have covered a few more weeks of professional services for when we first deployed code into the development and testing environments. About a month after we installed the BPMS and ESB, we deployed our first build on these environments. We encountered all kinds of environmental issues and various integration issues between the various components of the stack, especially the portal and BPMS. Since we didn't have budget for any more professional services we tried to figure it all out ourselves. Eventually we brought the vendor in and ate some incremental costs but we lost about a month which delayed the project.
   
2. Needed more vendor references - This is not for the selection criteria but for implementation. We should have sought out more companies that had been through an implementation using these products. We are networking with a few local companies now but it would have been nice to have heard their experiences earlier and we might have prevented some of the setbacks or resolved issues quicker.
   
3. Dove in too fast - This one we didn't have much control over. The downside of having the business fund SOA is that they want results immediately. We started our first project with no governance, no testing strategy, no automated build process, etc. We are trying to implement governance along the way but it always takes a back seat to due dates. Fortunately, a new budget starts in January and we can get the resources we need to get these important tasks done.
4. Threw the kitchen sink at the first iteration - I wrote a lessons learned piece a few weeks back which prompted Todd Biske to write a good article called SOA and the Kitchen Sink. We tried to do too many things currently and had a few instances where we were thrashing.
   
5. Not enough internal resources involved early enough - We partnered with a consulting firm to help teach us how to properly architect software in the new world of SOA. At the same time I have been working with the leadership team in IT and the business to free up more internal resources so we can start doing this ourselves. This has two bad side effects. First, very few of our own people were recipients of the knowledge transfer and two, the vendor does not know our business which has led to some gaps in requirements and some architectural decisions that were made that we have had to change.
   
6. Needed more change management - I did spend a lot of time focusing on communications & organization impacts but there was still a lot of "we vs. they", resistance, and confusion about the overall direction. The core team was has always been on the same page, but many spectators are still in the dark. This often leads to resistance and negatively which we deal with immediately, but it would be nice not to have to spend as much time resolving this issues.
   
7. Didn't research SOA testing enough - One of the things we did right was research the heck out of SOA and BPM. But one area that we didn't invest much time in was the testing. We are actively researching the heck out of it now but my advice to others is don't under estimate the changes required to test SOA.
   
8. Occasionally got away from project management best practices - We fell behind by a month because of numerous issues with our environments and some bugs in the stack products that the vendor had to patch. Once we resolved all of the issues we only had two weeks left to deliver our new B2B customer facing portal to our client. We threw everything we had at the remaining tasks but somehow forgot about some of the basics of project management like risk mitigation and we had several communication breakdowns. We coined it as "running around with our hair on fire". On the positive side, we all acknowledged our mistakes and are in the process of fixing this.
   
9. Taming requirements - Requirements have been too volatile and it is hard to be agile when you can't draw that line in the sand until the next iteration. With some of the business services, there does not appear to be enough ownership and accountability. This has caused a lot of rework and has been a big source of frustration.
   

If you read this article without reading part 1, it probably sounds like we failed miserably. In reality, we were highly successful. But with all of the issues I listed above, it was a lot harder then it should have been. Part of the challenge was that we were learning SOA, BPM, and agile on the fly while trying to meet highly aggressive dates. My goal here is to share my experiences, good and bad so others can learn from our success and mistakes as they prepare to take on their SOA implementation. Feel free to contact me if you ever want to discuss these experiences or offer suggestions.

I have been blogging about our BPM and SOA initiative all year long. Yesterday, we deployed our first application, a customer facing B2B portal that spans our sales order process. Although this is just the first iteration of many and we are still a few months away from delivering the features that will give us the big ROI, this was a major milestone for us. So I would like to take this opportunity to share a few things we did right. In part 2, I will discuss a few things I would do different next time.

Things we did right

1. Focused on business not technology - we performed a 90 day business process assessment where we documented current state, brain stormed on future state, and defined a portfolio of projects that each had their own ROI. This allowed us to sell the business on BPM and SOA and helped us secure the funds required to take on SOA.
   
2. We did our homework - We attended conferences, read blogs, researched web sites, collaborated with experience architects, and a got our hands on anything or anyone that had any information about SOA.
3. Thorough POC - We invested a lot of time in an extensive vendor assessment for both the BPM and SOA tools. When we narrowed it down to the short list of vendors we brought them each in for a full day proof of concept. I wrote a piece on CIO.com called the Proof is in the Pudding and another called Beware of Eye Candy that talks about some of the scary things you might find when you start going beyond the Power Point slides and start actually trying to execute processes and services.
4. Surrounded ourselves with talent - We put several of our best people on this project on both the IT and business side. Not only did we look for talent, but we looked for motivated people who embrace change and bought into the vision. We also partnered with an implementation company with years of SOA experience including working on some of the largest SOA implementations in the world. We embedded our architects with our SOA partners to accelerate the learning curve.
   
5. Strong executive support - Our executive sponsor is from the business and the person who went to the top to get the funding for the project. Like a couple of us in IT, she put her career on the line by signing up to deliver substantial ROI numbers over the next four years. You can bet that we have all the support we need. Recently, the CEO declared this one of the key initiatives over the next four years so now the whole company is behind this.
6. Squash resistance instantly - Any time we encountered or were told of resistance or negativity, we immediately addressed it head on. Some of these occurrences turned out to be great learning opportunities while others were the result of lack of buy in. In each case, we tried to understand the root cause and resolve the issue before it spread like a fungus.
7. Short, iterative deliverables - Many companies get themselves into trouble by trying to tackle too much out of the gates. We took a look at all of the projects that we identified and used a combination of portfolio management and a SOA roadmap to determine the priority and size of the scope for each project. By laying out the long term vision, we were able to prioritize which services to build first to maximize reuse and reduce the overall cost of deployment.
8. Sound architecture - We are laying the foundation for a loosely coupled, service enabled architecture that will give us a platform to quickly react to change. We are leveraging both Portal and BPM technology to create new tools that empower our users with self service capabilities, visibility into the workflow, and access to KPIs through Business Activity Monitoring (BAM). We are also leveraging SOA to connect this new front end to our years of legacy systems while hiding the complexity from the users.
9. Having Fun - We are all under intense pressure to deliver the numbers we promised in aggressive time frames. The work is challenging and we had a few "coming to Jesus" meetings along the way, but overall all we are extremely excited and focused on future successes. At the same time we are enjoying being part of an initiative that will transform the company.
   

I know this sounds like a bed of roses but this success has not come easy. In my next post I will describe some of the things that occurred that we will may sure never happen again. The list is as long as the one above.