Enterprise Initiatives

I just posted my top 10 posts for 2007 and all of them were related to Linux and open source (thanks Digg!). But my real passion for writing is enterprise initiatives like SOA, BPM, Enterprise Architecture, Project and Portfolio Management, Outsourcing, and any initiative that requires a combination of leadership, technical knowledge, and people skills. So here is my top 10 posts for all non open source topics in 2007:

1. How did we become a Dilbert cartoon?
2. What does it all mean?
3. Selling SOA - a True Story
4. Top 5 Bone Head Moves
5. Outsourcing vs. Offshoring
6. Adobe Flex beats Silverlight every time
   
7. Zap Think's Who's Killing SOA
8. Another Benefit of SOA - Career Path
9. Why are you still generating reports?
10. SOA and BPM Consolidation

I would also like to list a few of my lesson learned posts because I believe these types of posts add the most value to the community.

1. SOA Lessons Learned
2. Lessons Learned from our first SOA implementation - part 1
   
3. Lessons Learned from our first SOA implementation - part 2

Happy Holidays!

I was looking back at my first year of blogging to see what my best posts were. For those of you who have read several of my posts you know that I have two favorite topics: open source and SOA. I started blogging in March of this year and have written 120 posts. The top 10 were all dominated by open source articles. The reason is simple, Windows vs. Linux is a hot topic on sites like Digg, Del.icio.us, and Reddit. There are a lot of flame wars out there and not a lot of facts. Most of my posts were related to real work experience with open source and a few were dedicated to an experiment I did at home with several Linux distributions. So for 2007, here are my top 10 posts:

1. Open Source and Microsoft Free (54,000 hits)
2. Comparing Linux Distributions - Final Results
   
3. Dumping Vista - a Divorce with a Happy Ending
4. Open Source and Loving it!
5. Linux Mint is....Mint!
6. Eating my own Dog Food
7. Another easy Linux Install, Kubuntu Style
8. Review of Linux Distributions - Part 2
9. 10 Reasons why you need an Open Source Strategy
10. Review of Linux Distributions - Part 1

The top post was a hot ticket on Digg, Slashdot, Craigslist, Stumble, Del.icio.ous, and every Linux website known to mankind. Even though I wrote it several months ago, it continues to get a lot of traffic. Obviously, I am a big advocate of Linux. I have been using Kubuntu at work at a Microsoft shop for most of the year now and have been very productive without a single piece of Microsoft software on my laptop. Linux is ready for primetime. People aren't ready for Linux.

There isn't a day that goes by were I don't encounter an EA blogger who states his dislike for process. Earlier in my career I was very anti-process. I was working on small teams and sometimes I was simply the only one on the team. There was no established architecture and no standards. Since this was the norm and nobody cared, why bother me with tons of documentation. As I moved through my career I worked on larger projects that spanned multiple user groups and application silos. Suddenly, the lack of process became a recipe for failure. Now I am working on enterprise BPM and SOA initiatives that touch the entire company and practically every application under the sun. Process and Governance are now critical success factors.

For those who don't like process, I can meet you half way. I believe in a lightweight process that supports agile development but not process that creates bureaucracy and results in 12-18 month projects that are doomed for failure. My current project requires more process then any project I have ever worked on before. In the past I would typically scoff at project charters, team structure documents, accountability matrices, and all of the non technical type of artifacts. But this project consists of a consulting partner, that is half on-site and half remote, multiple internal application teams working on integrating various legacy systems to the new business workflows, an offshore support organization who will take responsibility for the code once it is approved through UAT (user acceptance testing), and business representatives from multiple departments. In addition we are working towards 12 week deliverables where we work on multiple workflows concurrently that need to plug in together at the end. Try that without process. Are you kidding me? There are over 50 business and IT people involved, there has to be some order to the chaos.

The trick is to not create so much process that you stifle creativity and make technical decisions based on process instead of business need. At the same time we are new to SOA and must closely govern the architecture so we don't create a long term mess and not realize the benefits of EA and SOA. So does this make me a process weenie? I don't think so. I do know that our first attempt at this I blew off the charter, the structure, and the accountability matrix and that led to confusion towards the end of the project. Next time around we will invest a few days nailing those steps down to prevent chaos down the road. From that point on this stuff will be boiler plate material that will just need minor tweaks as we move from release to release.

I know James McGovern is going to beat me up on this for both the process stuff and the outsourcing. Outsourcing is a decision our company has made so my job is to make it work regardless of what I think about outsourcing. I can complain about it and allow us to fail or I can make it work. The more external people you have working on the project (especially remotely) and the more areas of the business your requirements span, the more process you need. I never thought I would be pro-process but when you are responsible for large initiatives with large budgets associated with it, you better deliver on time and on budget or you will be introduced to the exit interview process!

James always promotes people over process and I totally agree with him. However, the people must have the tools they need to do the job. One of the most important tools is clear direction and good communication. When you have 50 or so people on a project you need some process to supply these tools. I agree with James that CMM is overkill (unless you are sending a space shuttle to the moon), but there needs to be a compromise. So I'll lob this one back to my EA peers out there. I am sure some heated debate will follow. Your thoughts?

I read an interesting article from ITBusinessEdge's Lorraine Lawson today called Coping with Disruptive Technologies. She discusses some of the resistance to "Enterprise 2.0" technologies like wiki's and blogs and how the enterprise users are demanding more empowerment. Those people who resist these technologies and laugh at wikis need to first get out of their boxes before they can think outside the box.

As an architect, my job is to build for the future. To do this, I must not constrain myself with the present state. My philosophy has always been to start with a clean slate, remove all rules and constraints, and then think outside of the box. Once the vision is formed, then you can start looking at constraints and start chipping away at the vision until you form something that is attainable.

I am working on a BPM and SOA initiative which challenges us every day to forget about our years of legacy business processes and system constraints and build a future state that reduces costs, improves speed to market, increases efficiencies, and ultimately improves customer satisfaction. On the business side, there are huge opportunities to change the way we collaborate with our customers. On the IT side, this is a drastic shift in the way we approach software development. On both sides, we need to get out of our boxes to create our future state roadmap. Everybody is buying in to the reasons why we need to do this but many are struggling with the future state vision.

Most of the resistance is due to the inability to let go of how things are done today, the rest is fear of change. Today we were listening to requirements on how to deliver a particular business service. The requirements were extremely dynamic almost to the point where it required artificial intelligence to satisfy it. These requirements are all based on the manual way we meet the current business need with spreadsheets. Automating manual spreadsheet manipulation and archiving is probably not the best strategy. Unfortunately, the current business process is extremely hard to automate due to lack of standards and the emphasis on exceptions and customization. This process has evolved through 20 years of business and probably needs to be totally revamped. Everytime I mention this I am informed of the many constraints that prevent this from happening. I don't disagree with the arguments, but those arguments are based on the box that we live in today. To reengineer these processes we must leave our box and derive a solution without constraints. Once we have done that we can brainstorm the new processes and modify them until they satisfy the business needs. Without leaving our box, we will never be able to overcome the limitations of the existing processes and will struggle to accomplish any automation that can improve on our current manual processes.

On the IT side, there is concern for the new SOA governance processes, agile methodology, short delivery cycles, and the shift from text heavy artifacts to model/visual based artifacts. Most of the feedback is based on today's constraints. I totally agree with the concerns. If we constrain ourselves with these new processes the way we constrain ourselves with the old processes, we will fail. However, if we leave our box of constraints behind and think outside the box, we can be successful with the new processes.

As I look at the emerging technologies like Enterprise 2.0, BPM, SOA, and others, the companies that will take advantage of these will be the ones not living inside their box. It is time we started paying attention to social networking, software as a service (SAAS), mobile workforces, open source, and other technologies. It's time to get out of our boxes.

In the day and age of cheap disk, fast processors, high bandwidth, and cheap memory, companies are analyzing data more then ever before. As the components of the infrastructure become more cost effective and data is becoming more accessible, IT is being asked to create reports like never before. This is an area where IT can either be an enabler for the business or a serious bottleneck.

As partners to the business, we must educate our users on the power of business intelligence and self service. The business will typically ask for reports which is a reactive approach to solving problems. We need to offer proactive solutions that provide alerts and KPIs to address issues before they become critical. In other words, lets use information to optimize our business instead of creating more non-value add processes where countless clerks dig through hundred page printouts looking for issues. Here is my philosophy on reports. Static or canned reports should only be generated if the information is an output of the system. Here are some basic examples of when systems should create canned reports:

• Bills
• Invoices
• Purchase orders
• Confirmations
• Receipts

These reports are fairly static and should not change too often. When gathering requirements, the reports above would have been identified as outputs from the system. Notice that these are not really reports, they are outputs or artifacts of the system.

So what is the problem with reports? I have seen many applications that contain upwards of 70-100 reports over my years in IT. Usually, you could boil these down to half a dozen subject areas. Over time, different users come and go and they all have their own ideas on how they want data presented to them. So IT is asked to create more reports by adding a field here, grouping a field there, shrinking the font and printing landscape, and so on. All of these requests are a waste of money and a drain on IT resources. Often, these requests go into the black hole of the backlog of simple requests that IT can never find time to get to.

This is where business intelligence kicks in. IT should create metadata to represent the complex relationships of various databases and present the user with a simple view of data in business terms. Then through tools like Microstrategy, SAS, Cognos, Business Objects, etc. we should empower the users to create their own reports. Through self service, the users no longer have to wait months or even years for IT to get around to developing yet another report that must be maintained. The users can now react to business demands and can tailor the output to meet their needs.

This is a mindset that must be pushed from IT. When users ask for a report, the business analyst must ask the user, "What problem are you trying to solve?" I have seen all too often where the user wants a report as a quality check in response to issues found in the data. My response is to fix the root cause and eliminate the data quality issue as opposed to adding another non-value added process that may live on for years before someone realizes that the step is a waste of time.

So I ask, "Why are you still generating all of those reports?" If you follow the trends in IT you will notice that today's users are becoming more empowered then ever before. Users are leveraging various Web 2.0 technologies to service their needs and to build their own composite applications. We need to provide them with that same easy access to data. Don't give them reports, give them information!

There are many critical success factors for implementing SOA. They range from strong executive sponsorship and business alignment to acquiring the proper talent and implementing the right amount of governance. But even if you have all of those ducks in a row, it all comes down to the basics of project management and leadership. The biggest challenges that I see implementing SOA are the same challenges that I have seen throughout my career implementing any other large initiative. This article called Avoiding Project Failure: It's not Rocket Science, gives an overview of the things that typically go wrong:

Typical problems here are scope creep, poor work-plan, lack of change control, poor communication and poor management of risks and issues.

The article goes on to state a few more killers:

There are many occasions during the lifecycle of a project for issues that may lead to failure. Examples of these include:

• Failure to define the requirements clearly, resulting in expectations not being met
• Cutting edge or new technology that causes unforeseen problems
• A poor technical design preventing the solution from being changed or scaled in the future
• Poor change control allowing change requests to cause the project to drift
• Changing priorities diverting attention away from core work

Since implementing SOA can be a culture changing event, the basic change management principles must be applied to prevent derailing the entire initiative. Here is a summary of the high points of this article:

1. Address the human-side systematically
   
2. Top level support
3. Involve every layer
4. Clearly communicate the business case
5. Create ownership, install change agents
6. Clearly and continually communicate the message
7. Assess the cultural impacts
8. Address cultural issues
9. Prepare for the unexpected
10. Define WIIFM (What's in it for me?) for each individual

I know that early in my SOA initiative, I spent most of my time digging into the technology. Once we defined what our target architecture was going to be, my role changed dramatically. I turned most of my architecture duties over to my enterprise architects and started focusing my energy on the project deliverables and the change management. I can tell you from personal experience that figuring out the target architecture was a cakewalk compared to what I am dealing with now.

So while many people lose sleep over the technical challenges of SOA, I stay awake at night worrying about scope creep, top level support, cultural issues, risk lists, and action items. If I put enough smart people in a "war room" I know we can tackle any technology problem, but coordinating the work of all these people is where the challenge is. The big challenge in my eyes is that SOA forces many different departments (inside and outside of IT) to work together as one cohesive team. This is not a technology issue, it is a people issue. So for those of you early on in your quest for SOA, don't under estimate the importance of project management and change management.

I have been pushing wikis at work the last few months as I mentioned in a previous post called another wiki use case. Every time I mention the word to anyone outside of my architect team I either get blank stares or I get the rolling eyes followed by laughter. Today I even got a "I guess we'll see a blog about wikis" followed by laughter. What is so funny about Wiki anyways?

We are rolling out a new version of our brand new B2B portal in the near future. We were discussing online help and I recommended we use a wiki help system. We are currently creating another large user manual that nobody will ever read. We can take that manual, open it with OpenOffice Word 2.3 and export it to MediaWiki format. Then we can host it from an install of MediaWiki specifically for our B2B portal. Then from the Help function on the B2B portal we can link right to our wiki and just like that we have clickable online help at no additional cost.

Ok, I hear you laughing. Well if this is so funny then how come some of the most popular websites in the world have wiki-style help? They may be using more sophisticated technology like Adobe RoboHelp 7, but the look and feel is similar.
Here are a few screen shots of wiki-style help from some popular web sites.



So if wiki is not funny at Amazon, eBay, and Southwest, why is it so funny at my shop? I guess we need to get out more!

I have read a lot of articles recently claiming that BPM & SOA are nothing more then hype. I am here to tell you that if you align these technologies with the proper business case, you can transform your company. I am watching this transformation happen right before my very own eyes. We had an all day future state meeting today and the eighteen people from a cross function of the business were "building the new company from scratch". It was a great moment for me. Just over a year ago, IT sold the business on BPM and SOA. Last week we delivered our first implementation of a BPM/SOA enabled B2B portal. When we did our first round of process reengineering, the company was new to the concepts and skeptical of the investment in the technology. Now, one year later, we have a dedicated architecture team, a dedicated business process department residing in operations, BPM and SOA technology proven in a production environment, and BPM is one of the companies top five initiatives in our four year corporate strategy.

In today's future state sessions, there were eighteen highly motivated and deeply engaged people who believe in the vision and the technology. Our company is transforming from being reactive and heavily labor intensive to proactive and process oriented. We have extremely aggressive growth goals that our existing processes cannot sustain. By using BPM and SOA, we will totally reengineer the way we do business.

At the beginning of the session I explained how SOA will allow us to wipe the slate clean and build our new processes from scratch. I explained in business terms how SOA is the bridge between our brand new business processes and our legacy systems. In our current state findings meeting, there was a lot of concern about "blowing up" all of our systems. In order for us to not constrain our thinking, I thought it was important to start today by explaining that we do not need to touch our legacy systems, instead we simply "connect" them to our business processes with SOA. Below is a slide similar to the one I showed them:



The way I explained it was that in the past we had a separate UI for each one of the systems in the blue circles. Each UI looked different, had redundant data across the applications, and were disconnected from each other. In the new world, there will be one composite UI that is driven by our new business processes and "talks" to the legacy systems through adapters (really web services) while hiding the complexity of all the legacy systems. In other words, they no longer need to care about the order entry system, the accounts payable system, the inventory system, etc. Now they can focus on the process which is designed by the business in the BPM tool and IT can magically connect their processes to our existing systems with a minimal amount of effort (once we built the infrastructure).

Once they understood this, they started building our new company from scratch. It was almost like someone freed their minds and took the handcuffs off. At the end of the day, we had a vision of what our new world might look like and it looks good. No paper, portals, dashboards, workflow, alerts, KPIs, self service, real time data, electronic approvals, etc. That sure beats the sticky notes, emails, phone calls, reams of paper, and rows of filing cabinets. This is all possible because of BPM and SOA. So the next time someone who is afraid to change starts calling the hype card, send him/her this post. Better yet, have them email me. These technologies are the real deal!

I stumbled across a web 2.0 company today called MizPee and just shook my head. I can't make this stuff up.

MizPee finds the closest, cleanest toilets in your area. You can add and review toilets, get some cool deals in your area and challenge your knowledge of toilet trivia.

When I see useless sites like this and sites like Facebook being valued at $15B I start to remind myself of the .com and real estate bubbles of recent memory. Another red flag is when you start seeing hilarious videos about the bubble bursting like the one below.

IT driven SOA
Over the past year I have collaborated with literally hundreds of IT professionals about SOA either at conferences, through networking, via blogging, or in person. From what I have learned, most SOA initiatives are driven by IT and the biggest challenge is getting the business to buy into the technology. The advantages of IT driven SOA is IT can take a top down approach to SOA, build the governance in from the start, and gradually ramp up their staff's knowledge of SOA. The downside is the business may not want or understand SOA and all you wind up with is an exercise on "what could have been". In summary, IT-driven SOA gives you the opportunity to do it right, but you risk the business rejecting it.

Business driven SOA
Business-driven SOA has a completely different set of pros and cons. Business-driven SOA gives you the strong sponsorship and the business resources required to align the technology to the business. It also makes it easier to tie BPM to your SOA initiative which greatly increases the value of the project to your company. Not only can you make great improvements in time to market, reusability, and reduced maintenance, but you also get operational efficiencies in the business by reengineering business processes.

So what are the cons? First, the business doesn't understand or care about governance. Once you buy the SOA stack, the expectation is to start delivering ASAP. In these scenarios, IT is typically forced to attack SOA from a bottom up or middle out approach. The risk here is that over time, you may never fully achieve a true service oriented architecture if you don't bake in the governance at some point. You run the risk of building JABOWS (just a bunch of web services). In some business-driven projects that I have seen, IT does not fully support the initiative. This creates a lot of wasted energy for the architect team who has to constantly deal with negativity, resistance, and even personal attacks by those who fear change or don't believe in the value of SOA. In cases like this, IT is its own worst enemy.

Resistance to Change
So how do we overcome resistance to change? Robbins and Judge, two thought leaders in organizational behavior, describe 7 tactics for dealing with resistance:

1. Education and Communication - By constantly communicating, evangelizing, and training, resistance can be reduced by minimizing misinformation and rumors and can actually help in the selling process.
2. Participation - Getting people involved and giving them accountability makes it harder for them to resist change. If they are any good, they will quickly see the value in SOA.
3. Building Support and Commitment - You can't be everywhere all the time so you need change agents spread throughout. Their job is to evangelize the benefits of SOA and the value it will bring to the company. At the same time they must address any resistance immediately otherwise it will spread like cancer. The biggest reason for resistance is fear. The change agents need to find out what is causing the fear and communicate that to the person driving SOA so the fears can be addressed across the organization.
4. Negotiation - Sometimes people just don't believe in the vision and there is nothing you can do to change their mind. This is where you hear the cries, "That will never get done here". For these people you must identify what is in it for them. If they are non management, you can explain the potential career path benefits and point to reduction in production support and other monotonous chores. If they are on the network side you can point to centralized security, the value of the ESB, and many other things that appeal to the networking folks.
5. Manipulation and Cooptation - Steer far away from this one. This will destroy your integrity and will make you unable to deliver any project once this tactic is exposed.
6. Selecting people who accept change - This is key. There are enough people who will throw daggers at the project from outside. The last thing you need is someone inside fueling the fire. As I mentioned in a previous post, we look for motivated people who embrace change and buy into the vision.
7. Coercion - File this under unethical and see bullet 5 for consequences.

The higher up you have buy in, the easier it is to deal with resistance. If you don't have active support from the top, you can spend an enormous amount of energy fighting resistance. When I say active I am talking about participation. That means the IT leaders, even those who are not directly involved, must be actively involved in some form or fashion to show their support of the SOA initiatives. Even if the involvement is as little as inviting architects into team meetings to give updates. Any IT shop should be able to figure out the technical challenges of SOA. The hard part is dealing with people and change.

A couple of days ago I wrote part 1 of this series and highlighted the things we did that made us successful on our first SOA implementation. Today, I will discuss the things that we should have done differently. Before I start, I would like to note that we are working on six different projects concurrently under the parent program. So some of these "what not to do's" apply to all of these projects not just our initial one that we just rolled.

What not to do...

1. Not enough professional services - Our budgeting process only included enough hours to install the stack. We should have covered a few more weeks of professional services for when we first deployed code into the development and testing environments. About a month after we installed the BPMS and ESB, we deployed our first build on these environments. We encountered all kinds of environmental issues and various integration issues between the various components of the stack, especially the portal and BPMS. Since we didn't have budget for any more professional services we tried to figure it all out ourselves. Eventually we brought the vendor in and ate some incremental costs but we lost about a month which delayed the project.
   
2. Needed more vendor references - This is not for the selection criteria but for implementation. We should have sought out more companies that had been through an implementation using these products. We are networking with a few local companies now but it would have been nice to have heard their experiences earlier and we might have prevented some of the setbacks or resolved issues quicker.
   
3. Dove in too fast - This one we didn't have much control over. The downside of having the business fund SOA is that they want results immediately. We started our first project with no governance, no testing strategy, no automated build process, etc. We are trying to implement governance along the way but it always takes a back seat to due dates. Fortunately, a new budget starts in January and we can get the resources we need to get these important tasks done.
4. Threw the kitchen sink at the first iteration - I wrote a lessons learned piece a few weeks back which prompted Todd Biske to write a good article called SOA and the Kitchen Sink. We tried to do too many things currently and had a few instances where we were thrashing.
   
5. Not enough internal resources involved early enough - We partnered with a consulting firm to help teach us how to properly architect software in the new world of SOA. At the same time I have been working with the leadership team in IT and the business to free up more internal resources so we can start doing this ourselves. This has two bad side effects. First, very few of our own people were recipients of the knowledge transfer and two, the vendor does not know our business which has led to some gaps in requirements and some architectural decisions that were made that we have had to change.
   
6. Needed more change management - I did spend a lot of time focusing on communications & organization impacts but there was still a lot of "we vs. they", resistance, and confusion about the overall direction. The core team was has always been on the same page, but many spectators are still in the dark. This often leads to resistance and negatively which we deal with immediately, but it would be nice not to have to spend as much time resolving this issues.
   
7. Didn't research SOA testing enough - One of the things we did right was research the heck out of SOA and BPM. But one area that we didn't invest much time in was the testing. We are actively researching the heck out of it now but my advice to others is don't under estimate the changes required to test SOA.
   
8. Occasionally got away from project management best practices - We fell behind by a month because of numerous issues with our environments and some bugs in the stack products that the vendor had to patch. Once we resolved all of the issues we only had two weeks left to deliver our new B2B customer facing portal to our client. We threw everything we had at the remaining tasks but somehow forgot about some of the basics of project management like risk mitigation and we had several communication breakdowns. We coined it as "running around with our hair on fire". On the positive side, we all acknowledged our mistakes and are in the process of fixing this.
   
9. Taming requirements - Requirements have been too volatile and it is hard to be agile when you can't draw that line in the sand until the next iteration. With some of the business services, there does not appear to be enough ownership and accountability. This has caused a lot of rework and has been a big source of frustration.
   

If you read this article without reading part 1, it probably sounds like we failed miserably. In reality, we were highly successful. But with all of the issues I listed above, it was a lot harder then it should have been. Part of the challenge was that we were learning SOA, BPM, and agile on the fly while trying to meet highly aggressive dates. My goal here is to share my experiences, good and bad so others can learn from our success and mistakes as they prepare to take on their SOA implementation. Feel free to contact me if you ever want to discuss these experiences or offer suggestions.