Enterprise Initiatives

The promise of SOA is flexibility and agility. I define agility as the ability to adapt to change at the speed of business. In today's global economy which has been fueled by collaboration and Internet technologies, businesses change at a much faster rate than ever before. So how does SOA help companies become agile? Let's look at a logical view of a typical SOA.



You can see in this diagram how each layer of the architecture has been abstracted and is mutually exclusive or "loosely coupled" from the other layers of the architecture. Why is this important? The answer is simple.....ease of change! Here are some advantages of this approach.

• Share services, components, rules, etc. across the enterprise (reuse)


• Isolate changes and reduce dependencies (speed to market)


• Minimize impact of business changes (speed to market)


• Easier to maintain (maintainability)

Let me give an example of how this approach helps companies become agile.

Use Case: New customer data from a new client


Let's say your company provides a service for customers in the retail industry. You have a website that offers online services for consumers and your white label solution is tailored to look like it is hosted by the individual retailers. The problem is that each retailer has their own customer database. In the past you would have to write a ton of code for each retailer that you signed up to use your services. With SOA, now it is a simple data mapping exercise. By abstracting data services, the other layers of the architecture use the logical representation of customer, not the physical implementation. Behind the scenes, the data services layer is translating the request for customer data from logical view to physical implementation. So when you bring on new clients, you simply use a tool in the data services layer to map the new clients customer data to the standard logical definition as defined by the architecture.



You can see from this example that all three retailers have an entirely different implementation of their customer database including different naming conventions and even different attributes. In the data services layer, you can map all of these physical implementations to one standard customer definition. You can also see how the business processes all use the logical view of customer. This allows us to add and change customer definitions on the back end without changing code on the front end. If two more retailers were to sign up tomorrow, we can map their definitions to the logical customer view and be done. No Code!!! That is agile!

I will give examples of agility within the other layers in future posts. If you would like help establishing a data services strategy, give me a shout!

For those of you who have been reading my blog for the last two years, you might know that after 13 years at my previous job I left the company this summer to pursue multiple opportunities that were available to me. I have been doing some consulting and freelance writing in the mean time but have finally found my new gig. I will be the CTO/Chief Architect for a startup (more details forthcoming in the future) and will have the opportunity to work on some of the newer technology initiatives. So here are some of the future blog topics that I will start covering in 2009 once I get rolling:

1. SOA - can't seem to let it go! As I wrote in a post at CIO.com, I believe SOA is not just for legacy systems and can be the foundation that startups can leverage as a competitive advantage (see the reasons why)
2. Cloud Computing - Startups typically have low budgets and leveraging platform (PaaS) and/or Infrastructure as a Service (IaaS) can be a great way to both minimize costs and reduce time to market. As I go through the requirements analysis, risk analysis, vendor analysis, proof of concepts, prototypes, and deal with issues such as stability and privacy, I will share my lessons learned here.
3. Open Source - We will likely leverage many open source solutions and I will discuss those experiences and decision points here.
4. Social Software - The company is based in the North East and I live in Florida. It is highly possible that the team I build will be dispersed all over the country and possibly even the world. There will be great lessons learned discussions as well as evaluations of tools to help enable working remotely (virtual whiteboards, live meetings, SaaS solutions for defect tracking, etc.).
5. Agile - One of our goals is to deliver quickly and at low cost. This involves a lot of the above mentioned topics and will also lead us into the world of Agile Development. I will discuss our challenges and lessons learned in this area as well.
6. C-Level IT topics - In my new role I will be working side by side with other C-Level people within the company as well as with C-Level people of partner and customer companies. I will have to hit the road and sell and/or promote our products/services and will try to shed some light on some of the interesting topics I come across.
7. Misc - Other topics that might be worthy of discussing (examples: impacts of financial crisis, VC funding, selling technology to business people and customers, etc.)

In addition, I still do some freelance writing and blogging from time to time. I have a series of discussions coming up on enterprise mashups and will blog about it in the next week or two. I am attending and speaking at the EDM Summit in Orlando next week and will definitely be sharing my presentation on Business Intelligence as a Service as well as covering topics like business rules management, data warehousing, IT governance, and others. I also have two books from Packt Publishing to review:

• SOA Governance by Todd Biske
• SOA Cookbook by Michael Havey

Once I finish reading these I will post a review on my blog. I am half way through Todd's book so far and really like it.

And finally, I am open to covering any topic that any of my readers would like me to discuss whether it is on this blog, over the phone, or in person. Whenever I travel I send a Tweet on Twitter to let people know where I can be found. I will be in Pittsburgh from Saturday 10/25 through Monday 10/27 if anybody lives there and would like to meet. Don't bug me from 4:15-7:15pm on Sunday because I'll be at Heinz Field pulling for my Giants against a tough Steeler team. The rest of next week I am at the EDM Summit in Orlando.

I look forward to sharing these topics with you all. Over the next two months my focus will be on the business plan and creating a team. We officially launch the company in the December-January time frame. At that point expect to see more discussions on the topics I mentioned above.

Over the last 18-24 months I have researched and taken on a number of Enterprise Initiatives such as Enterprise Architecture, BPM, SOA, and agile development to name a few. As I researched each topic I found tons of articles recommending to steer far away from each initiative. The common theme is that it is expensive, too hard to do, and most initiatives fall. Today I was researching Enterprise Metadata and many of the search results pointed towards doom and failure.

Why is there so much failure with these all important initiatives? I did some research and found that a large amount of normal projects fail at equally high rates. The enterprise initiatives are just more visible and usually require a significant upfront investment. So what are the common reasons for failed projects? An article at Gantthead.com provides the following list:

• Inadequately trained and/or inexperienced project managers
• Failure to set and manage expectations
• Poor leadership at any and all levels
• Failure to adequately identify, document and track requirements
• Poor plans and planning processes
• Poor effort estimation
• Cultural and ethical misalignment
• Misalignment between the project team and the business or other organization it serves
• Inadequate or misused methods
• Inadequate communication, including progress tracking and reporting

For enterprise initiatives I would add these:

• Lack of strong executive sponsorship
  
• Unrealistic scope ("boiling the ocean")
  
• Lack of governance
  

So the next time you start researching a large IT initiative and you see all the negative publicity about your chances of success, ask yourself these questions:

1. Do we have the leadership required to take on this project?
2. Do we have enough change agents to change our culture?
3. Will we be able to get enough funding to adequately meet our objectives?
4. Will we get the support from the executives and/or the business?

If the answer to any of these questions is "no", then you might want to listen to the warnings from the articles. If the answers to all of these are "yes", then it all comes down to how good of a plan you put together and how well you execute the plan. Don't be risk adverse and let fear get in the way.

I read this article today about outsourcing that claims that outsourcing is not just about cost anymore. The author points out these four reasons for outsourcing:

-Accessibility to right talent
-Geographic expansion
-Reinvention of their business model
-Promote innovation

After I finished choking on my cornflakes I decided I must offer a different opinion on this matter. First I want to differentiate between outsourcing and offshoring. Offshoring is obviously when you send work to other countries. Outsourcing is when you send work to companies outside of your corporation that may or may not be in the same country. Since the article was based on this link from The Times of India I know that they are really talking about offshoring.

Here is my take (from a US point of view). Offshoring is a cost savings exercise, period. There is a lot of overhead involved with offshoring due to language barriers, cultural barriers, and time zone challenges. To successfully offshore development, the paying customer must provide a maximum amount of oversite and process to overcome these barriers. Outsourcing is different. With outsourcing, I can bring a team of consultants on site for any amount of time without paying the expenses of flying them in from the other side of the planet. You still need oversite and process but not at the same level because the language and cultural barriers do not exist and the time zone differences are manageable (if there even are any).

I wrote this article a while back about agile development and consulting. In summary, if you want to be agile, don't engage with an offshore partner. Agile development requires face to face interactions and loosely defined requirements due to the iterative nature. The requirements get nailed down over time by cycling through requirements and protoyping. This model is a recipe for disaster if you are using offshore resources. For offshore to work you need to have a more waterfall type approach were the requirements are fairly static.

The article that I came across did not link back to the original research provided by PWC. If you read the PWC article you will see that over 50% of the service providers surveyed are US companies like PWC. So the findings in the research make a lot more sense to me because we are really talking about outsourcing and not totally focusing on offshoring. US companies are definitely leveraging consulting companies like IBM, Accenture, and others to foster innovation, reinvent their business models, and access the right talent. They don't use these companies to save money (they cost an arm and a leg). Saving money is all about offshore development.

The lesson learned for me is be careful what you read on the net. The PWC article is a very good analysis of what is happening in the marketplace. The problem is all of the offshoring blogs are using it to say, "See we are more then a cheap alternative", which does not represent the facts of the PWC research. For those who just read the headlines and don't take the time to read the details, beware!

Virtualization is another hot topic in IT today. It allows you to run multiple virtual or logical representations of physical machines, side by side on the same physical server. Many companies are putting strategies in place to consolidate servers and to reduce the cost of hardware, maintenance, power, and floor space. At my work we have a cluster of physical machines using VMWare that have allowed us to run 115 virtual servers on it. There is room for more and we will continue to move to that platform. Virtualization makes a lot of sense in the eyes of the system administrator.

So why is a software guy like me so excited about it? To me, it is much more then a way to control costs. It also allows us to be more agile, to react to capacity issues faster, deploy easier, and create faster applications. Allow me to expand on each one of these points. The following statements assume that you have a cluster of boxes with available capacity to add additional virtual machines.

Agile - How long does the procurement process take in your company? I have seen companies' server procurement cycles take from two weeks to six months. The norm in my world is about a month. A virtual machine can be created in 15-20 minutes without having to purchase anything.

React to capacity issues - On Friday at 10pm, your web site generates a ton of unexpected traffic which causes your servers to exceed their capacity and fail. Your systems administrator logs on from home and quickly adds five more virtual machines using a snapshot of an existing production virtual machine. In under an hour he has the system back up and running.

Deploy easier - Your application can be configured as a virtual appliance. This allows you to create a virtual machine that consists of the operating system and application software for all the tiers of your application and bundle it as a single file. Microsoft is leveraging this approach as a way to allow people to try out the latest version of Outlook without having to install hardware and software. Think about being able to deploy your multiple tier environment as a single file. Take that snap shot and deploy it at your DR site and you have knocked a huge chunk of time off your implementation phase.

Create faster applications - This is the topic that gave me the urge to write this post. Think about this architectural strategy for your web sites. Put all of your web servers, application servers, and database servers on virtual machines and create your virtual appliance. Since all of these virtual machines are on the same physical server, you could actually leverage the PCI bus instead of the Gigabit Ethernet. The PCI Bus is over 400 times faster then Gigabit Ethernet which should give you incredible response times for your application. What's even cooler, is that you can have your virtual web servers "sitting" outside the firewall in the DMZ while your application servers and database server is "sitting" inside the firewall. You read that right! Even though these virtual machines are physically installed on the same server, you can configure them to logically be located in different areas.

Think of all the possibilities. We all know the power of distributed systems. Distributed systems allow you to perform parallel processing across multiple nodes to achieve more work in shorter intervals. The downside of distributed systems is the cost of the physical servers and the management of software across the nodes. Enter virtualization. You can install several virtual machines at minimal cost and easily manage the software by updating one virtual machine and taking a snapshot. Then apply the snapshot to the other virtual machines.

To take that concept one step further, you can leverage this distributed model to create multiple virtual machines to act as an application server farm. If you are implementing an SOA and have an ESB in place, you can configure the ESB to load balance your services across these virtual machines to maximize your throughput.

The possibilities are endless. The point is that virtualization is more then just a great solution for consolidation and server management. Virtualization should be included as part of your application architecture strategy.

I have written a couple of articles (here and here) recently about offshore development and how the success or failure of offshoring depends heavily on the US companies' ability to manage their offshore development partners. I may have come across as a strong supporter of offshore development, which was not my intent. My intent was to point out that if you are going to send work overseas you better have good processes and controls in place to manage it.

This takes me to my next point. I am a huge fan of agile development. One of the many things I like about agile development is that it does not require you to spend countless hours analyzing and documenting requirements and designs. Instead you cycle through many iterations of requirements, design, and prototypes with your key stakeholders and deliver functionality in small chunks.

Now lets look at the offshore model. Because of the communication barrier, the lack of business knowledge offshore, and often the limited experience level of the development team, the amount of documentation required to make this work increases immensely. Without the "war room" mentality of agile development, the projects tend to lean more towards a waterfall approach. Waterfall projects are rarely discussed in the same breath as "speed to market".

On my current BPM/SOA implementation, we have about a dozen projects that we wish to implement in the next 12 months. As we work with our SOA consulting partners (a US company with an offshore partner), we put together two different pricing models for our executive sponsor for each project. One price is for the most cost effective approach which leverages offshore development resources, and one is for the fastest to market approach which requires more on-site collaboration. So far the business has only chosen the speed to market solutions. Why is that you may ask? Well, all of these projects have a huge ROI if we replace our 20 year old business processes with new streamlined processes. Every day that we continue to do business with our legacy processes we are wasting valuable corporate dollars. For this reason, our sponsor is willing to pay a premium to get the work done quickly.

So if speed to market is your goal, agile development could be your answer. If you want to be agile and deliver functionality every 10 weeks or so, you must have great collaboration amongst your team and should not follow an extremely rigid methodology. In my opinion, this is when you should not rely on offshore development.

In part 1 of this series I asked the question, "Are you running IT like it's your business?" Then I highlighted five barriers for preventing IT leaders from being able to transform their IT shop into a well oiled, cost effective machine?

• Resistance to change
• Lack of resources (time, money, and human capital)
  
• Lack of tools
• Lack of metrics
• Lack of process

In part 2 I will focus on Lack of Resources.

What are some of the most common phrases you hear when attempting to promote change within your organization? The top one I hear is "I don't have time", followed by "I don't have money", followed by "I need more resources". These are the biggest cop outs in the world. The reason why nobody has time, money, or resources is because they don't put any initiatives in place that allow them to do anything other then put fires out.

At some point you should reach a threshold of pain and realize that there must be a better way. Everybody is working 50-60 hours, all the projects are late, production support continues to increase, and the users are screaming. Everybody is working hard, but nobody is working smart.

If this sounds like your shop, you might want to take a step back and put a plan together to stop the madness. If you owned your own trucking business and you spent most of your time repairing your fleet of trucks you would probably be out of business. So why do so many IT shops spend countless hours every day in repair mode? It's time to make time, time to spend your shareholders' dollars effectively, and time to maximize your precious resources.

Let's start with time. Where are you spending your time? Do you have any metrics that allows you to proactively manage and control your projects or your production support? In many shops, the main cause of all these issues is lack of a sound project prioritization process coupled with a lack of a resource allocation model. If projects get accepted at will, priorities change like the seasons, and you have no data to show that your resources are over allocated, you will become a reactive culture. If you are not practicing Portfolio Management, you should make that a priority (view this article I wrote on Portfolio Management). If you have no visibility into how much time you are spending in production support, what categories of issues you are encountering, and how long it takes to fix issues then you should make change management a priority. If you are meeting with the business to discuss the next wave of new projects and do not have a clear picture of your resource capacity, you need to start practicing human resource planning.

Next is money. Obviously, if you are spending too much time keeping the lights on then you are wasting money. But are you wasting money on infrastructure? Are you still pouring millions of dollars into big name vendors like Microsoft, IBM, Oracle, and HP? Are you paying huge amounts of money for "Gold Support"? You need to start evaluating Open Source alternatives. If you believe in the myths of Open Source, it's time you start doing your homework and read about these 50 Open Source success stories. Oh, and that Gold Support you are paying for each and every product...there are many Open Source Support Providers who can support a variety of open source products at a fraction of the cost. What is better is that these folks earn their stripes supporting customers and are actually extremely interested in helping you resolve your issues. This is their core competency. I wonder how much effort Big Blue would put forth to help a small $100M company resolve its problems. The days of saying, "Nobody every lost their job buying Microsoft and IBM" are long gone. If that is all you are buying you are definitely not seeing the big picture and your CEO might start giving you that look.

Are you still buying rack after rack of servers to support your test, development, staging, and production environments? Have you embraced Virtualization Software like VMWare yet?

And what about human capital? We already talked about spending too much time keeping the lights on and spending too much time maintaining rack after rack of servers. But what about development and testing? Do you have an architecture that allows rapid deployment and reusability? Have you separated your business rules from your applications? If your business rules are embedded in your applications then you are creating additional work to maintain and test your applications thus increasing your chances of creating defects.

Let's say you are a retailer with 1000 stores and you have a business rule for defining what a high volume store is. You have several different systems that use this business rule (billing, inventory, eCommerce site, financials, etc.). Now you want to change the definition of a high volume store. You now have to make that change in each of these systems and implement all of the systems at the same time. This simple change has turned into a project. Wouldn't it be better to go to one place, make the change, test it in one place, and roll? You don't need to practice agile methodologies to be agile. You just need to be smart! My recommendation is invest in your architecture. Create a business rule layer that sits between your applications and your presentation layer. I would go one step further and create a service layer to orchestrate your business rules and allow other applications and/or customers to access your business rules.

Well, I have rambled a lot longer then I usually do so I will pause until part 3 which focuses on Lack of Tools. I will add a conclusion after part 5 that tells you how to prioritize these initiatives and how to get started.

Nick Carr created one of the longest running questions I have ever seen when he wrote his book Does IT Matter back in 2004. There has been so much discussion arguing for both sides of the answer including this article from Harvard's Andrew McAfee. I believe that a key differentiator for a company is the business processes. For IT to Matter, IT must create an architecture that supports the business's need to dynamically change and customize their processes. This allows the business to be more self sufficient and agile, which leads to a better user experience and faster time to market.

This is why the BPMS and SOA vendors are cashing in right now. Many IT executives are starting to understand that IT needs to be more of a partner to the business then a cost of doing business. BPMS solutions allow IT to provide tools for the business to rapidly deploy new systems that can have a huge impact on the bottom line by:

• reducing costs through process reengineering
• identifying bottlenecks and providing what-if analytics for ongoing process improvement
• providing robust, AJAX & web enabled user interfaces
  
• providing visibility into performance metrics with built in business intelligence tools
• providing self service capabilities for the end users, thus reducing the dependency on IT resources

The smart IT executives are leveraging SOA to allow the BPMS tools to talk to the existing legacy systems by providing a service layer that acts as a bridge between the user interface and the backend systems. This allows IT to rapidly deploy new systems without having to throw away the huge investments made over the years on their existing systems.

So to make a long story short. Does IT Matter? It can matter if IT recognizes that "Process is King" and that the business and IT need to work hand in hand to provide solutions that create a competitive advantage.

A good buddy of mine forwarded me this article from eWeek by Deborah Perelman. The following quote from the article summarizes the content: “In the simplest terms: too many IT workplaces have become Dilbert-ized—micromanaged, bureaucratic and stifled creatively. It's become an environment where busy work is praised and morale is low.” The article talks about IT as a commodity with trends in outsourcing. Flextronics CEO, Michael Marks, goes one step further in this Businessweek Online article Design is a Commodity. He recommends outsourcing the engineering process for electronics.

How did we get here? In my opinion, IT has done this to itself through the years due to the following reasons:

1) Not working closely with the business

2) Inability to successfully manage projects

Let’s talk about the first point. In the 60’s and 70’s, the business was dependent on IT for information. There were no high powered PCs and the Internet was not for commercial use. Most of what IT worked on in the public sector was business enabling applications. During the 80’s and 90’s, huge advancements in processor speed, memory, and disk technology enabled personal computers to do the work of the massive mainframes from the previous decades. Then the internet came of age which changed the way people and businesses interact with one another. These two important technology advancements changed business for the better but not without consequences. The days of IT being in control with centralized and reliable systems gave way to the complex, distributed, and multi platform environments that we live in today. This in turn, directed a lot of IT’s attention towards infrastructure projects. In today’s world, a large portion of IT budgets go into projects and services that keep the lights on for the company (email, voice & telecommunications, security, compliance, etc.) and do not contribute to additional revenue. In addition, software vendors started delivering shrink wrapped solutions (ERP, CRM, Financial applications, etc.) that was not feasible for companies to build internally. I believe these factors have all contributed to the fact that many IT shops have become disconnected and/or out of touch or alignment with the business. IT has become perceived more as a cost center then an enabler. Employees have become known as what Catbert calls “Headcount”.

Point #2. The PMI Institute states that 72% of IT projects they studied were late, over budget, lacking functionality, or never delivered. Of the 28% “successful” projects, 45% were over budget and 68% took longer then planned. These numbers are frightening! Lack of project management best practices have caused many companies to lose faith in IT. Many business units have started buying their own software packages or paying outside vendors to solve their business problems. This is another reason why IT and the business have become unaligned.

When a company views IT as an expense and not as an enabler, the IT shop becomes a poster child for Dilbert cartoons. Companies tend to look for ways to reduce or eliminate expenses. Once you view your employees as “headcount”, the creativity, passion, and drive gets drained right out of you.

So is IT doomed? Many experts believe that in order for companies to stay competitive and survive in the upcoming years, IT needs to focus on business processes. In the article, The How, Why, and Where of Future I.T., Mark Gibb’s states that, “I.T. has to be able to show that it delivers a real return on investment.” To accomplish that, I believe that IT should start embracing:

1) Project management – to improve delivery and communication

2) Portfolio management – to maximize IT investments, align priorities w/business, and control workloads

3) Business process management – to optimize and automate business processes

4) Enterprise architecture – to align technology with corporate goals and strategies

5) Change management - to manage change and impacts on people and processes

6) Agile development – to deliver value early and often

What are your thoughts?

How many times have we seen high profile people do things that contradict the messages that they send to us? We have seen a homeland security official stalking teenagers on MySpace, we have seen athletes deny taking performance enhancing drugs only to be proved guilty, and we have seen laws like the Patriot Act, which was supposed to protect us, actually violate our civil liberties. What does this have to do with technology you might ask? Well the lesson learned here is if you say one thing but do the other, the people who follow you lose faith and your initiatives fail.

The same holds true in corporate America. When a visionary leader attempts to implement a culture changing initiative like business process management, agile development, or a new project management methodology, they must practice what they preach. I have seen several attempts to implement standard software methodologies, like CMM, fail. Why do they fail? Because the visionary leader didn't practice what he preached. In this example, the visionary preached that by adopting CMM, IT would be more successful in delivering projects. Unfortunately, implementing CMM was not carried out like a project. No scope was defined, no communication plan was developed, no stakeholders or executive sponsor was named. Instead , a boat load of documents were thrust on the IT staff. Of course, there was mass resistance and the initiative failed. Had the visionary practiced what he preached and used CMM to implement CMM his chances for success would have greatly increased.

Practice what you preach applies to those trying to implement SOA for the first time. You probably sold management that some of the advantages of SOA are speed to market and agility. The CEO was so excited by your presentation that he freed up a boat load of funds to launch your SOA initiative. Guess what? It's now time to get agile. Practice what you preach. Throw away your old waterfall approach. Gone are the days of 6-8 month deliverables. Start delivering early and often. If you don't change your ways, the CEO will quickly lose faith in this initiative quicker the you can say WMD.