Enterprise Initiatives

Tomorrow at 4pm I am presenting at the annual EDM Summit in Orlando on the topic of Business Intelligence as a Service. Check out the presentation below.

This presentation is based on a real life project that I once worked on. We had a scenario where the business had a very complicated set of business rules required to determine what inventory was available to sell. Inventory for a loyalty marketing company is very dynamic and is not a physical thing like a widget. Instead it is a data mining exercise comprised of many "What-if" scenarios.

The old way of doing things was to get data from many different sources, some from systems, some from spread sheets, and some from somebody's head, and go through an ugly and error-prone process to determine what was available to sell. The process took many days which puzzled the customer why we couldn't tell them right away if we could run the program or not. Through some analysis, we defined new and improved business processes and data services that would allow our sales people to enter numerous parameters on a new web UI and let the systems return information to the screen with potential sales opportunities.

When the designers brought their solution to the architecture team for review we saw a huge opportunity to change the approach that was being recommended for the user interface. The designers were Java guys so naturally they recommended a Java web based UI (which infuriated the .Net community). But as I looked at the multiple screens that they story boarded it became obvious that building a what-if type UI was not best served by building proprietary code. Business Intelligence tools make a living doing just that. So I put a hold on the design and recommended that we brought in our BI partner for a proof of concept.

To make a long story short, we proved with our BI partner, that we could leverage the BI tool to create a robust what-if style GUI which we could talk to all of the layers of our SOA (see slides for details). In other words, we abstracted the BI tool as our presentation layer. What that gave us was all of the bells and whistles that come with BI tools like:

1. Flash enabled output
2. Subscription services
3. Mobile capabilities
4. Alerts
5. Great scalability
6. Logging
7. and much more

All of these things I mentioned above were out of the box features of the BI tool that we did not have to code in Java. In fact, we offered to the business a much more efficient solution. Instead of submitting data and reviewing rows on the screen, we offered to automate the business rules and allow them to subscribe to categories. This means that they only needed to go to the system to tell it what to look for and the system would alert them when categories were available. Now they could go into a sales meeting knowing in advance what was available to sell. When certain categories became available they could get an alert and immediately schedule a call to the appropriate customer. A future step could be to tie the alert into the CRM system.

So the key point to this story is that you can leverage BI as an abstracted layer within your SOA which will help maximize the value of your existing BI investment.

I have been blogging about our SOA project for over a year now. We are closing in on our first year of development and are ready to deliver eight different projects over the next two months. A few months ago we started our second process reengineering effort in another line of business. In this initiative we were able to access the process from pre-sales through contracts/proposals all the way through delivery. In the pre-sales area there is plenty of data mining and data discovery processes that take place so the sales team can go after the best opportunities. The pre-sales process is a business problem that is perfectly suited for our business intelligence (BI) tools. Historically, our BI tools were delivered as stand alone solutions. Now we have a need to allow the users to drill into data, select one to many result sets, and pass the information onto the next business process. In short, this means that we need to connect our BI results to our BPM tool.

In our current implementation, our SOA stack looks like this...



Our BI architecture (we use Microstrategy) looks like this...



But if treat our BI architecture as just another abstracted presentation layer, we get this...



In the previous picture, you can see that our SOA stack does not need to understand or even interface with the complexities of the BI platform. Instead, the selected result set that the user chooses from the BI user interface can trigger a web service to pass the results as an XML message to the SOA stack where the BPM engine takes over and moves the data into the next step of the process. Below is a example of a BI interface that I "borrowed" from the Microstrategy's web site.



Imagine drilling into the subcategory of computers to find an opportunity to sell a loyalty campaign to a customer based on certain performance metrics and demographics. Once the sales person selects a potential opportunity and hits the submit button, the data can be fed into the proposal process and much of the order entry data can be prepopulated to improve quality and save time.

So why does this matter? The point is that when we leverage architectures that abstract the presentation, data, business processes, and business rules, it becomes really simple to integrate applications whether they are home grown or commercially built. Because of the abstraction layers in both our SOA and Microstrategy's architecture, both solutions can treat the other as a black box and simply communicate via services. What makes this even more attractive is if our customers or partners have their own BI tools that we want to integrate with our systems we can simply provide them with an XSD so they know how to format the XML message (of course this assumes that the proper security is in place).

Another benefit of leveraging our BI tools as an abstracted presentation layer is that we can take advantage of many out-of-the-box features from the BI platform. Features like subscription services, alerts, flash enabled emails, mobile support, scorecards, and dashboards are just a few of the many rich features that you do not have to build from scratch. But the big bang is that your CIO will be happy to see you leverage the company's large investments in both BI and SOA while wowing your customers at the same time.

I wrote a piece called Why are you still generating reports that spawned a few other posts from fellow bloggers James Taylor, Todd Biske, and Robert McIIree. Robert had an interesting point of view which I would like to discuss. In his post he stated:

Excel and similar tools are used for two reasons: first, because it is very familiar to most users and "good enough" to generate reports and analytic output; and second, such use is quick and effective without running through the IT gauntlet to enlarge the effort by "gathering requirements," making it a full-blown project, and waiting a long time for "results."

I agree that Excel is user friendly and a favorite of most business users. Excel is an acceptable tool to manipulate data but should not be used as a system of record. I have seen too many occurrences of key financial data/reports residing on someone's C: drive with no way of validating the integrity of the data. I have also seen many business users create their own "systems" by pulling data out of various databases and building critical reports that are most likely inaccurate. Why are they inaccurate? First, most users don't understand the concepts of data integrity, don't have access to data dictionaries, and don't understand the physical implementation of the data which means they have limited means of validating what they have created is accurate. Second, since IT usually has no idea that these systems exist or how they are updated, changes to the source data are made without the knowledge of the end users' home grown systems. So I say Excel is a great tool for dicing and slicing data but the business community should not be driving the business off of home grown Excel systems.

Further down in the article Robert states:

Instead of asking "What problem are you trying to solve?" we should be stating "Here's what we have available, what is missing or needs to be altered for your needs, and will this work in other parts of the business?" The former question implies that IT knows more about the business and its specific issues than the business does.

Which it doesn't, and never did.

I understand Roberts point but this is where I ask the question, "Are you customer centric or a customer servant?" If you are a mere servant for the business, when they ask for stuff you simply build it the way they want. At the end of the day, you are letting the users design systems. If you are customer centric, your goal is to provide the business with accurate tools while at the same time ensuring that the business processes being requested are value added. One of the main reasons why BPM is so hot these days is that we have been servants to the business for years. The business knows what they need to do their jobs, IT should know how to provide solutions to meet those needs. My company just reevaluated its business processes in a certain area of the business and found that over 50% of the processes were non-value add processes, also known as waste. Why is this? For years the users asked for reports and we created them. The users knew what they needed but they did not ask us to build the right solution, nor should they. Here is an example:

We were asked to create a "data quality check" report to ensure that a certain attribute in a legacy order entry system matched the appropriate value of an attribute in a legacy financial system. The real problem was that the order entry system needed additional quality checks in the user interface to prevent erroneous data from getting into the database. So if I did not ask the question, "What problem are you trying to solve?", we create a new report accompanied by a new non-value add business process which requires manual labor to validate and to correct and does not guarantee that the problem will get fixed. If I do ask the question, I get the answer, "We need to be proactive to find and fix discrepancies between the financial system and the order entry system before we expose bad data to our customers". That response leads us to a simple UI fix which allows the systems to handle the problem and eliminate the need for a data entry clerk to perform more mind numbing, non-value added processes.

I keep going back to Nick Carr's theory that IT does not matter. If IT is simply taking orders and letting the users design their systems, then Nick is right. Instead, IT needs to help the business meet its objectives by delivering feasible solutions that provide quality, ease of use, and business value. If IT's role is to be servants, then there is no reason for the company to have it's own IT staff. Simply hire cheaper labor to follow orders. Just my 2 cents!

In the day and age of cheap disk, fast processors, high bandwidth, and cheap memory, companies are analyzing data more then ever before. As the components of the infrastructure become more cost effective and data is becoming more accessible, IT is being asked to create reports like never before. This is an area where IT can either be an enabler for the business or a serious bottleneck.

As partners to the business, we must educate our users on the power of business intelligence and self service. The business will typically ask for reports which is a reactive approach to solving problems. We need to offer proactive solutions that provide alerts and KPIs to address issues before they become critical. In other words, lets use information to optimize our business instead of creating more non-value add processes where countless clerks dig through hundred page printouts looking for issues. Here is my philosophy on reports. Static or canned reports should only be generated if the information is an output of the system. Here are some basic examples of when systems should create canned reports:

• Bills
• Invoices
• Purchase orders
• Confirmations
• Receipts

These reports are fairly static and should not change too often. When gathering requirements, the reports above would have been identified as outputs from the system. Notice that these are not really reports, they are outputs or artifacts of the system.

So what is the problem with reports? I have seen many applications that contain upwards of 70-100 reports over my years in IT. Usually, you could boil these down to half a dozen subject areas. Over time, different users come and go and they all have their own ideas on how they want data presented to them. So IT is asked to create more reports by adding a field here, grouping a field there, shrinking the font and printing landscape, and so on. All of these requests are a waste of money and a drain on IT resources. Often, these requests go into the black hole of the backlog of simple requests that IT can never find time to get to.

This is where business intelligence kicks in. IT should create metadata to represent the complex relationships of various databases and present the user with a simple view of data in business terms. Then through tools like Microstrategy, SAS, Cognos, Business Objects, etc. we should empower the users to create their own reports. Through self service, the users no longer have to wait months or even years for IT to get around to developing yet another report that must be maintained. The users can now react to business demands and can tailor the output to meet their needs.

This is a mindset that must be pushed from IT. When users ask for a report, the business analyst must ask the user, "What problem are you trying to solve?" I have seen all too often where the user wants a report as a quality check in response to issues found in the data. My response is to fix the root cause and eliminate the data quality issue as opposed to adding another non-value added process that may live on for years before someone realizes that the step is a waste of time.

So I ask, "Why are you still generating all of those reports?" If you follow the trends in IT you will notice that today's users are becoming more empowered then ever before. Users are leveraging various Web 2.0 technologies to service their needs and to build their own composite applications. We need to provide them with that same easy access to data. Don't give them reports, give them information!

I attended the Event Processing Summit in Orlando last week and probably the most interesting presentation was given by Dr. Mani Chandi from the California Institute of Technology. Mani did a great job of explaining the important role that event processing plays today but it was his vision of the future that really caught my interest.

Mani described what he said would be the next big thing in IT that the analysts would be talking about in 2017. He took us through history and explained how innovation focuses on the most scarce resources. In the 60's the most scarce resource was the mainframe, namely the computation power. From that, innovation led us to the distributed model of client server computing. The next wave of innovation dealt with the scarce resource of communication and bandwidth which led to the great internet revolution and high speed access. The current wave of innovation is in the area of integration and information, both internal and external. There is such a high demand for systems to talk to one another that SOA has become mainstream.

So what is the next scarce resource? According to Mani, it is time and attention. Now that we have solved the issues with computing power, communication and bandwidth, and application integration, the next hurdle to overcome is dealing with the enormous amount of information we have at our fingertips. The next wave of innovation may be focused on some combination of Web 2.0 technologies, complex event processing, and business intelligence. Here are Mani's predictions:

• Systems will predict your needs without explicit work on your part
• "CEO need in 2017 is what kids use today"
• We will move from a reactive web to a proactive web

Mani marveled over how incredible kids' sensors are today and how they easily consume the continuous flow of information they subscribe to. He sees a move towards user contexted BAM (Business Activity Monitoring). Mani left us with some good advice:

Focus your development effort on the scarcest resource.

I wrote an article several months ago named Built to last is a thing of the past. Yesterday, at BEA World in San Francisco, a speaker mentioned the phrase...

Built to change, architected to last

I think that this is a great motto and one that teams starting SOA implementations should use as a their vision. Built to change, architected to last is all about agility and flexibility. BEA is touting their own new acronym called Dynamic Business Applications. This new term refers to an architectural mindset based on four principles:

1. Built to change (flexible)
   
2. Embody business processes
3. adaptive
4. agile

This mindset is BEA's response to today's business environment where the business is changing faster then IT can keep up with. The solution is to provide a platform of business processes and services which allow users to design and/or assemble their own composite applications (mashups). The days of buying enterprise third party applications and forcing the users to use the canned processes and features are over. The business changes so fast that third party packages become a crutch over time due to their long release cycles. Simply put, the packages can't keep up with business needs. The answer is take the best of breed features and services from the combination of third party packages, SaaS solutions, and custom code and present a common interface that integrates all of the functionality together into a single view of the users' workflow.

Dynamic Business Applications is a shift to a world where applications are human made and designed for the way people work. The IT team delivers business processes, business services, core infrastructure, and integration services. The business can then create their own composite application by assembling all of the pieces together. Think of it as the Lego approach. The IT team builds the rectangles, squares, wheels, and various connectors in all different shapes, sizes, and colors. If the users want to make a boat, the choose all the pieces necessary to make the boat. If they want to make a plane, they choose the pieces necessary to make a plane. The users did not have to wait for IT to build them a boat or plane.

To accomplish this, IT must work closely with the business to identify the core business processes and services. This is where it is extremely helpful to establish a SOA roadmap. Once you have identified the collection of business processes and services in some key area of the business, you can then prioritize your deliverables by delivering the processes and services with the highest chance of reuse first. This greatly improves your speed to market in the long run.

Another shift in thinking that should be mandated in organizations is to move away from creating reports and move to creating information. My philosophy on this topic goes like this....

You should only develop reports if it is an output of the system (ie. bill, invoice, purchase order, etc.). Otherwise, IT should leverage business intelligence and deliver data as content to the users with a tool that allows them to format the output and drill down into the data as needed.

Why do we continue to pay developers to write custom reports only to get frequent change requests to change or add new columns, create multiple output formats to please different individuals' preferences, and make changes to group or sort things differently? Not only is this expensive and an inefficient use of people's time, but the users typically have to wait for IT to make these changes. Although these changes may be quick to make, it may be a long time before the change is high enough on the priority list to get worked on. Wouldn't it be better to present data to the users and give them self service capabilities to define their own reports, create their own dashboards, and do their own what-if analysis?

IT needs to become an enabler instead of a bottleneck. We should stop building custom applications and start building the building blocks that can be assembled into Dynamic Business Applications.

In part 1 of this series I asked the question, "Are you running IT like it's your business?" Then I highlighted five barriers for preventing IT leaders from being able to transform their IT shop into a well oiled, cost effective machine?

• Resistance to change
• Lack of resources (time, money, and human capital)
  
• Lack of tools
• Lack of metrics
• Lack of process

In part 4, I will focus on Lack of Metrics.

Many IT shops are stuck in maintenance mode and live the life of fire fighting and working hard instead of working smart. These shops struggle to meet the demands of the business and spend most of their time and money keeping the lights on. The business is reluctant to release more funds to IT to fix their issues because they don't see the value that they are getting for their current spend in IT. Your CEO is telling you to do more with less and you are asking for more resources to stay afloat. What gives?

I have heard the grumbling within IT wondering why the business "doesn't get it" or why "they can't see that we need resources". The business has a full time job and expects IT to do their job. They don't have the time to sympathize with IT because they have their own battles to fight. What you have to do to get their attention is to tell a meaningful story that is based on facts (metrics) and discussed in business terms (dollars). So before spending another day complaining about the current state of affairs, start brainstorming about what metrics you should be collecting to allow yourself to paint a picture for the business.

Metrics can be used a couple of ways. First of all, I am not a big fan of coming up with metrics to try to measure the productivity of developers (function points, lines of code, etc.). These types of measurements do more harm then good. I am not looking for metrics that show a distrust in your teams ability to do their jobs. I am looking for metrics that can capture opportunities to make improvements in the enterprise. Tracking where time is being spent can be a valuable exercise. You should know how much time you are spending on product support and maintenance and put plans in place to reduce these numbers by some percentage each year. You should also measure defect types and severity by application so you can put task forces in motion to improve the problem areas. If you ever want to get out of support hell, you must be proactive.

If the business does not think that IT is providing value, you can use metrics to show them otherwise. Metrics about availability and uptime, SLAs, change requests implemented, money saved due to process or performance improvements, and many others can highlight the things that IT brings to the table. Create executive level dashboards that sell the value of IT real time. One of IT's biggest faults is that IT rarely celebrates its success stories. IT falls victim to the "what have you done for me lately" routine. If you got it, flaunt it!

Once you have your metrics in place, you can start the long journey of continuous improvement. Attack your top problem areas and tie your metrics to dollars. The business listens and understands money. They typically don't listen to or understand technology. When you do need to go to the well to ask for resources, you should have plenty of facts that you can tie to dollars to sell your story. Despite some of the griping within the gallows of IT, the business is not blind. IT has just never provided them with enough data to see the light.

Nick Carr created one of the longest running questions I have ever seen when he wrote his book Does IT Matter back in 2004. There has been so much discussion arguing for both sides of the answer including this article from Harvard's Andrew McAfee. I believe that a key differentiator for a company is the business processes. For IT to Matter, IT must create an architecture that supports the business's need to dynamically change and customize their processes. This allows the business to be more self sufficient and agile, which leads to a better user experience and faster time to market.

This is why the BPMS and SOA vendors are cashing in right now. Many IT executives are starting to understand that IT needs to be more of a partner to the business then a cost of doing business. BPMS solutions allow IT to provide tools for the business to rapidly deploy new systems that can have a huge impact on the bottom line by:

• reducing costs through process reengineering
• identifying bottlenecks and providing what-if analytics for ongoing process improvement
• providing robust, AJAX & web enabled user interfaces
  
• providing visibility into performance metrics with built in business intelligence tools
• providing self service capabilities for the end users, thus reducing the dependency on IT resources

The smart IT executives are leveraging SOA to allow the BPMS tools to talk to the existing legacy systems by providing a service layer that acts as a bridge between the user interface and the backend systems. This allows IT to rapidly deploy new systems without having to throw away the huge investments made over the years on their existing systems.

So to make a long story short. Does IT Matter? It can matter if IT recognizes that "Process is King" and that the business and IT need to work hand in hand to provide solutions that create a competitive advantage.