Enterprise Initiatives

I have been blogging for quite some time about SOA and Event Processing and have recently been getting more experience with Cloud Computing. The last few weeks I did a series of posts on agile SOA:

• Agile SOA: Empower the Business with Business Rules Engines
  
• Agile SOA: The Flexibility of Business Processes
  
• Agile SOA: Leveraging Data Services
  
• Enterprise Mashups: The Icing on your SOA
  

Today I completed a nice presentation on Slideshare called Agile Architecture which combines SOA, EDA, and Cloud Computing in a strategy to support the ever demanding needs of our dynamic business environment. I have the luxury of starting from scratch because we are in startup mode and do not have the burden of years of legacy systems and legacy cultures. The goals of our architecture are many, but here are a few key goals that speak to the topic of agile:

If you have been following my blog recently you would have seen my previous posts that discuss the benefits of the various layers of the stack (mashups, data services, business processes). Although many IT folks talk about the benefits of SOA in terms of code reuse, the real benefit is in the ability for IT to adapt their systems at the speed that the business is changing. One of the best ways to accomplish this feat is to put tools in the hands of the business to empower them to dynamically alter the business rules as they change in real time.

In the old days we built systems with rigid rules and users were forced to adopt to the "laws of the system". Today, users are king and they expect systems to adapt to the way they need to use them. In order to satisfy today's tech savvy users we must build dynamic and customizable systems. We also need to make sure that IT does not become a bottle neck and hinder the rate of change that occurs within the business. The answer is simple, abstract the business rules into its own layer within the architecture and empower your users with the appropriate permissions to make the necessary changes in a timely manner.

Let's discuss the following order process example that I used in the post about business processes.



Now what if we had different rules on what a good credit score is? What if those rules were dynamic? Do we want to hard code these rules in the system and accept change requests from the business to every time they change or would we rather abstract them and give the business a tool to change them as needed? I'll vote for the later! Here is the same business process integrated with a business rule engine (BRE).



Here is a simple example of how the business can set up rules to apply different criteria to different customers.



In this example you can see that Gold customers get special treatment and will be allowed to proceed regardless of their credit score. Other tiers have higher standards applied to them. Now let's say that due to the credit crisis the company decides to apply stricter criteria on their customers and raise the minimum at each level by 20 points. The business can simply make these changes from the business rules engine interface which is most likely done from a browser. If the company wanted to add a Platinum user type, they simply add a record to the customer type table and the appropriate rule for Platinum customers in the BRE. Done! No change request, no priority setting, no change control board, no development, no testing, no rollout!

I also used a callout to point to a business rule lookup at the fulfillment process. Let's say that our company is an eCommerce seller for many different wholesalers. We may sell books, music, appliances, clothes, etc. but we are simply the middle man and don't own the inventory. Instead we sell, take a cut, and trigger transactions to the appropriate wholesaler for distribution. Each wholesaler has a unique set of business rules for their fulfillment process. In addition, we may add and remove wholesalers several times during the year. We have a standard business process for dealing with fulfillment but different wholesalers may have different rules due to industry specific or geographic requirements (age limitations, state laws, tax rules, etc.). Once again, an appropriate business person with the proper permissions should be able to make these changes as necessary without having to deal with IT.

Business rules have their place and purpose but they also come with some baggage. Like any other layer within the architecture, business rules need to be administered, maintained, and governed. Business rules need some level of change control, backup/recovery, and auditing like any other artifact. Most BREs either provide that functionality as well or hook up with other tools that manage artifacts (CMDBs, governance tools, etc.). But business rules are only worth the hassle if the business environment is dynamic. If the rules are static in nature, it is probably not worth the effort from an architectural stand point to create an abstract business rules layer complete with the appropriate security, auditing capabilities, etc.

The higher up the stack we go the more opportunities we create for the business to become agile. Business Rules create the ultimate flexibility and agility that most of today's businesses require. Implementing a BRE in your SOA is another great step towards IT becoming an enabler instead of a cost center!

As I have mentioned in many posts before, SOA built right can provide business agility and flexibility. Yesterday I discussed how to leverage data services and a few days back I discussed enterprise mashups. Today the focus is on business processes.



Separating business processes from code is an ideal way to build an architecture that enables systems to adapt more readily to change. Let's make believe that we are working for a large eCommerce site that sells many different categories of products from multiple suppliers and warehouses each with their own distinct business processes. Abstracting the business processes gives us some very important advantages:

• Creates a set of standard business processes for a consistent user experience
• Allows us to quickly bring on additional suppliers
• Allows us to change a core business process once and apply it to all suppliers
• Allows the business to make modifications to its processes with minimal or even no coding

Here is an example of an order fulfillment process.



This work flow speaks to my first point "Creates a set of standard business processes for a consistent user experience". The complexities of working with multiple suppliers, each with their own processes and systems is hidden from the end user. Regardless of what products the customer is buying and which supplier is fulfilling the order, the user experience is exactly the same. On the other side of the equation, bringing on a new supplier can be as simple as collaborating on a set of services to integrate the supplier's back end system to the core business processes of the eCommerce site. Had we not separated the business processes from the code, we would have to make modifications to our existing code base which is already running in production and would have to go through a complete testing cycle. Instead, we simply tell the business processes which set of new services to integrate with for the new supplier. All of the existing code is not impacted and does not need to be retested. Instead, we can just validate the new services and the new supplier flow.

Here is another benefit. Let's say that we just signed agreements with three new suppliers and have contracted to launch them all in 4 weeks. Without the separation of business processes from code, all three supplier integration efforts would have to be coordinated and most likely bundled into one build. This creates huge project risks because slippage in any one integration effort can delay all three. By separating the business processes from the code, each supplier integration can be implemented whenever it is ready. We can stub out a call for each supplier and assign the appropriate WSDL when the services are completed. For example, the "Fulfill Order" process could be an integration with SAP for supplier 1, PeopleSoft for supplier 2, a custom built fulfillment process written in Java for supplier 3, and a custom built fulfillment system written in .Net for supplier 4. The business and the business processes have no need to be concerned with the technologies behind the fulfillment process. They are only concerned with the inputs, the outputs, and the quality of service (QoS).

As we move up the stack it gets even better! Next post I will discuss the importance of business rules. Stay tuned.

I have discussed the role of the SOA Evangelist and the importance of the architects. Today I will discuss the different developer roles and the characteristics that make a successful SOA developer.

In a traditional 3-tier architecture it was common that there was a presentation layer, a middle tier or application layer, and a data layer. In some cases, a single developer role was responsible for all three layers. In larger companies, there may have been specialized roles such as a UI developer, an application developer, and a database developer. In an SOA, the concept of an application is no longer relevant except from the standpoint of integrating to existing applications. With SOA, we should be building business services that are application independent. The following chart is one I have used in previous posts which shows a typical breakdown of some of the development roles required for SOA.

From SOA Slides

So let's talk about business services. A business service is the compilation of all of the work done by the collection of developers within each layer. For example, let's discuss a "Shopping Cart" business service like the one we use on Amazon.com. It is most likely made up of services and/or components that live within each of the various layers of the architecture. The presentation layer obviously contains the physical implementation which the user sees and interacts within the browser. The business process layer contains the logical flow of steps that walk the user through the checkout process from start to finish. The business rule layer contains rules about tax codes, discounts, memberships, etc. and the underlying data elements and structures are handled in the data layer. In many instances, companies have multiple data structures that serve similar functions due to mergers, acquisitions, years of legacy systems, 3rd party application purchases, and many other reasons. To abstract these different data structures and present a common data interface that hides the complexity of the physical implementation of the data, the data services layer exists (think master data management).

So to develop this shopping cart business service, all of the people working within the different layers of the architecture must collaborate to deliver the service in a way that it both meets the business requirements and the technical requirements as defined by the SOA Governance model that the company has adopted. Examples of the technical requirements might be...

• Adheres to WS-* security standards
• Data encryption policies
• Platform neutral implementation
• Meets specific performance requirements

Why do I bring all of this up? Because developers that will succeed developing in a service-oriented architecture need to have the following characteristics:

• Flexible, open to change
• Collaborative
• Able to work in a setting with peers reviewing their work
• Able to see the big picture
  
• Not married to a certain technology
• Able to withstand constructive criticism
• Innovative

Developers who are protective of their own work or of their technology of choice will likely struggle working on SOA. One of the goals of SOA is to build platform neutral software that is highly flexible, maintainable, and loosely coupled. To build software like this we must shift away from software developing and move to software engineering. In simple terms, we must move from drag and dropping to planning and modeling. We must embrace collaboration, peer reviews, and governance. If your developers don't like these requirements, they either need to change or leave. Otherwise they will be a cancer and will continously feed the forces of resistance.

Ok, off my soap box now. So let's discuss the roles. But before we do let me note that we are talking about roles and not individuals. In smaller companies, a developer may perform multiple roles. In larger companies we may actually see very specialized developers living within a single layer of the architecture. One last disclaimer: I am assuming that an architecture team exists and some form of standards and governance is in place within each layer.

UI developers
If your company can afford to specialize, this is the perfect layer for it. Developers in this layer do not necessarily have to be extremely technical. It is more important that they understand usability, UI standards, and web interface best practices. These people should be comfortable working in prototype mode where they can start with mockups and work closely with the business or business analyst to reiterate through the various visuals until an agreed upon format is approved. These developers must be able to talk to the business in business terms and understand how typical business users interact with web technology.

Business Process developers
There are two distinct roles in this business process layer. One role deals with modeling business processes, the other role deals with integrating business processes with the underlying services and systems. In some companies this role may be served by one person but more likely it will be two because of the difference in the skills required. The business process modeler may not even be an IT person. Some companies have a department that exists within the business that focuses on business process improvement and business process automation. This is the norm for companies that practice Six Sigma or Total Quality Management (TQM).

The business process integrator is a technical role which requires expert experience working with Web services, REST, JMS queues or several other technical areas of expertise. The integrator is the person who connects business processes to the backend technologies which controls the overall flow of the business services and overall composite business application, often coined orchestration.

Business Rules developer
This one is a little fuzzier. Not all architectures have a distinct business rule layer. In some instances, the business rules are controlled in the data layer. For companies who have very dynamic business rules, especially rules that are customer centric and could be updated by end users and even customers, it makes a great deal of sense to abstract the business rules. If a company has a business rule layer and implemented a tool to manage business rules, it is highly likely that you will need technicians who govern this layer much like DBAs govern the database layer. In some case this might be an additional role for a DBA but it does not have to be. Whoever it is they must understand the concepts of abstraction and be looking for ways to empower the end user to react quicker to changes in the business. For example, let's say that a loan approval process requires state specific business rules about what percentage of capital is required from a loan applicant up front before the loan application can be expected. The frequency which this value changes varies by state and must be updated as soon as possible. The best scenario is to take IT out of the process and enable certain users (or systems) with the appropriate authority to update the values as necessary. Whoever is responsible for the business rules must be able to work with the business and/or business analysts to understand the frequency of change, the rights and permissions, and the impacts of the various business rules. There is an amount of overhead associated with creating and managing rules and the person in charge must understand the trade offs.

Data Services developer
Maybe a better name for this person is an information architect. The person in charge of this function is responsible for abstracting the underlying data layer and exposing it the other layers of the architecture and even externally to other systems. For example, let's say that our shopping cart business service allows multiple forms of payment (American Express, Visa, and Pay Pal). In addition, the different products purchased (books, DVDs, clothing, etc.) are managed from different inventory systems. To hide that complexity and to allow us to add additional payment services and inventories with ease, we leverage data services to present the shopping cart service with a logical view of the data. In other words, we create a standard payment and inventory message that the shopping cart communicates with. As long as the shopping cart communicates with these standard messages, the underlying physical implementation of the different payment and inventory services are irrelevant. The shopping cart service simply communicates these messages in the standard format it knows and the translation from the standard message to the physical layout of the receiving systems or services is performed in the data services layer.

From SOA Slides

(This slide is from BEA)

Developers or architects within this layer must have exceptional technical knowledge in the area of data modeling and database design. If the company has a tool to manage this layer (which is highly recommended) then an administrator is also needed.

Database developers
This data layer is typically an area that already exists today within each enterprise. This is where the DBAs live. With SOA, the DBAs must work extremely close with the developers within the other layers of the architecture. They must understand the performance and security requirements of each layer and ensure that the underlying data model meets the needs. Since legacy integration is common in many SOA implementations, DBAs are often called upon to create views or structures or even establish new ETL processes to provide a common view of data to expose to the other layers of the architecture.

Security developers
The security experts may not actually do the development but there must be a person or group of people who understand the new security challenges that SOA introduces. SOA can introduce the following new threats:

• Expose components of legacy systems that were never designed to live outside the firewall
• Require trusts between multiple internal/external systems without additional sign-ons
• Send a single message to multiple partners each with their own distinct security requirements
• Expose credit card and other privacy related data out on the web

From SOA Slides

(This slide is borrowed from the following presentation from Mamoon Yunus of Forum Systems)

Todays n-tier security models are not enough when dealing with B2B type SOA implementations. The developers in this area must have a deep understanding of communication protocols, security best practices, web architecture, regulation (ie. HIPAA, SOX, PCI) and WS-* or other standards.

Summary
As you can see there are many distinct developer skill sets required to implement SOA properly. Trying to find a person that can master each layer is not realistic. If you have this person, it is likely that he or she is on your architecture team. The beauty of this architecture is the ability to divide and conquer the problem within each layer. A single business service can be worked on by numerous people within each layer. It requires solid governance, strong collaboration skills, and teamwork to work this way which is why it is critical that the right team of people is assembled to tackle SOA. My next post will discuss the testers and their relationship with the development team.

In part 1 of this series I asked the question, "Are you running IT like it's your business?" Then I highlighted five barriers for preventing IT leaders from being able to transform their IT shop into a well oiled, cost effective machine?

• Resistance to change
• Lack of resources (time, money, and human capital)
  
• Lack of tools
• Lack of metrics
• Lack of process

In part 2 I will focus on Lack of Resources.

What are some of the most common phrases you hear when attempting to promote change within your organization? The top one I hear is "I don't have time", followed by "I don't have money", followed by "I need more resources". These are the biggest cop outs in the world. The reason why nobody has time, money, or resources is because they don't put any initiatives in place that allow them to do anything other then put fires out.

At some point you should reach a threshold of pain and realize that there must be a better way. Everybody is working 50-60 hours, all the projects are late, production support continues to increase, and the users are screaming. Everybody is working hard, but nobody is working smart.

If this sounds like your shop, you might want to take a step back and put a plan together to stop the madness. If you owned your own trucking business and you spent most of your time repairing your fleet of trucks you would probably be out of business. So why do so many IT shops spend countless hours every day in repair mode? It's time to make time, time to spend your shareholders' dollars effectively, and time to maximize your precious resources.

Let's start with time. Where are you spending your time? Do you have any metrics that allows you to proactively manage and control your projects or your production support? In many shops, the main cause of all these issues is lack of a sound project prioritization process coupled with a lack of a resource allocation model. If projects get accepted at will, priorities change like the seasons, and you have no data to show that your resources are over allocated, you will become a reactive culture. If you are not practicing Portfolio Management, you should make that a priority (view this article I wrote on Portfolio Management). If you have no visibility into how much time you are spending in production support, what categories of issues you are encountering, and how long it takes to fix issues then you should make change management a priority. If you are meeting with the business to discuss the next wave of new projects and do not have a clear picture of your resource capacity, you need to start practicing human resource planning.

Next is money. Obviously, if you are spending too much time keeping the lights on then you are wasting money. But are you wasting money on infrastructure? Are you still pouring millions of dollars into big name vendors like Microsoft, IBM, Oracle, and HP? Are you paying huge amounts of money for "Gold Support"? You need to start evaluating Open Source alternatives. If you believe in the myths of Open Source, it's time you start doing your homework and read about these 50 Open Source success stories. Oh, and that Gold Support you are paying for each and every product...there are many Open Source Support Providers who can support a variety of open source products at a fraction of the cost. What is better is that these folks earn their stripes supporting customers and are actually extremely interested in helping you resolve your issues. This is their core competency. I wonder how much effort Big Blue would put forth to help a small $100M company resolve its problems. The days of saying, "Nobody every lost their job buying Microsoft and IBM" are long gone. If that is all you are buying you are definitely not seeing the big picture and your CEO might start giving you that look.

Are you still buying rack after rack of servers to support your test, development, staging, and production environments? Have you embraced Virtualization Software like VMWare yet?

And what about human capital? We already talked about spending too much time keeping the lights on and spending too much time maintaining rack after rack of servers. But what about development and testing? Do you have an architecture that allows rapid deployment and reusability? Have you separated your business rules from your applications? If your business rules are embedded in your applications then you are creating additional work to maintain and test your applications thus increasing your chances of creating defects.

Let's say you are a retailer with 1000 stores and you have a business rule for defining what a high volume store is. You have several different systems that use this business rule (billing, inventory, eCommerce site, financials, etc.). Now you want to change the definition of a high volume store. You now have to make that change in each of these systems and implement all of the systems at the same time. This simple change has turned into a project. Wouldn't it be better to go to one place, make the change, test it in one place, and roll? You don't need to practice agile methodologies to be agile. You just need to be smart! My recommendation is invest in your architecture. Create a business rule layer that sits between your applications and your presentation layer. I would go one step further and create a service layer to orchestrate your business rules and allow other applications and/or customers to access your business rules.

Well, I have rambled a lot longer then I usually do so I will pause until part 3 which focuses on Lack of Tools. I will add a conclusion after part 5 that tells you how to prioritize these initiatives and how to get started.