by Colin White

January 2003

Building the Intelligent Business

Business intelligence and data warehousing systems were once considered nice to have, but they are now so essential to business success that many companies cannot manage their business effectively with them.

These systems are no longer standalone and separate from operational processing — they are becoming more and more integrated into the overall business process. This integration allows business intelligence to be used, not only for strategic and tactical planning, but also for actively driving day-to-day business operations and decisions. To support operational decision making, the business intelligence system must provide easy-to-use and self-service enterprise analytical applications that enable real-time decision-making. It must also include a methodology (either formal or informal) that helps organizations assign responsibilities to specific business users for handling business intelligence and define actions to be taken when the business intelligence identifies business problems that require urgent attention. Such a system can be thought of as supporting integrated and actionable business intelligence, which enables an organization to become an intelligent business, rather than just a business with a business intelligence and data warehousing system.

A Historical Perspective

Software and hardware for delivering accurate business information to executives and managers for corporate decision making has evolved from early centralized batch reporting systems to data warehousing and business intelligence processing, and more recently to enterprise analytics applications.
Users of first-generation centralized batch reporting systems had to deal with complex tools, poor data quality, no data integration, inconsistent data, and limited historical information. Data warehousing was introduced to help improve the quality, integration, and consistency of data, and to store historical data that helped business users track business trends. The problem with early data warehousing implementations was that the development group felt that once the data warehouse was built the job was done, and they were surprised when the data warehouse did not provide the expected ROI. The problem here was that business users did not have the tools to fully exploit the value of the data in the warehouse. Vendors then introduced business intelligence tools that extended the use of data warehousing from simple reporting to advanced analytical processing and data mining.
Although the introduction of business intelligence tools allowed organizations to improve the quality and depth of the reports and analyses coming out of data warehousing systems, several problems still remained. These systems were still passive in that value could only be obtained from the business intelligence produced if business users constantly monitored the results to look for business problems, and then took action to fix them. Even if business users identified a problem they often did not know how to analyze the problem in more detail, or what actions should be taken to resolve it. Although very powerful, many of the new business intelligence tools were complex and time consuming to use, and business users had to rely on power users and the IT group to help them develop new reports and analyses. Complexity also prevented many grass roots employees from having access to the business intelligence they needed to do their jobs more effectively. Call center staff, for example, when handling a customer problem either did not have access to business intelligence about the customer they were talking to, or had to use several different systems to find it.

A New Generation of Enterprise Analytical Solutions

To solve the complexity problems with business intelligence tools, vendors have been introducing new software solutions and products that are at last beginning to improve the usability, and therefore the ROI of business intelligence and data warehousing investments by organizations.
These solutions fall into three groups. The first is aimed at faster and easier application development through the use of analytic application development tools and packaged analytic applications. These applications may support high-level business performance management (BPM) dashboards and scorecards (that may or may not support a business methodology) and/or domain-specific enterprise analytics. The second group of solutions provides personalized access to business information through the use of internal corporate and e-business portals. The third group is designed to support real-time decision-making through the use of near real-time data warehousing, an on-demand analysis server, and an automated rules engine.
It is important to emphasize that each new generation of product builds on, rather than replaces, solutions provided in earlier generations. This means that as you select batch reporting products, data warehousing and business intelligence tools, and enterprise analytic applications solutions they must fit into an overall framework as shown in Figure 1. Otherwise you simply end up with a bunch of non-integrated stovepipe applications.

Near Real-Time and Automated Decision Making

The trend toward using BI to drive business operations is resulting in it becoming more integrated into operational processing. This integration can be achieved in several ways. One approach is to build a closed-loop decision-making system where actionable analytics produced by BI applications are used to generate recommended actions (product pricing changes, marketing campaign modifications, identification of fraudulent transactions, and so forth) to address specific business issues. In the e-business environment, many companies are looking toward extending this closed-loop process to the automatic adjustment of business operations, based on decisions generated by the BI system. In fact, some companies would like this automated closed-loop processing to occur in close to real-time.
CIOs often blanch at the mere mention of the term “real-time.” This is because many of the vendors and analysts jumping on the real-time bandwagon don’t fully understand the business requirements behind real-time processing, and often leave CIOs with the impression that real-time is about performance, rather than about improving the speed of business decision making. Real-time BI enables business users to react rapidly to changing business conditions – it helps reduce the latency between a business event occurring and the time it takes for the business to react to the event. Ideally, this latency would be zero, i.e., the business would be able to react in real-time to changing business situations and requirements.
From a technology perspective, a business intelligence system for making real-time decisions and taking automated actions consists of three components (see Figure 2): a data integration server, an on-demand analysis server, and a rules engine. These components may be used together, or independently of each other. It is unlikely, however, that any given application would require all three components.
data integration server captures and transforms operational data, and loads it into a near real-time data store, i.e., a data warehouse whose data currency is close to that of operational systems. This technology could be used, for example, to build an operational data store (ODS) that integrates customer data from all the customer touch points throughout an organization. Data latency in the ODS may vary from a few seconds to many hours, depending on business requirements. The ODS could be used in a front-office application such as a call center to provide current customer status information for making day-to-day operational decisions. An example would be to identify high-value customers who are to be given priority service or a better discount.
An on-demand analysis server reacts to requests from users and applications for business analytics for tactical and strategic planning and decision-making. The analytics produced by the server can also be used as input to a rules engine. An analysis server may be implemented using vendor-provided packaged analytical applications, or by in-house applications built using OLAP and data mining tools, or by a component-based BI development tool. Commonly requested analytics will be pre-built by the application, but less commonly requested analytics might be calculated dynamically. The main objective of an analysis server is to convert raw analytics stored in a data warehouse to actionable analytics that put the warehouse information into a business context that can be acted upon. For example, the raw analytic,

widget sales today = $20,000

could become the actionable analytic,

today’s widget sales were 30% lower than forecast

rules-engine is used for operational decision-making. The engine employs actionable analytics and business rules to generate and deliver personalized alerts and associated business intelligence to business users, or to generate and deliver action messages for processing by operational applications. The alerts and messages may contain notifications, warnings, recommendations, or suggested actions for handling a particular business situation. If widget sales fall by thirty percent, for example, the appropriate business users could be informed, or the rules engine could generate an operational message to lower the price of widgets by ten percent.

Near Real-Time Data Warehousing

Data warehouses are typically maintained by batch jobs that take periodic non-volatile snapshots of operational data, and clean, transform, and load the data into a warehouse database. To support real-time data integration, the present batch snapshot approach to extracting operational data must be replaced by processes that continuously monitor source systems and capture and transform data changes as they occur, and then load those changes into a data warehouse in as close to real-time as possible. Target Stores is using such technology in a CRM initiative that involves a multi-terabyte real-time data warehouse that contains details about customer interactions across its 900 stores, Web sites, call centers, and catalog sales.
A data integration server can capture data changes from operational systems by monitoring either data events or applications events in the source systems. Data event monitoring involves familiar database technologies such as database triggers, data replication, and database recovery log processing. To support application event monitoring, data warehousing vendors have begun building interfaces to integration brokers and application middleware, and adding support for Web Services. Examples of products and their support include Ascential Software DataStage XE (IBM WebSphere MQ), Acta ActaWorks (Java JMS), Informatica PowerCenterRT (IBM WebSphere MQ, TIBCO Rendezvous), IBM DB2 Warehouse Manager (IBM WebSphere MQ), and the PeopleSoft Integration Broker (Informatica inbound, and IBM WebSphere, JMS and Web Services for outbound messages). These vendors no longer talk about their ETL platforms, but instead market their data integration platforms. Key distinguishing factors when selecting products will be metadata integration, parallel processing, workflow facilities, bi-directional processing (inbound and outbound messages), support for industry standards, scalability, change data capture, and the provision of an interface development kit.
Recent announcements from data warehouse vendors show there is significant interest in real-time data integration. This technology promises significant business benefits, but potential users should be aware that it is very much in its infancy, and poses some interesting challenges in areas such as data quality, for example.

The Importance of Rules

A rules engine is normally invoked based on actionable analytics produced by an analysis server, but it may also be called dynamically in real-time by a user or application for help in making business decisions, whether to grant a client a loan or credit card, or to assess the risk of a specific business transaction, for example.
The business rules used to drive the rules engine may be entered and maintained by business users and analysts, or may be generated by a tool such as a data mining product. Rules engines are embedded in a number of software products including Web application servers and business intelligence tools (where they are sometimes called intelligent agents). There are also several vendors who develop and market sophisticated stand-alone rules engines. Examples here include Computer Associates (CleverPath Aion BRE), HNC (Blaze Advisor), and Pegasystems (PegaRULES).
To fully exploit the power of a decision-making system, the analytics, recommendations, and actions must be related and integrated with the overall business process. One way to achieve this integration is through Business Process Automation (BPA). The best way to explain the use of BPA in a business intelligence environment is by using an example.
The top part of Figure 3 shows a simplified operational workflow for processing a customer order. This workflow can be used to determine the points in the operational process where business activity needs to be monitored by a BI system. Application and data events can then be captured at these points and used to populate an ODS and/or a data warehouse. Event capture can be achieved directly in the application itself, in an integration broker, or at application and data interfaces (database API, application API, EJB interface, user interface, etc.).
If an analytics workflow in a rules engine in the BI system identifies a business situation that requires action, the business user could be alerted and passed an action workflow to help them diagnose the problem and determine what action to take. If automation is required, the action workflow could be embedded in the rules-engine, which would generate the appropriate action messages to be sent to the operational environment. Gartner Research uses the term Business Activity Monitoring, or BAM, to describe this integrated and dynamic real-time environment.
We are likely to see increasing use of workflow and associated business rules to help integrate BI into the overall business process. This is one reason why workflow and rules engines are being added to integration brokers and application middleware — for some examples look at Compaq’s ZLE solution, the Mercator Integration Broker, and Vitria BusinessWare. We are also seeing these technologies being integrated with BI tools. For an example, take a look at how products from SeeRun Corporation are integrated with data warehousing and BI tools from Microsoft and ProClarity.
From a business intelligence and data warehousing perspective it is important to realize that the term real-time is the latest industry buzzword to become abused in vendor marketing literature and campaigns. The move toward real-time BI processing for supporting an intelligent business must be related to business needs, and to what can be achieved by current BI technology. In general, “real-time” should be considered to be a marketing term, and realistically you should not equate real-time BI with the sub-second response times we have become used to in operational transaction processing.

Optional Sidebar: What Do We Mean by Real-Time?

Real-time can be looked at from both a business viewpoint and a technology one. From a business perspective, real-time signifies the need for business users and business applications to be able to respond rapidly to specific business situations. To satisfy this need, the business user (or business application) requires easy access to accurate business information from any place at any time. How current the required information has to be will vary based on business circumstance. In some cases, the information may need to be as current as possible (ideally within a split second of a specific event occurring), whereas in other situations, a latency of a few minutes, or even hours, maybe sufficient.
From a technology and business intelligent viewpoint there are three technology components that need to be considered with respect to real-time: a data integration server, an on-demand analysis server, and a rules-engine.
data integration server is used to build a near real-time data store in a data warehouse. Current technology can reduce the latency of this data store to within a few seconds of the events occurring that caused data to change in associated operational systems. Information in the data store can equally be delivered to business users and applications in a matter of seconds. Here then near real-time time means a matter of seconds.
An on-demand analysis server delivers actionable analytics to business users and applications. These analytics can be delivered based on an event occurring in a data warehouse (push), or based on an information request from a user or application (pull). If the analytics have been pre-computed then delivery can be achieved in a matter of seconds. If the analytics have to be dynamically calculated then the response time will vary based on the amount of processing that has to be done, and on the amount of data that has to be accessed. It is important to point out that the analysis may or may not require near-current data.
A rules engine delivers messages, alerts, recommendations, or business intelligence to business users and applications. As with an analysis server, information delivery from a rules engine can occur on a push or pull basis. (Note that in some cases the rules engine may be embedded in an analysis server). The output from a rules engine is generated based on business rules, and on data supplied by either a business user or application, or by an analysis server. In general, the processing done by a rules engine can be performed rapidly, and a response can therefore be delivered in a matter of seconds. If, however, the rules engine has to dynamically build or dynamically modify a complex model, then the response will take longer.