Barry Devlin

June 2013

Developments in big data and other technology over the past few years suggest a growing convergence of business and IT and a fundamental change in the way we manage and use information.

Developments in big data and other technology over the past few years suggest a growing convergence of business and IT and a fundamental change in the way we manage and use information.

Over the past quarter century, businesses of every shape and size have pursued an information policy that could be summarized in a single phrase: “single version of the truth” or SVOT. As the name implies, this concept suggests that a business can and should be described and operated according to the principle that there is one single set of information that fully and accurately describes how the business operates. We see this policy in action in operations, replacing multiple transactional applications by integrated ERP, CRM and similar systems. We see it in business intelligence, with the implementation of all-encompassing data warehouse environments. We observe integrated data governance and process rationalization initiatives. The hope is that—if only we can get all our information and procedures aligned—we will finally get our businesses under control and get on with the process of maximizing profit, or whatever the underlying purpose of the business is seen to be.
Sadly, the world doesn’t work like that. And strangely, while we observe and admit that the world doesn’t work like that, we insist that business should. Of course, there are parts of the business where SVOT is important. Delivering financial results to the market or reporting profits to the tax authorities should be accurate, complete and verifiable (Mafia excepted, of course). Research or marketing activities, however, do not have the same requirement; multiple versions of the truth are much more interesting. Even within the finance department, there may be a valid but small number of varieties of truth in the run-up to closing the books for a financial period. And, that’s just focusing on numerical data. When it comes to softer information, from e-mails to images, truth is a much more fluid affair.
Early adopters of big data techniques and tools have focused on such softer information. This includes sentiment information from the likes of Twitter and Facebook, behaviour information from clickstreams and web logs, as well as text mining from e-mails and conversation transcriptions. Deep analytics creates models that predict future behaviour—whether a mobile phone customer is likely to churn, if a certain offer might increase propensity to buy or whether a particular transaction may be fraudulent. Such information is very far from the concept of SVOT. However, big data usage is maturing as we discovered in a recent survey . Business users are combining such information with traditional operational and informational data sources, from telco call detail records and retail till receipts to financial performance information from the data warehouse. This is game-changing for business. And demands significant technological shifts by IT. It turns out that big data is just the forerunner of what’s to come. Let’s look at just two aspects in particular.

Operational Analytics
Operational processes have been highly structured and heavily regulated since they were first computerized on mainframes. The flexibility to handle novel circumstances—characteristic of human interaction—was and is notable only by its absence. Operational analytics dramatically changes that. Starting with large samples of behavioural and transactional data, a predictive model is created. So far, this is just data mining. In operational analytics, the model is automatically applied in the operational environment—a retail website or call centre—to change the offer made in real time to the customer, based on the actual behaviour of the customer during the current interaction. The result is fed back into the predictive modelling environment to further optimize the model.
This application breaks the barrier between operational and informational processing that has been a feature of computing since the earliest days. Operational procedures gain a new level of flexibility, perhaps not personalized service, but something far more humanlike. The impact for IT is enormous. A fundamental architectural principle is broken. The infrastructure, application and database are required to handle two very different types of processing hand in glove. This demands the elimination of batch processing and a re-architecting of the underlying database, features implicit in a move to in-memory processing, as seen in SAP HANA, for example.

Mobile Business
When retail moved to the Web, pundits predicted the end of the physical store. They were correct only to an extent. Now as the Web moves to tablets and smartphones, the clicks and bricks balance has shifted again. Like operational analytics, the net business effect is the acceleration of operational activities and their closer integration with informational processes. A smartphone-equipped shopper in store can find lower rivals’ prices, possibly postponing that purchase; so real-time competitive information is mandatory for every retailer. On the other hand, knowing who that shopper is and where she is standing in the store allows the retailer to make that special, personalized offer; returning us immediately to operational analytics. And it’s not only consumers; business users also find their lives changed. Executives get real-time access to their KPI dashboards wherever they are. Field personnel get instant access to the operational and informational applications needed to perform every task.
Operational analytics, mobile business—and it’s extension, the Internet of Things, when every manufactured article, every domesticated animal and potentially every human is equipped with one or more sensors and permanently online—generate a veritable superstorm of information. Only a small percentage of this information will exist in the highly structured formats we use in relational databases today. Much of it will be transient, multiple truths representing fleeting glimpses of reality that require massively parallel, highly flexible and deeply algorithmic processing, only the beginnings of which we see in Hadoop. The processing speed, networking, information volumes and personal flexibility required cannot be handled with today’s IT architecture and technologies. New tools, both hardware and software, are already available, in early versions in many cases. A new IT architecture is emerging, evolving from data warehousing, service oriented architecture (SOA) and Web 2.0, to name but a few. Evaluating, adapting and implementing these new technologies and architectures should be high on the agenda of every forward-looking IT department.