Complex Event Processing Is A Technique
Complex Event Processing (CEP) is useful for big data because it is intended to manage data in motion. Complex Event Processing is a technique for tracking, analyzing, and processing data as an event happens. This information is then processed and communicated based on business rules and processes.
The idea behind CEP is to be able to establish the correlation between streams of information and match the resulting pattern with defined behaviors such as mitigating a threat or seizing an opportunity. CEP is an advanced approach based on simple event processing that collects and combines data from different relevant sources to discover events and patterns that can result in action.
Here is an example. A retail chain creates a tiered loyalty program to increase repeat sales — especially for customers who spend more than $1,000 a year. It is important that the company creates a platform that could keep these critical customers coming back. Using a CEP platform, as soon as a high-valued customer uses the program, the system triggers a process that offers the customer an extra discount.
Another process rule could give the customer a surprise — an extra discount or a new product sample. The company also adds a new loyalty program that links to a mobile application. When a loyal customer walks near a store, a text message offers the customer a discounted price. If that loyal customer writes something negative on a social media site, the customer care department is notified and issues an apology.
It is quite likely that you are dealing with a huge number of customers with a significant number of interactions. But it would not be enough to simply stream the data and analyze that data. To achieve the business goals the retailer wanted to achieve would require executing a process to respond to the results of the analysis.
Many industries take advantage of CEP. Credit card companies use CEP to better manage fraud. When a pattern of fraud emerges, the company can shut off the credit card before the company experiences significant losses. The underlying system will correlate the incoming transactions, track the stream of event data, and trigger a process. CEP is also implemented in financial-trading applications, weather-reporting applications, and sales management applications, to name a few.
What all these applications have in common is that the applications have a predefined norm for temperature, pressure, size of the transaction, or value of the sale. A change in state will trigger an action. If you drive a late-model car, you probably have noticed that when a tire’s pressure has dropped, the car will trigger a dashboard indicator that notifies the driver to take action (getting the tire fixed).
Many vendors offer CEP solutions. Many of the CEP tools on the market allow the creation of real-time, event-driven applications. These applications might ingest data from streams, but they can also ingest data from traditional database sources. Most of the offerings include common capabilities, including a graphical development environment that is typically Eclipse-based, connectivity to real-time data flows, as well as APIs to historical data sources.
Most of these products include a graphical event flow language and support SQL. Key vendors in this space include Espier (open-source vendor), IBM with IBM Operational Decision Manager, Informatics with Rule Point, Oracle with its Complex Event Processing Solution, Microsoft’s Stream Insights, and SAS Data Flux Event Stream Processing Engine, and Stream base’s CEP. Numerous startups are emerging in this market.
CEP is used in operational intelligence (OI) solutions to provide insight into business operations by running query analysis against live feeds and event data. OI solutions collect real-time data and correlate against historical data to provide insight into and analysis of the current situation. Multiple sources of data can be combined from different organizational silos to provide a common operating picture that uses current information. Wherever real-time insight has the greatest value, OI solutions can be applied to deliver the information needed.
In network management, systems management, application management, and service management, people usually refer instead to event correlation. As CEP engines, event correlation engines (event correlates) analyze a mass of events, pinpoint the most significant ones, and trigger actions. However, most of them do not produce new inferred events. Instead, they relate high-level events with low-level events.
Inference engines, e.g. rule-based reasoning engines typically produce inferred information in artificial intelligence. However, they do not usually produce new information in the form of complex (i.e., inferred) events.
Most CEP solutions and concepts can be classified into two main categories:
- Aggregation-oriented CEP
- Detection-oriented CEP
An aggregation-oriented CEP solution is focused on executing on-line algorithms as a response to event data entering the system. A simple example is to continuously calculate an average based on data in the inbound events.
Detection-oriented CEP is focused on detecting combinations of events called event patterns or situations. A simple example of detecting a situation is to look for a specific sequence of events.
There also exist hybrid approaches.
A more systemic example of CEP involves a car, some sensors and various events and reactions. Imagine that a car has several sensors—one that measures tire pressure, one that measures speed, and one that detects if someone sits on a seat or leaves a seat.
In the first situation, the car is moving and the pressure of one of the tires moves from 45 psi to 41 psi over 15 minutes. As the pressure in the tire is decreasing, a series of events containing the tire pressure is generated. In addition, a series of events containing the speed of the car is generated. The car’s Event Processor may detect a situation whereby a loss of tire pressure over a relatively long period of time results in the creation of the “lossOfTirePressure” event. This new event may trigger a reaction process to note the pressure loss into the car’s maintenance log and alert the driver via the car’s portal that the tire pressure has reduced.
In the second situation, the car is moving and the pressure of one of the tires drops from 45 psi to 20 psi in 5 seconds. A different situation is detected—perhaps because the loss of pressure occurred over a shorter period of time, or perhaps because the difference in values between each event was larger than a predefined limit. The different situation results in a new event “blowout Tire” being generated. This new event triggers a different reaction process to immediately alert the driver and to initiate onboard computer routines to assist the driver in bringing the car to a stop without losing control through skidding.
In addition, events that represent detected situations can also be combined with other events in order to detect more complex situations. For example, in the final situation, the car is moving normally and suffers a blown tire which results in the car leaving the road and striking a tree, and the driver is thrown from the car. A series of different situations are rapidly detected. The combination of “blowout Tire”, “zero Speed” and “driverLeftSeat” within a very short space of time results in a new situation is detected: “occupantThrownAccident”. Even though there is no direct measurement that can determine conclusively that the driver was thrown, or that there was an accident, the combination of events allows the situation to be detected and a new event to be created to signify the detected situation. This is the essence of a complex (or composite) event. It is complex because one cannot directly detect the situation; one has to infer or deduce that the situation has occurred from a combination of other events.