Complex Event Processing John Plummer, Jeff Johnson Agenda

Complex Event Processing John Plummer, Jeff Johnson Agenda

Complex Event Processing John Plummer, Jeff Johnson Agenda Introduction What is CEP ? Typical Application and Architecture Event Query Languages Event Processing Examples NEsper

BizTalk RFID Demo What is CEP ?

Complex Event Processing (CEP) is a set of techniques and tools to help understand and control event-driven Information Systems Lets look at some of the concepts... Conceptual Description Event examples: Church bells ringing, appearance of a man in a suit, a woman in flowing white gown and people throwing

confetti !! A complex event is inferred from simple events A wedding is happening System Examples RFID events What is a Complex Event ? An event that can only happen if lots of other events happened ie Car in Showroom that you like is only there because of a

number of previous events - events in inventory control of factory and dealer - shipping events - customs events - etc History of Event Processing http://complexevents.com/wp-content/uploads/2008/02/1-a-short-history-of-cep-part-1.pdf David Luckham

What is an Event ? Oxford Dictionary defines an event as something that happens or is thought of as happening In CEP an event is an object that is a record of an activity in a system. It signifies the activity and has three features:Form: Form of an event is an object, may have attributes or data components. Can be as simple a string or more often a series of data items Significance: Events signifies an activity. Relativity: An activity is related to other activities by time, causality and aggregation. Events have the same

relationship to one another as the activities they signify. Examples of Events Order Process Class InputEvent { Name NewOrder; EventId E_Id; Customer Id;

OrderNo OrdNo; Order (CD x, Book ...); Time T; Causality (Id1, Id2); } Class OutputEvent { Name CDOrder; EventId E_Id1;

Customer Id; OrderNo OrdNo; SubOrder O_Id1; Order (CD x, Book ...); SubOrders (O_Id2, ...); Time T1; Causality (E_Id); } Event Models

Streaming Large, dense data streams Eg. Financial trading information 000s of events / second Non-Streaming Business events Eg. New Order, BAM Time

How Events are Created We need to be able to create events that signify the activities that are happening in the system. Observation Step: Access and Observe the activities at any level and it MUST NOT change system behaviour (ie it must be benign) Adaptation Step: Observations need to be transformed into event objects that can be processed by CEP (typically via Adapters)

Sources can be from: IT Layer (components, MOM, databases etc) Instrumentation (heartbeats, network mgmt, application etc) CEP (events created by CEP in course of processing events) Time, Causality and Aggregation The Three most common and important relationships between events: Time: this is a relationship that orders events ie: event A happened before event B

Cause: This is a dependence relationship between activities in a system ie: if the activity that signified event A had to happen in order for the activity that signified event B, then A caused B Aggregation: this is an abstraction relationship ie: if Event A signifies an activity that consists of the activities of a set of events, B1, B2, B3 then A is an aggregation of all the events in B.

Typical Application and Architecture CEP Part of Event Driven Architecture EDA Definition: Notable thing happens in business Event might signify a problem, opportunity, threshold, variance etc Event pushed to all interested parties Characteristics:

Loose coupling creator of event no knowledge of consumption Event Processing styles Simple Event Processing event occurs; action initiated Stream Event Processing stream of ordinary and notable events; filtered to raise significant business event Complex Event Processing notable and ordinary events; different event types, longer time spans. Correlation may be causal, temporal or spatial

Example EDA Architecture http://elementallinks.typepad.com/bmichelson/2006/02/eventdriven_arc.html Typical CEP Applications

BPM Monitoring, BAM, report exceptions Finance (trade analysis, detect fraud, risk analysis) Network (SLA monitoring, intrusion detection) Sensor (RFID, air traffic, schedule & control) CEP Comparison to traditional App SQL standard query language

CEP engines are like a RDBMS turned upside down Data generally more static Store queries and run data through them Complex queries rarer

Continuous execution model, rather than when a query is submitted Not suited to 000s queries / second Event pattern languages Triggers can respond to

events but relatively slow Event stream queries CEP Platform Characteristics Event sampling Storing Routing

Enrichment Parsing Matching Transformation Generalised Event Language Notation:

Xi, Yi Events need order number, so can understand order Xi(a) Event need attributes, so we can compare values and match etc or compare specific events within the set Xi(a)=Yi(b) T - time interval important

Operators: Logical operators: and, or and not. Time operator: within T (Z). Sequence operator: ->.

Example expressions: X and Y within T(40 seconds) A -> B (event B has to arrive after A) http://dist.codehaus.org/esper/CEP_MasterThesis_PaulDekkers_200709.pdf Important Operators Time Within n seconds (...) Sequence of Events insider trader detection

Within 10 days (sellShares(amount>10000) -> stockPriceChange(..) ) -> operator significance Detects where larger share sales have occurred after significant price change, which might indicate insider trading http://dist.codehaus.org/esper/CEP_MasterThesis_PaulDekkers_200709.pdf Filter Sliding Window Example select * from Withdrawal(amount>=200).win:length(5)

Events are filtered into the sliding window http://www.espertech.com Filter events within the window

select * from Withdrawal.win:length(5) where amount >= 200 Events passed onto the Listener are filtered http://www.espertech.com

SOA and CEP CEP / EDA augments and enhances SOA Event-Driven SOA Notable event occurs that can trigger a service invocation Service Generation of Events Service invocation generates an event which is dispatched to all subscribers who have registered an interest

Event Processing Examples Nesper BizTalk RFID CEP Example - NEsper (N)ESPER Architecture Listeners

http://www.espertech.com ESP and CEP Sliding windows, Aggregation, Causality http://www.espertech.com NEsper & BAM Demo

Contextual Architecture Demo Scope BizTalk BAM NEsper Event Streams

RFID Events Filtered Events BizTalk RFID WCF, WF, BizTalk BAM Events BAM

Portal Market Data Feed Scenario 1s window 10s Windows Detect an event rate fall off. Checking if count in a 10 second

window is < 75% of the average count. Data Feed A Data Feed B select event count in 1 sec window. Insert into TicksPerSecond

Alert raised if detected and BAM event written TicksPerSecond Market Data Feed Run the simulation 2 threads Drop probability 60% 10 second interval

Populate TicksPerSecond Feed Selects the event count from the Market Data Event stream in 1 second windows Inserts the number of ticks per second in the Ticks Per Second feed Detecting a Fall Off in Rate EQL statement to detect fall-off rate Selects from TicksPerSecond which has 10 second

windows of counts Checks if count is < 75% of average count indicating a fall off BAM Event Data Event Feed Rates Feed A

Feed B BizTalk Server R2 RFID Event Processing BizTalk RFID Support services for RFID at the edge Device plug-n-play and management Filtering / transformation / aggregation, data cleansing

and validation Reacting to RFID events Alerts (HW / SW) & tag processing rules Inferring business relevant information Integration of RFID into business process server RFID events as messages in BizTalk Standards based interop through XML Web services

Commands can be pushed using connector architecture Example Flow Event Processing Engine BizTalk RFID Event Processing Application model for Synchronous and Asynchronous event processing

Declarative specification of an Event Processing Tree Design and Deployment separation BRE Event Handler Summary & Q&A Defined CEP and history Relationship To SOA Types of challenges of CEP Provide demonstration of event stream processing

integrated to BizTalk BAM Review event processing capabilities in BizTalk RFID Thank you

Recently Viewed Presentations

  • Oklahoma City Public Schools FY20 Beginning of Year

    Oklahoma City Public Schools FY20 Beginning of Year

    After ILD signs, claim will be emailed to appropriate Title I Staff Member, and cc: Principal & Fin. Sec. Title I will review, sign, and email/contact Fin. Sec. to GR and then Title I will email claim to AP for...
  • «Основні класи неорганічних сполук»

    «Основні класи неорганічних сполук»

    За вмістом Оксигену кислоти поділяються на… Складні речовини, що складаються із двох елементів, один із яких є Оксиген, називаються…
  • Ground Rules, exams, etc. (no &quot;make up&quot; exams) Text: read ...

    Ground Rules, exams, etc. (no "make up" exams) Text: read ...

    Download Syllabus from above site (Contract) Exams, ground rules, discussion sections, textbook, etc. ... analgesics diuretics hormone mimics laxatives tranquilizers Fer-de-Lance Bothrops asper Harry Greene Nature versus Nurture Controversy "Environmentally induced" polymorphism Kathy ...
  • Presentation Title

    Presentation Title

    Includes On-Net, Unlimited local and unlimited LD calling (Customer does not receive Unlimited LD with an IP PBX w/ IP Flex Reach), available on AT&T dedicated transport or bring-your-own-broadband (cable or DSL).
  • Algorithms Tradeoff: Execution speed vs. solution quality Solution

    Algorithms Tradeoff: Execution speed vs. solution quality Solution

    Maximum Cut Maximum cut problem: Given a graph, find a partition of the vertices maximizing the # of crossing edges. ... NSPACE NTIME Occam's razor Octonions One-to-one correspondence Open problems Oracles P completeness P vs. NP Parallel postulate Parallel simulation...
  • Sarcopenia - minsal.cl

    Sarcopenia - minsal.cl

    Definición. Consenso EUGMS,ESPEN e IAGG-ER. La sarcopenia es «un síndrome caracterizado por una progresiva y generalizada pérdida de masa y fuerza muscular esquelética con riesgo de resultados negativos como discapacidad física, peor calidad de vida y muerte».
  • Unit 1: Introduction to Social Studies

    Unit 1: Introduction to Social Studies

    You are a merchant in India in about 240 BC. You travel from town to town on your donkey, carrying bolts of colorful cloth. In the heat of summer, you are grateful for the banyan trees along the road. They...
  • Kroenke-Auer-DBP-e14-PPT-Chapter-03

    Kroenke-Auer-DBP-e14-PPT-Chapter-03

    Albert, Bruce, Cathy, David, Edith, and so forth. All values of . FirstName. must . come from the names in that domain. Columns in different relations may have the same name. 3-Alternative Terminology. Although not all tables are relations, the...