StreamInsight Complex Event Processing (CEP) Platform John Plummer
StreamInsight Complex Event Processing (CEP) Platform John Plummer Technical Specialist Data Platform Microsoft Ltd What is CEP? Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Database Applications Event-driven Applications Query Paradigm Ad-hoc queries or requests Continuous standing queries Latency Seconds, hours, days Milliseconds or less Data Rate Hundreds of events/sec Tens of thousands of events/sec or more request response
3 Event input stream output stream 3 Example CEP Scenarios Power, Utilities: Energy consumption Outages Smart grids 100,000 events/sec Financial Services: Stock & news feeds Algorithmic trading Patterns over time Super-low latency 100,000 events /sec Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds Stream Data Store & Archive 4 Web Analytics: Click-stream data
Data Data Stream Stream Data Data Stream Stream Manufacturing: Sensor on plant floor React through device controllers Aggregated data 10,000 events/sec Event Processing Engine Lookup Lookup Threshold Threshold queries queries Event Event correlation correlation from from multiple sources multiple sources Pattern Pattern queries queries 4
Sweet Spot for Complex Event Processing Months CEP Target Scenarios Days Relational Database Applications Hours Data Warehousing Applications Minutes Seconds 100 ms Monitoring Applications < 1 ms 0 10 Operational Analytics Applications (e.g., Logistics) Web Analytics Applications Manufacturing Applications 100 1000
10000 Aggregate Data Rate (Events/sec) 5 Financial Trading Applications 100000 higher Complex Event Processing SQL SERVER 2008 R2 STREAMINSIGHT SITUATION TODAY Pressure to handle large volumes Process large volume of events across of complex data in realintime multiple data streams less than a second The
size your and frequency of data historical make it Manage business through challenging store for data insights. mining and data mining to and continuous analysis Growing need tofor monitor, analyze and Built in support different types of event act on theand data in query motionsemantics handling rich "Intelligence "Intelligence isn't isn't just just about about knowing
knowing what what is is happening. happening. It's It's about about looking looking at at the the patterns patterns in in real real time. time. If If business business people people want want to to truly truly optimize their their resources, resources, they optimize they must must examine examine patterns patterns in in real real time. time. -- Mike Mike Gualtieri Gualtieri -- Forrester Forrester
6 StreamInsight Platform from Microsoft for continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with low latency EVENT SOURCES Event stores & Databases Bloomberg.co m Stock tickers & News feeds 7 MSFT MSFT CEP CEP Engine Engine Output Output Adapters Adapters Web servers Input Input Adapters Adapters Devices, Sensors
APPLICATION EVENT TARGETS Pagers & Monitoring devices KPI Dashboards, SharePoint UI Trading stations Reference data Event stores & Databases Events Events expose different temporal characteristics Point in time events Interval events with fixed duration Interval events with initially unknown duration (Edge) Payload/ value Rich payloads capture all properties of an event a t1 8 b t2
c t3 Time d t4 e t5 Event Types Events in Microsofts CEP platform use the .NET type system Events are structured and can have multiple fields Fields are typed using the .NET framework types CEP engine provisioned timestamp fields capture all the different temporal event characteristics Event sources populate time stamp fields Timestamps/ Metadata 9 Long pumpID String Type
String Location Double flow Double pressure Event Streams & Adapters A stream is a possibly infinite sequence of events Insertions of new events Changes to event durations Examples: time series sensor data, log entries Stream characteristics: Event/data arrival patterns Steady rate with end-of-stream indication Examples: files, tables Intermittent, random, or in bursts Example: retail scanners, web, weather telemetry Order of arrival of events does not match the order of their application timestamps Input & Output Design Time for dealing with Event Streams 10 10 Core CEP Query Engine
Queries are composable Query results are computed incrementally Event CEP Engine Standing Queries Event Event Event Event Event Event Event Event Output OutputAdapters Adapters Operators consume and produce streams Input InputAdapters Adapters Hosts standing queries
Query instance management: Submit, start, stop Runtime statistics Takeaway: CEP engine does the heavy lifting for you when processing temporal event data 11 11 Developer Experience with the CEP Platform CEP development environment Builds on top of .NET Microsoft Visual Studio as IDE Applications written in C# Queries written in LINQ Supports various deployment models for CEP applications Embedded applications Server applications 12 LINQ: var var queryFilter queryFilter == from from cc in in TestEventStream TestEventStream where c.Field1 where c.Field1 >> 1
1 select select c; c; 12 StreamInsight Demo 13 StreamInsight More Information StreamInsight - http:// www.microsoft.com/sqlserver/2008/en/us/R2-complex-eve nt.aspx Team Blog - http://blogs.msdn.com/streaminsight/ Forum - http:// social.msdn.microsoft.com/Forums/en-US/streaminsight/th reads 14 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Penyebaran Islam. Abu Bakar mengirim kekuatan ke luar Arabia, yaitu wilayah-wilayah kekuasaan Bizantium. Misalnya, Khalid bin Walid dikirim ke Iraq dan dapat menguasai al-Hirah pada tahun 634 M. Sementara itu, ke Syria dikirim ekspedisi di bawah pimpinan empat jenderal, yaitu...
REA margin assessment. Chapter 3. Other activities. Chapter 4. CONTENTS. Sept 11, 2016. Tractebel Activities with FRAPCON-FRAPTRAN. ... good estimation of the test section inlet coolant boundary conditions and validation of the FRAPTRAN heat transfer model.
Improve achievement and graduation rates for all students . Eliminate attendance and learning barriers . Improve attendance . ... reform by identifying what matters for student success and school improvement and creating the critical indicators to chart progress.
Process Less than Container Load * LCL Seafreight - sales presentation Booking by supplier Delivery at CFS Delivery to container terminal Vanning of container Customs handling Damage check Departure of vessel Transfer container to CFS Devanning of container Customs handling...
each task (e.g., Renzo Piano is a famous architect. Find out. the name of the building that is Piano's most famous work) paired with a link comprising one or more query words that. would launch a search when clicked (e.g.,...
Ready to download the document? Go ahead and hit continue!