StreamInsight Complex Event Processing (CEP) Platform John Plummer

StreamInsight Complex Event Processing (CEP) Platform John Plummer

StreamInsight Complex Event Processing (CEP) Platform John Plummer Technical Specialist Data Platform Microsoft Ltd What is CEP? Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Database Applications Event-driven Applications Query Paradigm Ad-hoc queries or requests Continuous standing queries Latency Seconds, hours, days Milliseconds or less Data Rate Hundreds of events/sec Tens of thousands of events/sec or more request response

3 Event input stream output stream 3 Example CEP Scenarios Power, Utilities: Energy consumption Outages Smart grids 100,000 events/sec Financial Services: Stock & news feeds Algorithmic trading Patterns over time Super-low latency 100,000 events /sec Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds Stream Data Store & Archive 4 Web Analytics: Click-stream data

Online customer behavior Page layout 100,000 events /sec Visual Visual trend-line trend-line and and KPI KPI monitoring monitoring Batch & product management Batch & product management Automated Automated anomaly anomaly detection detection Real-time customer segmentation Real-time customer segmentation Algorithmic Algorithmic trading trading Proactive condition-based Proactive condition-based maintenance maintenance Asset Specs & Parameters

Data Data Stream Stream Data Data Stream Stream Manufacturing: Sensor on plant floor React through device controllers Aggregated data 10,000 events/sec Event Processing Engine Lookup Lookup Threshold Threshold queries queries Event Event correlation correlation from from multiple sources multiple sources Pattern Pattern queries queries 4

Sweet Spot for Complex Event Processing Months CEP Target Scenarios Days Relational Database Applications Hours Data Warehousing Applications Minutes Seconds 100 ms Monitoring Applications < 1 ms 0 10 Operational Analytics Applications (e.g., Logistics) Web Analytics Applications Manufacturing Applications 100 1000

10000 Aggregate Data Rate (Events/sec) 5 Financial Trading Applications 100000 higher Complex Event Processing SQL SERVER 2008 R2 STREAMINSIGHT SITUATION TODAY Pressure to handle large volumes Process large volume of events across of complex data in realintime multiple data streams less than a second The

size your and frequency of data historical make it Manage business through challenging store for data insights. mining and data mining to and continuous analysis Growing need tofor monitor, analyze and Built in support different types of event act on theand data in query motionsemantics handling rich "Intelligence "Intelligence isn't isn't just just about about knowing

knowing what what is is happening. happening. It's It's about about looking looking at at the the patterns patterns in in real real time. time. If If business business people people want want to to truly truly optimize their their resources, resources, they optimize they must must examine examine patterns patterns in in real real time. time. -- Mike Mike Gualtieri Gualtieri -- Forrester Forrester

6 StreamInsight Platform from Microsoft for continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with low latency EVENT SOURCES Event stores & Databases Bloomberg.co m Stock tickers & News feeds 7 MSFT MSFT CEP CEP Engine Engine Output Output Adapters Adapters Web servers Input Input Adapters Adapters Devices, Sensors

APPLICATION EVENT TARGETS Pagers & Monitoring devices KPI Dashboards, SharePoint UI Trading stations Reference data Event stores & Databases Events Events expose different temporal characteristics Point in time events Interval events with fixed duration Interval events with initially unknown duration (Edge) Payload/ value Rich payloads capture all properties of an event a t1 8 b t2

c t3 Time d t4 e t5 Event Types Events in Microsofts CEP platform use the .NET type system Events are structured and can have multiple fields Fields are typed using the .NET framework types CEP engine provisioned timestamp fields capture all the different temporal event characteristics Event sources populate time stamp fields Timestamps/ Metadata 9 Long pumpID String Type

String Location Double flow Double pressure Event Streams & Adapters A stream is a possibly infinite sequence of events Insertions of new events Changes to event durations Examples: time series sensor data, log entries Stream characteristics: Event/data arrival patterns Steady rate with end-of-stream indication Examples: files, tables Intermittent, random, or in bursts Example: retail scanners, web, weather telemetry Order of arrival of events does not match the order of their application timestamps Input & Output Design Time for dealing with Event Streams 10 10 Core CEP Query Engine

Queries are composable Query results are computed incrementally Event CEP Engine Standing Queries Event Event Event Event Event Event Event Event Output OutputAdapters Adapters Operators consume and produce streams Input InputAdapters Adapters Hosts standing queries

Query instance management: Submit, start, stop Runtime statistics Takeaway: CEP engine does the heavy lifting for you when processing temporal event data 11 11 Developer Experience with the CEP Platform CEP development environment Builds on top of .NET Microsoft Visual Studio as IDE Applications written in C# Queries written in LINQ Supports various deployment models for CEP applications Embedded applications Server applications 12 LINQ: var var queryFilter queryFilter == from from cc in in TestEventStream TestEventStream where c.Field1 where c.Field1 >> 1

1 select select c; c; 12 StreamInsight Demo 13 StreamInsight More Information StreamInsight - http:// www.microsoft.com/sqlserver/2008/en/us/R2-complex-eve nt.aspx Team Blog - http://blogs.msdn.com/streaminsight/ Forum - http:// social.msdn.microsoft.com/Forums/en-US/streaminsight/th reads 14 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Recently Viewed Presentations

  • Islam Masa Khulafaur Rashidin

    Islam Masa Khulafaur Rashidin

    Penyebaran Islam. Abu Bakar mengirim kekuatan ke luar Arabia, yaitu wilayah-wilayah kekuasaan Bizantium. Misalnya, Khalid bin Walid dikirim ke Iraq dan dapat menguasai al-Hirah pada tahun 634 M. Sementara itu, ke Syria dikirim ekspedisi di bawah pimpinan empat jenderal, yaitu...
  • RIA margin assessment using Tractebel multiphysics code package

    RIA margin assessment using Tractebel multiphysics code package

    REA margin assessment. Chapter 3. Other activities. Chapter 4. CONTENTS. Sept 11, 2016. Tractebel Activities with FRAPCON-FRAPTRAN. ... good estimation of the test section inlet coolant boundary conditions and validation of the FRAPTRAN heat transfer model.
  • How to Register - Arbiter

    How to Register - Arbiter

    Step Four: Submit Test or Save and Exit. Once you complete your exam click Submit Test. If you do not finish it, click save and exit and you can come back and pick up where you left off.
  • Vaka sunumu - Kocaeli Üniversitesi

    Vaka sunumu - Kocaeli Üniversitesi

    Kocaeli Üniversitesi Tıp Fakültesi Çocuk Sağlığı ve Hastalıkları Anabilim Dalı Alerji-İmmünoloji Bilim Dalı Olgu Sunumu 8 Eylül 2011
  • Diseases of the Joint - University of Ottawa

    Diseases of the Joint - University of Ottawa

    OTHER Tendonitis, Bursitis, Ligament Tear, Meniscus. PEDIATRIC SCFE, Legg-Calve-Perthes, Osgood . Schlatter, Henoch- Schönleinpurpura, Juvenille. Idiopathic Arthritis *POLYARTHRITIS IS RED. Hemarthrosis-Bleeding into joint spaces-Coagulopathy (secondary hemostasis disorders like hemophillia, Vit K def, liver disease, DIC)-Trauma
  • www.AttendanceInstitute.org Welcome Absenteeism and GPA: Exploring the top

    www.AttendanceInstitute.org Welcome Absenteeism and GPA: Exploring the top

    Improve achievement and graduation rates for all students . Eliminate attendance and learning barriers . Improve attendance . ... reform by identifying what matters for student success and school improvement and creating the critical indicators to chart progress.
  • PowerPoint-Präsentation - Kuehne

    PowerPoint-Präsentation - Kuehne

    Process Less than Container Load * LCL Seafreight - sales presentation Booking by supplier Delivery at CFS Delivery to container terminal Vanning of container Customs handling Damage check Departure of vessel Transfer container to CFS Devanning of container Customs handling...
  • PowerPoint 演示文稿 - Illinois

    PowerPoint 演示文稿 - Illinois

    each task (e.g., Renzo Piano is a famous architect. Find out. the name of the building that is Piano's most famous work) paired with a link comprising one or more query words that. would launch a search when clicked (e.g.,...