Streaming over Internet 1st Generation Progressive 2nd Generation UDP based Real-Time Protocols (RTP, RTCP, RTSP) 3rd Generation TCP based Proprietary Adobe
Adaptive HTTP based Challenges TCP/UDP/IP , best-effort, with no guarantees on expectation or variance of packet delay Streaming applications delay of 5 to 10 seconds is typical and has been acceptable, but performance deteriorate if links are congested Real-Time Interactive requirements on delay and its jitter have been satisfied by overprovisioning (providing plenty of bandwidth), what will happen when the load increases?...
On-Demand Streaming Important and growing application due to reduction of storage costs, increase in high speed net access from homes, enhancements to caching Interactive control by user (but often with long response time) Ubiquitous on the web:
YouTube, Netflix, Vimeo Television networks, Hollywood, etc. Most local radio & TV stations Virtually everywhere on websites 4 First Generation: HTTP Download A simple architecture is to have the Browser request the object(s) and after their reception pass them to the player for display No pipelining
Helper Application i.e. media player Displays content, which is typically requested via a Web browser; typical functions: Decompression Jitter removal Error correction: use redundant packets to be used for reconstruction of original stream GUI for user control Examples: RealPlayer
Adobe Flash Player Windows Media Player QuickTime DivX Web Player First Gen: HTTP Progressive Download (2) Alternative: set up connection between server and player; player takes over Web browser requests and receives a Meta File (a file describing the object) instead of receiving the file itself; Browser launches the appropriate Player and passes it
the Meta File; Player sets up a TCP connection with Web Server and downloads or streams the file Meta file requests 8 HTTP Progressive Download With helper application doing the download, playback can start immediately... Or after sufficient bytes are buffered
Sender sends at maximum possible rate under TCP; retransmit when error is encountered; Player uses a much larger buffer to smooth delivery rate of TCP HTTP Progressive Download Buffering Continuous Media Jitter = variation from ideal timing Media delivery must have very low jitter Video frames every 30ms or so Audio: ultimately samples need < 128ms jitter
But network packets have much more jitter that that! Solution: buffers Fill them with best effort Drain them via low-latency, local access Buffer Duration "Good" Region: smooth playback
Buffer Size Ti m e ay b th
Pl = allowable jitter Sm oo "Bad": Buffer overrflows ac k
Max Buffer Duration Max Buffer Size File Position Streaming, Buffers and Timing "Bad": Buffer underflows and playback stops
Buffer almost empty Time The myth of SBR Any single bitrate you choose for a video is by definition the wrong video for 99% of users. Average Throughput for your Site Visitors 100 90 80 70
These users will rebuffer These have lower quality than they could sustain 60 50 40 30 20 10 0
100 300 500 700 900 1100 1300 1500 1700 1900 2100 2300 2500 2700 2900 3100 3300 3500 3700 3900 4100 4300 4500 4700 4900 5100 Drawbacks of HTTP Progressive Download (2) HTTP connection keeps data flowing as fast as possible to user's local buffer May download lots of extra data if you do not watch the video TCP file transfer can use more bandwidth than necessary Mismatch between whole file transfer and stop/start/ seek playback controls.
However: use file range requests to seek to video position Cannot change video quality (bit rate) to adapt to network congestion 2nd Generation: Real-Time Streaming This gets us around HTTP, allows a choice of UDP vs. TCP and the application layer protocol can be better tailored to Streaming; many enhancements options are possible Legacy Video Streaming
Approaches RTSP and RTP Example: Real Time Streaming Protocol (RTSP) For user to control display: rewind, fast forward, pause, resume, etc Out-of-band protocol (uses two connections, one for control messages (Port 554) and one for media stream) RFC 2326 permits use of either TCP or UDP for the control messages connection, sometimes called the
RTSP Channel As before, meta file is communicated to web browser which then launches the Player; Player sets up an RTSP connection for control messages in addition to the connection for the streaming media RTSP Operation 18 RTSP Media Stream Stateful Server keeps track of client's state
Client issues Play, Pause, ..., Close Steady stream of packets UDP - lower latency TCP - may get through more firewalls, reliable Credit: some content adapted from Alex Zambelli 19 RTSP Exchange Example C: SETUP rtsp://audio.example.com/xena/audio RTSP/1.0 Transport: rtp/udp; compression; port=3056; mode=PLAY
S: RTSP/1.0 200 1 OK Session 4231 C: PLAY rtsp://audio.example.com/xena/audio.en/lofi RTSP/1.0 Session: 4231 Range: npt=0 (npt = normal play time) C: PAUSE rtsp://audio.example.com/xena/audio.en/lofi RTSP/1.0 Session: 4231 Range: npt=37 C: TEARDOWN rtsp://audio.example.com/xena/audio.en/lofi RTSP/1.0 Session: 4231 S: 200 3 OK
20 Example 2: RTMP - Real-Time Messaging Protocol Proprietary Adobe protocol Runs over TCP Manages audio, video, and other
Multiplex multiple streams over TCP connection 21 Flash Streaming Server Flash Streaming Server communicates with its clients using the Adobe patented RTMP over TCP, which manages a twoway connection, allowing the server to send and receive,video, audio and data between client and server Typically used with FLV encoded files
Variations of RTMP RTMP- Standard,unencrypted RTMP. The default port is 1935. RTMPT- RTMP tunneled over HTTP The default port is 80 RTMPS- RTMP sent over an SSL. SSL enables secure TCP/IP connections. The default port is 443
RTMPE- Enhanced and encrypted version of RTMP. RTMPTE- Encrypts the communication channel, tunneling over HTTP Drawbacks of RTSP, RTMP Web downloads are typically cheaper than streaming services offered by CDNs and hosting providers Streaming often blocked by routers UDP itself often blocked by firewalls HTTP delivery can use ordinary proxies and
caches Conclusion: rather than adapt Internet to streaming, adapt media delivery to the Internet 24 Bandwidth is never constant Even preselecting bitrates will lead to rebuffering problems as bandwidth fluctuates over the duration of the vide, especially for long form content Ideally we would like to switch bitrates constantly to always give the user the
highest quality they can sustain at any point in time. Buffering Bandwidth is never constant Even preselecting bitrates will lead to rebuffering problems as bandwidth fluctuates over the duration of the vide, especially for long form content Ideally we would like to switch bitrates constantly to always give the user the
highest quality they can sustain at any point in time. Adaptive delivery 1500kbps 1000kbps Buffering 500kbps
3rd Generation: HTTP Streaming Other terms for similar concepts: Adaptive Streaming, Smooth Streaming, HTTP Chunking Client-centric architecture with stateful client and stateless server Standard server: Web servers Standard Protocol: HTTP Session state and logic maintained at server Video is broken into multiple chunks
Chunks begin with keyframe so independent of other chunks A series of HTTP progressive downloads of chunks Playing chunks in sequence gives seamless video Adaptive Bit Rate with HTTP Streaming Encode video at different levels of quality/bandwidth Client can adapt by requesting different sized chunks Chunks of different bit rates must be synchronized
All encodings have the same chunk boundaries and all chunks start with keyframes, so you can make smooth splices to chunks of higher or lower bit rates How does segmentation work? 1 3 Incoming video
500 kbps 1000 kbps 2 is split by an encoder 2000 kbps into multiple short blocks. Each block holds the same section of video, encoded at a
different size and bitrate. How does adaptive delivery work? The segmented video is stored on a server, along with a text file which describes the names of each segment. This text file is called a manifest. A player downloads the manifest
and then begins requesting individual segments of video. It makes its choice based on bandwidth conditions, grabbing the best quality it can at the time. SERVER CLOUD PLAYER
gives developers incredible control over playback, buffering and adaption logic of in-page video. Pareto principle in video popularity Also known as the 80-20 (or 90-10) rule [http:// en.wikipedia.org/wiki/Pareto_principle ] Associated with highly skewed distributions Caused by positive-feedback effects (the rich get richer) Does non-UGC content (say movies from NetFlix) follow a similar pattern?
A practical implication of this skewness: caching applies! Caching works Lets build CDNs Take advantage of locality and relativity A practical implication of this skewness: caching works! Akamai You tube Netflix based on Amazon infrastracture
Adaptive HTTP Streaming System (Protocol) Server Can be standard web server Media segment can be prepared in-line or offline Client Sends series of HTTP GET segment requests and receives segments
Performs rate adaptation before sending a new GET segment request Advantages of HTTP Streaming Easy to deploy: it's just HTTP, work with existing caches/proxies/CDN/Firewall Fast startup by downloading lowest quality/smallest chunk Bitrate switching is seamless Many small files
Small with respect to the movie size Large with respect to TCP 5-10 seconds of 1Mbps 3Mbps 0.5MB 4MB per chunk 35 Which adaptive streaming formats exist today? HTTP Live Streaming (HLS) - Controlled by Apple - Known as HLS, it is supported well by iOS, Safari and half-heartedly by Android.
HTTP Dynamic Streaming (HDS) - Controlled by Adobe - Known as HDS, it is played back only by Flash clients with custom apps. Smooth Streaming (SMOOTH) - Controlled by Microsoft - Played back via Silverlight clients, Xbox, MPEG-DASH (DASH) - An ISO standard - Playback via MSE in browsers, native apps on iOS, Android and WInOS.
Simple HLS example Example of HLS Meta Data Other Video Streaming Approaches RTMP Adaptive video streaming over HTTP Several players support it Several commercial content providers use it
Microsoft Smooth Streaming Player Netflix Player Adobe OSMF/Zeri Player Apple Player HTTP Adaptive Video Streaming HTTP Adaptive Video Streaming: Manifest File and Fragments Smooth Streaming Player
Smooth Streaming Player Sample HTTP Request: GET /mediadl/iisnet/smoothmedia/Experience/BigBuckBun ny720p.ism/QualityLevels(2040000)/ Fragments(video=400000000) HTTP/1.1 Adobe OSMF/Zeri Player Adobe OSMF Player Sample HTTP Request: GET /content/inoutedit-mbr/inoutedit\_h264\
_3000Seg1-Frag5 HTTP/1.1 What Makes Netflix Interesting? Commercial, feature-length movies and TV shows and not free; subscription-based Nonetheless, Netflix is huge! 25 million subscribers and ~20,000 titles (and growing) consumes 30% of peak-time downstream bandwidth in North America
A prime example of cloud-sourced architecture Maintains only a small in-house facility for key functions Majority of functions are sourced to Amazon cloud (EC2/S3)
e.g., subscriber management (account creation, payment, ) user authentication, video search, video storage, DNS service is sourced to UltraDNS Leverage multiple CDNs (content-distribution networks) for video delivery Akamai, Level 3 and Limelight
Netflix Architecture Netflix has its own data center for certain crucial operations (e.g., user registration, billing, ) Most web-based user-video interaction, computation/storage operations are cloud-sourced to Amazon AWS Video delivery is out/cloud-sourced to 3 CDNs Users need to use MS Silverlight player for video streaming Netflix Videos and Video Chunks Netflix uses a numeric ID to identify each movie IDs are variable length (6-8 digits): 213530, 1001192,
70221086 video IDs do not seem to be evenly distributed in the ID space these video IDs are not used in playback operations Each movie is encoded in multiple quality levels, each is identified by a numeric ID (9 digits) various numeric IDs associated with the same movie appear to have no obvious relations 49
Netflix Player Netflix Player Sample HTTP Request: GET /sa2/946/1876632946.wmv/range/22120592252058?token=1283923056\ _d6f6112068075f1fb60cc48eab59ea55\ &random=1799513140 HTTP/1.1 Netflix Videos and Video Chunks Videos are divided in chunks (of roughly 4 secs), specified using (byte) range/xxx-xxx? in the URL path:
Limelight: http://netflix-094.vo.llnwd.net/s/stor3/384/534975384.ismv/range/0-57689? p=58&e=1311456547&h=2caca6fb4cc2c522e657006cf69d4ace Akamai: http://netflix094.as.nflximg.com.edgesuite.net/sa53/384/534975384.ismv/ range/0-57689?token=1311456547_411862e41a33dc93ee71e2e3b3fd8534 Level3: http://nflx.i.ad483241.x.lcdn.nflximg.com/384/534975384.ismv/range/057689? etime=20110723212907&movieHash=094&encoded=06847414df0656e697c bd Netflix uses a version of (MPEG-)DASH for video streaming 52
Terms and Definitions of Adaptive HTTP Streaming Need Media Presentation Description (MDP) which provides metadata For requesting (GET request) media segments For rate adaptation purpose Segment which may include media data or metadata to decode
Use DASH an example in the few slides Example: DASH Client Thomas Stockhammer, Qualcomm, DASH Design Principles and Standards , Presentation at MMSys 2011 Meta Data DASH uses MPD (Media Presentation Descriptor) and Index Information as metadata for DASH Access Client Initialization and Media Segments for Media Engine Reuse of existing container format
Source: Stockhammer, Qualcomm, DASH Design Principles and Standards , Presentation at MMSys 2011 Media Presentation Data Model MDP - description of accessible segments and corresponding timing Source: Stockhammer, Qualcomm, DASH Design Principles and Standards , Presentation at MMSys 2011
A Social Story was created for each of the facial expressions that the children were not able to label, with illustrations that highlighted facial expressions. The Social Stories and quiz questions were created to match the specific emotions, narrative structure,...
Facultative Anaerobe. Other bacteria such as E. coli are facultative anaerobe i.e. they utilize oxygen if present to generate energy by respiration but they can use the fermentation path way to syntheses ATP in the absence of oxygen.
Succinctly describe and/or illustrate your proposed approaches to the three APT objectives (stimulus-response, resources, ecology) that you will pursue and the technical challenges that must be overcome. Include program phase structure, as necessary. Impact. What is the anticipated impact of...
How to write a "hamburger" paragraph Click on me to learn about good paragraphs! The Topic Sentence (Top Bun) Very first sentence of your paragraph. Always needs to be indented. States what your paragraph is going to be about. There...
Types of Clouds. There are five categories of clouds based on a cloud's physical structure and process of formation. The cloud types are: cirriform, cumuliform, cumulonimbiform, stratocumuliform, and . stratiform. Supplies to Make Cloud in a Bottle. 1 liter clear...