HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
Real-Time Transport Protocol (RTP)
? 2005 J?rg Ott
1
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
Real-time Transport Protocol (1)
` RTP Functionality (RFC 3550)
y framing for audio/video information streams y preserve intra- and inter-stream timing y mechanisms for awareness of others in a conference ? RTP sessions
Media streams (RTP)
Control Flows (RTCP)
? 2005 J?rg Ott
2
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
Real-time Transport Protocol (2)
` Standard RTP packet header
y Independent of payload type y Possibly seconded by payload header
` Mechanisms
y Detect packet loss, cope with reordering
? sequence number per media stream
y Determine variations in transmission delays
? media specific time stamp (e.g., 8 kHz for PCM audio) ? allows receiver to adapt playout point for continuous replay
y Source identification
? possibly mixed from several sources
y Payload type identifier
? 2005 J?rg Ott 3
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
RTP Header
0 4 8 16 31 V P X CC M Payload Sequence number 12 Bytes
Time stamp SSRC Identifier Contributing Sources (CSRC)
Max. 16 entries, 32 bits each
Extension Header Payload Padding ‘0’ # Bytes
64k - header
? 2005 J?rg Ott
4
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
RTP Header Fields (1)
V: Version P: Padding X: eXtension bit Extension header — — — — version 2 defined in RFC 1889 indicates padding # bytes indicated in last byte extension header is present single additional header (TLV coded)
CC: CSRC count — # of contributing sources CSRC: contributing sources — which sources have been “mixed” to produce this packet’s contents
? 2005 J?rg Ott
5
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
RTP Header Fields (2)
M: Marker bit — marks semantical boundaries in media stream (e.g. talk spurt) indicates packet content type of the packet in the media stream (strictly monotonically increasing) indicates the instant when the packet contents was sampled (measured to media-specific clock)
Payload type Sequence #
— —
Timestamp
—
SSRC: synchronization source — identification of packet originator
? 2005 J?rg Ott 6
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
Real-time Transport Control Protocol
Mechanisms: ` Receivers constantly measure transmission quality
y delay, jitter, packet loss
` Regular control information exchange between senders and receivers
y feedback to sender (receiver report) y feed forward to recipients (sender report)
` Allows applications to adapt to current QoS ` Overhead limited to a small fraction (default: 5% max.) of total bandwidth per RTP session
y members estimate number of participants y adapt their own transmission rate
Obtaining sufficient capacity: outside of RTP
? 2005 J?rg Ott 7
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
RTCP Sender Report
` Enable cross-media stream synchronization
y Relate stream-specific RTP time stamp to wall clock time y NTP timestamp + RTP timestamp y Playout adjustment to be performed by the receivers
` Provide data point for RTT measurement
y NTP timestamp
` Provide feed forward about data transmitted
y Transmit sender’s packet and byte count y Enable receiver to do proper loss calculation
` Include Receiver Reports for the sender as well
? 2005 J?rg Ott 8
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
0 1 2 3
RTCP Sender Report (SR) 7 8 15 16
RC RT = SR = 200 Sender SSRC 64 Bit NTP Timestamp (MSW) 64 Bit NTP Timestamp (LSW) RTP Timestamp Cumulative number of packets sent Cumulative number of octets sent Receiver Report blocks (0 – 31) Length
31
V P
Sender Info
? 2005 J?rg Ott
9
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
RTCP Receiver Report
` Feedback timing for RTT estimation
y SR Timestamp
? Middle 32 bits taken from the last SR’s NTP timestamp
y Delay since last SR
? Local delay at receiver between receiver SR and sending the RR block ? Measured in units of 1 / 65556 seconds
` Provide per-sender reception statistics
y y y y Total number of packets lost Fraction of packets lost (in units of 1 / 256) Highest sequence number received so far Jitter of received packets
` Enable adaptive sender behavior
y Adjust codecs, codec parameters, transmission rate, etc.
? 2005 J?rg Ott 10
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
RTCP Receiver Report (RR)
0 1 2 3 7 8 15 16 31
V P
RC
RT = RR = 201 Sender SSRC
Length
SSRC of sender reported in this block Fraction lost Cumulative number of packets lost
Receiver Report block
Extended highest sequence number received Interarrival jitter Last SR Delay since last SR Further receiver report blocks (up to 31 in total)
? 2005 J?rg Ott
11
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
RTCP Source Description (SDES)
` Persistent Identification of an endpoint: Canonical Name
y CNAME — y Mandatory! globally unique identifier (id@host)
y Binding across RTP sessions y Identification across changes in the SSRC in an RTP session
` Providing additional information about an endpoint
y y y y y y y NAME EMAIL PHONE LOC TOOL NOTE PRIV — — — — — — — Name of user (or system) mailto: address phone number location (no format defined) (software) client in use brief to other participants (e.g. “on the phone”) private extensions
12
? 2005 J?rg Ott
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
0 1 2 3
RTCP 7Source Description (SDES) 8 15 16 31
SC RT = RR = 202 Sender SSRC of chunk #1
SDES chunk
V P
Length
SDES items chunk #1
Further SDES chunks (up to 31)
Item Type
Item Length SDES Item Data
SDES chunk
? 2005 J?rg Ott
13
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
RTCP Statistics Collection (Sender)
` Round-Trip Time (sender only)
y Derived from time stamps in RR y Simple formula: RTT = t1 – t0 – DSL_SR y RTT may be asymmetric!
t1 t0 Delay since Last SR Last SR
` Byte count ` Packet count
? 2005 J?rg Ott
14
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
RTCP Statistics Collection (Receiver)
` Packet Loss
y Calculated from gaps in sequence number space
? First (lowest packet sequence number) received
y Expected number of packets = current – lowest y Received number of packets
? Count duplicates, out-of-order, and late packets as received!
y Absolute # of lost packets = expected – received
? May be negative!
y Fraction of lost packet
? Loss since last SR or RR packet was sent
y Loss of all packets not detected!
` Extended highest sequence number received (32 bits) ` Time of last SR reception ` Jitter
? 2005 J?rg Ott 15
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
RTCP Interarrival Jitter Estimation
` Receiver measures in time units of the
media clock
` Relates it to local real-time clock ` Initialized through first packet received ` Derives expected reception time ` Calculates deviation D upon packet
Sender Packet Timestamp TS=240 TS=280 TS=320 TS=360 TS=400 TS=440 TS=480
Receiver Reference Clock Reference Time for Jitter meas.
reception ` Sampled for each packet ` Jitter derived for each peer of successively received packets
y Ordering is not relevant
` Weighing function:
J = J’ + (D – J’) / 16
? 2005 J?rg Ott
16
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
Other RTCP Packets
` BYE
y Announce that an entity will be leaving a session y Provide a reason phrase
` APP
y Application-specific extensions
` Further extensions being defined
y Timely feedback extensions y Single source multicast extensions y More detailed reception statistics
? 2005 J?rg Ott
17
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
RTP / RTCP Bandwidth Calculation
` Senders and receivers estimate group size (independently!)
y # senders from SRs y # receivers from RRs y Consider BYE packets
` RTCP bandwidth: 5% of RTP session bandwidth
y 5% for both senders and receivers y 1.25% for senders and 3.75% for receivers if >25% senders otherwise
` Sample of RTCP packets sent and received
y Average number of bytes per time unit currently transmitted y Consider whether you are sender or receiver
` Last time own packet was sent
y y y y Calculate expected next time to send based upon above data rate Use a minimum of 5 seconds (initially, 2.5 seconds) Dither with random function: 0.5 .. 1.5 of calculated interval Timer reconsideration: double check values when timer expires
18
? 2005 J?rg Ott
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
RTCP Transmission Interval
` Must scale with the number of group members
y Must not take up too much network capacity (rate-limited!)
` Overall “RTP session bandwidth”
y Includes UDP and IP header overhead y Provided by the application (i.e. not measured dynamically)
` Default: 5% of the session bandwidth for RTCP
y Takes role (sender or receiver) into account y Up to 25% of session members are senders
? 3.75% for receivers, 1.25% for senders
y More than 25% of session members are senders
? Share data rate proportionally
` May be modified by profiles
y Parameters S and R to indicate relative share for senders/receivers
` Scalable RTCP transmission interval
y Based upon the group size, RTCP data rate, average RTCP packet size
? 2005 J?rg Ott 19
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
RTCP Variables for Bandwidth Calculation
`
Data rate
y Session bandwidth y R, S: Receiver, sender bandwidth share y Average RTCP packet size (moving average)
`
Time
y Tp y tc y tn last time an RTCP packet was sent current time next scheduled transmission of an RTCP packet # members when tn was last computed current # members # senders in the session relevant # of members (depending on role, etc.) Deterministic calculated interval Calculated interval minimal interval between RTCP packets
20
`
Membership
y y y y pmembers members senders n
`
Intervals
y Td y T y Tmin
? 2005 J?rg Ott
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
Basic Operation
` Determine role (sender or receiver)
y Derive n as # of relevant members for calculation y Derive relevant bandwidth share
` C = average RTCP size / relevant bandwidth share ` Td = max (Tmin, n*C) ` T = Random [0.5 – 1.5] * Td
? 2005 J?rg Ott
21
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
Basic RTCP Interval Calculation
Calculated interval T
Deterministic interval Td
tp
tn
Range for T Time of calculation
? 2005 J?rg Ott
Time of transmission
22
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
Timer Reconsideration
` The group size may change between tp and tn ` Particularly during startup and shutdown phase
y Many users may join / leave during a short period of time
` Many joining parties: risk of RTCP implosion ` Algorithm for joining members
y y y y Validate the group size at time tn before transmission Recalculate T as above If tp + T <= tc transmit RTCP packet and update variables If tp + T > tc set tn = tp + T and set timer to expire at tn
` Algorithm for leaving members
y Adjust tp, tn according to the observed membership change
? Factor: members / pmembers
y Run every time a member leaves or times out
? 2005 J?rg Ott 23
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
Extended Operation
` Determine role (sender or receiver)
y Derive n as # of relevant members for calculation y Derive relevant bandwidth share
` C = average RTCP size / relevant bandwidth share ` Td = max (Tmin, n*C) ` T = Random [0.5 – 1.5] * Td ` T = T / e^-1.5
(T = T / 1.21828)
y Correction factor for timer reconsideration
? 2005 J?rg Ott
24
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
RTP Payloads
? 2005 J?rg Ott
25
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
RTP Payload Types
` 7-bit payload type identifier
y Some numbers statically assigned y Dynamic payload types identifiers for extensions – mapping to be defined outside of RTP (control protocol, e.g. SDP “a=rtpmap:”)
Payload formats defined for many audio/video encodings
` Conferencing profile document RFC 3551
y Audio: G.711, G.722, G.723.1, G.728, GSM, CD, DVI, …
` In codec-specific RFCs
y Audio: Redundant Audio, MP-3, ... y Video: JPEG, H.261, MPEG-1, MPEG-2, H.263, H.263+, BT.656 y Others: DTMF, text, SONET, ...
` Generic formats
y Generic FEC, (multiplexing)
? 2005 J?rg Ott 26
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
Media Packetization Schemes (1)
General principle: ` Payload specific additional header (if needed) ` Followed by media data
y Packetized and formatted in a well-defined way y Trivial ones specified in RFC 3551 y RFC 2029, 2032, 2035, 2038, 2190, 2198, 2250, 2343, 2429, 2431, RFC 2435, 2658, 2733, 2793, 2833, 2862, and many further ones y Guidelines for writing packet formats: RFC 2736
` Functionality
y Enable transmission across a packet network y Allow for semantics-based fragmentation y Provide additional information to simplify processing and decoding at the recipient y Maximize possibility of independent decoding of individual packets
? 2005 J?rg Ott 27
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
Audio over RTP: PCM
0 1 2 3 7 8 15 16 31
V PX
CC
M
PT = 0
Sequence Number
Timestamp (8 KHz clock) Sender SSRC
Audio Data
? 2005 J?rg Ott
28
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
Video over RTP: H.261
Additional payload-specific header preceeds payload ` To avoid expensive bit shifting operations
y Indicate # invalid bits in first (SBit) and last (EBit) octet of payload
` Indicate Intra encoding (I bit) ` Indicate the presence of motion vector data (V bit) ` Carry further H.261 header information to enable decoding in the presence of packet losses
Further mechanisms for video conferencing ` FIR: Full Intra Request
y Ask sender to send a full intra encoded picture
` NACK: Negative Acknowledgement
y Indicate specific packet loss to sender
? 2005 J?rg Ott 29
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
Video over RTP: H.261 (2)
0 1 2 3 7 8 15 16 31
V PX
CC
M
PT = 0
Sequence Number
Timestamp (8 KHz clock) Sender SSRC SBit EBit I V GOBN …
Video Data
MBAP
QUANT
HMVD
VMVD
…
? 2005 J?rg Ott
30
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
Media Packetization Schemes (2)
Error-resilience for real-time media
` Input: Observation on packet loss characteristics ` Generic mechanisms (RFC 2354)
y Retransmissions
? in special cases only (e.g. with no interactivity!)
y Interleaving y Forward Error Correction (FEC)
? media-dependent vs. media-independent ? Generic FEC: RFC 2733
` Feedback loops for senders
y based upon generic and specific RTCP messages y adapt transmission rate, coding scheme, error control, ...
? 2005 J?rg Ott 31
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
RTP Interleaving
` Distribute packets or packet contents for transmission
y Avoid consecutive packet erasures in case of (burst) losses y Avoid loss of large consecutive data portions in case of single packet losses
` Motivations
y Human perception tolerates individual losses better y Make simple FEC schemes work better with burst losses (e.g. XOR)
` Drawback
y Re-ordering causes additional delay at the receiver
1 2 3 4 5 6 7 8 9
1
4
7
2
5
8
3
6
9
? 2005 J?rg Ott
32
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
RTP FEC (RFC 2733)
` Forward Error Correction scheme for RTP packets
y Media-independent, flexible FEC (that can be enhanced)
` Simple XOR-based (parity) FEC
y P_fec = P1 XOR P2 XOR P3 XOR … XOR Pn
? Allows reconstruction of any single missing packets of P1, …, Pn, P_fec
` RTP FEC stream transmitted independently of RTP stream
y Separate transport address (IP address, port number) y Different SSRC
RTP stream FEC stream #1 #2 + F(1,2) #3 #4 + F(3,4)
` Recovery
? 2005 J?rg Ott
#1
#2
+ F(1,2)
#2
33
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
RTP Parity FEC Packet format
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |V=2|P| FMT | PT | length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SSRC of packet sender | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SSRC of media source | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SN base | length recovery | =1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |E| PT recovery | mask = 10110000 00000000 00000000 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | TS recovery | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ : : : XOR of Payloads indicated by SN Base and mask : : : +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
RTP stream FEC stream
#1
#2
#3
#4 F(1,3,4)
? 2005 J?rg Ott
34
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
Audio Redundancy Coding (1)
` Audio Packets are small!
y have to be because of interactivity
? avoid large packetization delay
y packet loss primarily depends on packet rate
? rather than packet size
` Payloads for multiple time slots in one packet
y send redundant information in packet n to reconstruct packets k, ..., n-1 in packet n y redundant information typically sent at lower quality y details defined in RFC 2198 y uses dynamic payload type
` Format specification, e.g. using SDP
y m=audio 20002 RTP/AVP 96 0 0 0 y a =rtpmap:96 red/8000/1
? 2005 J?rg Ott 35
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
Audio Redundancy Coding (2)
t 1 2 3 4 Primary Encoding: Packet #1 … PCM
Secondary Encoding: GSM
Packet #2
Header
PCM
Header
GSM PCM
Header
Packet #3
GSM PCM GSM
36
Packet #4
? 2005 J?rg Ott
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
0
Audio Redundancy Coding (3) 8 16 24 31
PT=96 Sequence # Timestamp = 6400 SSRC
V P X 0000M
Redundant 2 1 Redundant 1 1 Primary 0
PT=0 PT=0 PT=0
Timestamp offset = 320 Timestamp offset = 160
Len = 160 Len = 160 Header
Data Block “Redundant 2” (160 Bytes)
Data Block “Redundant 1” (160 Bytes)
Data Block “Primary” (160 Bytes)
? 2005 J?rg Ott 37
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
Video Redundancy Coding (1)
` Video redundancy coding
y For H.263+ video streams y Transmit several interleaved sequences of predicted frames (threads) instead of one
? improves error resilience against packet loss
` Principle
y create several (n) independently decodable streams y achieved by choosing different reference pictures y decode only streams with no packet losses
? reduces temporal resolution by 1/n-th per affected stream
y bit rate penalty due to larger deltas between frames y RFC 2429, revised version in progress
? 2005 J?rg Ott
38
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
Video Redundancy Coding (2)
1
2
3
4
5
6
U
1 2 3 4 5 6
U
1
? 2005 J?rg Ott
Intra Frame
2
3
4
5
6
39
HELSINKI UNIVERSITY OF TECHNOLOGY NETWORKING LABORATORY
Video Redundancy Coding (3)
1.1 REF 2.1 1 2 3 4 2.2 5 6 1.2 REF
1.1 REF
1.2 REF
U
2.1 3 1
? 2005 J?rg Ott
2.2 4 5 6
40
2