Video Streaming, Under the Hood

From the camera sensor to the viewer's screen: codecs, containers, transport protocols, adaptive bitrate and the trade-offs between quality, latency and cost.

The Streaming Pipeline

Every streaming service, from a Twitch broadcast to a Netflix movie, follows roughly the same chain of stages.

Capturecamera, screen
→
EncodeH.264 / HEVC / AV1
→
IngestRTMP / SRT / WHIP
→
TranscodeABR ladder
→
PackageHLS / DASH / CMAF
→
CDNedge caching
→
Playerdecode & render

Encoding

Raw 1080p60 video is roughly 3 Gbit/s. Encoders compress it 300–1000× using intra-frame (spatial) and inter-frame (motion-compensated) prediction.

Packaging

Encoded streams are split into short segments (typically 2–6 s) and described by a manifest (.m3u8 or .mpd) that the player reads.

Delivery

Segments are plain HTTP objects, so CDNs can cache them at the edge and serve millions of viewers without hitting the origin.

Video Codecs

CodecYearEfficiency vs H.264Notes
H.264 / AVC2003baselineUniversal hardware support; still the safe default.
H.265 / HEVC2013~40–50% smallerCommon for 4K/HDR; complex patent licensing.
VP92013~35–45% smallerRoyalty-free, widely used by YouTube.
AV12018~50% smallerRoyalty-free (AOMedia); hardware decode spreading fast.
H.266 / VVC2020~50% smaller than HEVCNext generation; limited adoption so far.

Frames come in three types: I-frames (self-contained keyframes), P-frames (predicted from previous frames) and B-frames (predicted from both directions). The distance between keyframes is the GOP size; segment boundaries must start on a keyframe.

Transport Protocols

RTMP

Legacy Flash-era protocol over TCP. Still the most common way to push a live stream from OBS to a platform.

SRT

Secure Reliable Transport over UDP with retransmission and encryption. Built for contribution over unreliable networks.

HLS

Apple's HTTP Live Streaming. Segments plus .m3u8 playlists; plays nearly everywhere. LL-HLS cuts latency to ~2–3 s.

MPEG-DASH

Open ISO standard, codec-agnostic, with an XML manifest (.mpd). Popular on Android and smart TVs.

WebRTC

Peer-to-peer real-time media over UDP with sub-second latency. Used for calls, cloud gaming and interactive live; WHIP/WHEP standardize ingest and playback.

CMAF

Common Media Application Format: one set of fragmented MP4 segments that serves both HLS and DASH, halving storage.

Example HLS master playlist

#EXTM3U
#EXT-X-STREAM-INF:BANDWIDTH=800000,RESOLUTION=640x360,CODECS="avc1.4d401e,mp4a.40.2"
360p/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=2800000,RESOLUTION=1280x720,CODECS="avc1.4d401f,mp4a.40.2"
720p/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=5000000,RESOLUTION=1920x1080,CODECS="avc1.640028,mp4a.40.2"
1080p/index.m3u8

Adaptive Bitrate (ABR)

The source is transcoded into a bitrate ladder: several renditions at different resolutions and bitrates. The player measures throughput and buffer level and switches renditions at segment boundaries, so playback continues smoothly as network conditions change.

RenditionResolutionTypical H.264 bitrate
Low640×3600.8 Mbit/s
SD854×4801.4 Mbit/s
HD1280×7202.8 Mbit/s
Full HD1920×10805.0 Mbit/s
4K3840×216015–20 Mbit/s

Typical FFmpeg command for one rendition of an HLS ladder:

ffmpeg -i input.mp4 -c:v libx264 -preset veryfast -b:v 2800k -maxrate 3000k \
  -bufsize 5600k -vf scale=1280:720 -g 48 -keyint_min 48 -sc_threshold 0 \
  -c:a aac -b:a 128k -f hls -hls_time 4 -hls_playlist_type vod 720p/index.m3u8

Latency Tiers

TierGlass-to-glassTypical techUse case
Standard15–45 sHLS / DASH, 6 s segmentsVOD, linear TV
Low2–5 sLL-HLS, LL-DASH, chunked CMAFSports, live events
Ultra-low< 1 sWebRTCVideo calls, auctions, cloud gaming

Latency comes from encoder look-ahead, segment duration, player buffer and network round trips. Shorter segments lower latency but raise request overhead and reduce CDN cache efficiency.

Bitrate & Storage Calculator