Part one, common to every codec

How video encoding works

Most video codecs are built out of the same handful of building blocks. This part covers those so that part two can focus on the H.264 specific coding tools.

The bit budget

A second of uncompressed 1080p60 video, 8-bit 4:2:0, is about 187 megabytes. A good quality stream of the same second is about 6 Mbit. That is 99.6% of the bits thrown away. Nobody notices. The rest of this page is how the encoder chooses what to discard.

Four kinds of redundancy

All compression comes down to finding redundancy and not paying for it twice. Every standardized codec chases the same four kinds.

• Spatial. Neighboring pixels look alike. Predict a block from pixels already decoded above and to the left and then code only the difference.
• Temporal. Frame x+1 is mostly frame x but displaced. We record the motion instead of the image.
• Statistical. After prediction, transform and quantization, most of what's left are zeros. Common values are assigned short codes, rare ones long codes.
• Perceptual. Vision is far more sensitive to brightness than color, and to gradients than to fine texture. Spend bits where they are seen.

The encode loop

Encoding works on blocks of the video frame. Every block goes around one loop.

Motion estimation

Frame-to-frame repetition gives us the biggest opportunity for savings. We will exploit it millions of times a second. Where did this block go? The encoder takes a block from the frame it is coding, slides it around the previous frame, and keeps the position where the pixels differ least. That difference measure is a SAD, the sum of absolute differences.

Dark violet on the cost map is a good match. The basin around it is why the shortcut works. The surface is smooth enough that a pattern which walks downhill usually finds the same minimum as testing everything. Usually. The percentage in the last box is what the shortcut costs you when it doesn’t. Choosing that trade is most of what a preset is.

Quantization

Prediction, transform and entropy coding are all reversible. Quantization is the one step that destroys information. Rate control below has the mechanics and the knob that sets them.

If the compression is pushed too hard the image will break up into a visible grid of squares. This is the same grid the transform uses. “Blocking” is one of the more common compression artifacts viewers notice and complain about. It is why every codec adds a deblocking filter inside the decoding loop.

The decision

Until now every stage had one obvious way to do it. From here on, encoding is a choice. A block can be skipped. It can be predicted with one motion vector. It can be split into four blocks with four vectors. It can be coded from scratch. The cheaper a choice is to describe, the worse it usually looks. So the encoder prices both sides together: cost = D + lambda x R, distortion plus lambda times rate. It takes the smallest.

Lambda is the price of a bit in that trade. When lambda is small, bits are cheap and the encoder spends them to cut error. When it’s large, bits are dear and the encoder accepts more error to save them. The quantizer sets lambda. That one number steers every decision in the encoder at once. Drag it to watch the trade move.

The constants

Every encoder carries constants that someone tuned by hand. The lambda law behind the decision above traces to the Lagrange-multiplier work of the late 1990s and early 2000s. Its authors swept lambda against QP on the test clips of the day and fit a curve through the pairs that won. Every H.264-era encoder inherited that fit, ours included. The constants around it were swept once on small clip batteries and frozen.

The usual assumption is that one setting suits all content. We tested that. We flipped six settings we already ship, one at a time, across the whole test set, and read the per-clip results. Each one turned out best for some clips and wrong for others.

The average across the test set hides this. A talking head and a handheld street scene want different settings. Picking the right one per clip would save about 1.3% on our twelve clips.

Nothing in the encoder picks per clip yet. One gate comes close: it spots flat, cartoon-like frames and raises the psychovisual strength for them. It fires on one clip in the test set and no other.

Finding a cheap measurement that predicts the right setting turned out to be hard. We labeled 239 clips from an outside set and tested four features. None of them predicts. Most clips probably have nothing to win anyway: on that set a tenth of the clips held most of the gain.

Rate control

Quantization is the step that throws detail away: each transform coefficient is divided by a step size and rounded. A bigger step means fewer bits and more loss.

Quantization can be changed per block. Rate control chooses that value thousands of times a second. It has to hit a bitrate target it can’t see far enough ahead to plan for. Viewers never see it working. They see a blurry face or a video that stalls.

An encoder can’t fix quality and bitrate at the same time. You pick one and let the other adjust. A talking head sitting still is cheap to compress. A detailed action scene is expensive. Lock in the quality and the bitrate jumps around with what is on screen. Lock in the bitrate and the quality takes the hit instead. Every mode below picks which one to let go.

Constant QP

QP is the number that sets the quantization step. In H.264 it runs from 0 to 51. Every six steps double the step size.

Set one QP and never change it. Quality stays even. The bitrate follows the content and runs up to five times higher in the action than in the calm. It’s rarely used for delivery. Encoder experiments, though, get measured with it.

CRF

Hold perceived quality roughly steady and let the bitrate go where it must. QP still moves with the content. It moves much less than the complexity does, because the eye cannot follow detail in fast motion.

That is why CRF is the right default whenever file size is not capped. Constant QP holds the quantizer steady instead of the look. It overspends on frames nobody is scrutinizing. It starves the still ones where errors show. Single-pass ABR chases a bitrate number through a feedback loop. The quality wobbles, worst at the scene changes a viewer is most likely to notice. CRF chases nothing. It pays what each scene actually costs. At the same bits that buys a steadier picture than single-pass ABR. Two-pass also gets to a steady picture and is what you want when you need a known output size. It costs you the second pass.

What you give up is any say over how big the file comes out. That is what the next mode is for.

Capped CRF

A buffer ceiling on top of a quality target is a common choice for VOD ladders. Quality leads. An easy title codes cheaply and comes out small. The cap only ever takes bits away. Below it you get exactly the CRF encode you asked for, bit for bit. Above it the frame gets bounded. It stays deliverable.

That gives you constant quality’s cheapness on the easy half of a catalog and a buffer constraint’s safety on the hard half. Single-pass ABR is now rare in on-demand work.

Average bitrate

ABR must hit a number over the whole file, so it runs a feedback controller. If it has overspent so far it tightens. If it has underspent it relaxes. The trouble with feedback alone is that it only learns about a hard section after paying for the first frames of it. It overshoots into the cut and over-corrects afterwards. The quality lurches either side of a scene change. That lurch is the rate-control failure viewers notice.

A lookahead fixes this. The encoder keeps a queue of the next few dozen frames and looks at them before coding. That tells it what is coming: where the cuts are, which stretches will be expensive, and which blocks later frames will copy from. So it can pick frame types, hold bits back for a spike, and spend extra where it pays off.

Two-pass fixes it the other way. A first pass encodes the whole file only to learn how hard each part is. The second pass then plans the budget with everything known. The total lands exactly and no scene surprises it. A lookahead sees a few dozen frames ahead, while two-pass sees the whole file. The price is a second encode and an input you can seek in. That rules out live.

VBV

A decoder reads from a buffer that fills at the channel rate and empties by one frame’s worth each time a frame is decoded. If a frame needs more bits than the buffer holds at that moment, playback stalls. VBV makes that constraint explicit. No frame may take more than the buffer holds when its turn comes. A big frame is allowed only if the frames before it left room, and the frames after it must stay small until the buffer refills. So the cap is on a run of frames over the buffer’s length, usually one to two seconds, and not on each frame alone. Through a hard section the quality dips instead of the stream breaking. Broadcast profiles require it. Most adaptive-streaming authoring specs ask for it too. That is why a live encode of a hard scene looks worse than the same scene encoded offline.

VBV and HRD

The two names get used for each other. They are not the same thing.

HRD, the hypothetical reference decoder, is the model in the H.264 specification. It describes an idealized decoder with a buffer in front of it. Conformance is defined against that model: a stream is conformant when it never overflows or underflows the buffer. Those parameters can also be written into the sequence header. A real decoder is then told what buffering to expect.

VBV, the video buffering verifier, is the encoder side of the same constraint. It is the buffer simulation a rate controller runs while encoding so it never emits a frame the model could not decode. The name is MPEG-1 and MPEG-2’s own term for their buffer model. H.264 specifies its own model instead, the HRD with a coded picture buffer at its front. Encoders kept the older name for the option.

You configure VBV. The decoder cares about HRD. They describe one constraint from opposite ends.

Both encoders write HRD parameters into the sequence header when you ask for them with --nal-hrd. Neither writes them by default. A stream constrained by --vbv-maxrate and --vbv-bufsize alone carries no signaled buffer. For most delivery nobody notices. Broadcast profiles that require signaled buffering, Blu-ray and ATSC among them, will reject a stream that has none even when the encode itself was properly constrained.

Measuring it

The only real measure of video quality is a person watching it. The ground truth in this field is a MOS, a mean opinion score. You seat a panel of viewers in a controlled room and show them clips in a randomized order. Each viewer rates what they saw from 1 (bad) to 5 (excellent). Average the scores and you have the MOS for that clip at that bitrate. The procedure is standardized, down to the room lighting and the viewing distance, by ITU-R BT.500 and ITU-T P.910.

MOS is also slow, expensive, and impossible to put in a build. Every metric we use is a stand-in for it: a way to predict what viewers would say without asking them. PSNR measures squared error. It is cheap and only loosely tracks what viewers report. SSIM compares local structure and does better. VMAF is a model trained on MOS scores to predict them. It has become the standard the streaming industry judges quality by.

It is the metric this project runs on. Every comparison against x264, every tuning decision, and every quality gate is read on VMAF. We use its NEG variant so that sharpening tricks can’t inflate the score. PSNR stays underneath as a floor. A change can’t buy VMAF by making the picture less accurate.

Next

Part two covers what H.264 adds. Macroblocks, intra modes, transforms, CABAC, deblocking, and the rate-control knobs.