How H.264 works

1What H.264 adds

Part one covers the fundamentals every block-based codec shares. The bit budget, the kinds of redundancy worth finding, the predict-transform-quantize-entropy loop and the reconstruction that feeds back into it, rate control, and how quality gets measured. This page is about what H.264 specifically does with it. It assumes you have read the other.

H.264 processes each frame in 16x16 macroblocks. It brings a unique tool to each kind of redundancy part one describes:

Spatial → guess a block from the pixels beside it. Nine directions at 4x4 and 8x8, four more at 16x16. The integer transform underneath is exact (intra, transform).
Temporal → find where this block was in an earlier frame. Quarter-pixel accuracy, up to sixteen frames to search, variable partitions and B-frame pyramids (frames, motion).
Statistical → spend the fewest bits on whatever happens most. CABAC keeps re-estimating how likely each symbol is as it codes (entropy coding).
Perceptual → do not spend bits where nobody will look. This one is not in the standard at all. It lives in the encoder's decisions. That is why two conformant H.264 encoders at the same bitrate can look so different. It is also most of what this project is about (the size of the choice).

Part one's loop ends by reconstructing the frame the next one predicts from. H.264 puts a deblocking filter inside that loop (deblocking). Encoder and decoder must filter identically or they drift apart.

2Frames and GOPs

Not all frames are equal. H.264 has three kinds. What separates them is what each one is allowed to predict from:

I frames predict only from themselves (spatial prediction). They are expensive, maybe 5 to 10x the bits of a P frame. They earn it by being random-access points where a decoder can start cold. Seeking and channel switching jump to I frames. Recovery from packet loss restarts on one.
P frames predict from earlier frames. The workhorses.
B frames predict from earlier and later frames and average both directions. That lets them use even fewer bits than P frames. The catch is that a B frame needs its future reference decoded first. Frames are transmitted out of display order and the decoder reorders them. That adds latency.

One GOP (group of pictures) in display order. Coding order differs. Frame 3 (P) is coded before frames 1 and 2 (B) because they need it as a reference. A typical streaming GOP runs one I frame every 2 to 10 seconds.

The distance between I frames is the keyframe interval. It is a straight trade. Long GOPs compress better because there are fewer expensive I frames. They also make seeking coarser and recovery from packet loss slower. Streaming services put their segment boundaries at I frames. That is why adaptive players can only switch quality every couple of seconds.

A special I frame called an IDR (instantaneous decoder refresh) additionally forbids any later frame from referencing anything before it. That fully resets decoder state. The point is then truly seekable. One of the oldest encoder-side smarts is placing IDRs at scene cuts. A full refresh was going to be nearly free there anyway, because nothing predicts well across a cut.

3Intra prediction

Even a frame with no past can avoid coding raw pixels. Intra prediction builds each block from its already-decoded neighbors. Those are the row of pixels above and the column to the left. The encoder projects them across the block in one of several directions.

Three of the nine 4x4 intra modes. The others are diagonals at various angles. The encoder tries them all and keeps the one whose prediction error is cheapest. Signalling which one it picked takes a couple of bits.

Luma blocks come in two intra flavors: 16x16 with four simple modes (vertical, horizontal, DC, plane) for flat regions, and 4x4 with nine modes for detailed ones. High profile adds 8x8. The prediction is rarely exact. It doesn't have to be. If a gradient sky is predicted within a few code values, the leftover error is tiny and nearly free to code.

4Motion compensation

Inter prediction is where the largest opportunity lies. For each block, the encoder searches the reference frame for the patch that matches best. Then it transmits just a motion vector, the (x, y) offset to that patch, and the residual error. A panning shot of a stadium crowd holds tens of thousands of "detailed" pixels and can cost almost nothing. Every block is "same as last frame, shifted 4 pixels left."

Inter prediction. The dashed box is the motion search window. Vectors have quarter-pixel precision. The encoder interpolates sub-pixel positions with a 6-tap filter, because real motion doesn't align to the pixel grid.

A few details make this far better than the naive version:

Variable block sizes. A 16x16 macroblock can split into 16x8, 8x16, 8x8, and each 8x8 further down to 4x4. Every partition gets its own vector. Smooth backgrounds use one big cheap vector. The boundary of a moving arm gets a fistful of small precise ones.
Quarter-pixel precision. Vectors resolve to 0.25 px. Half-pel positions come from a 6-tap filter and quarter-pel positions from averaging. This alone is worth a large slice of H.264's advantage over older codecs.
Multiple reference frames. Up to 16. Useful when content occludes and reappears. The wall behind a passing truck is still sitting, perfectly intact, in a frame from half a second ago.
Predicted vectors. Neighboring blocks usually move together. Each vector is therefore coded as a difference from the median of its neighbors' vectors. Uniform motion costs almost zero bits.
Skip mode. The best block is the one you never send. If the predicted vector with no residual is good enough, a macroblock costs roughly one bit. In quiet scenes most of the frame is skips.

Motion search is also where encoder CPU time goes. The spec constrains how you encode the vector and says nothing about how you found it. That makes search strategy (diamond, hexagon, exhaustive) a pure speed-versus-quality knob. It is one of the main things separating fast presets from slow ones.

5The integer transform

Part one covers what a transform does. Two things are specific to H.264.

Its transform uses whole numbers. Older standards used floating point, so no two decoders computed it quite the same way. Those small differences built up until a long sequence drifted away from what the encoder intended. Whole numbers give every decoder the same answer, exactly. That also lets this project test itself. It compares its own reconstructed picture against a real decoder's output bit for bit.

There are two block sizes and the encoder picks one per macroblock. The small one follows edges without blurring them. The big one packs smooth areas into fewer numbers. Neither wins everywhere. That is one more decision to get right.

QP runs from 0 to 51. The step doubles every 6 points, so QP 28 quantizes four times as coarsely as QP 16. Everything called rate control is a policy for choosing it.

6Entropy coding

Everything so far turned the picture into symbols. Which prediction mode, how far a block moved, what survived quantization. Entropy coding writes those symbols out as the actual bitstream, losslessly, by giving the likely ones short codes and the rare ones long ones. H.264 ships two engines and the profile decides which you get.

CAVLC looks each symbol up in a table and spends a whole number of bits on it. Which table it uses depends on the neighbors. One such input is how many coefficients the blocks nearby had. It is simple and fast. Every symbol costs at least one bit.
CABAC is the interesting one. It breaks every symbol into yes-or-no decisions and keeps a running estimate of how likely each answer is. It holds those estimates across more than 400 separate situations. They update as the stream is coded, so the model sharpens as it goes. An arithmetic coder then writes each decision at what it is actually worth. A decision that is 95% predictable costs about 0.07 bits. That is well under the whole bit CAVLC has to spend. Most decisions in video are that lopsided.

CABAC is worth roughly 10 to 15% of the bitrate at the same quality. That is one of the largest single wins in the standard. It is paid for at the other end. Decoding it is a serial, branch-heavy loop that resists parallelism. That tension between compressing well and decoding fast has shaped every codec since. H.265 restructured its own CABAC specifically so hardware could keep up at 4K.

7The deblocking filter

Block-based coding has a signature failure. Every block is quantized on its own, so the seams between them show. That is the classic blocky video look. H.264 builds the fix into the standard. A smoothing filter runs along every block edge. How hard it smooths depends on the conditions. Intra blocks get more. High QP gets more. The clever part is the test that tells a quantization seam apart from the real edge of an object. The filter softens the first and leaves the second alone.

The word that matters is in-loop. The filtered frame is the one that goes into the reference buffer, so later frames predict from cleaned-up pixels and the improvement compounds. It is worth several percent of the bitrate and a much larger subjective gain at low rates. It is also why an H.264 picture that runs out of bits turns blurry, where MPEG-2 in the same situation broke up into visible squares. The filter smooths the block edges away, so starvation shows as softness instead of a grid.

8The size of the choice

Part one covers rate-distortion optimization. Every candidate encoding has a cost and the encoder takes the cheapest. The number of candidates H.264 offers is why two encoders following the same standard differ so much.

One macroblock alone offers intra or inter, nine intra directions, seven ways to partition it, sixteen reference frames to search, quarter-pixel vectors, two transform sizes, and a choice of which coefficients survive. That is hundreds of combinations. A 1080p frame has 8160 macroblocks.

No encoder searches that space. A preset is a set of decisions about what not to look at. Fast presets guess cheaply and throw most candidates away. Slow ones encode candidates to find out what they really cost. This is the space this project works in.

9What the standard does not say

Rate control, mode decision, motion search, lookahead, adaptive quantization, psychovisual tuning. None of it is in H.264. The specification defines what a conformant bitstream decodes to and says nothing about how an encoder chose it. Part one covers the five rate-control modes and how quality gets measured. Neither topic is H.264-specific.

That silence is the whole opportunity. Every decoder already deployed keeps working no matter how much better the encoder gets. Twenty-three years after the format froze there is still room to move. That is where all of this project's work happens. The measured results are what came of it.

10What yah264 ships, and what comes next

H.264 is twenty-three years old and is still the most played video format on Earth. That is where the opportunity comes from. Every bit saved at the same quality, and every second saved encoding it, is multiplied by an audience no other codec has. Its successors H.265, AV1, and VVC buy better compression with far more expensive tools. What gets underappreciated is that the standard only fixes the decoder. Inside the fixed syntax the encoder's decisions are wide open. Where to spend, what to predict, what to throw away. Most of what changed video quality in the last decade was encoder-side intelligence, not new syntax. CRF, mbtree, psy-RD, per-title encoding. That is the thesis of yah264. It already does the full pipeline above on arm64 and x86-64. I/P/B frames, quarter-pel motion, RD mode decision, deblocking, and both entropy coders, CAVLC and CABAC. What follows is where we think the remaining headroom is. Each item carries a label: Shipped means you can run it today and the README describes it, In progress means the work has started, and Not started means it is an idea from the literature with nothing built yet.

Shot-based encoding in one pass Shipped

Per-shot keyframes and per-shot CRF ship. Per-shot resolution is orchestrator work, with a first version in progress

A movie is a few hundred shots, and they do not all cost the same to encode. A static close-up might look perfect at 1 Mb/s while a rain-soaked chase needs 8. Netflix proved the value of exploiting this. Their dynamic optimizer encodes every shot at many resolution and QP combinations and scores every result, so each shot gets a convex hull of its best rate-quality tradeoffs. One point per shot then makes the whole title optimal. Netflix reports 17% bitrate savings against fixed-QP encoding and over 50% against ordinary two-pass VBR. Both figures are measured on their own catalogue. Both include resolution switching. The catch is dozens of trial encodes per shot. A cluster-scale orchestrator runs them. The research since says most of those are avoidable. RCN-Hull predicts a shot's convex hull from its content with a small model. It gives up 0.26% BD-rate and skips 62% of the encodes.

Inside yah264 that collapses to two flags. --cut-split pre-scans the source and starts every shot on its own keyframe. --shot-crf gives each shot its own quality setting from that same scan. Nothing is encoded twice and no trial encode runs, though the pre-scan reads the whole input, so the mode needs a file and does nothing on a pipe. An orchestrator that does want to search gets the shot table, a plan of quality offsets over frame ranges, per-frame statistics and one output file per shot. Re-encode a shot on its own at a fixed thread count and you get back the bytes it had inside the film. On three five-shot test sequences the per-shot setting reads 6 to 11% better than splitting at the cuts alone. Against the ordinary default it keeps 1 to 4%. On ten real titles at 1080p it needs a median 1% fewer bits for the same VMAF-NEG score, and 5% fewer for the same PSNR-Y. Per-shot resolution and a measured hull are the stages after this one. The full plan lives in shot-based.md.

Why resolution is a per-shot decision. One shot encoded at three resolutions across a QP sweep. Quality is measured after upscaling everything back to 1080p. Below about 1.8 Mb/s, 480p upscaled beats both larger encodes, and 1080p does not win until about 6. At starvation bitrates the 1080p encode spends everything on structure and still shatters, while the 480p encode is clean and merely soft. The dashed hull is the menu an optimal encoder orders from. Hover the points.

Film grain synthesis Not started

Large savings on film content

Film grain is the most expensive thing a codec can face. It is high-energy noise, different every frame, that motion compensation can't predict and quantization visibly destroys. AV1 made the fix famous. Denoise the source, encode the clean video cheaply, transmit a statistical description of the grain, and have the decoder resynthesize it. Almost nobody knows H.264 has supported this since 2004 through the film grain characteristics SEI message. FFmpeg's H.264 decoder has parsed the SEI and applied the grain since 2021, so FFmpeg-based players render it. No mainstream open-source H.264 encoder writes it, as far as we can find. x264 does not. The best-known real-world use was Thomson's Film Grain Technology, which HD DVD adopted two decades ago. Writing that SEI would put yah264 among the first open-source H.264 encoders to do so. Everywhere else the stream degrades gracefully to clean video with no grain.

Perception-driven bit placement Not started

Reported 7 to 22% bitrate savings in recent research

Adaptive quantization spends bits where errors are visible. The next step is spending them where people actually look. Saliency models can steer per-block QP so the face in focus gets the bits the out-of-focus background was hoarding. Even cheap heuristic models built from motion contrast will do it. Recent work on newer codecs reports 7 to 22% bitrate savings at equal subjective quality from saliency combined with just-noticeable-difference models: PAVEN at the bottom of that range and SJ-PVC at the top. Both are measured on VVC. The mechanism is per-macroblock QP offsets. That drops straight into H.264.

Learned encoder decisions Not started

Speed and consistency, no bitstream changes

A small trained model beats a hand-tuned heuristic in three places the decoder never sees. Predicting coded frame size from lookahead features tightens rate control and calms QP oscillation. Google shipped an imitation-learned rate controller for VP9. Pruning the mode search to the candidates likely to win is reported to speed newer codecs up 30 to 50% at about 1% quality cost, again on VVC. And a prefilter trained end-to-end through a differentiable codec proxy makes frames cheaper to encode before the codec ever sees them. That one follows Google's sandwiched compression work. On Apple Silicon these run on the neural engine while the CPU encodes.

Cleaner references, borrowed from AV1 Shipped

Standard-compliant, encoder-side only

SVT-AV1 motion-compensation-filters a frame before encoding it as a reference. Predictions then come from a temporally denoised image and residuals shrink everywhere downstream. Nothing about that trick is AV1-specific. It never touches the bitstream. Ported to H.264 it pairs beautifully with grain synthesis. Temporal filtering removes exactly the noise the grain SEI puts back. In yah264 it's --mcstf. It filters every fourth picture against its motion-compensated neighbors and leaves pictures that read clean alone. It's off by default, and denoise.md has the numbers.

The syntax of H.264 was settled in 2003. What's left to improve is the decisions an encoder makes inside that syntax: which mode, which reference, how many bits, where. Those decisions can now be judged with perceptual metrics and guided by lookahead and learned models, and every improvement ships as software that a 2004 decoder plays. That's the bet: a twenty-three-year-old format with a better encoder behind it.