How image to video AI works: animating a photo with attention
An image to video model takes one photo and returns a few seconds of moving footage. The hard part is not the motion but keeping the person or object recognisable while it moves. This page explains how image to video AI works using the same attention maths as our attention mechanism explainer, then works one step by hand and lets you switch the key trick on and off.
On this page
What has to stay the same, and what has to change
Give a model a portrait and ask for five seconds of the person turning their head. Two things must happen at once. The appearance must stay fixed: the same eyes, the same hair, the same background. The geometry must change: the head turns, the light shifts across the cheek, the background slides a little. A model that gets the motion right but lets the appearance wander produces a clip in which the person slowly becomes someone else.
Text-to-video models only have to be consistent with themselves. An image to video model also has to be consistent with a picture it was given, pixel for pixel, at the very first frame. Attention is the tool that makes that possible. If queries, keys and values are new to you, read the attention mechanism first; everything below reuses its formula.
Frames as grids of patches
Transformers work on sequences of tokens, so a video has to become a sequence. The usual way is to cut each frame into a grid of small square patches and turn each patch into one vector, an embedding that describes its colours and texture. Real models do this on a compressed version of the frame rather than raw pixels, but the idea is the same.
Our toy uses a 3×3 grid, so each frame is 9 tokens. Three frames make tokens. Number the patches 1 to 9 row by row, so patch 5 is the centre. Frame 0 is the photo; frames 1 and 2 are the ones the model has to produce.
Each token also carries its position: which frame it belongs to and where in the grid it sits. Without that, attention could not tell “the patch above” from “the same patch, one frame later”, because attention by itself ignores order.
The reference frame trick
In ordinary self-attention every token attends to every other token in the sequence. The design this page describes, used in several image to video models, gives frame 0 a special role. When the model builds frame , each patch’s query is compared with keys from two places: the patches of frame itself and the patches of frame 0. The values come from the same two places:
Stacking the rows of on top of just makes the score matrix wider: each query now has 18 scores instead of 9, and softmax turns all 18 into weights that sum to 1. Frame 0 itself is never rewritten. Its patches are clean, so they act as a fixed library of “what this subject looks like” that every later frame can copy from.
When a patch in frame 2 is part of the subject’s cheek, its query will match the keys of the cheek patches in frame 0 best, and most of its output will be the photo’s own cheek values. That is why the person in the photo is still the same person five seconds later. Other models reach the same goal differently, for example by feeding the photo in as extra input channels or through cross-attention on an image embedding, and many use more than one of these at once.
How it is trained
The weight matrices that produce the queries, keys and values are learned. During training the model is shown the first frame of a real clip and asked to reconstruct the rest, usually from noisy versions of those frames, and gradient descent adjusts the weights to make the reconstruction closer to the real footage. Over millions of clips, the weights learn to produce keys that match when two patches show the same part of the same thing.
Worked example by hand
Step 1: one query against nine keys
Take one patch of frame 2 with query , and the nine keys of frame 0, row by row:
To keep the arithmetic short we leave out frame 2’s own keys; adding them would only add more columns of the same kind.
Step 2: scores
Each score is , since the vectors have length . The raw dot products are 0, 2, 0, 1, 5, 1, 0, 2, 0. For the centre patch, . Dividing by gives
Step 3: softmax
Exponentiate each score: , , and . The nine terms add up to about 50.60. Dividing each by that total gives the weights
The centre patch of the photo takes 0.678 of the attention. Its four neighbours share about 0.243, and the corners about 0.079.
Step 4: the weighted sum of values
Give the frame-0 patches (red, blue) colour values: the centre is red , the top and bottom middles are purple , the left and right middles are blue , and the corners are black . Then
Here 0.163 is the two purple patches together, 0.080 the two blue ones, and the black corners add nothing. The new patch comes out mostly red, the colour of the photo patch it matched, with a little blending from its neighbours. In a trained model, that output is what lets the patch keep the photo’s look while the rest of the network moves it.
Try it: anchor on or off
The widget runs a toy version of the whole loop. Frame 0 is a 3×3 picture of nine colours. In each new frame the outer ring of patches turns one step around the centre, and each patch is copied from the previous frame with a little random noise added, standing in for an imperfect generator. With reference attention on, every patch also attends to frame 0, using its own colour as the query, and is pulled towards the photo patch it matches best. With it off, the patches only ever see the previous frame.
Press Play and watch the strip of frames. With the anchor off, the errors add up and the colours wander further from the photo with every frame. With it on, they stay close for all 16 frames. Click a patch to see its nine attention weights over frame 0; the outlined square is the photo patch it actually started as. Push the noise above about 0.1 and even the anchored version sometimes snaps to the wrong photo patch, such as orange to red: the photo only helps while each frame is close enough to match the right part of it.
The toy cheats in one way: its keys are built straight from colours, so a patch matches the photo patch with the nearest colour. A real model learns its keys, so it can match “the same part of the same object” even when the lighting or angle has changed.
Why this costs so much
Attention compares every query with every key, so its cost grows with the square of the number of tokens. That is the term from Big-O notation, and video is where it bites. Our toy has 27 tokens, so full attention across all three frames needs scores.
Now take a five-second clip at 720p and 24 frames a second, and make some typical assumptions: the frames are compressed 8 times in each direction and 4 times in time before the model sees them, and cut into 2×2 patches. That still leaves 30 frames of 3,600 patches each, or 108,000 tokens. Full attention over all of them would need about scores, for every attention head in every layer, at every denoising step.
That is why generated clips are short, why models split attention into a spatial part (within a frame) and a temporal part (across frames), and why the reference frame is so useful: attending to the patches of one clean frame is far cheaper than attending to every patch of every frame, yet it carries most of the information about what the subject looks like.
Where you’ll see this
The reference frame buys two things. First, identity holds: a face, a product or a pet keeps its features from start to finish, because every frame keeps copying them from the same source. Second, the motion is free to change: since appearance is anchored, the rest of the network can concentrate on where things move, guided by a text prompt such as “the camera slowly pushes in” or by a separate motion module.
Because every frame looks back at the photo, the person or object in it stays recognisable while it moves. That is the main job of the image to video AI services that let you upload one picture, such as HighReach, and turn it into a short clip.
Limits
- Hands and fingers. A hand changes shape a lot between frames and often shows parts the photo never did, so there is nothing in frame 0 to copy. Extra or merging fingers are still common.
- Text in the image. Letters on a sign or a T-shirt span several patches, and small errors in each one turn words into near-words as the clip goes on.
- Large camera moves. When the camera turns far enough to reveal what was behind the subject, frame 0 has no information about it, and the model has to invent the new area. That is where scenes change most.
- Long clips. Because of the quadratic cost, long videos are made in chunks, each starting from the last frame of the one before. Small errors can then build up from chunk to chunk, which is the drift you saw with the anchor off.
Further reading
To go deeper, start with the attention mechanism page, then search for these papers by title:
- A. Blattmann et al., “Stable Video Diffusion: scaling latent video diffusion models to large datasets”, 2023, arXiv:2311.15127.
- S. Zhang et al., “I2VGen-XL: high-quality image-to-video synthesis via cascaded diffusion models”, 2023, arXiv:2311.04145.
- A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić and C. Schmid, “ViViT: a video vision transformer”, ICCV 2021, arXiv:2103.15691.