Runway ML and AI Video Generation Prompts

Runway is the most widely-used AI video platform — and prompting for video is its own discipline. Static images forgive vague prompts; video does not. A prompt has to describe what is in the shot and what moves, when, and how. This guide shows you how to write video prompts that actually do what you want.

Note: Runway evolves its model line (Gen-2, Gen-3, Gen-4 and successors) and feature set quickly. Concepts here are stable; specific UI labels may change.

1. Introduction

Runway started as a video-editing-with-AI tool and grew into a generative video studio. Today it covers text-to-video, image-to-video, video-to-video restyling, and a host of "magic tools" like green-screen, motion brush, and camera control. The platform is genuinely powerful — but the difference between an amateur clip and a usable one is almost entirely in the prompt and the reference inputs.

This tutorial covers the three main generation modes, the vocabulary that controls motion and camera, and the storyboard-driven workflow that lets you build longer pieces without the output feeling like a slideshow.

2. The Three Generation Modes

Most workflows pick one of these three starting points. Each rewards a different prompt style.

• Text-to-video. You describe the scene and motion in words. Best for quick exploration; weakest for character or location consistency.
• Image-to-video. You provide a starting frame (often generated in Midjourney or Stable Diffusion) and a short prompt describing the motion. By far the most reliable mode for polished output.
• Video-to-video. You provide a source clip plus a style prompt; Runway restyles the clip while preserving motion. Excellent for animation looks and consistent characters.

If you care about brand or character consistency, default to image-to-video. Generate the look you want as a still first, then animate.

3. Prompt Anatomy for Motion

A video prompt has two parts most beginners under-write: what moves and how the camera behaves. Get those in every prompt and your clips stop feeling like jittery photos.

Scene: a young woman walks through a quiet Tokyo alley
at night, neon signs reflecting in puddles on the pavement.

Subject motion: she walks slowly from left to right,
glancing up briefly toward a sign.

Camera: slow dolly forward, slight low angle, shallow
depth of field. No shake. Hold the framing.

Style: cinematic, anamorphic, soft teal-and-magenta colour
palette, gentle film grain.

Duration: 5 seconds.

Notice how each line answers a different question — subject, motion, camera, style, duration. Drop any one and the model has to guess.

4. Camera Control Vocabulary

Runway exposes explicit camera-control sliders, and even when you're using text-only prompts the model recognises a clear cinematic vocabulary. The words below are the ones to memorise.

• Dolly in / dolly out — camera moves toward or away from the subject along its axis.
• Truck left / truck right — camera slides sideways.
• Pedestal up / pedestal down — camera moves vertically.
• Pan left / pan right — camera rotates horizontally on its tripod.
• Tilt up / tilt down — camera rotates vertically.
• Orbit — circling motion around the subject.
• Static / locked off — no movement at all (often the most professional choice).

Combining two of these is fine — "slow dolly in with a gentle tilt up" — but three or more usually fight each other and produce mushy motion.

5. Image-to-Video: The Reliable Workflow

For client work, almost everyone serious about Runway uses the same workflow.

1. Generate or curate the starting frame elsewhere — Midjourney with a consistent --sref, Stable Diffusion with a LoRA, or a real photo.
2. Upload it to Runway as the first frame.
3. Write a short motion prompt — what moves, how, camera behaviour.
4. Use motion brush (where available) to paint which regions should move, and how strongly.
5. Generate three or four variants at the same duration and pick the best.
6. Extend from the last frame to grow the clip if needed.

This pipeline turns Runway from a "slot machine" into a controlled studio tool. Visual consistency is solved upstream (in your image model); Runway only has to handle motion.

6. Patterns That Work Especially Well on Runway

• Constrain motion explicitly. "Subject is static; only the trees in the background sway in a light breeze" produces calmer, more usable clips than vague "things move".
• Specify what should not happen. "No camera shake. No zoom. Subject does not turn their head." Negative directives prevent the most common artefacts.
• Storyboard first, generate second. Plan a 30-second piece as six 5-second shots. Write a separate prompt for each. Generate, pick winners, edit together. This is how pros avoid the "weird AI video" feel.
• Use video-to-video for animation looks. Film yourself doing the motion, then restyle. The motion lands naturally and the style stays consistent.
• Hold framing for dialogue and beats. Static, locked-off shots feel more professional than constant camera motion. Use movement deliberately, not by default.

7. Practice Exercises

8. Key Takeaways