Teaching AI to Edit Videos with 49 Real Films

VideoEdit

When Fine-Grained Credit Assignment Helps and Hurts

Multimodal Agentic Reinforcement Learning through Video Editing

¹ University of Pennsylvania ² Philo Labs

Part of this work was completed while Lei Zhao was an intern at Philo Labs.

Paper ForthcomingCode Models Watch the edits

We turn movies and their trailers into video-editing tasks, then train multimodal agents to inspect footage, reason about editing decisions, and use editing tools. On held-out films, we study when finer-grained feedback helps—and when it hurts task completion. The project brings together our scalable task-generation pipeline, RL training code, and outcome graders.

Click a preview to compare the edits. Watch with sound in the comparison viewer.

Showcase

Same footage. A different edit.

Click a preview to compare the edits.
Watch with sound in the comparison viewer.

Task generation

Turn any movie–trailer pair into RL tasks: A scalable data-generation pipeline

The pipeline matches trailer shots to the source movie, annotates the footage, and builds tasks with editing briefs, media inputs, and rubric-based reward functions. Applied to 49 real films, it generates 2,352 tasks; filtering retains 797 (716 training / 81 validation), with no films shared between splits.

The four independent task types are Content Selection (choose shots), Narrative Sequencing (order shots), Rough Cut (set clip boundaries), and Fine Cut (edit dialogue, music, and transitions over fixed picture timing).

Posters from the 49 real films used to construct the editing tasks. — Film credits ↗

Four independent task types; Fine Cut keeps picture timing fixed.

Long-horizon multimodal agents

RL Training

Adapted for long-horizon multimodal RL with the Qwen3.5 series. The runtime combines SkyRL training, vLLM inference, and an OpenCode agent harness for visual inspection, reasoning, and editing-tool use across up to 100 turns and 128K tokens of context.

Run Qwen3.5 training Environment setup, task preparation, and launch commands

ONE TRAJECTORY · UP TO 100 TURNS

Inspect the footageObserve frames and tool outputs.

Reason about the editPlan, check evidence, and revise decisions.

Use editing toolsSelect, sequence, trim, and mix.

Submit the resultReceive task-outcome feedback.

↶ Repeat with intermediate observations

Credit assignment

TRAJECTORY-LEVEL

Outcome-only GRPO

A single final task reward supplies the learning signal for all actions in the trajectory.

TURN-LEVEL + TRAJECTORY-LEVEL

GRPO + PRM

An LLM judge scores intermediate actions at each turn, alongside the final task reward. Process-reward weight can decay during training.

TOKEN-LEVEL

On-policy self-distillation (OPSD)

Each generated token receives a distillation signal from a frozen teacher conditioned on reference answers or turn suggestions.

Main experiments: Qwen3.5-9B. Additional GRPO/PRM experiments: Qwen3.6-35B-A3B.

Experiment settings

Run training

Full training guide

After setup, choose GRPO, GRPO+PRM, or OPSD in your configuration, then check it and run a two-update smoke test.

export EDITING_TRAINING_CONFIG=/path/to/training.toml
python -m modal run training/modal_train.py --action check
python -m modal run training/modal_train.py --action smoke --steps 2

The smoke test uses 8 H200 GPUs. The released GPU launch configuration has not yet been verified end to end.

Findings

When does finer feedback help—and hurt?

Three findings from our Qwen3.5-9B experiments.

01 / Task-dependent gains

Fine-grained feedback helps more complex tasks.

Decayed process rewards bring the largest benefit on Fine Cut. On the other three task types, outcome-only GRPO remains competitive, with additional gains of at most 10.4%.

+45.4%relative score gain on Fine Cut

Relative score changes for decayed GRPO+PRM versus outcome-only GRPO: NS +10.4%, RC −2.2%, CS −2.1%, FC +45.4%. — Decayed GRPO+PRM vs. outcome-only GRPO. Relative changes use the audited four-decimal evaluation means.

02 / Reward hacking

Process rewards can lead to reward hacking.

With a fixed weight, process reward keeps rising while outcome reward declines late in training. Decaying the weight to zero over updates 20–30 mitigates this pattern and improves completion.

44.1%6.2%validation truncation · fixed → decayed weight

Training rewards and the lambda schedule. With a fixed process weight, process reward increases while outcome reward declines; decay mitigates the pattern. Shading marks updates 20–30. — Training rewards: recorded observations and centered five-observation means. Shading marks the decay window; the lower strip shows λ. Truncation percentages are separate held-out validation results.

03 / Agent behavior

Finer-grained feedback encourages desirable behaviors.

Decayed process rewards encourage more visual inspection, longer reasoning, and more self-correction markers. These changes accompany the best overall score in the main comparison: 0.447, versus 0.413 for outcome-only GRPO.

1.5–1.9×more inspection, reasoning, and self-correction markers

Validation behavior · Qwen3.5-9B · four evaluation repetitions
Behavior	GRPOoutcome-only	+ PRMfixed weight	+ PRMdecayed weight
Visual inspection
Frames seenimage attachments per rollout	84.2	143.6	129.8
Blind rollouts ↓no images returned	4.6%	0.3%	0.0%
Reasoning
Reasoning lengthcharacters per completed turn	618	761	1,035
Self-correction markersshare of completed turns	19.0%	24.8%	36.0%
Interaction & completion
Interaction turnsmean per rollout	45.1	82.4	56.9
Tool callsrequested invocations per rollout	52.2	165.8	99.5
Truncated rollouts ↓stopped by a budget limit	0.3%	44.1%	6.2%

Frames count returned image attachments, not distinct film frames. More turns or tool calls are not inherently better. Reasoning length and self-correction markers do not establish reasoning quality.

Scope: main Qwen3.5-9B experiments; held-out scores use avg@4. Task structure, scoring criteria, and horizon vary together. Reward-hacking findings concern our LLM-judged GRPO+PRM setup. Experimental details ↗

Models

Two sizes. Built to edit.

Full BF16 policy checkpoints,
with processor, tokenizer, and chat template.

DENSE VISION–LANGUAGE MODELPrivate preview

VideoEdit-Qwen3.5-9B

GRPO + PRM · decayed process weight

Overall task score higher is better
Untrained base	0.2841
Outcome-only GRPO	0.4133
VideoEdit	0.4467

avg@4 · 4 complete evaluation repetitions

Evaluated context: 131,072 tokens
BF16 weights: 18.8 GB

View on Hugging Face

MIXTURE-OF-EXPERTS MODELPrivate preview

35B3B active

VideoEdit-Qwen3.6-35B-A3B

GRPO + PRM · fixed process weight

Overall task score higher is better
Untrained base	0.2923
Outcome-only GRPO	0.5079
VideoEdit	0.5314

avg@3 · 3 complete evaluation repetitions

Evaluated context: 262,144 tokens
BF16 weights: 70.2 GB

View on Hugging Face

Scores use 81 held-out tasks. Repetitions evaluate the same checkpoint, not independent training seeds. The 35B comparisons use three complete repetitions per policy, with different retained subsets; see each model card for its protocol. Model and code access currently require permission.

Citation

Build on this work.

The task-generation pipeline, training implementations,
and outcome graders live in the companion repository.

Explore the project on GitHub

SOFTWARE CITATION

@software{zhao2026videoediting,
  title  = {When Fine-Grained Credit Assignment
            Helps and Hurts: Multimodal Agentic
            Reinforcement Learning through Video Editing},
  author = {Zhao, Lei and Cao, Zongheng and Hu, Xinyu},
  year   = {2026},
  url    = {https://github.com/leizhao7/video-editing-credit-assignment}
}