Edit-A-Video<span>:</span> Single Video Editing with Object-Aware Consistency

→

"a bear is playing a guitar"

"a monkey is playing a guitar"

"a man is playing a guitar, cartoon style"

"a man is playing a guitar, Matisse style"

→

"a man is on the swing"

"a Iron Man is on the swing"

"a woman is on the swing, Van Gogh style"

"a watercolor painting that a woman is on the swing"

→

"a Lionel Messi is dribbling a baksetball"

"a racoon is dribbling a baksetball"

"a oil painting that a man is dribbling a baksetball"

"a man is dribbling a baksetball, Renoir style"

→

"a zombie is running"

"a werewolf is running"

"a man is running, pixar style"

"a pencil sketch that a man is running"

→

"a bull is walking"

"a zebra is walking"

"a cow is walking on the snow"

"a cow is walking on the desert"

With advancements in text-to-image (TTI) models, text-to-video (TTV) models have recently been introduced. Motivated by approaches on TTV models adapting from diffusion-based TTI models, we suggest the text-guided video editing framework given only a pretrained TTI model and a single <text, video> pair, which we term Edit-A-Video. The framework consists of two stages: (1) inflating the 2D model into the 3D model by appending temporal modules and tuning on the source video (2) inverting the source video into the noise and editing with target text through attention map injection. Each stage enables the temporal modeling and preservation of semantic attributes of the source video. One of the key challenges for video editing is a background inconsistency problem, where the regions unrelated to the edit suffer from undesirable and inconsistent temporal alterations. To mitigate this issue, we also introduce a novel mask blending method, termed as temporal-consistent blending (TC Blending). We improve previous mask blending methods to reflect the temporal consistency, ensuring that the area where the editing is applied exhibits smooth transition while also achieving spatio-temporal consistency of the unedited regions. We present extensive experimental results over various types of text and videos, and demonstrate the superiority of the proposed method compared to baselines in terms of background consistency, text alignment, and video editing quality.

Baseline Comparisons

"a Bat Man is boxing"