Project page · 2026

UniSwap Streaming Audio-Visual Identity Swapping for Talking Videos

Yuxuan Zhang^1,2 Haozhong Xiong² Jiayi Song² Jinpeng Yu² Yang Shi² Jiaming Liu² Ruihua Huang² Liwei Wang¹

¹ The Chinese University of Hong Kong ² Qwen Applications Business Group of Alibaba

UniSwap transfers a reference appearance and voice to a source talking video in a real-time streaming pipeline. — **UniSwap performs streaming joint audio-visual identity replacement.** Given a source talking video and a reference identity (an image and voice clip), UniSwap jointly transfers appearance and vocal timbre while preserving the source motion, background, and linguistic content. Blockwise autoregressive generation with 3-step sampling runs at 13.6 FPS on one NVIDIA H100 and supports stable hour-scale long-form generation.

Abstract

Talking-video character replacement requires coordinated transfer of appearance and voice while preserving the source motion, scene, linguistic content, and audio-video timing. Existing methods use separately optimized models for the two modalities, making audio-visual consistency difficult to enforce. We present UniSwap, the first framework for streaming joint audio-visual identity replacement in talking videos. Given a source video, a reference image, and a reference voice clip, UniSwap transfers the reference appearance and vocal timbre within a single audio-visual diffusion transformer while preserving the source content and dynamics. A swap-and-reconstruct pipeline turns ordinary talking clips into aligned cross-identity training pairs, and a three-stage adaptation converts a bidirectional backbone into a block-causal, three-step streaming generator that remains stable over hour-scale rollouts.

Motivation

Swapping a character in a talking video should change how the speaker looks and how they sound, while leaving the source motion, background, and speech untouched. Today's tools handle only one side: video character-replacement methods stay silent, and voice-conversion systems never see the face. Cascading the two leaves no joint objective to keep converted speech consistent with lip motion, and most diffusion-based replacements must ingest the entire clip before generating anything — ruling out low-latency, interactive use.

One-sided tools

Video replacement ignores the voice; voice conversion ignores the face.

Fragile cascades

Independently optimized stages cannot keep speech and lip motion consistent.

Offline pipelines

Full-clip, many-step diffusion blocks streaming and interactive workflows.

Method

UniSwap starts from LTX-2.3, an audio-video diffusion transformer with native cross-modal attention, and progressively converts it into a causal, three-step streaming generator. Training pairs come from the data itself: each real clip is identity-stripped into a source and then reconstructed, so no cross-identity recordings are needed.

UniSwap training pipeline: in-context pretraining, conditional streaming adaptation, and efficient self-forcing DMD. — **Overview of UniSwap.** The three-stage pipeline comprises (a) In-Context Pretraining for joint audio-video replacement, (b) Conditional Streaming Adaptation with block-causal masking and KV-cached inference, and (c) Efficient Self-Forcing DMD, which reduces denoising to 3 steps per block. Feature-RoPE Decomposition bounds cached positions while preserving cross-modal physical-time alignment.

Feature-RoPE decomposition inference with a fixed reference, sink block, rolling clean cache, and FIFO shifts. — **Overview of UniSwap.** The three-stage pipeline comprises (a) In-Context Pretraining for joint audio-video replacement, (b) Conditional Streaming Adaptation with block-causal masking and KV-cached inference, and (c) Efficient Self-Forcing DMD, which reduces denoising to 3 steps per block. Feature-RoPE Decomposition bounds cached positions while preserving cross-modal physical-time alignment.

Swap-and-reconstruct supervision

Aligned cross-identity pairs are impractical to collect. UniSwap synthesizes an identity-altered source from each real clip — an identity-reduced motion proxy plus a timbre-converted voice — and trains the model to reconstruct the original appearance and voice.

Stage 1–2 · In-context pretraining → streaming adaptation

Source, reference, and noisy target latents are concatenated with aligned coordinates so full-sequence attention learns the joint swap. A Decoupled Streaming Conditioning Mask then restricts each token to its inference-time receptive field, yielding a block-causal generator with KV-cached autoregressive inference.

Stage 3 · Efficient self-forcing DMD

The student rolls out its own predictions to see inference-like histories, while distribution matching cuts sampling from 30 to 3 denoising steps per block. Efficient Multi-LoRA Switching lets the teacher, generator, and critic share one frozen backbone instead of three model copies.

Feature-RoPE decomposition

Cached features are separated from their rotary coordinates and re-anchored to bounded positions, keeping every cached index inside the training range while preserving cross-modal temporal alignment — the key to stable hour-scale streaming.

Demo

Each example shows the source talking video, the reference identity — an image plus a short voice clip — and UniSwap's joint audio-visual output. The videos carry their own audio tracks: click any video to play it with sound, and use the small player under each reference image to hear the reference voice.