PEVA: Predicting Egocentric Video from Whole-Body Actions
Berkeley Artificial Intelligence Research (BAIR)
Researchers from BAIR (Berkeley AI) introduce PEVA, a model that predicts egocentric video frames from human whole-body actions. It uses an autoregressive conditional diffusion transformer trained on the Nymeria dataset, enabling atomic action synthesis, counterfactual simulation, and long video generation. PEVA can also be used for visual planning by optimizing action sequences.
PEVA (Predicting Ego-centric Video from human Actions) is a model developed by BAIR that generates future egocentric video frames conditioned on past frames and a sequence of whole-body actions represented as 48-dimensional kinematic pose trajectories. It extends the Conditional Diffusion Transformer (CDiT) with random timeskips, sequence-level training, and action embeddings. Trained on the Nymeria dataset, which pairs egocentric video with body pose capture, PEVA can simulate atomic actions, counterfactuals, and long 16-second video rollouts. It also supports planning by optimizing action sequences using the Cross-Entropy Method to minimize perceptual similarity (LPIPS) to a goal image. The model outperforms baselines in perceptual quality and shows scaling properties with model size.
- Сокращения
- PEVA = Predicting Ego-centric Video from human Actions — Предсказание эгоцентричного видео по действиям человека
- CDiT = Conditional Diffusion Transformer — Условный диффузионный трансформер
- LPIPS = Learned Perceptual Image Patch Similarity — Обучаемое перцептуальное сходство изображений
- CEM = Cross-Entropy Method — Метод кросс-энтропии
- VAE = Variational Autoencoder — Вариационный автоэнкодер
- FID = Fréchet Inception Distance — Расстояние Фреше по Inception
Source: BAIR (Berkeley AI) —
original