VEFX-Bench: A Holistic Benchmark for Generic Video Editing and Visual Effects
Summary: arXiv:2604.16272v1 Announce Type: cross
Introduction
As artificial intelligence continues to transform various industries, the realm of video editing is experiencing significant advancements. AI-assisted video creation techniques are becoming more practical, necessitating effective instruction-guided video editing to refine both generated and captured footage. However, the current landscape of video editing evaluation is fraught with limitations, particularly the lack of a comprehensive human-annotated dataset and standardized evaluation metrics.
The Limitations of Existing Resources
While there are numerous resources available, they often fall short in various ways:
- Limited scale, with few examples to draw from.
- Missing edited outputs that hinder comprehensive evaluations.
- Absence of human quality labels, making it difficult to assess editing quality accurately.
- Reliance on expensive manual inspection methods or generic vision-language models that fail to specialize in editing quality.
Introducing VEFX-Dataset
To address these challenges, we present the VEFX-Dataset, a human-annotated dataset comprising 5,049 video editing examples across nine major editing categories and 32 subcategories. Each example is meticulously labeled along three distinct dimensions:
- Instruction Following: Evaluates how well the editing aligns with the provided instructions.
- Rendering Quality: Assesses the visual fidelity and overall quality of the edited video.
- Edit Exclusivity: Measures how unique or distinctive the edits are in relation to the original footage.
VEFX-Reward: A Specialized Assessment Model
Building upon the VEFX-Dataset, we introduce VEFX-Reward, a reward model explicitly designed for assessing video editing quality. VEFX-Reward utilizes a joint processing approach, analyzing the source video, the editing instructions, and the edited output to predict quality scores across the three dimensions mentioned above. This model employs ordinal regression techniques to enhance the precision of quality assessments.
VEFX-Bench: A Standardized Benchmark
In addition to the dataset and the reward model, we release VEFX-Bench, a benchmark consisting of 300 curated video-prompt pairs. This benchmark enables standardized comparisons of different editing systems, facilitating a more transparent evaluation process. Our experiments indicate that VEFX-Reward exhibits a stronger alignment with human judgments compared to generic vision-language model judges and previous reward models. This is evident in both standard Image Quality Assessment (IQA) and Video Quality Assessment (VQA) metrics, as well as group-wise preference evaluations.
Benchmarking Video Editing Systems
Using VEFX-Reward as an evaluation tool, we conducted a benchmarking exercise on various representative commercial and open-source video editing systems. The results revealed a persistent gap in performance across three critical areas:
- Visual Plausibility: The degree to which edited videos appear realistic and visually appealing.
- Instruction Following: How effectively the systems adhere to the provided editing instructions.
- Edit Locality: The relevance and context of the edits in relation to the source material.
Conclusion
VEFX-Bench, along with the VEFX-Dataset and VEFX-Reward, represents a significant advancement in the evaluation of video editing systems. With these tools, researchers and practitioners can now better assess and improve the quality of AI-assisted video editing, paving the way for more sophisticated and user-friendly editing solutions in the future.