| --- |
| license: cc-by-nc-sa-4.0 |
| language: |
| - en |
| tags: |
| - video-generation |
| - vision-language-navigation |
| - embodied-ai |
| - pytorch |
| --- |
| |
|  |
| # SparseVideoNav: Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigation |
|
|
| ## Model Details |
|
|
| ### Model Description |
|
|
| SparseVideoNav introduces video generation models to real-world beyond-the-view vision-language navigation for the first time. It pioneers a paradigm shift from continuous to sparse video generation for longer prediction horizons. By guiding trajectory inference with a generated sparse future spanning a 20-second horizon, it achieves sub-second inference (a 27× speed-up). It also marks the first realization of beyond-the-view navigation in challenging night scenes. |
|
|
| - **Developed by:** Hai Zhang, Siqi Liang, Li Chen, Yuxian Li, Yukuan Xu, Yichao Zhong, Fu Zhang, Hongyang Li |
| - **Shared by:** The University of Hong Kong & OpenDriveLab |
| - **Model type:** Video Generation-based Model for Vision-Language Navigation |
| - **Language(s) (NLP):** English (Instruction prompts) |
| - **License:** CC BY-NC-SA 4.0 |
| - **Finetuned from model:** Based on UMT5-XXL (text encoder) and Wan2.1 VAE. |
|
|
| ### Model Sources |
|
|
| - **Repository:** [https://github.com/OpenDriveLab/SparseVideoNav](https://github.com/OpenDriveLab/SparseVideoNav) |
| - **Paper:** [arXiv:2602.05827](https://arxiv.org/abs/2602.05827) |
| - **Project Page:** [https://opendrivelab.com/SparseVideoNav](https://opendrivelab.com/SparseVideoNav) |
|
|
| ## Uses |
|
|
| ### Direct Use |
|
|
| The model is designed for generating sparse future video frames based on a current visual observation (video) and a natural language instruction (e.g., "turn right"). It is primarily intended for research in Embodied AI, specifically Vision-Language Navigation (VLN) in real-world environments. |
|
|
| ### Out-of-Scope Use |
|
|
| The model is a research prototype and is not intended for deployment in safety-critical real-world autonomous driving or robotic navigation systems without further extensive testing, safety validation, and fallback mechanisms. |
|
|
| ## How to Get Started with the Model |
|
|
| Use the code below to get started with the model using our custom pipeline. |
|
|
| Ensure you have cloned the [GitHub repository](https://github.com/OpenDriveLab/SparseVideoNav) and installed the requirements. |
|
|
| ```python |
| from omegaconf import OmegaConf |
| from inference import SVNPipeline |
| |
| # Load configuration |
| cfg = OmegaConf.load("config/inference.yaml") |
| cfg.ckpt_path = "/path/to/models/SparseVideoNav-Models" # Path to your downloaded weights |
| cfg.inference.device = "cuda:0" |
| |
| # Initialize pipeline |
| pipeline = SVNPipeline.from_pretrained(cfg) |
| |
| # Run inference (Returns np.ndarray (T, H, W, C) uint8) |
| video = pipeline(video="/path/to/input.mp4", text="turn right") |
| ``` |
|
|
| ## BibTeX |
| ```python |
| @article{zhang2026sparse, |
| title={Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigation}, |
| author={Zhang, Hai and Liang, Siqi and Chen, Li and Li, Yuxian and Xu, Yukuan and Zhong, Yichao and Zhang, Fu and Li, Hongyang}, |
| journal={arXiv preprint arXiv:2602.05827}, |
| year={2026} |
| } |
