ASID-Captioner-7B

ASID-Captioner-7B is an audiovisual captioning model (based on Qwen2.5-Omni) fine-tuned for attribute-structured and quality-verified video understanding. It is designed to generate fine-grained captions that cover both visual and audio signals, with controllable prompting over multiple attributes.

[🏠 Homepage] [📖 Arxiv Paper] [🤗 Models & Datasets] [💻 Code]

Introduction

Modern video MLLMs often describe long and complex audiovisual content with a single caption, which can be incomplete (missing audio or camera details), unstructured, and weakly controllable.

ASID-Captioner-7B is trained to follow attribute-specific instructions and produce more organized, fine-grained descriptions. It is built upon Qwen2.5-Omni and fine-tuned on ASID-1M, which provides structured supervision over multiple attributes (scene, characters, objects, actions, narrative elements, speech, camera, emotions) with quality verification and refinement.

Key Features

• Audiovisual captioning: uses both video frames and audio (when available).
• Attribute-structured instruction following: supports prompts targeting specific attributes (e.g., speech-only, camera-only).
• High-quality supervision: trained on attribute-structured, quality-verified instructions from ASID-1M.
• Standard Transformers interface: load with transformers and the Qwen2.5-Omni processor/model classes.

What’s in this repo

Typical files include:

• config.json
• generation_config.json
• preprocessor_config.json
• chat_template.jinja
• added_tokens.json / special_tokens_map.json
• model-*.safetensors and model.safetensors.index.json

Prompting (recommended)

ASID-Captioner-7B works best with explicit attribute prompts, for example:

• Describe the scene in the video in detail. Write your answer as one coherent paragraph.
• Describe the characters in the video in detail. Write your answer as one coherent paragraph.
• Provide a comprehensive description of all the content in the video, leaving out no details, and naturally covering the scene, characters, objects, actions, narrative elements, speech, camera, and emotions in a single coherent account.

Usage (minimal, single GPU)

Install

pip install -U transformers accelerate

Optional: faster attention

If you want faster attention (optional), install FlashAttention2 following its official instructions.

You must also have qwen_omni_utils.process_mm_info available in your environment (same as your reference script).

Run inference

import os
import torch
from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor
from qwen_omni_utils import process_mm_info

# Constants (same spirit as reference)
VIDEO_MAX_PIXELS = 401408        # 512*28*28
VIDEO_TOTAL_PIXELS = 20070400    # 512*28*28*50
USE_AUDIO_IN_VIDEO = True

# Some pipelines use this env var
os.environ["VIDEO_MAX_PIXELS"] = str(VIDEO_TOTAL_PIXELS)

model_id = "AudioVisual-Caption/ASID-Captioner-7B"

model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="cuda",
    attn_implementation="flash_attention_2",  # optional; remove if not available
    low_cpu_mem_usage=True,
)
model.disable_talker()

processor = Qwen2_5OmniProcessor.from_pretrained(model_id)

file_path = "/path/to/video.mp4"
prompt = "Provide a comprehensive description of all the content in the video, leaving out no details, and naturally covering the scene, characters, objects, actions, narrative elements, speech, camera, and emotions in a single coherent account."

conversation = [
    {
        "role": "system",
        "content": [
            {
                "type": "text",
                "text": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech."
            }
        ],
    },
    {
        "role": "user",
        "content": [
            {"type": "video", "video": file_path, "max_pixels": VIDEO_MAX_PIXELS},
            {"type": "text", "text": prompt},
        ],
    },
]

text = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=False,
)

# IMPORTANT: reference-style multimodal extraction
audios, images, videos = process_mm_info(
    conversation,
    use_audio_in_video=USE_AUDIO_IN_VIDEO,
)

inputs = processor(
    text=text,
    audio=audios,
    images=images,
    videos=videos,
    return_tensors="pt",
    padding=True,
    use_audio_in_video=USE_AUDIO_IN_VIDEO,
)

device = "cuda"
inputs = inputs.to(device).to(model.dtype)

with torch.no_grad():
    text_ids = model.generate(
        **inputs,
        use_audio_in_video=USE_AUDIO_IN_VIDEO,
        do_sample=False,
        thinker_max_new_tokens=4096,
        repetition_penalty=1.1,
        use_cache=True,
    )

decoded = processor.batch_decode(
    text_ids,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
)[0]

answer = decoded.split("\nassistant\n")[-1].strip()
print(answer)

Notes (important)

• If you do not use process_mm_info, you may get missing/incorrect audiovisual inputs in some environments.
• use_audio_in_video=True enables audio-conditioned captioning when your runtime supports extracting audio from the video container.
• thinker_max_new_tokens is used in the reference script. If your environment does not recognize it, replace with max_new_tokens.

Training Data

This model is fine-tuned using ASID-1M (attribute-structured and quality-verified audiovisual instructions). Dataset: AudioVisual-Caption/ASID-1M

Citation

If you use our model in your research, please cite our paper:

@misc{asid2026,
  title={Towards Universal Video MLLMs with Attribute-Structured and Quality-Verified Instructions},
  author={Yunheng Li and Hengrui Zhang and Meng-Hao Guo and Wenzhao Gao and  Shaoyong Jia and Shaohui Jiao and Qibin Hou1 and Ming-Ming Cheng},
  year={2026}
}

Contact

Please open a Discussion on the Hugging Face page for usage questions or issues.