Track Anything: Segment Anything Meets Videos
Paper
• 2304.11968 • Published
YAML Metadata Warning: The pipeline tag "video-object-tracking" is not in the official list: text-classification, token-classification, table-question-answering, question-answering, zero-shot-classification, translation, summarization, feature-extraction, text-generation, fill-mask, sentence-similarity, text-to-speech, text-to-audio, automatic-speech-recognition, audio-to-audio, audio-classification, audio-text-to-text, voice-activity-detection, depth-estimation, image-classification, object-detection, image-segmentation, text-to-image, image-to-text, image-to-image, image-to-video, unconditional-image-generation, video-classification, reinforcement-learning, robotics, tabular-classification, tabular-regression, tabular-to-text, table-to-text, multiple-choice, text-ranking, text-retrieval, time-series-forecasting, text-to-video, image-text-to-text, image-text-to-image, image-text-to-video, visual-question-answering, document-question-answering, zero-shot-image-classification, graph-ml, mask-generation, zero-shot-object-detection, text-to-3d, image-to-3d, image-feature-extraction, video-text-to-text, keypoint-detection, visual-document-retrieval, any-to-any, video-to-video, other
Track-Anything: Optimized for Qualcomm Devices
Track-Anything is a video based machine learning model to track an object in a video.
This is based on the implementation of Track-Anything found here. This repository contains pre-exported model files optimized for Qualcomm® devices. You can use the Qualcomm® AI Hub Models library to export with custom configurations. More details on model performance across various devices, can be found here.
Qualcomm AI Hub Models uses Qualcomm AI Hub Workbench to compile, profile, and evaluate this model. Sign up to run these models on a hosted Qualcomm® device.
Getting Started
There are two ways to deploy this model on your device:
Option 1: Download Pre-Exported Models
Below are pre-exported model assets ready for deployment.
| Runtime | Precision | Chipset | SDK Versions | Download |
|---|---|---|---|---|
| ONNX | float | Universal | QAIRT 2.42, ONNX Runtime 1.24.1 | Download |
| TFLITE | float | Universal | QAIRT 2.43, TFLite 2.17.0 | Download |
For more device-specific assets and performance metrics, visit Track-Anything on Qualcomm® AI Hub.
Option 2: Export with Custom Configurations
Use the Qualcomm® AI Hub Models Python library to compile and export the model with your own:
- Custom weights (e.g., fine-tuned checkpoints)
- Custom input shapes
- Target device and runtime configurations
This option is ideal if you need to customize the model beyond the default configuration provided here.
See our repository for Track-Anything on GitHub for usage instructions.
Model Details
Model Type: Model_use_case.video_object_tracking
Model Stats:
- Model checkpoint: xmem
- Input resolution: 320x568
- Number of parameters (TrackAnythingEncodeKeyWithShrinkage): 9.72M
- Model size (TrackAnythingEncodeKeyWithShrinkage) (float): 37.1 MB
- Number of parameters (TrackAnythingEncodeValue): 23.3M
- Model size (TrackAnythingEncodeValue) (float): 88.8 MB
- Number of parameters (TrackAnythingEncodeKeyWithoutShrinkage): 9.71M
- Model size (TrackAnythingEncodeKeyWithoutShrinkage) (float): 37.1 MB
- Number of parameters (TrackAnythingSegment): 30.1M
- Model size (TrackAnythingSegment) (float): 115 MB
Performance Summary
| Model | Runtime | Precision | Chipset | Inference Time (ms) | Peak Memory Range (MB) | Primary Compute Unit |
|---|---|---|---|---|---|---|
| TrackAnythingEncodeKeyWithShrinkage | ONNX | float | Snapdragon® X2 Elite | 2.824 ms | 18 - 18 MB | NPU |
| TrackAnythingEncodeKeyWithShrinkage | ONNX | float | Snapdragon® X Elite | 5.571 ms | 19 - 19 MB | NPU |
| TrackAnythingEncodeKeyWithShrinkage | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 3.67 ms | 0 - 203 MB | NPU |
| TrackAnythingEncodeKeyWithShrinkage | ONNX | float | Qualcomm® QCS8550 (Proxy) | 5.166 ms | 0 - 20 MB | NPU |
| TrackAnythingEncodeKeyWithShrinkage | ONNX | float | Qualcomm® QCS9075 | 9.491 ms | 9 - 12 MB | NPU |
| TrackAnythingEncodeKeyWithShrinkage | ONNX | float | Snapdragon® 8 Elite For Galaxy Mobile | 2.941 ms | 0 - 166 MB | NPU |
| TrackAnythingEncodeKeyWithShrinkage | ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | 2.378 ms | 1 - 176 MB | NPU |
| TrackAnythingEncodeKeyWithShrinkage | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 4.086 ms | 0 - 213 MB | NPU |
| TrackAnythingEncodeKeyWithShrinkage | TFLITE | float | Qualcomm® QCS8275 (Proxy) | 31.193 ms | 3 - 181 MB | NPU |
| TrackAnythingEncodeKeyWithShrinkage | TFLITE | float | Qualcomm® QCS8550 (Proxy) | 5.947 ms | 3 - 26 MB | NPU |
| TrackAnythingEncodeKeyWithShrinkage | TFLITE | float | Qualcomm® SA8775P | 41.728 ms | 3 - 181 MB | NPU |
| TrackAnythingEncodeKeyWithShrinkage | TFLITE | float | Qualcomm® QCS9075 | 11.551 ms | 2 - 29 MB | NPU |
| TrackAnythingEncodeKeyWithShrinkage | TFLITE | float | Qualcomm® QCS8450 (Proxy) | 11.767 ms | 3 - 198 MB | NPU |
| TrackAnythingEncodeKeyWithShrinkage | TFLITE | float | Qualcomm® SA7255P | 31.193 ms | 3 - 181 MB | NPU |
| TrackAnythingEncodeKeyWithShrinkage | TFLITE | float | Qualcomm® SA8295P | 10.376 ms | 3 - 165 MB | NPU |
| TrackAnythingEncodeKeyWithShrinkage | TFLITE | float | Snapdragon® 8 Elite For Galaxy Mobile | 3.32 ms | 0 - 180 MB | NPU |
| TrackAnythingEncodeKeyWithShrinkage | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 2.439 ms | 0 - 191 MB | NPU |
| TrackAnythingEncodeKeyWithoutShrinkage | ONNX | float | Snapdragon® X2 Elite | 3.939 ms | 65 - 65 MB | NPU |
| TrackAnythingEncodeKeyWithoutShrinkage | ONNX | float | Snapdragon® X Elite | 6.966 ms | 66 - 66 MB | NPU |
| TrackAnythingEncodeKeyWithoutShrinkage | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 5.069 ms | 64 - 276 MB | NPU |
| TrackAnythingEncodeKeyWithoutShrinkage | ONNX | float | Qualcomm® QCS8550 (Proxy) | 6.815 ms | 58 - 78 MB | NPU |
| TrackAnythingEncodeKeyWithoutShrinkage | ONNX | float | Qualcomm® QCS9075 | 11.463 ms | 61 - 63 MB | NPU |
| TrackAnythingEncodeKeyWithoutShrinkage | ONNX | float | Snapdragon® 8 Elite For Galaxy Mobile | 4.356 ms | 61 - 253 MB | NPU |
| TrackAnythingEncodeKeyWithoutShrinkage | ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | 3.492 ms | 0 - 216 MB | NPU |
| TrackAnythingEncodeKeyWithoutShrinkage | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 5.366 ms | 19 - 241 MB | NPU |
| TrackAnythingEncodeKeyWithoutShrinkage | TFLITE | float | Qualcomm® QCS8275 (Proxy) | 33.133 ms | 20 - 204 MB | NPU |
| TrackAnythingEncodeKeyWithoutShrinkage | TFLITE | float | Qualcomm® QCS8550 (Proxy) | 7.514 ms | 18 - 20 MB | NPU |
| TrackAnythingEncodeKeyWithoutShrinkage | TFLITE | float | Qualcomm® SA8775P | 11.394 ms | 20 - 206 MB | NPU |
| TrackAnythingEncodeKeyWithoutShrinkage | TFLITE | float | Qualcomm® QCS9075 | 12.953 ms | 20 - 63 MB | NPU |
| TrackAnythingEncodeKeyWithoutShrinkage | TFLITE | float | Qualcomm® QCS8450 (Proxy) | 14.159 ms | 20 - 222 MB | NPU |
| TrackAnythingEncodeKeyWithoutShrinkage | TFLITE | float | Qualcomm® SA7255P | 33.133 ms | 20 - 204 MB | NPU |
| TrackAnythingEncodeKeyWithoutShrinkage | TFLITE | float | Qualcomm® SA8295P | 12.751 ms | 20 - 186 MB | NPU |
| TrackAnythingEncodeKeyWithoutShrinkage | TFLITE | float | Snapdragon® 8 Elite For Galaxy Mobile | 4.391 ms | 20 - 206 MB | NPU |
| TrackAnythingEncodeKeyWithoutShrinkage | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 3.553 ms | 0 - 226 MB | NPU |
| TrackAnythingEncodeValue | ONNX | float | Snapdragon® X2 Elite | 6.037 ms | 39 - 39 MB | NPU |
| TrackAnythingEncodeValue | ONNX | float | Snapdragon® X Elite | 12.514 ms | 38 - 38 MB | NPU |
| TrackAnythingEncodeValue | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 9.904 ms | 0 - 206 MB | NPU |
| TrackAnythingEncodeValue | ONNX | float | Qualcomm® QCS8550 (Proxy) | 15.636 ms | 0 - 45 MB | NPU |
| TrackAnythingEncodeValue | ONNX | float | Qualcomm® QCS9075 | 20.008 ms | 6 - 14 MB | NPU |
| TrackAnythingEncodeValue | ONNX | float | Snapdragon® 8 Elite For Galaxy Mobile | 8.398 ms | 7 - 175 MB | NPU |
| TrackAnythingEncodeValue | ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | 6.63 ms | 2 - 168 MB | NPU |
| TrackAnythingEncodeValue | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 8.311 ms | 2 - 292 MB | NPU |
| TrackAnythingEncodeValue | TFLITE | float | Qualcomm® QCS8275 (Proxy) | 51.388 ms | 3 - 225 MB | NPU |
| TrackAnythingEncodeValue | TFLITE | float | Qualcomm® QCS8550 (Proxy) | 11.88 ms | 0 - 2 MB | NPU |
| TrackAnythingEncodeValue | TFLITE | float | Qualcomm® SA8775P | 17.427 ms | 3 - 219 MB | NPU |
| TrackAnythingEncodeValue | TFLITE | float | Qualcomm® QCS9075 | 18.655 ms | 0 - 55 MB | NPU |
| TrackAnythingEncodeValue | TFLITE | float | Qualcomm® QCS8450 (Proxy) | 22.491 ms | 3 - 284 MB | NPU |
| TrackAnythingEncodeValue | TFLITE | float | Qualcomm® SA7255P | 51.388 ms | 3 - 225 MB | NPU |
| TrackAnythingEncodeValue | TFLITE | float | Qualcomm® SA8295P | 19.789 ms | 3 - 233 MB | NPU |
| TrackAnythingEncodeValue | TFLITE | float | Snapdragon® 8 Elite For Galaxy Mobile | 7.135 ms | 3 - 200 MB | NPU |
| TrackAnythingEncodeValue | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 6.274 ms | 1 - 210 MB | NPU |
| TrackAnythingSegment | ONNX | float | Snapdragon® X2 Elite | 15.069 ms | 38 - 38 MB | NPU |
| TrackAnythingSegment | ONNX | float | Snapdragon® X Elite | 23.479 ms | 37 - 37 MB | NPU |
| TrackAnythingSegment | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 17.456 ms | 25 - 270 MB | NPU |
| TrackAnythingSegment | ONNX | float | Qualcomm® QCS8550 (Proxy) | 23.435 ms | 0 - 45 MB | NPU |
| TrackAnythingSegment | ONNX | float | Qualcomm® QCS9075 | 36.822 ms | 21 - 45 MB | NPU |
| TrackAnythingSegment | ONNX | float | Snapdragon® 8 Elite For Galaxy Mobile | 13.542 ms | 5 - 216 MB | NPU |
| TrackAnythingSegment | ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | 10.474 ms | 26 - 239 MB | NPU |
| TrackAnythingSegment | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 17.924 ms | 0 - 290 MB | NPU |
| TrackAnythingSegment | TFLITE | float | Qualcomm® QCS8275 (Proxy) | 103.816 ms | 2 - 198 MB | NPU |
| TrackAnythingSegment | TFLITE | float | Qualcomm® QCS8550 (Proxy) | 23.485 ms | 2 - 4 MB | NPU |
| TrackAnythingSegment | TFLITE | float | Qualcomm® SA8775P | 33.692 ms | 2 - 202 MB | NPU |
| TrackAnythingSegment | TFLITE | float | Qualcomm® QCS9075 | 38.353 ms | 0 - 83 MB | NPU |
| TrackAnythingSegment | TFLITE | float | Qualcomm® QCS8450 (Proxy) | 54.127 ms | 2 - 285 MB | NPU |
| TrackAnythingSegment | TFLITE | float | Qualcomm® SA7255P | 103.816 ms | 2 - 198 MB | NPU |
| TrackAnythingSegment | TFLITE | float | Qualcomm® SA8295P | 38.674 ms | 2 - 209 MB | NPU |
| TrackAnythingSegment | TFLITE | float | Snapdragon® 8 Elite For Galaxy Mobile | 13.286 ms | 1 - 210 MB | NPU |
| TrackAnythingSegment | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 10.788 ms | 0 - 226 MB | NPU |
License
- The license for the original implementation of Track-Anything can be found here.
References
Community
- Join our AI Hub Slack community to collaborate, post questions and learn more about on-device AI.
- For questions or feedback please reach out to us.
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support