| | --- |
| | tags: |
| | - vector-database |
| | - benchmarks |
| | - faiss |
| | - weaviate |
| | - chroma |
| | - multimodal |
| | - clip |
| | - retrieval |
| | license: apache-2.0 |
| | --- |
| | |
| | # Vector Database Benchmarks: FAISS vs Chroma vs Weaviate |
| |
|
| | This repository contains experiments benchmarking popular vector databases on **multimodal embeddings** generated from the [Flickr8k dataset](https://huggingface.co/datasets/jxie/flickr8k). |
| | We focused on four key evaluation dimensions: |
| |
|
| | 1. **Latency per query** |
| | 2. **Recall@5 vs Flat (accuracy tradeoffs)** |
| | 3. **Queries per second (QPS throughput)** |
| | 4. **Ingestion scaling performance** |
| |
|
| | All experiments were run on **Google Colab** (T4 GPU for embedding generation, CPU backend for databases). |
| |
|
| | --- |
| |
|
| | ## Methodology |
| |
|
| | - Dataset: 6k images and 30k captions from Flickr8k. |
| | - Embeddings: CLIP (OpenAI ViT-B/32). |
| | - Workload: Caption-to-image retrieval (cross-modal). |
| | - Baseline: FAISS Flat index used as the ground-truth for recall calculations. |
| |
|
| | Each vector database was tested under the same conditions for ingestion, search, and recall. |
| |
|
| | --- |
| |
|
| | ## Results Summary |
| |
|
| | | Metric | FAISS | Chroma | Weaviate | |
| | |--------------------------|------------------|------------------|------------------| |
| | | **Avg Latency per Query** | 0.19 ms | 0.76 ms | 1.82 ms | |
| | | **Recall@5 (Flat Baseline)** | 1.00 | 0.002 | 0.918 | |
| | | **QPS Throughput** | 1929.94 | 719.01 | 598.40 | |
| | | **Ingestion Scaling (20k)** | 0.024s | 2.806s | 4.000s | |
| |
|
| |
|
| |  |
| |
|
| | --- |
| |
|
| | ## Key Takeaways |
| |
|
| | - **FAISS** is fastest, leveraging in-memory array ingestion and customizable indexing strategies. |
| | - **Chroma** offers simplicity and ease of integration but struggles at scale due to batching and internal constraints. |
| | - **Weaviate** provides a more feature-rich ecosystem (schema, hybrid search, persistence) but at higher ingestion and query overhead. |
| |
|
| | At the million-vector scale, speed alone will not decide your choice; **engineering tradeoffs, developer productivity, and system features** will. |
| | Benchmarks tell one part of the story, your use case tells the rest. |
| |
|
| | --- |
| |
|
| | ## Usage |
| |
|
| | You can reproduce these experiments using the provided notebook and Hugging Face dataset. |
| | See full code here: [rag-experiments/VectorDB-Benchmarks](https://huggingface.co/rag-experiments/VectorDB-Benchmarks). |
| | Dataset used: Flickr8k (train split — 6k images, 30k captions, multimodal — images and text), CLIP Embeddings. Dataset Author: Johnathan Xie |
| |
|
| | --- |
| |
|
| | ## Citation |
| |
|
| | If you find this useful, please cite this repository: |
| |
|
| |
|
