Cohere-10M and 144.1M-vector AWS EC2 benchmarks.
Two reproducible evaluations on AWS EC2: single-node 16-vCPU Zilliz VectorDBBench (Cohere-10M, 768-d) in RAM and Direct NVMe modes, and 4-node horizontal scale across all 21 Wikipedia languages (144,137,311 unique 1024-d vectors, 288.3M replicas at RF=2).
1. Single-node Cohere-10M (Official VectorDBBench)
Executed with unmodified upstream zilliztech/VectorDBBench on a single 16-vCPU AWS EC2 instance (10M × 768-d, Cosine, RF=1, 8 local shards) in both 64 GiB RAM (m6id.4xlarge) and 32 GiB Direct NVMe (c6id.4xlarge) modes.
| VectorDBBench Case | 16c64g RAM (m6id.4xlarge) | 16c32g NVMe (c6id.4xlarge) | Serial p99 / p95 | Recall@100 | Official 16c64g Baselines |
|---|---|---|---|---|---|
| Performance768D10M (Unfiltered, Load + Search) | 441.6 QPS (11.1m load) | 452.2 QPS (11.3m load) | 6.2 / 5.7 ms | 0.9833 | 2.4–15.7h load • 136–146 QPS unmerged HNSW |
| Performance768D10M1P (Int Filter 1% • 99% Match) | 313.4 QPS | 320.9 QPS | 8.3 / 7.8 ms | 0.9836 | 9.9M matching docs |
| Performance768D10M99P (Int Filter 99% • 1% Match) | 1,122.8 QPS | 1,133.0 QPS | 3.2 / 3.1 ms | 0.9582 | 100K matching docs on 32 GiB NVMe node |
| LabelFilterPerformanceCase (1.0% Scalar Label) | 504.2 QPS (11.3m load) | 511.2 QPS (11.1m load) | 5.6 / 5.3 ms | 0.9653 | Qdrant 273 • OpenSearch 115 • Milvus 76 • Pinecone 32 QPS |
| StreamingPerformanceCase (100K @ 1,000 writes/s) | 1,668.6 live / 4,708 post | 1,797.1 live / 4,463 post | 2.4 / 1.8 ms | 0.9854 | Qdrant 348 • Milvus 156 QPS • Pinecone 5,992 ms p99 |
2. 144.1M-vector topologies, ingest speed & monthly cost
Full 21-language Upstash/wikipedia-2024-06-bge-m3 corpus (144,137,311 unique 1024-d vectors, 288,274,622 replicas at RF=2, 1.18 TB raw float32) indexed across two 4-node AWS EC2 fleets with zero post-load segment compaction stalls, compared against published VectorDBBench ingest rates and HNSW memory requirements.
| System & Fleet | Cluster RAM (RSS) | Load + Index Time | Unique Docs / sec | AWS EC2 / Cloud Cost |
|---|---|---|---|---|
| MassEngine In-Memory (4 × r6id.8xlarge, 128 vCPUs, RF=2) | 510.1 GiB (127.5 GiB/node) | 15.0 min (144.1M measured) | 160,129 docs/s | $6,685 / mo EC2 |
| MassEngine Direct NVMe (4 × c6id.4xlarge, 64 vCPUs, RF=2) | 62.7 GiB (15.7 GiB/node) | 17.9 min (144.1M measured) | 134,219 docs/s | $2,053 / mo EC2 |
| TurboPuffer (VectorDBBench Cohere-10M rate) | Object storage + cache | 61.6m / 10M (~14.8h est.) | 2,705 docs/s | Usage-metered |
| Milvus 16c64g-sq8 (VectorDBBench rate) | ~1,850 GiB est. at 288M | 2.38h / 10M (~34.3h est.) | 1,168 docs/s | ~$13,370 / mo (8 × r6id.8xlarge) |
| Qdrant 16c64g-payload-index (VectorDBBench rate) | ~1,850 GiB est. at 288M | 3.97h / 10M (~57.3h est.) | 699 docs/s | ~$13,370 / mo (8 × r6id.8xlarge) |
| Pinecone p2.x8 / OpenSearch 16c128g (VectorDBBench rate) | Managed pods / ~1,850 GiB | 7.9–10.2h / 10M (4.8–6d est.) | 273–351 docs/s | $13,370–$20,000+ / mo |
3. Search latency, streaming writes & failover (144.1M vectors)
Every query regime below was verified against exact brute-force FP32 ground truth across all 144,137,311 records, including concurrent write streams and hard data-node termination (RF=2).
| Workload | Cluster Tier | Measured Throughput | Latency (p50 / p99) | Recall@10 / SLA |
|---|---|---|---|---|
| Filtered 1.42% (2.05M matching docs) | In-Memory (4 × r6id.8xlarge) | 3,937.8 QPS peak (3,736.6 @ c=16) | 1.29 / 2.41 ms (c=1) | 0.9478 |
| Filtered 1.42% (2.05M matching docs) | Direct NVMe (4 × c6id.4xlarge) | 2,114.6 QPS peak (2,069.6 @ c=16) | 1.52 / 3.53 ms (c=1) | 0.9480 |
| Filtered 33.1% (47.7M English docs) | In-Memory (4 × r6id.8xlarge) | 563.8 QPS (c=32) • 200.6 QPS (c=1) | 4.20 / 13.65 ms (c=1) | 0.9703 |
| Unfiltered Top-10 (Fast / Balanced) | In-Memory (4 × r6id.8xlarge) | 383.3 QPS fast • 204.8 QPS balanced | 9.16 / 28.11 ms (balanced c=1) | 0.9456 / 0.9774 |
| Unfiltered Top-10 (High / Max Recall) | In-Memory (4 × r6id.8xlarge) | 139.9 QPS high • 103.4 QPS max (c=32) | 213.3 / 439.4 ms (high c=32) | 0.9867 / 0.9894 |
| Streaming Writes + 32 Concurrent Readers | In-Memory (4 × r6id.8xlarge) | 24,300 writes/s + 129.3 Read QPS | 234.0 / 551.6 ms (c=32) | 48,600 replicas/s |
| Hard Node Kill Failover (kill -9, RF=2) | In-Memory & Direct NVMe | 0 failed queries during failover | 3.4s NVMe / 43.3s RAM restart | 100.0% availability |
4. Architectural tradeoffs
No single index structure is optimal for every access pattern. Here is where MassEngine outperforms graph indexes and where a static pre-merged HNSW graph holds an advantage.
Where MassEngine wins
- Metadata-filtered & multi-tenant search: Selective filters break HNSW graph connectivity and waste distance hops on non-matching nodes. MassEngine evaluates compact filter bitmaps first, accelerating to 2,115–3,938 QPS under 2 ms as filters tighten.
- Fast ingestion & live streaming writes: HNSW requires random graph-edge mutations and multi-hour background segment compactions. MassEngine indexes in parallel streams (11.1 min for 10M; 15.0 min for 144.1M) and sustains 24,300 writes/sec under concurrent reads.
- Memory footprint & VPC cost: Serving 288.3M 1024-d replicas takes 510 GiB RAM in-memory or 62.7 GiB RAM in Direct NVMe mode ($2,053/mo EC2), compared to ~1.8 TiB RAM ($13,370+/mo) for uncompressed HNSW.
Where static pre-merged HNSW wins
- Write-frozen, unfiltered 1M–10M datasets: If your dataset rarely changes, your queries never apply metadata filters, and you can afford a 2.5 to 15-hour blocking force-merge into a single RAM-resident HNSW segment, HNSW visits fewer candidates per unfiltered query. Once live writes resume, unmerged segments form and HNSW QPS drops back down.
- Choosing RAM vs. NVMe Mode above 100M vectors: Direct NVMe mode on c6id.4xlarge is ideal for filtered search (2,115 QPS at 1.52 ms p50) and high-speed ingestion at a fraction of RAM cost. For workloads dominated by 100% unfiltered queries at 100M+ scale, use In-Memory mode (r6id.8xlarge) so full-precision reranking stays in RAM.