Local Embedding Models in 2026: Which to Run, What Hardware They Need, and What They Cost
*Published 13 October 2026 · Model cards, runtime docs and API prices checked on 13 October 2026* A local embedding model turns your text into vectors on hardware you control, so no document or query leaves your machine to get embedded. As of 13 October 2026, the one to start with on a GPU is [Qwen3-Embedding-0.6B](huggingface.co) (Apache 2.0, 32K-token context, 1,024 dimensions, 2.49 GiB of peak VRAM in a published test), and the one to start with on a laptop CPU is [EmbeddingGemma 2](huggingface.co), which Google released on 6 October under Apache 2.0 with a 270M-parameter text backbone and an 8,192-token context. If you have a 16 GB GPU and quality is the only thing you care about, run Qwen3-Embedding-8B. If you are on a Raspberry Pi or prototyping, all-MiniLM-L6-v2 still works, and it is also the model most likely to be holding your retrieval back. Picking the model is the easy decision, though, and the ranking guides from Local AI Zone and AIMultiple stop there. Local setups go wrong in four other places: a missing query prefix that silently costs retrieval quality, a context limit that silently truncates your chunks, a vector index four times bigger than it needed to be, and a model switch that forces you to re-embed the whole corpus. Every figure below comes from a model card, a runtime's own documentation, or a benchmark with its hardware named, all checked on 13 October 2026. Leaderboard ranks for open and hosted models are in our [embedding model comparison from 26 September](maximem.ai). ## Which local embedding model to run: Qwen3-Embedding-0.6B on a GPU, EmbeddingGemma 2 on a laptop Pick by the hardware you have, then by language and licence. The table below covers the models worth running locally today; parameter counts, context limits and licences come from each model's card or from the spec table in [Atomic Chat's 25 September benchmark](atomic.chat), and peak VRAM comes from that same benchmark (one NVIDIA RTX A6000, mostly bfloat16). | Model | Licence | Params | Dimensions | Max tokens | Peak VRAM (A6000) | Ollama download | Best for | |---|---|---|---|---|---|---|---| | all-MiniLM-L6-v2 | Apache 2.0 | 22.7M | 384 | 256 | 0.24 GiB | 46 MB | Prototypes, edge devices | | nomic-embed-text v1.5 | Apache 2.0 | 136.7M | 768 (to 64) | 8,192 | 0.47 GiB | 274 MB | Fast CPU default, long chunks | | EmbeddingGemma 2 (text) | Apache 2.0 | 270M text backbone, 740M with all encoders | 768 (to 128) | 8,192 | not yet measured | 270m to 740m tags | Laptops and phones, multimodal | | EmbeddingGemma-300M (v1) | Gemma terms | 300M class | 768 (to 128) | 2,048 | 0.96 GiB | 622 MB | Superseded by v2 for most uses | | BGE-M3 | MIT | 567.8M | 1,024 plus sparse and multi-vector | 8,192 | 1.30 GiB | ~1.2 GB | Multilingual and hybrid search | | Qwen3-Embedding-0.6B | Apache 2.0 | 595.8M | 1,024 (to 32) | 32K | 2.49 GiB | 639 MB | Default on any GPU | | Qwen3-Embedding-4B | Apache 2.0 | 4B | 2,560 | 32K | not measured | 2.5 GB | Mid-size GPUs | | Qwen3-Embedding-8B | Apache 2.0 | 7.57B | 4,096 (to 32) | 32K | 16.60 GiB | 4.7 GB (Q4_K_M) | Highest quality you can host | Ollama download sizes come from the [Ollama library](ollama.com) and [Morph's Ollama guide](morphllm.com), and they are quantized files, which is why they sit well below the bfloat16 VRAM figures. The Qwen3-Embedding-8B download grows to 8.0 GB at q8_0 and 15 GB at fp16. On quality, Atomic Chat's run on the full [BEIR](beir-cellar/beir) SciFact set (5,183 documents, 300 queries) put Qwen3-Embedding-8B first at 0.7953 nDCG@10, EmbeddingGemma-300M second at 0.7861, nomic-embed-text v1.5 at 0.7156, Qwen3-Embedding-0.6B at 0.7011, BGE-M3 at 0.6510 and a BM25 keyword baseline at 0.6402. Two cautions apply to that run: it is English only and dense only, and the authors did not record batch size, library versions or model revisions, and reported no significance test. Treat close scores as ties. ### EmbeddingGemma 2 brings an 8K context and Apache 2.0 to the laptop tier [EmbeddingGemma 2](blog.google) is the release that changes the laptop tier. Google published it on 6 October 2026 as a 740M-parameter model that maps text, code, images, video and audio into one embedding space, and the part that matters for local use is that you can load only the 270M text backbone; Google reports about 191 MB of active RAM for the quantized text-only weights on a Pixel 11 Pro and about 567 MB for the full multimodal model. The context window grew from 2,048 tokens in version 1 to 8,192, Google reports MTEB Code rising from 68.76 to 78.68, and it already runs in [Ollama](ollama.com), [llama.cpp](ggml-org/llama.cpp), [sentence-transformers](sbert.net), [MLX](ml-explore/mlx), [vLLM](vllm-project/vllm) and [transformers.js](huggingface.co) in the browser. No independent retrieval benchmark of version 2 existed when we checked, so the SciFact score quoted above (0.7861) is for version 1. ### Apache 2.0 and MIT cover the strong local models, with exceptions to check Qwen3-Embedding, EmbeddingGemma 2 and nomic-embed-text v1.5 are Apache 2.0, and BGE-M3 is MIT, so all of them can go into a commercial product. EmbeddingGemma version 1 sits under Google's Gemma terms and its prohibited-use policy, which is the main practical reason to move to version 2, since version 2 changed to Apache 2.0. Read the card before you ship anything else: some strong open-weight models, such as jina-embeddings-v5-text-small, are CC BY-NC 4.0 and cannot be used commercially without a separate licence. ### Published model lists often compare MTEB scores that are not comparable Morph's Ollama guide quotes MTEB scores from different tracks side by side and then warns that they cannot be compared directly. MTEB English v1, English v2, Multilingual and the BEIR retrieval-only figures are different test sets. [Pinecone's guide](pinecone.io) adds the second caution: self-reported scores can be inflated when a model has been trained on data close to the benchmark. On the MTEB leaderboard, sort by the retrieval score instead of the overall average, compare models on the same track, and let a sample of your own queries make the final call. ## Most local embedding models need under 2.5 GiB of VRAM; Qwen3-Embedding-8B needs 16.60 GiB You need much less than a chat model needs. Every measured model in the table except Qwen3-Embedding-8B peaked under 2.5 GiB of VRAM at bfloat16, and the quantized Ollama files for those smaller models are about 1.2 GB or less, so a laptop with 8 GB of RAM runs any of them. Qwen3-Embedding-8B needed 16.60 GiB of VRAM at bfloat16 in Atomic Chat's run, which means a 24 GB card at full precision or the 4.7 GB Q4_K_M file on a smaller one. CPU-only is a real option for sub-billion models. Our own retrieval study ran without a discrete GPU: all-MiniLM-L6-v2 embedded 50,000 documents in 161.6 seconds (309 documents per second) on an Apple M4 with 16 GB of RAM through [ChromaDB](trychroma.com), in a single run on 14 January 2026. Parameter count is a poor guide to speed, though: in Atomic Chat's run EmbeddingGemma-300M indexed slower than the larger BGE-M3 (315.0 against 433.0 chunks per second), so measure throughput on your own hardware. Indexing time is where model size shows up. Using Atomic Chat's measured throughput on one RTX A6000 (chunks of about 250 words), here is how long one million chunks would take: | Model | Chunks per second (A6000) | Time for 1M chunks | |---|---|---| | all-MiniLM-L6-v2 | 1,074.7 | about 16 minutes | | nomic-embed-text v1.5 | 727.3 | about 23 minutes | | BGE-M3 | 433.0 | about 38 minutes | | EmbeddingGemma-300M | 315.0 | about 53 minutes | | Qwen3-Embedding-0.6B | 169.9 | about 1.6 hours | | Qwen3-Embedding-8B | 26.1 | about 10.6 hours | Atomic Chat did not record batch size, so read these times as order of magnitude; a tuned batch can move them noticeably. Choosing between the 0.6B and 8B Qwen models is a trade: 0.0942 nDCG@10 on SciFact in exchange for an indexing job about six and a half times longer and nearly seven times the peak VRAM. ## Ollama is the fastest way to get local vectors; sentence-transformers, llama.cpp, TEI and FastEmbed suit other stacks Five runtimes cover almost every setup, and the choice depends on what the rest of your stack already uses. [Ollama](docs.ollama.com) is the shortest path from nothing to vectors. Pull a model, then call the `/api/embed` endpoint, which takes a single string or an array of strings and returns an `embeddings` array along with a `prompt_eval_count` of tokens processed: ```bash ollama pull qwen3-embedding:0.6b curl localhost -d '{ "model": "qwen3-embedding:0.6b", "input": ["first chunk of text", "second chunk of text"], "truncate": false }' ``` Use `/api/embed`, not the older `/api/embeddings`, which takes a `prompt` field, returns one vector and does not batch. Ollama also serves an OpenAI-compatible `/v1/embeddings` endpoint that accepts a `dimensions` parameter, so most OpenAI client code points at it with a base-URL change. [Sentence-transformers](sbert.net) is the Python default for anything on [Hugging Face](huggingface.co), and it also ships the `quantize_embeddings` helper for shrinking stored vectors: ```python from sentence_transformers import SentenceTransformer model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B") doc_vecs = model.encode(chunks, batch_size=64, normalize_embeddings=True) query_vecs = model.encode(queries, prompt_name="query", normalize_embeddings=True) ``` Qwen3-Embedding needs sentence-transformers 2.7.0 or later and transformers 4.51.0 or later, according to its card; older transformers versions fail with `KeyError: 'qwen3'`. [llama.cpp](ggml-org/llama.cpp) runs GGUF files with the lowest overhead. Start `llama-server -m model.gguf --embeddings --pooling mean` (match `--pooling` to the method the model was trained with, which its card states) and it serves an OpenAI-compatible `/v1/embeddings` endpoint with normalized vectors. [Text Embeddings Inference](huggingface.co) (TEI) from Hugging Face is the production server, shipped as a Docker image for GPUs and buildable for x86 CPUs or Apple Metal, and its own CPU guide uses Qwen3-Embedding-0.6B as the example model. [FastEmbed](qdrant.github.io) from [Qdrant](qdrant.tech) runs models on ONNX Runtime without PyTorch, which makes it the lighter choice for serverless functions and small containers. ### A missing query prefix lowers retrieval quality without any error Most modern embedding models expect a different prefix on queries than on documents (Qwen3's card calls it an instruction), and nothing fails when you leave it out; retrieval simply gets worse. [nomic-embed-text v1.5](huggingface.co) states that every input must start with `search_document: ` or `search_query: `. Qwen3-Embedding wants queries as `Instruct: {task description}\nQuery:{query}` and documents with no instruction, and its card credits that prefix with a 1% to 5% gain on most tasks. EmbeddingGemma uses `task: search result | query: {query}` for queries and `title: none | text: {content}` for documents. Morph points out that the [LangChain](langchain.com) and [LlamaIndex](llamaindex.ai) Ollama wrappers do not add these prefixes for you, and reports that a 5 to 10 point gap between published and observed retrieval quality often traces back to that. Write the prefix into your own embedding function so it cannot be forgotten. ### Ollama truncates chunks past the context limit unless you set `truncate: false` Context limits range from 256 tokens for all-MiniLM-L6-v2 and 512 for mxbai-embed-large and multilingual-e5-large to 8,192 for nomic, BGE-M3 and EmbeddingGemma 2 and 32K for Qwen3-Embedding. Anything longer is cut off without an error by default; Ollama's `truncate` parameter defaults to `true`, and setting it to `false` turns overflow into an error you can catch. Ollama also loads nomic-embed-text with a shorter context than the model supports, so set `num_ctx` to 8192 if your chunks are long. To catch silent truncation, compare the `prompt_eval_count` Ollama returns against your own token count for each chunk. ## A million 1,024-dimension vectors take 4.1 GB, and quantization cuts that 4x to 32x Raw storage is the number of vectors times the dimensions times four bytes for float32. For one million chunks, that is 1.5 GB at 384 dimensions, 3.1 GB at 768, 4.1 GB at 1,024 and 16.4 GB at 4,096, before the extra memory your vector database spends on its search graph. Atomic Chat's SciFact index shows the same ratio in practice: 38.1 MiB for 1,024-dimension models against 152.5 MiB for Qwen3-Embedding-8B's 4,096. Matryoshka truncation and vector quantization both cut that down, and you can use both. Matryoshka-trained models let you keep only the first part of each vector. Qwen3-Embedding-8B scored 0.7960 on SciFact at 4,096 dimensions, 0.7905 at 1,024 and 0.7672 at 256 in Atomic Chat's tests, so cutting to 1,024 dimensions gives up 0.0055 nDCG@10 for a quarter of the storage; nomic-embed-text v1.5 drops from 62.28 to 61.04 MTEB average going from 768 to 256 dimensions, by Morph's table. Re-normalize after truncating, and use the same dimension for queries and documents. Quantizing the stored vectors shrinks them further. Hugging Face's [embedding quantization study](huggingface.co) (March 2024, mxbai-embed-large-v1 on 15 MTEB retrieval datasets) found int8 vectors are 4x smaller and keep about 97% of retrieval quality, rising to about 99% when the top candidates are rescored with full-precision query vectors, while binary vectors are 32x smaller and keep about 92.5%, or about 96% with rescoring. At the scale of their worked example, 250 million 1,024-dimension vectors take 953.67 GB as float32 and 29.80 GB as binary. Sentence-transformers ships the `quantize_embeddings` helper for both. ## A local embedding model is rarely much cheaper than an API: embedding 1M chunks once on text-embedding-3-small costs about $6.66 Local saves less than the API price suggests, and cost is rarely the best reason to go local. [OpenAI's text-embedding-3-small](developers.openai.com) costs $0.02 per million tokens and text-embedding-3-large $0.13. Suppose you have one million chunks of about 330 tokens each, which is 333 million tokens: embedding all of them once costs about $6.66 on the small model and about $43.29 on the large one. A local GPU does the same job for the price of electricity, but only if the GPU already exists, and somebody still owns the server and the on-call rota. Four situations tip the arithmetic toward local: you re-index often (every chunking change means paying again on an API), your query volume is high enough that per-call latency and cost add up, you need the exact same vectors a year from now, or the text cannot leave your network. Reproducibility and privacy matter most. A local model file gives identical embeddings for as long as you keep it, while a hosted model can change versions behind the same name. If the text cannot leave your network, an API is ruled out before cost comes up. Local is the wrong call when your volume is small and nobody on the team wants to run inference. It is also wrong when your hardware cannot hold a model good enough for your hardest queries, since multi-hop questions and rare jargon are where the largest hosted models still pull ahead, as [AI/TLDR](ai-tldr.dev) notes. Run both on a labelled sample of your own queries before deciding; it takes an afternoon. ## Switching embedding models means re-embedding everything Vectors from different models live in different spaces, even when the dimensions match, so you cannot query a Qwen3 index with an EmbeddingGemma query vector or mix the two in one collection. Changing models means re-embedding every document, and the indexing table above is your cost estimate; for a million chunks the move from all-MiniLM-L6-v2 to Qwen3-Embedding-8B goes from about 16 minutes to about 10.6 hours on one A6000. A migration without downtime is ordinary engineering. Write new documents to both indexes, backfill the new index in batches, compare recall on a labelled query set while reads still go to the old one, switch reads once the new index wins, and keep the old index until the rollback window passes. Store the model name, revision, prefix, dimension and normalization next to every vector, because that metadata is the only record of how the old vectors were made. ## The embedding model is often not your retrieval bottleneck, so test recall first Keyword search beat vector search in our own study once a single code dataset was taken out, which is a reason to measure before you swap models. In our [50,000-document retrieval study](maximem.ai), vector search led keyword search across five datasets on MRR@10 by 0.6320 to 0.5325, but almost all of that lead came from one natural-language-to-code dataset; remove it and keyword search led the remaining four, 0.5931 to 0.5614, and on science exam questions keyword search scored 0.8145 against 0.6142. The setup was all-MiniLM-L6-v2 at 384 dimensions, [Tantivy](quickwit-oss/tantivy) 0.22.0 and ChromaDB on an Apple M4, 5,000 queries scored by exact document-ID matching with no LLM judge, in one run on 14 January 2026. A stronger embedder would narrow keyword search's lead on those four datasets. It would not remove the class of query where the exact term (a product code, a drug name, an error string, a ticket ID) is the whole point, which is why hybrid search, with BGE-M3's sparse output or a separate keyword index, belongs in most local setups. Rerankers solve a narrower problem. A cross-encoder reranker reorders the candidates the embedder already found, so it helps when the right chunk sits in the top 50 but not the top 5, and it cannot recover a chunk the first stage missed. Finding your bottleneck takes about ten lines of code: label which chunk answers each of 50 to 100 real queries, embed them with two candidate models using the correct prefixes, and measure how often the right chunk lands in the top 10. ```python import numpy as np from sentence_transformers import SentenceTransformer def recall_at_k(model_id, chunks, queries, gold, k=10, query_prompt=None): m = SentenceTransformer(model_id) d = m.encode(chunks, normalize_embeddings=True, batch_size=64) q = m.encode(queries, prompt_name=query_prompt, normalize_embeddings=True) top = np.argsort(-q @ d.T, axis=1)[:, :k] return float(np.mean([g in row for g, row in zip(gold, top)])) ``` If both models land within 0.02 of each other (two percentage points of recall), the model is not your problem; look at chunking and query prefixes first. ## Running embeddings locally keeps text private, but the vectors can still leak it Running the model yourself removes a third-party processor from the embedding step, which shortens your list of sub-processors and keeps raw text inside your network. It does not make the vectors safe to treat as anonymous. In [Morris and colleagues' 2023 EMNLP paper](aclanthology.org), an inversion method recovered 92% of 32-token inputs exactly from their embeddings and recovered full names from a dataset of clinical notes. So the vector store needs the same access controls and residency rules as the documents it was built from, and a deletion request has to reach the vectors as well as the source rows. ## Where Maximem Synap fits: agent memory needs more than similarity A local embedding model finds passages that are similar to a query. For agent memory, similarity search leaves four questions open: it cannot tell which of two stored facts is current, that "my manager" in one conversation and "Sarah" in another are the same person, which user a fact belongs to, or when a fact should stop being returned at all. [Maximem Synap](maximem.ai) is a managed memory layer that takes care of those parts, and the embedding step runs on Maximem's own infrastructure, so text is not sent to a third party for it. Teams that need everything inside their own network can deploy Synap self-hosted or air-gapped on enterprise plans. If your retrieval feeds an agent that has to remember users across sessions, run the recall test above on your own queries first; it tells you whether the embedder or the memory around it is the part to fix. ## Frequently asked questions ### What is the best local embedding model in 2026? As of 13 October 2026, Qwen3-Embedding-0.6B is the best default if you have a GPU, Qwen3-Embedding-8B scored highest on SciFact in Atomic Chat's test and fits on one 24 GB GPU (16.60 GiB of VRAM at bfloat16), and EmbeddingGemma 2 is the strongest candidate for laptops and multimodal search, though no independent retrieval benchmark of it existed on that date. All three are Apache 2.0. Run the recall test above on two of them with 50 real queries before committing. ### Can I run an embedding model on CPU only? Yes. Sub-billion models such as all-MiniLM-L6-v2, nomic-embed-text v1.5 and EmbeddingGemma 2 run comfortably on a laptop CPU; all-MiniLM-L6-v2 embedded 50,000 documents in 161.6 seconds on an Apple M4 with no discrete GPU in our study. Keep the 4B and 8B Qwen3 models for machines with a GPU. ### What is the best Ollama embedding model? For most RAG work, use `qwen3-embedding:0.6b` (639 MB) if quality matters and `nomic-embed-text` (274 MB) if CPU speed matters; `embeddinggemma-2` is now the small multimodal option. Call `/api/embed`, add the model's query and document prefixes yourself, and set `num_ctx` for nomic if your chunks are long. ### How much storage do embeddings take? One million float32 vectors take 4.1 GB at 1,024 dimensions and 16.4 GB at 4,096, plus your vector database's index overhead. Int8 quantization makes that 4x smaller. Binary makes it 32x smaller, and Matryoshka truncation cuts it further on models trained for it. ### Do I need to re-embed my data if I change embedding models? Yes. Vectors from different models are not comparable even at the same dimension, so every document and query has to be embedded with the new model. Plan it as a backfill into a second index, and budget the time from your model's measured throughput. ### Are local embeddings private? The raw text stays on your hardware, which removes a third party from that step. The vectors themselves can be partly inverted back into text, so protect and delete them the way you would the original documents. ## Sources All retrieved 13 October 2026. - [Qwen3-Embedding-0.6B model card](huggingface.co) - [EmbeddingGemma 2 announcement, Google, 6 October 2026](blog.google) and [model card](huggingface.co) - [EmbeddingGemma-300M model card](huggingface.co) - [nomic-embed-text v1.5 model card](huggingface.co) - [Best Local Embedding Models for RAG: 7 Models Compared, Atomic Chat, 25 September 2026](atomic.chat) - [Best Ollama Embedding Models 2026, Morph, 9 June 2026](morphllm.com) - [Ollama embedding model library](ollama.com) and [/api/embed reference](docs.ollama.com) - [llama.cpp embeddings guide](mintlify.wiki) - [Text Embeddings Inference, local CPU guide](huggingface.co) - [FastEmbed documentation](qdrant.github.io) - [Binary and Scalar Embedding Quantization, Hugging Face, 22 March 2024](huggingface.co) - [OpenAI text-embedding-3-small model page](developers.openai.com) - [Choosing an Embedding Model, Pinecone](pinecone.io) - [Best Local Embedding Models to Run Without an API, AI/TLDR, 13 June 2026](ai-tldr.dev) - [Text Embeddings Reveal (Almost) As Much As Text, Morris et al., EMNLP 2023](aclanthology.org) - [File Search vs Vector Search for RAG, Maximem](maximem.ai)
Mentions
hello and happy november folks! anyone wanna help me test typeahead.waow.tech? i’ve swapped in tantivy (quickwit-oss/tantivy) for the search engine and initial results promising (~4x faster, smaller archive, same resource reqs) but...
should be fixed! i swapped the index to use quickwit-oss/tantivy so it should be about 4x faster and also cover these multi term cases i was missing lmk if you see anything goofy still
Quickwit is a search engine for logs and traces, written in Rust. Search indexes stored on S3, Azure Blob or Google Cloud Storage, and scale compute separately from storage. 11k+ stars quickwit-oss/quickwit
10億件の日本語全文検索、Quickwitならほぼ3秒以内・月額6万円台だった話 セーフィーの白柳です。普段はバックエンドからフロントエンド、インフラ構築などを幅広く行ってソリューションを開発しています。業務で大量のテキストを全文検索する方法を検討する機会があり、それが今回の検証のきっかけです。 この記事は、Elasticsearch / OpenSearch などの全文検索エンジンを触ったことがあるエンジニアを想定して書いています。Quickwit の前提知識は不要で、仕組みは本文の「Quickwit の仕組みをざっくり」で簡単に説明します。 忙しい人のための結論 「10億件の日本語テキストを全文検索したい。でも OpenSearch は高い」というところから、Quickwit を AWS 上で検証しました。結果はこうです。 項目 結果 データ 1件約1KBのプレーンテキスト × 10億件（約1TB） 月額費用 約$438 （約6.6万円。1ドル150円換算） 固定費 約$298 + 変動費 約$140 インデックス作成のスループット 2vCPU/8GBのままで約2,000件/秒 8vCPU/16GB + Lambda 300並列なら約20,000件/秒 インデックスサイズ 約592GB（元データ約1TBより小さい） 検索レイテンシー 毎回違うキーワードで p50 約0.5秒、ほぼ3秒以内 絞り込みありでも p99 は1秒を超える （10億件、Fargate 4vCPU/8GB、API Gateway 経由の往復） 目標は「3か月で10億件 = 平均129件/秒、ピークは平均の5倍の645件/秒」「絞り込みがあれば1秒以内、なくても3秒以内」でした。インデックス作成は余裕でクリアしました。検索は、3秒以内は「ほぼ」クリアしましたが、絞り込み時の1秒以内は一部未達です。 3秒以内の「ほぼ」の中身はこうです。1件ずつ検索するなら960件中959件が3秒以内で、残る1件も3.4秒でした。ただし5並列で同時に検索すると、p99 で3秒を超えます（最大3.9秒）。また、ヒット件数は正確な総数ではなく概数を返す設定にしています。Fargate を 8vCPU/16GB にすれば5並列も含めて全件3秒以内に収まりましたが、月額が約$180上がります。 絞り込み時の1秒以内は、p90 ではどの絞り込みでも達成しました。ただし p99 では、テナント指定で1.6秒、テナント+店舗で2.4秒と1秒を超えます。全件1秒以内に収まったのは、期間を1日に絞った場合だけでした。 どうして検証しようと思ったか 全文検索を実現する方法を議論したとき、無難に Elasticsearch / OpenSearch だろうけど、10億件となると高いよね〜という話になりました。 改めて調べてみると、S3 にインデックスを置いて検索できる、サーバーレスでも動くという全文検索エンジン Quickwit を見つけました。 ただ会議では「まぁでも怖いよね〜、性能もよくわからないしね〜、やっぱり ES かね〜」で終わりました。 それでも気になったので、自分で検証することにしました。 検証したワークロード 1件約1KBのプレーンテキストを 3か月で10億件、継続的に投入 平均129件/秒、ピークは平均の5倍の645件/秒に耐えられること 検索はテキスト中の商品名などのキーワード 絞り込みがあれば1秒以内、なくても3秒以内 最悪10億件すべてが対象 絞り込み用のメタデータは、3階層のIDと発行日時 Quickwit の仕組みをざっくり この記事に出てくる Quickwit の用語は次のとおりです。 split: インデックスを分けて保存する単位のファイルです。一定の件数ごとに作られて S3 に置かれ、検索も split ごとに行います Indexer: 投入されたデータを受け取り、split を作って S3 に書き出すノードです。受け取ったデータは、split になるまでの間、ローカルディスクの WAL（先行書き込みログ）に溜めます。小さい split をまとめて大きな split にするマージも、Indexer が行います Searcher: 検索リクエストを受け、S3 上の split を読んで検索するノードです。状態を持たないので、台数を増減しやすくなっています メタストア: どの split にどの期間のデータが入っているかなどを管理する DB です。今回は RDS（PostgreSQL）を使っています 今回の検証では、Indexer と Searcher を Fargate の1タスクにまとめて動かしました。 サーバーレスで行けると思いきや、1インスタンス必要だった 最初は、Lambda + S3 だけの完全サーバーレスで組めると思っていました。 ところが、最新版（v0.9.0）では常駐するノードが最低1台必要でした。 v0.9.0 で何が変わったのか リリースノートから、今回の話に関係するところを抜き出します。 Ingest V2 がデフォルトに: /api/v1/{index}/ingest が Ingest V2 で動くようになった AWS Lambda のサポート: 追加されたが、オフロードできるのは検索（leaf search）だけ メタストアの自動移行: 0.8.x からの初回書き込み時に移行され、ロールバックにはバックアップが必要 ここで効いてくるのが Ingest V2 です。投入したデータは、まず Indexer ノード（ingester）が持つローカルディスク上の WAL（先行書き込みログ）に積まれ、そこから split として S3 に出ていきます。コード上も、Ingester は WAL のディレクトリパスとディスク容量を受け取って起動する作りで、そのディスク容量には設定の max_queue_disk_usage が渡されています。max_queue_disk_usage は、設定の説明にある WAL のディスク使用量の上限（既定 4GiB）です。つまり、データを受け取ってから S3 に書き出すまでの間、ローカルディスクを持った常駐ノードが必要になります。 一方、Lambda 対応でオフロードできるのは検索だけで、インデックス作成は Lambda で完結しません。結局、検索の一部は Lambda に逃がせるものの、インデックス作成の面倒を見る常駐ノードは必要というのが v0.9.0 の現状で、完全なサーバーレスにはなりませんでした。 一瞬、意地でもサーバーレスにしようかとも思いましたが、せっかくの検証なので公式の構成に従うことにしました。公式ドキュメントの Cluster sizing では、検証や小規模な PoC で全サービスを1台に載せる場合、最低 2コア / メモリ8GB を推奨しています。今回は検証なので、まずこの構成（ECS Fargate 1タスク）で始めました。なお、同じページでは、本番などでサービスを分ける場合のメモリの目安として、Indexer は1コアあたり4GB（8GB未満は避ける）、S3 を使う Searcher は1コアあたり8GB を挙げています。 まずは Analyzer の設計 日本語なので、デフォルトのままというわけにはいかないだろうと思い、定番の以下を検討しました。 正規化 形態素解析 2-gram シノニム展開 ところが Quickwit 自体はシンプルで、これらをプラグインで足していく仕組みではありませんでした。そのため、自前のプログラムでデータを加工してからインデックス作成・検索することにしました。 最終的な変換は次のとおりです（インデックス作成時も検索時も同じ処理をかけます）。 正規化: NFKC + 小文字化（全角/半角、大文字/小文字を吸収） 形態素解析: SudachiPy の正規化形（送り仮名・かな漢字の表記違いを辞書の代表表記へ） トークン化: 半角英数字の連続（1000ml など）は1語、それ以外は2文字ずつスライドした2-gramにして、空白区切りで連結 Quickwit 側は whitespace トークナイザーで record: position にして、検索はフレーズクエリ（body:"欠き き氷 ..."）で発行します。 キーワード トークン 結果 かき氷バー 欠き / き氷 / 氷バ / バー ヒット（表記が違っても正規化形が揃う） 氷バー 氷バ / バー ヒット（語の途中からでも一致する） 牛乳 1000ml 牛乳 / 1000ml ヒット（牛乳1000ml と書いても同じ。牛乳 500ml には当たらない） カキゴオリバー カキ / キゴ / ゴオ / ... ヒットしない（別の語として解析されるため） こおり 氷（1文字） ヒットしない（2-gram にならないため） 英数字を2-gramにしないのは、検索を速くするためです（後述）。表記ゆれの吸収には限界がありますが、それはシノニム展開をやめたことによる割り切りです。 シノニム展開はやめた シノニム展開は、クエリが「大量のキーワードを OR でつないだもの」になるので遅くなります。 2vCPU/8GB、絞り込みあり（テナント・店舗・1か月）、単発実行で「かき氷バー」を検索すると、シノニム展開なしでは0.722秒、ありでは1.663秒と、2倍以上遅くなりました。 「バー」が「バーニングプロダクション」にまで展開されるなど、検索意図と関係ない語が混ざるのも効いていました。 今回の用途では表記ゆれ吸収の需要がそこまでないとメンバーから聞けたので、シノニム展開は外しました。 ダミーデータを作りながらインデックス作成のスループットを測定 インデックス作成時には正規化・形態素解析・トークン化の処理が必要なので、ダミーデータの生成とこれらの前処理を行いながら Quickwit に投入しました。ダミーデータは商品名の表記ゆれ（かな / 半角カナ / 長音脱落）を混ぜた1件約1KBのもので、負荷生成用の EC2 から、エラーが出ない最大の14並列で Indexer に接続して投入しています。 すでにこの時点で相当速く、2vCPU/8GB のままで平均 約2,000件/秒 出ました。 平均は1,993件/秒で、Indexer の CPU 使用率は平均58%（最大96%）、メモリ使用率は10%前後でした。 目標のピーク645件/秒の3倍です。 split のマージでスループットが落ちる ただ、しばらくすると split のマージが走り、スループットが低下しました。 これは merge_concurrency: 1 でマージに割り当てるコア数を制限して、事なきを得ました。 indexer: merge_concurrency: 1 もっといじめてみた もっといじめたくなりました。SQS に 1ドキュメント = 1メッセージで積み、batch_size: 100 で Lambda を起動して、前処理から Quickwit への投入までを任せます。Lambda の同時実行は 300（上限）で並列処理させました。 さすがに 2vCPU/8GB では無理だろうと、Fargate を 8vCPU/16GB に上げたところ、さらにすごいスループットが出ました。 構成 平均スループット Indexer CPU 2vCPU/8GB（EC2から14並列で直接投入） 約2,000件/秒 平均58% / 最大96% 8vCPU/16GB + Lambda（1メッセージ=1件、batch_size なし） 約3,046件/秒 最大29% 8vCPU/16GB + Lambda（batch_size: 100） 約20,000件/秒（瞬間ピーク 約32,000件/秒） 平均41〜53% / 最大65% Lambda を1メッセージずつ起動していた間は、同時実行が上限に達し、SQS のポーリングも振動して約3,000件/秒止まりでした。バッチ内の全件を前処理して1回の NDJSON POST で投入するようにしたところ、同じ同時実行数のまま平均約7倍、最大約11倍になり、約2万件/秒に届きました。目標のピーク645件/秒の 平均約31倍、最大約50倍 です。負荷生成は c6i.4xlarge の EC2 から行っています。 代償もあります。バッチ化すると300並列が1ノードに集中して 503 no shards available が散発しました（300万件中38件）。ただ、SQS のリトライで全件成功し、DLQ は0件でした。 Lambda の同時実行数は、アカウント上限が400で、予約できるのは300まででした。上限緩和の申請はめんどくさくて、ここまでしか検証していません。ただ Indexer の CPU 使用率には余裕があったので、本当はもっと出せそうです。 一括投入なら約7.4万件/秒 Quickwit の ingest API 経由では、Indexer 1台が受けられる量に上限があります。8vCPU/16GB の Indexer に EC2 から直接 POST しても、WAL への書き込み（persist）のロック待ちで約2.6万件/秒が頭打ちでした。 検証データの一括投入には quickwit tool local-ingest を使いました。EC2 の中で split を作って S3 に置き、メタストア（RDS）に直接登録するコマンドで、Indexer を経由しません。 時間順に並んだ400万件を1回の local-ingest に流し込み、split を1つずつ作ります。これを6並列で動かしました。最初から最終サイズの split を作るので、あとからマージする必要がありません。1つの split を一度に組み立てられるのは約490万件までで、それを超えると勝手に分割されて1件失敗するので、1回あたり400万件にしています。 c7a.8xlarge（32vCPU）1台で 約7.4万件/秒で投入できました。 インデックスのサイズと圧縮効率 10億件で、インデックスの合計サイズは 約592GB でした。元データは約1TB（1件約1KB × 10億件）なので、元データを大きく下回りました。 1件あたりで見ると、split の圧縮サイズが約592バイトです。前処理後（トークン化後）の投入ペイロードは非圧縮で約2.0KBまで膨らんでいるので、それが約1/3に圧縮された計算になります。 検索のレイテンシー 計測条件 10億件（400万件 × 250 split）、Fargate 4vCPU/8GB API Gateway → Lambda → Quickwit の往復時間 毎回違うキーワード（商品名192種を1回ずつ）、各条件192件 partial_request_cache_capacity: 0、count_all=false、上位10件 同じキーワードを繰り返し投げて測るのはおすすめしません。Quickwit の Searcher がリクエストをまたいで持つキャッシュは、split footer（hotcache）、fast field、partial request cache（split 単位の検索結果）の3つです。このうち partial request cache は「split + クエリ + max_hits + 期間」がキーなので、同じキーワードの連投だけがヒットして数msで返り、実際よりずっと速く見えます。ポスティングリスト自体は、リクエストの中でしか保持されません。毎回違うキーワードで検索する実運用ではほぼヒットしないので、無効にして測っています。 結果 条件 p50 p90 p99 最大 絞り込みなし 0.46秒 0.78秒 1.65秒 2.13秒 テナント指定 0.46秒 0.75秒 1.60秒 2.24秒 テナント+店舗 0.57秒 0.94秒 2.39秒 3.37秒 期間1週間 0.37秒 0.56秒 1.05秒 1.39秒 期間1日 0.26秒 0.33秒 0.44秒 0.63秒 5並列（絞り込みなし） 0.94秒 2.26秒 3.65秒 3.86秒 1件ずつなら、960件中959件が3秒以内でした。一番遅かったのは「テナント+店舗」の3.37秒です。ヒットが少なくなる検索ほど、上位10件がなかなか揃わずに多くの split を読むので、遅くなります。 絞り込み時に1秒以内という目標に対しては、p90 まではどの絞り込みでも1秒以内でしたが、p99 では期間1日以外の条件で1秒を超えました。 5並列で同時に検索すると、p99で3秒を超えます。同時アクセスが多い用途なら、8vCPU/16GB にすれば5並列でも全件3秒以内（最大2.98秒）に収まります。 Fargate のタスクを入れ替えた直後の最初の1件は、エラーになることがあります。起動直後に何件か検索を流して温めておくのが無難です。 遅さを決めているのは「1回の検索で S3 から読む量」 Quickwit の /metrics（quickwit_storage_object_storage_download_num_bytes）の差分を見ると、1回の検索で S3 から何バイト読んだかが分かります。CPU にはかなり余裕があり（4vCPU で約30%）、遅い検索は S3 から読む量とネットワーク帯域で決まっていました。 英数字も2-gramにすると、1000ml は 10 / 00 / 00 / 0m / ml になります。「00」や「ml」は金額や容量としてほぼ全レシートに出てくるので、フレーズ検索のためにその出現位置の情報を split ごとに丸ごと読むことになります。10億件で「牛乳 1000ml」をテナントと店舗で絞り込むと、1回で S3 から約7.3GB を読み、8vCPU の Fargate でも毎秒約1.2GB（約10Gbps）で頭打ちになりました。 効いた設定 効いた順に並べます。 1. 英数字の連続は1語にする 1000ml を1語として索引すれば、ほとんど出てこない語になります。同じ期間の1億件分で比べると、S3 から読む量は1/9〜1/150になりました。 キーワード（テナント+店舗の絞り込み） 英数字も2-gram 英数字の連続は1語 牛乳 1000ml 3.02秒 / 2,634MB 0.29秒 / 17MB 缶ビール 350ml 2.10秒 / 1,767MB 0.28秒 / 54MB トイレットペーパー 4ロール 2.05秒 / 1,668MB 0.53秒 / 181MB 代わりに、数字の部分一致（「100」で「1000ml」を探す）はできなくなります。今回の用途では不要なので、割り切りました。 2. ヒット件数を数え切らない（count_all=false） Quickwit は既定で、ヒット件数の正確な総数を数えるために、全 split を最後まで検索します。count_all=false にすると、上位10件が揃った時点で残りの split を打ち切ります。英数字も2-gramにした10億件・2vCPU/8GB でテナント指定の検索を比べると、中央値が7.9秒から0.6秒になりました。代わりに、返ってくるヒット件数は「打ち切った時点までの件数」になり、正確な総数ではなくなります。 3. Fargate のサイズを上げる 遅い検索は S3 からの読み込み（ネットワーク帯域）で決まり、Fargate はタスクのサイズでネットワーク帯域も変わります。上の結果は4vCPU/8GB で、8vCPU/16GB にすると5並列も含めて全件3秒以内になりました。2vCPU は、この設定では測っていません。 効かなかったこと 同時に検索する split 数（max_num_concurrent_split_searches）は、32から64に増やしても変わらず、160にすると 8vCPU/16GB でもメモリ不足で落ちる。メモリの少ない 4vCPU/8GB では、もっと低い値で落ちる可能性がある 新しい順の並べ替え（sort_by）を足すと、ヒットした全件の日時を読むので、大幅に遅くなる split のサイズを揃えたり小さくしたりする方法は、試していません。split を小さくすると、期間による読み飛ばしや count_all=false の打ち切りが細かく効く一方で、S3 への GET の回数は増えます。そのため、効果は未検証です。 月額費用の内訳（ap-northeast-1） 固定費 リソース 設定 月額 Fargate 4vCPU/8GB $179.93 RDS（インスタンス） db.t4g.micro $18.25 RDS（ストレージ） gp2 20GB $2.76 S3 約592GB $14.80 VPC エンドポイント Interface 型4種 × 2AZ $81.76 合計 $297.50 Fargate が6割を占めています。VPC エンドポイントは、Fargate がインターネットに出ずにイメージ取得（ECR）やログ送信（CloudWatch Logs）、シークレット取得（Secrets Manager）をするためのもので、意外と効いてきます。 変動費（リクエスト課金） 上の固定費にはリクエスト課金が入っていません。10億件を3か月で投入する場合の概算です。 項目 3か月合計 API Gateway（1,000万リクエスト） 約$13 Lambda ingest-enqueue 約$35 Lambda ingest-worker 約$250 SQS 約$120 S3 リクエスト（約170万 PUT） 約$8 合計 約$420（月あたり約$140） 固定費の約$298にこの変動費の約$140（3か月の月平均）を足して、月額は約$438（約6.6万円）です。8vCPU/16GB にすると、約$617（約9.3万円）になります。 検索は100万回あたり、API Gateway が約$1.3、Lambda が約$8.5です。 比較: OpenSearch で商用構成を組むといくらか 比較のために、同じ10億件・約1TB を Amazon OpenSearch Service で商用向けに組んだ場合の月額も、ざっくり見積もってみました。実際に構築して測ったものではなく、以下の前提を置いた概算です。 3AZ 構成で、レプリカは2（各シャードが3つの AZ に1つずつ載る）。AWS が本番向けに推奨する構成（Multi-AZ with Standby）の最小で、3AZ・専用マスター3台・データノード3の倍数台・レプリカ2以上が条件 データノードは AWS が本番向けに推奨する最小の台数、3AZ × 1 = 3台（r6g.2xlarge.search、8vCPU / 64GB）。レプリカ2なので、各ノードが全データのコピーを1つずつ持つ 専用マスターノードは、これも推奨どおり m6g.large.search を3台 インデックスサイズは元データの約1.2倍（約1.2TB）と仮定し、レプリカ込みの約3.6TB に空き容量を見込んで、EBS（gp3）は合計約4.5TB（1ノードあたり1.5TB） 料金は東京リージョンのオンデマンド（RI や Savings Plans なし）で、1ドル150円換算 リソース 構成 月額（概算） データノード r6g.2xlarge.search × 3 約$1,850 専用マスターノード m6g.large.search × 3 約$400 EBS gp3 約4.5TB 約$620 合計 約$2,900（約44万円） 幅を持たせると、月額 40〜50万円 くらいです。 Quickwit も3AZ構成にして比べる ここまでの Quickwit の約$438は Fargate 1台なので、AZ 障害には耐えられません。OpenSearch と条件を揃えるため、Quickwit も3AZ に1タスクずつ置く構成で見積もり直します。 構成 月額（概算） Quickwit（1タスク、AZ障害には非対応） Fargate 4vCPU/8GB × 1 + RDS + S3 + VPC エンドポイント 約$438（約6.6万円） Quickwit（3AZ） Fargate 4vCPU/8GB × 3 + RDS Multi-AZ + S3 + VPC エンドポイント（3AZ） 約$860（約12.9万円） OpenSearch（3AZ、推奨最小構成） r6g.2xlarge.search × 3 + 専用マスター × 3 + EBS 約$2,900（約44万円） Quickwit の3AZ版は、Fargate を3AZ に1タスクずつ置いて（AZ が1つ落ちても残りの2タスクで動かせる）$540、RDS を Multi-AZ にして（インスタンスもストレージも2倍）$42、S3 は標準クラスならもともと複数 AZ に冗長化されるのでそのままの$15、VPC エンドポイントを3AZ 分にして$123、これに変動費の約$140を足した合計です。 同じ3AZ で比べても、Quickwit は OpenSearch の 約1/3 です。 この差には理由があります。OpenSearch はインデックスをノードのローカル（EBS）に持つので、ストレージがレプリカ分だけ増え、検索性能のためにメモリと CPU を積んだノードを常に3AZ に並べておく必要があります。一方 Quickwit は、インデックスを S3 に1コピーだけ置けば耐久性が足りるので、ストレージも計算資源も小さく済みます。 ただし、Quickwit の3AZ構成は今回の検証では動かしていません。複数ノードのクラスタ構成や、インデックス作成時の WAL の複製（Ingest V2 のレプリケーション設定）を含めて、実際にはこの見積もり以上の検討や追加費用が必要になる可能性があります。また、OpenSearch のほうが機能は豊富で、実績もあります。ヒット件数を正確に数える、数字の部分一致で探すといった要件があるなら、今回のチューニングはそのまま使えません。 なお、料金の単価は概算です。実際に検討するときは AWS Pricing Calculator で、東京リージョンの最新の単価で見積もってください。 筆者の独り言 また公式が、完全なサーバーレス構成に対応してくれたりしないかな、と思っています。Indexer / Searcher を Lambda だけで動くようにして、そのうえで S3 を Metastore にしたりして、完全サーバーレスみたいな。まぁ、複数のノードが動く中での書き込みの競合とかを考えると難しいんだろうけど……。 やっぱりサーバーレスにはあこがれちゃいますよね。 まとめ Quickwit は、Fargate 4vCPU/8GB の1台で、1件約1KB・10億件の日本語全文検索を、毎回違うキーワードでもほぼ3秒以内（中央値約0.5秒）でこなしてくれました。ただし、絞り込み時に1秒以内という目標は、p99 では届きませんでした。月額は約$438（約6.6万円）で、OpenSearch で同じ規模を組むことを考えると、かなり魅力的だと思います。 また、そのまま使えるわけではありません。完全なサーバーレスにはならず、投入されたデータを受ける常駐ノードが必要です。日本語は自前の前処理（正規化・形態素解析・トークン化）が必要で、インデックス作成時と検索時に同じ処理をかける必要があります。そして何より、同じキーワードを繰り返して測ると、キャッシュのおかげで実際よりずっと速く見えます。毎回違うキーワードで測ってみて、1回の検索で S3 から何バイト読んでいるかを見るのがおすすめです。 「怖いよね〜、性能もよくわからないしね〜」で終わらせず、手を動かして測ってみてよかったです。Quickwit が気になっている方の参考になれば幸いです。
CÔNG CỤ TÌM KIẾM DỮ LIỆU ĐÁM MÂY TỐC ĐỘ DƯỚI 1 GIÂY TRÊN S3: QUICKWIT VIẾT BẰNG RUST THÁCH THỨC ELASTICSEARCH Elasticsearch rất mạnh mẽ nhưng đòi hỏi dữ liệu phải được lưu trên ổ cứng cục bộ (Local SSD) đắt đỏ và ngốn bộ nhớ RAM khổng lồ để duy trì chỉ mục. Khi khối lượng logs và dữ liệu phân tích cán mốc hàng trăm terabyte, chi phí lưu trữ phần cứng trở thành một gánh nặng tài chính khổng lồ. Quickwit là công cụ tìm kiếm và phân tích dữ liệu phân tán mã nguồn mở (Distributed Search Engine) được viết hoàn toàn bằng Rust, được thiết kế chuyên biệt để lập chỉ mục và tìm kiếm trực tiếp trên dịch vụ lưu trữ đám mây (Amazon S3, Google Cloud Storage, MinIO). Tech Stack: Được xây dựng 100% bằng Rust trên nền tảng thư viện tìm kiếm Tantivy (tương đương Lucene của thế giới Rust). Tương thích với Elasticsearch Query API và Jaeger gRPC API. Tại sao nên dùng? - Tách rời tính toán và lưu trữ (Decoupled Compute & Storage): Lưu trữ 100% dữ liệu và chỉ mục trên object storage giá rẻ, các node tìm kiếm có thể co giãn (auto-scale) linh hoạt theo nhu cầu mà không lo mất mát dữ liệu. - Tốc độ tìm kiếm sub-second trên S3: Nhờ kỹ thuật lập chỉ mục thông minh và bộ nhớ đệm đa tầng, Quickwit có thể thực hiện các câu truy vấn phức tạp xuyên suốt hàng petabyte dữ liệu trên S3 với thời gian phản hồi dưới 1 giây. - Thay thế hoàn hảo cho Elasticsearch trong bài toán Log Management: Hỗ trợ nạp dữ liệu trực tiếp từ Kafka, Vector, Fluentbit và tích hợp trực tiếp với Grafana làm Data Source. Dự án sở hữu hơn 8.500 stars trên GitHub và đang là giải pháp tìm kiếm dữ liệu lớn được nhiều tập đoàn công nghệ lựa chọn để cắt giảm 80% chi phí vận hành. Link tham khảo: quickwit-oss/quickwit #OpenSourceVN #VNOSS #opensourcevn #vnoss #DevCommunity #Quickwit #Rust #SearchEngine #BigData #S3 #ElasticsearchAlternative
tantivy ⭐ 16k Tantivy is a full-text search engine library that Rust developers can embed in their applications. 🔗 quickwit-oss/tantivy #opensource t.co
If u use Tantivy/Quickwit, please patch, since there is a RCE: github.com/quickwit-oss/tantivy/security/advisori… - also credits to Kyle Elliot for finding this initially
Local Embedding Models in 2026: Which to Run, What Hardware They Need, and What They Cost *Published 13 October 2026 · Model cards, runtime docs and API prices checked on 13 October 2026* A local embedding model turns your text into vectors on hardware you control, so no document or query leaves your machine to get embedded. As of 13 October 2026, the one to start with on a GPU is [Qwen3-Embedding-0.6B](huggingface.co) (Apache 2.0, 32K-token context, 1,024 dimensions, 2.49 GiB of peak VRAM in a published test), and the one to start with on a laptop CPU is [EmbeddingGemma 2](huggingface.co), which Google released on 6 October under Apache 2.0 with a 270M-parameter text backbone and an 8,192-token context. If you have a 16 GB GPU and quality is the only thing you care about, run Qwen3-Embedding-8B. If you are on a Raspberry Pi or prototyping, all-MiniLM-L6-v2 still works, and it is also the model most likely to be holding your retrieval back. Picking the model is the easy decision, though, and the ranking guides from Local AI Zone and AIMultiple stop there. Local setups go wrong in four other places: a missing query prefix that silently costs retrieval quality, a context limit that silently truncates your chunks, a vector index four times bigger than it needed to be, and a model switch that forces you to re-embed the whole corpus. Every figure below comes from a model card, a runtime's own documentation, or a benchmark with its hardware named, all checked on 13 October 2026. Leaderboard ranks for open and hosted models are in our [embedding model comparison from 26 September](maximem.ai). ## Which local embedding model to run: Qwen3-Embedding-0.6B on a GPU, EmbeddingGemma 2 on a laptop Pick by the hardware you have, then by language and licence. The table below covers the models worth running locally today; parameter counts, context limits and licences come from each model's card or from the spec table in [Atomic Chat's 25 September benchmark](atomic.chat), and peak VRAM comes from that same benchmark (one NVIDIA RTX A6000, mostly bfloat16). | Model | Licence | Params | Dimensions | Max tokens | Peak VRAM (A6000) | Ollama download | Best for | |---|---|---|---|---|---|---|---| | all-MiniLM-L6-v2 | Apache 2.0 | 22.7M | 384 | 256 | 0.24 GiB | 46 MB | Prototypes, edge devices | | nomic-embed-text v1.5 | Apache 2.0 | 136.7M | 768 (to 64) | 8,192 | 0.47 GiB | 274 MB | Fast CPU default, long chunks | | EmbeddingGemma 2 (text) | Apache 2.0 | 270M text backbone, 740M with all encoders | 768 (to 128) | 8,192 | not yet measured | 270m to 740m tags | Laptops and phones, multimodal | | EmbeddingGemma-300M (v1) | Gemma terms | 300M class | 768 (to 128) | 2,048 | 0.96 GiB | 622 MB | Superseded by v2 for most uses | | BGE-M3 | MIT | 567.8M | 1,024 plus sparse and multi-vector | 8,192 | 1.30 GiB | ~1.2 GB | Multilingual and hybrid search | | Qwen3-Embedding-0.6B | Apache 2.0 | 595.8M | 1,024 (to 32) | 32K | 2.49 GiB | 639 MB | Default on any GPU | | Qwen3-Embedding-4B | Apache 2.0 | 4B | 2,560 | 32K | not measured | 2.5 GB | Mid-size GPUs | | Qwen3-Embedding-8B | Apache 2.0 | 7.57B | 4,096 (to 32) | 32K | 16.60 GiB | 4.7 GB (Q4_K_M) | Highest quality you can host | Ollama download sizes come from the [Ollama library](ollama.com) and [Morph's Ollama guide](morphllm.com), and they are quantized files, which is why they sit well below the bfloat16 VRAM figures. The Qwen3-Embedding-8B download grows to 8.0 GB at q8_0 and 15 GB at fp16. On quality, Atomic Chat's run on the full [BEIR](beir-cellar/beir) SciFact set (5,183 documents, 300 queries) put Qwen3-Embedding-8B first at 0.7953 nDCG@10, EmbeddingGemma-300M second at 0.7861, nomic-embed-text v1.5 at 0.7156, Qwen3-Embedding-0.6B at 0.7011, BGE-M3 at 0.6510 and a BM25 keyword baseline at 0.6402. Two cautions apply to that run: it is English only and dense only, and the authors did not record batch size, library versions or model revisions, and reported no significance test. Treat close scores as ties. ### EmbeddingGemma 2 brings an 8K context and Apache 2.0 to the laptop tier [EmbeddingGemma 2](blog.google) is the release that changes the laptop tier. Google published it on 6 October 2026 as a 740M-parameter model that maps text, code, images, video and audio into one embedding space, and the part that matters for local use is that you can load only the 270M text backbone; Google reports about 191 MB of active RAM for the quantized text-only weights on a Pixel 11 Pro and about 567 MB for the full multimodal model. The context window grew from 2,048 tokens in version 1 to 8,192, Google reports MTEB Code rising from 68.76 to 78.68, and it already runs in [Ollama](ollama.com), [llama.cpp](ggml-org/llama.cpp), [sentence-transformers](sbert.net), [MLX](ml-explore/mlx), [vLLM](vllm-project/vllm) and [transformers.js](huggingface.co) in the browser. No independent retrieval benchmark of version 2 existed when we checked, so the SciFact score quoted above (0.7861) is for version 1. ### Apache 2.0 and MIT cover the strong local models, with exceptions to check Qwen3-Embedding, EmbeddingGemma 2 and nomic-embed-text v1.5 are Apache 2.0, and BGE-M3 is MIT, so all of them can go into a commercial product. EmbeddingGemma version 1 sits under Google's Gemma terms and its prohibited-use policy, which is the main practical reason to move to version 2, since version 2 changed to Apache 2.0. Read the card before you ship anything else: some strong open-weight models, such as jina-embeddings-v5-text-small, are CC BY-NC 4.0 and cannot be used commercially without a separate licence. ### Published model lists often compare MTEB scores that are not comparable Morph's Ollama guide quotes MTEB scores from different tracks side by side and then warns that they cannot be compared directly. MTEB English v1, English v2, Multilingual and the BEIR retrieval-only figures are different test sets. [Pinecone's guide](pinecone.io) adds the second caution: self-reported scores can be inflated when a model has been trained on data close to the benchmark. On the MTEB leaderboard, sort by the retrieval score instead of the overall average, compare models on the same track, and let a sample of your own queries make the final call. ## Most local embedding models need under 2.5 GiB of VRAM; Qwen3-Embedding-8B needs 16.60 GiB You need much less than a chat model needs. Every measured model in the table except Qwen3-Embedding-8B peaked under 2.5 GiB of VRAM at bfloat16, and the quantized Ollama files for those smaller models are about 1.2 GB or less, so a laptop with 8 GB of RAM runs any of them. Qwen3-Embedding-8B needed 16.60 GiB of VRAM at bfloat16 in Atomic Chat's run, which means a 24 GB card at full precision or the 4.7 GB Q4_K_M file on a smaller one. CPU-only is a real option for sub-billion models. Our own retrieval study ran without a discrete GPU: all-MiniLM-L6-v2 embedded 50,000 documents in 161.6 seconds (309 documents per second) on an Apple M4 with 16 GB of RAM through [ChromaDB](trychroma.com), in a single run on 14 January 2026. Parameter count is a poor guide to speed, though: in Atomic Chat's run EmbeddingGemma-300M indexed slower than the larger BGE-M3 (315.0 against 433.0 chunks per second), so measure throughput on your own hardware. Indexing time is where model size shows up. Using Atomic Chat's measured throughput on one RTX A6000 (chunks of about 250 words), here is how long one million chunks would take: | Model | Chunks per second (A6000) | Time for 1M chunks | |---|---|---| | all-MiniLM-L6-v2 | 1,074.7 | about 16 minutes | | nomic-embed-text v1.5 | 727.3 | about 23 minutes | | BGE-M3 | 433.0 | about 38 minutes | | EmbeddingGemma-300M | 315.0 | about 53 minutes | | Qwen3-Embedding-0.6B | 169.9 | about 1.6 hours | | Qwen3-Embedding-8B | 26.1 | about 10.6 hours | Atomic Chat did not record batch size, so read these times as order of magnitude; a tuned batch can move them noticeably. Choosing between the 0.6B and 8B Qwen models is a trade: 0.0942 nDCG@10 on SciFact in exchange for an indexing job about six and a half times longer and nearly seven times the peak VRAM. ## Ollama is the fastest way to get local vectors; sentence-transformers, llama.cpp, TEI and FastEmbed suit other stacks Five runtimes cover almost every setup, and the choice depends on what the rest of your stack already uses. [Ollama](docs.ollama.com) is the shortest path from nothing to vectors. Pull a model, then call the `/api/embed` endpoint, which takes a single string or an array of strings and returns an `embeddings` array along with a `prompt_eval_count` of tokens processed: ```bash ollama pull qwen3-embedding:0.6b curl localhost -d '{ "model": "qwen3-embedding:0.6b", "input": ["first chunk of text", "second chunk of text"], "truncate": false }' ``` Use `/api/embed`, not the older `/api/embeddings`, which takes a `prompt` field, returns one vector and does not batch. Ollama also serves an OpenAI-compatible `/v1/embeddings` endpoint that accepts a `dimensions` parameter, so most OpenAI client code points at it with a base-URL change. [Sentence-transformers](sbert.net) is the Python default for anything on [Hugging Face](huggingface.co), and it also ships the `quantize_embeddings` helper for shrinking stored vectors: ```python from sentence_transformers import SentenceTransformer model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B") doc_vecs = model.encode(chunks, batch_size=64, normalize_embeddings=True) query_vecs = model.encode(queries, prompt_name="query", normalize_embeddings=True) ``` Qwen3-Embedding needs sentence-transformers 2.7.0 or later and transformers 4.51.0 or later, according to its card; older transformers versions fail with `KeyError: 'qwen3'`. [llama.cpp](ggml-org/llama.cpp) runs GGUF files with the lowest overhead. Start `llama-server -m model.gguf --embeddings --pooling mean` (match `--pooling` to the method the model was trained with, which its card states) and it serves an OpenAI-compatible `/v1/embeddings` endpoint with normalized vectors. [Text Embeddings Inference](huggingface.co) (TEI) from Hugging Face is the production server, shipped as a Docker image for GPUs and buildable for x86 CPUs or Apple Metal, and its own CPU guide uses Qwen3-Embedding-0.6B as the example model. [FastEmbed](qdrant.github.io) from [Qdrant](qdrant.tech) runs models on ONNX Runtime without PyTorch, which makes it the lighter choice for serverless functions and small containers. ### A missing query prefix lowers retrieval quality without any error Most modern embedding models expect a different prefix on queries than on documents (Qwen3's card calls it an instruction), and nothing fails when you leave it out; retrieval simply gets worse. [nomic-embed-text v1.5](huggingface.co) states that every input must start with `search_document: ` or `search_query: `. Qwen3-Embedding wants queries as `Instruct: {task description}\nQuery:{query}` and documents with no instruction, and its card credits that prefix with a 1% to 5% gain on most tasks. EmbeddingGemma uses `task: search result | query: {query}` for queries and `title: none | text: {content}` for documents. Morph points out that the [LangChain](langchain.com) and [LlamaIndex](llamaindex.ai) Ollama wrappers do not add these prefixes for you, and reports that a 5 to 10 point gap between published and observed retrieval quality often traces back to that. Write the prefix into your own embedding function so it cannot be forgotten. ### Ollama truncates chunks past the context limit unless you set `truncate: false` Context limits range from 256 tokens for all-MiniLM-L6-v2 and 512 for mxbai-embed-large and multilingual-e5-large to 8,192 for nomic, BGE-M3 and EmbeddingGemma 2 and 32K for Qwen3-Embedding. Anything longer is cut off without an error by default; Ollama's `truncate` parameter defaults to `true`, and setting it to `false` turns overflow into an error you can catch. Ollama also loads nomic-embed-text with a shorter context than the model supports, so set `num_ctx` to 8192 if your chunks are long. To catch silent truncation, compare the `prompt_eval_count` Ollama returns against your own token count for each chunk. ## A million 1,024-dimension vectors take 4.1 GB, and quantization cuts that 4x to 32x Raw storage is the number of vectors times the dimensions times four bytes for float32. For one million chunks, that is 1.5 GB at 384 dimensions, 3.1 GB at 768, 4.1 GB at 1,024 and 16.4 GB at 4,096, before the extra memory your vector database spends on its search graph. Atomic Chat's SciFact index shows the same ratio in practice: 38.1 MiB for 1,024-dimension models against 152.5 MiB for Qwen3-Embedding-8B's 4,096. Matryoshka truncation and vector quantization both cut that down, and you can use both. Matryoshka-trained models let you keep only the first part of each vector. Qwen3-Embedding-8B scored 0.7960 on SciFact at 4,096 dimensions, 0.7905 at 1,024 and 0.7672 at 256 in Atomic Chat's tests, so cutting to 1,024 dimensions gives up 0.0055 nDCG@10 for a quarter of the storage; nomic-embed-text v1.5 drops from 62.28 to 61.04 MTEB average going from 768 to 256 dimensions, by Morph's table. Re-normalize after truncating, and use the same dimension for queries and documents. Quantizing the stored vectors shrinks them further. Hugging Face's [embedding quantization study](huggingface.co) (March 2024, mxbai-embed-large-v1 on 15 MTEB retrieval datasets) found int8 vectors are 4x smaller and keep about 97% of retrieval quality, rising to about 99% when the top candidates are rescored with full-precision query vectors, while binary vectors are 32x smaller and keep about 92.5%, or about 96% with rescoring. At the scale of their worked example, 250 million 1,024-dimension vectors take 953.67 GB as float32 and 29.80 GB as binary. Sentence-transformers ships the `quantize_embeddings` helper for both. ## A local embedding model is rarely much cheaper than an API: embedding 1M chunks once on text-embedding-3-small costs about $6.66 Local saves less than the API price suggests, and cost is rarely the best reason to go local. [OpenAI's text-embedding-3-small](developers.openai.com) costs $0.02 per million tokens and text-embedding-3-large $0.13. Suppose you have one million chunks of about 330 tokens each, which is 333 million tokens: embedding all of them once costs about $6.66 on the small model and about $43.29 on the large one. A local GPU does the same job for the price of electricity, but only if the GPU already exists, and somebody still owns the server and the on-call rota. Four situations tip the arithmetic toward local: you re-index often (every chunking change means paying again on an API), your query volume is high enough that per-call latency and cost add up, you need the exact same vectors a year from now, or the text cannot leave your network. Reproducibility and privacy matter most. A local model file gives identical embeddings for as long as you keep it, while a hosted model can change versions behind the same name. If the text cannot leave your network, an API is ruled out before cost comes up. Local is the wrong call when your volume is small and nobody on the team wants to run inference. It is also wrong when your hardware cannot hold a model good enough for your hardest queries, since multi-hop questions and rare jargon are where the largest hosted models still pull ahead, as [AI/TLDR](ai-tldr.dev) notes. Run both on a labelled sample of your own queries before deciding; it takes an afternoon. ## Switching embedding models means re-embedding everything Vectors from different models live in different spaces, even when the dimensions match, so you cannot query a Qwen3 index with an EmbeddingGemma query vector or mix the two in one collection. Changing models means re-embedding every document, and the indexing table above is your cost estimate; for a million chunks the move from all-MiniLM-L6-v2 to Qwen3-Embedding-8B goes from about 16 minutes to about 10.6 hours on one A6000. A migration without downtime is ordinary engineering. Write new documents to both indexes, backfill the new index in batches, compare recall on a labelled query set while reads still go to the old one, switch reads once the new index wins, and keep the old index until the rollback window passes. Store the model name, revision, prefix, dimension and normalization next to every vector, because that metadata is the only record of how the old vectors were made. ## The embedding model is often not your retrieval bottleneck, so test recall first Keyword search beat vector search in our own study once a single code dataset was taken out, which is a reason to measure before you swap models. In our [50,000-document retrieval study](maximem.ai), vector search led keyword search across five datasets on MRR@10 by 0.6320 to 0.5325, but almost all of that lead came from one natural-language-to-code dataset; remove it and keyword search led the remaining four, 0.5931 to 0.5614, and on science exam questions keyword search scored 0.8145 against 0.6142. The setup was all-MiniLM-L6-v2 at 384 dimensions, [Tantivy](quickwit-oss/tantivy) 0.22.0 and ChromaDB on an Apple M4, 5,000 queries scored by exact document-ID matching with no LLM judge, in one run on 14 January 2026. A stronger embedder would narrow keyword search's lead on those four datasets. It would not remove the class of query where the exact term (a product code, a drug name, an error string, a ticket ID) is the whole point, which is why hybrid search, with BGE-M3's sparse output or a separate keyword index, belongs in most local setups. Rerankers solve a narrower problem. A cross-encoder reranker reorders the candidates the embedder already found, so it helps when the right chunk sits in the top 50 but not the top 5, and it cannot recover a chunk the first stage missed. Finding your bottleneck takes about ten lines of code: label which chunk answers each of 50 to 100 real queries, embed them with two candidate models using the correct prefixes, and measure how often the right chunk lands in the top 10. ```python import numpy as np from sentence_transformers import SentenceTransformer def recall_at_k(model_id, chunks, queries, gold, k=10, query_prompt=None): m = SentenceTransformer(model_id) d = m.encode(chunks, normalize_embeddings=True, batch_size=64) q = m.encode(queries, prompt_name=query_prompt, normalize_embeddings=True) top = np.argsort(-q @ d.T, axis=1)[:, :k] return float(np.mean([g in row for g, row in zip(gold, top)])) ``` If both models land within 0.02 of each other (two percentage points of recall), the model is not your problem; look at chunking and query prefixes first. ## Running embeddings locally keeps text private, but the vectors can still leak it Running the model yourself removes a third-party processor from the embedding step, which shortens your list of sub-processors and keeps raw text inside your network. It does not make the vectors safe to treat as anonymous. In [Morris and colleagues' 2023 EMNLP paper](aclanthology.org), an inversion method recovered 92% of 32-token inputs exactly from their embeddings and recovered full names from a dataset of clinical notes. So the vector store needs the same access controls and residency rules as the documents it was built from, and a deletion request has to reach the vectors as well as the source rows. ## Where Maximem Synap fits: agent memory needs more than similarity A local embedding model finds passages that are similar to a query. For agent memory, similarity search leaves four questions open: it cannot tell which of two stored facts is current, that "my manager" in one conversation and "Sarah" in another are the same person, which user a fact belongs to, or when a fact should stop being returned at all. [Maximem Synap](maximem.ai) is a managed memory layer that takes care of those parts, and the embedding step runs on Maximem's own infrastructure, so text is not sent to a third party for it. Teams that need everything inside their own network can deploy Synap self-hosted or air-gapped on enterprise plans. If your retrieval feeds an agent that has to remember users across sessions, run the recall test above on your own queries first; it tells you whether the embedder or the memory around it is the part to fix. ## Frequently asked questions ### What is the best local embedding model in 2026? As of 13 October 2026, Qwen3-Embedding-0.6B is the best default if you have a GPU, Qwen3-Embedding-8B scored highest on SciFact in Atomic Chat's test and fits on one 24 GB GPU (16.60 GiB of VRAM at bfloat16), and EmbeddingGemma 2 is the strongest candidate for laptops and multimodal search, though no independent retrieval benchmark of it existed on that date. All three are Apache 2.0. Run the recall test above on two of them with 50 real queries before committing. ### Can I run an embedding model on CPU only? Yes. Sub-billion models such as all-MiniLM-L6-v2, nomic-embed-text v1.5 and EmbeddingGemma 2 run comfortably on a laptop CPU; all-MiniLM-L6-v2 embedded 50,000 documents in 161.6 seconds on an Apple M4 with no discrete GPU in our study. Keep the 4B and 8B Qwen3 models for machines with a GPU. ### What is the best Ollama embedding model? For most RAG work, use `qwen3-embedding:0.6b` (639 MB) if quality matters and `nomic-embed-text` (274 MB) if CPU speed matters; `embeddinggemma-2` is now the small multimodal option. Call `/api/embed`, add the model's query and document prefixes yourself, and set `num_ctx` for nomic if your chunks are long. ### How much storage do embeddings take? One million float32 vectors take 4.1 GB at 1,024 dimensions and 16.4 GB at 4,096, plus your vector database's index overhead. Int8 quantization makes that 4x smaller. Binary makes it 32x smaller, and Matryoshka truncation cuts it further on models trained for it. ### Do I need to re-embed my data if I change embedding models? Yes. Vectors from different models are not comparable even at the same dimension, so every document and query has to be embedded with the new model. Plan it as a backfill into a second index, and budget the time from your model's measured throughput. ### Are local embeddings private? The raw text stays on your hardware, which removes a third party from that step. The vectors themselves can be partly inverted back into text, so protect and delete them the way you would the original documents. ## Sources All retrieved 13 October 2026. - [Qwen3-Embedding-0.6B model card](huggingface.co) - [EmbeddingGemma 2 announcement, Google, 6 October 2026](blog.google) and [model card](huggingface.co) - [EmbeddingGemma-300M model card](huggingface.co) - [nomic-embed-text v1.5 model card](huggingface.co) - [Best Local Embedding Models for RAG: 7 Models Compared, Atomic Chat, 25 September 2026](atomic.chat) - [Best Ollama Embedding Models 2026, Morph, 9 June 2026](morphllm.com) - [Ollama embedding model library](ollama.com) and [/api/embed reference](docs.ollama.com) - [llama.cpp embeddings guide](mintlify.wiki) - [Text Embeddings Inference, local CPU guide](huggingface.co) - [FastEmbed documentation](qdrant.github.io) - [Binary and Scalar Embedding Quantization, Hugging Face, 22 March 2024](huggingface.co) - [OpenAI text-embedding-3-small model page](developers.openai.com) - [Choosing an Embedding Model, Pinecone](pinecone.io) - [Best Local Embedding Models to Run Without an API, AI/TLDR, 13 June 2026](ai-tldr.dev) - [Text Embeddings Reveal (Almost) As Much As Text, Morris et al., EMNLP 2023](aclanthology.org) - [File Search vs Vector Search for RAG, Maximem](maximem.ai)
Tantivy (The Full-Text Search Engine Library for Rust) Often called the 'Lucene of Rust.' An ultra-high-performance search engine library delivering blazing-fast indexing and query execution with minimal memory consumption. → quickwit-oss/tantivy
@notpronsh @pcstyle53 we are entering new era when search engines are so shit it's easier to ask agents to implement library instead of find the already existing one quickwit-oss/tantivy