Vue 3 guide

Browser RAG in Vue 3 with altor-vec

Q: Which LLM can I use with browser-side RAG?

For fully offline RAG, use WebLLM (Llama 3.1, Phi-3, Mistral 7B quantized). For online RAG with a client-side retrieval step, call the OpenAI or Anthropic API with the retrieved context — only the LLM call is remote.

Q: How large should my document chunks be for RAG retrieval?

Chunk size of 200–400 tokens (roughly 150–300 words) works well for most use cases. Shorter chunks give more precise retrieval; longer chunks give more context. Use overlapping chunks (50–100 token overlap) to avoid cutting off mid-thought.

Q: How do I prevent the LLM from hallucinating if no relevant chunks are found?

Check the top result score: if the highest similarity score is below 0.6, tell the LLM 'No relevant information found in the document set' rather than injecting empty or low-relevance context. This significantly reduces hallucinations.

Use altor-vec to add browser rag to your Vue 3 app — entirely in the browser, with no server, no API keys, and zero per-query cost. Retrieval-Augmented Generation (RAG) entirely in the browser — retrieve relevant document chunks from a local vector index, then inject them as context into an LLM prompt, all without a server.

Install: npm install altor-vec @xenova/transformers

Implementation

Uses Composition API (setup + onMounted). Uses ref() for engine and results.

Performance

Retrieval from 10K chunks: <1ms. Total RAG latency dominated by LLM call (1–30s depending on model). Measured on M2 MacBook Pro, Chrome 124. Mobile is typically 2–4× slower — test on target devices before deploying.

Index size	Dimensions	Query p50	Memory
1,000 vectors	384	~0.1ms	~2MB
10,000 vectors	384	~0.4ms	~17MB
50,000 vectors	384	~0.9ms	~85MB

When this approach works best

Privacy-sensitive apps where document content must never leave the device
Offline-first AI assistants that run on cached content without network access
Demo or prototype RAG pipelines with no infrastructure budget

Limitations

Context window limits: you can only inject a finite number of retrieved chunks — chunk size and k must be tuned
Browser-side LLM inference (WebLLM) is slow on low-end devices; use an API-based LLM for production

Frequently asked questions

Which LLM can I use with browser-side RAG?

For fully offline RAG, use WebLLM (Llama 3.1, Phi-3, Mistral 7B quantized). For online RAG with a client-side retrieval step, call the OpenAI or Anthropic API with the retrieved context — only the LLM call is remote.

How large should my document chunks be for RAG retrieval?

Chunk size of 200–400 tokens (roughly 150–300 words) works well for most use cases. Shorter chunks give more precise retrieval; longer chunks give more context. Use overlapping chunks (50–100 token overlap) to avoid cutting off mid-thought.

How do I prevent the LLM from hallucinating if no relevant chunks are found?

Check the top result score: if the highest similarity score is below 0.6, tell the LLM 'No relevant information found in the document set' rather than injecting empty or low-relevance context. This significantly reduces hallucinations.

Related resources

framework

use case

reference