Node.js guide

Browser RAG in Node.js with altor-vec

Use altor-vec to add browser rag to your Node.js app — entirely in the browser, with no server, no API keys, and zero per-query cost. Retrieval-Augmented Generation (RAG) entirely in the browser — retrieve relevant document chunks from a local vector index, then inject them as context into an LLM prompt, all without a server.

Implementation

Server-side indexing script (Node 18+, ESM). Uses module-level variable for the engine.

// build-rag-index.mjs — Node.js: chunk documents + build RAG retrieval index
// Run at build time; browser loads the index for client-side retrieval
import { pipeline } from '@xenova/transformers';
import init, { WasmSearchEngine } from 'altor-vec';
import { readFileSync, writeFileSync, readdirSync } from 'fs';
import { join } from 'path';

function chunkText(text, chunkSize = 300, overlap = 50) {
  const words = text.split(/\s+/);
  const chunks = [];
  for (let i = 0; i < words.length; i += chunkSize - overlap) {
    chunks.push(words.slice(i, i + chunkSize).join(' '));
    if (i + chunkSize >= words.length) break;
  }
  return chunks;
}

const docsDir = './content';
const allChunks = [];
for (const file of readdirSync(docsDir).filter(f => f.endsWith('.md'))) {
  const text = readFileSync(join(docsDir, file), 'utf8');
  const chunks = chunkText(text);
  chunks.forEach(chunk => allChunks.push({ text: chunk, source: file }));
}

console.log(\`Embedding \${allChunks.length} chunks...\`);
await init();
const embedder = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2');
const DIM = 384;
const vectors = new Float32Array(allChunks.length * DIM);

for (const [i, chunk] of allChunks.entries()) {
  const out = await embedder(chunk.text, { pooling: 'mean', normalize: true });
  vectors.set(out.data, i * DIM);
}

const engine = WasmSearchEngine.from_vectors(vectors, DIM, 16, 200, 50);
writeFileSync('public/rag-index.json', engine.to_json());
writeFileSync('public/rag-chunks.json', JSON.stringify(allChunks));
console.log(\`RAG index built: \${allChunks.length} chunks, index size: \${
  Math.round(JSON.stringify(engine.to_json()).length / 1024) + 'KB'
}\`);

Performance

Retrieval from 10K chunks: <1ms. Total RAG latency dominated by LLM call (1–30s depending on model). Measured on M2 MacBook Pro, Chrome 124. Mobile is typically 2–4× slower — test on target devices before deploying.

When this approach works best

• Privacy-sensitive apps where document content must never leave the device
• Offline-first AI assistants that run on cached content without network access
• Demo or prototype RAG pipelines with no infrastructure budget

Limitations

• Context window limits: you can only inject a finite number of retrieved chunks — chunk size and k must be tuned
• Browser-side LLM inference (WebLLM) is slow on low-end devices; use an API-based LLM for production

Frequently asked questions

Which LLM can I use with browser-side RAG?

For fully offline RAG, use WebLLM (Llama 3.1, Phi-3, Mistral 7B quantized). For online RAG with a client-side retrieval step, call the OpenAI or Anthropic API with the retrieved context — only the LLM call is remote.

How large should my document chunks be for RAG retrieval?

Chunk size of 200–400 tokens (roughly 150–300 words) works well for most use cases. Shorter chunks give more precise retrieval; longer chunks give more context. Use overlapping chunks (50–100 token overlap) to avoid cutting off mid-thought.

How do I prevent the LLM from hallucinating if no relevant chunks are found?

Check the top result score: if the highest similarity score is below 0.6, tell the LLM 'No relevant information found in the document set' rather than injecting empty or low-relevance context. This significantly reduces hallucinations.

Related resources