code example

Build Content Deduplication with altor-vec

What this pattern solves: Near-duplicate detection for editorial pipelines, datasets, and CMS migrations.

Duplicate content rarely matches word for word. Marketing teams rewrite headlines, support teams copy old answers, and migrations create slightly different versions of the same article. Vector similarity is a good first pass for spotting semantic overlap.

For a moderate review batch, local comparison is useful because editors can run it privately in the browser without provisioning a database just to score overlap.

Install

npm install altor-vec

Concept explanation

In a content deduplication workflow, users usually describe intent in their own words. That is why vector search works well here: each record is turned into an embedding, the embeddings are indexed once, and later queries retrieve the nearest semantic neighbors instead of relying only on exact tokens. In practice this means the interface can respond to paraphrases, shorthand, and partial descriptions far better than a literal-only search box.

The browser is often the right place to do this when the corpus is moderate in size and safe to ship. The instant benefit is lower latency. The architectural benefit is that you remove a whole search service from the request path. That matters for keystroke-heavy interactions, offline-capable apps, and product surfaces where search should feel like a UI primitive rather than a network round trip.

This page uses a deterministic embedding helper so the sample is runnable with only altor-vec installed. That keeps the example honest and easy to paste into a demo project. Most teams use vector similarity for candidate generation, then run a stricter review step before auto-merging or deleting anything important.

Runnable JavaScript example

The following snippet indexes a small in-memory dataset, performs a semantic lookup for remote work stipend policy, and prints the nearest matches. It uses the real altor-vec API, including init(), WasmSearchEngine.from_vectors(), and search().

import init, { WasmSearchEngine } from 'altor-vec';

        const dims = 12;
        const records = [
  {
    "title": "Remote work reimbursement",
    "text": "Policy describing stipends for desks, chairs, and home office accessories.",
    "meta": "people"
  },
  {
    "title": "Home office allowance",
    "text": "Guidelines for claiming remote setup expenses during the first ninety days.",
    "meta": "people"
  },
  {
    "title": "Travel expense rules",
    "text": "Approved hotel, flight, and meal costs for business travel.",
    "meta": "finance"
  },
  {
    "title": "Laptop refresh process",
    "text": "How employees request a replacement laptop after the standard lifecycle.",
    "meta": "it"
  },
  {
    "title": "Equipment return checklist",
    "text": "Steps for collecting laptops and badges during offboarding.",
    "meta": "it"
  },
  {
    "title": "Manager onboarding notes",
    "text": "Guide for new managers running first-week team introductions.",
    "meta": "people"
  }
];

        function embedText(text) {
  const vector = new Float32Array(dims);
  for (const token of text.toLowerCase().split(/[^a-z0-9]+/).filter(Boolean)) {
    let hash = 2166136261;
    for (const char of token) {
      hash = Math.imul(hash ^ char.charCodeAt(0), 16777619);
    }
    const slot = Math.abs(hash) % dims;
    vector[slot] += 1;
    vector[(slot + token.length) % dims] += token.length / 10;
  }
  const magnitude = Math.hypot(...vector) || 1;
  return Array.from(vector, (value) => value / magnitude);
}

        async function main() {
          await init();

          const flat = new Float32Array(
            records.flatMap((record) => embedText(`${record.title} ${record.text} ${record.meta}`))
          );

          const engine = WasmSearchEngine.from_vectors(flat, dims, 16, 200, 64);
          const hits = JSON.parse(engine.search(new Float32Array(embedText('remote work stipend policy')), 4));

          const results = hits.map(([id, distance]) => ({
            ...records[id],
            similarity: Number((1 - distance).toFixed(3)),
          }));

          console.table(results);
          engine.free();
        }

        main();

React component version

The React version keeps the same index build but wires it into component state so the UI can query on input changes. That is usually how teams introduce semantic retrieval into an existing product: initialize once, keep the engine in memory, and map nearest-neighbor hits back to the original records.

import { useEffect, useState } from 'react';
        import init, { WasmSearchEngine } from 'altor-vec';

        const dims = 12;
        const records = [
  {
    "title": "Remote work reimbursement",
    "text": "Policy describing stipends for desks, chairs, and home office accessories.",
    "meta": "people"
  },
  {
    "title": "Home office allowance",
    "text": "Guidelines for claiming remote setup expenses during the first ninety days.",
    "meta": "people"
  },
  {
    "title": "Travel expense rules",
    "text": "Approved hotel, flight, and meal costs for business travel.",
    "meta": "finance"
  },
  {
    "title": "Laptop refresh process",
    "text": "How employees request a replacement laptop after the standard lifecycle.",
    "meta": "it"
  },
  {
    "title": "Equipment return checklist",
    "text": "Steps for collecting laptops and badges during offboarding.",
    "meta": "it"
  },
  {
    "title": "Manager onboarding notes",
    "text": "Guide for new managers running first-week team introductions.",
    "meta": "people"
  }
];

        function embedText(text) {
  const vector = new Float32Array(dims);
  for (const token of text.toLowerCase().split(/[^a-z0-9]+/).filter(Boolean)) {
    let hash = 2166136261;
    for (const char of token) {
      hash = Math.imul(hash ^ char.charCodeAt(0), 16777619);
    }
    const slot = Math.abs(hash) % dims;
    vector[slot] += 1;
    vector[(slot + token.length) % dims] += token.length / 10;
  }
  const magnitude = Math.hypot(...vector) || 1;
  return Array.from(vector, (value) => value / magnitude);
}

        export function ContentDeduplicationExample() {
          const [engine, setEngine] = useState(null);
          const [query, setQuery] = useState('');
          const [results, setResults] = useState([]);

          useEffect(() => {
            let cancelled = false;
            let instance;

            (async () => {
              await init();
              const flat = new Float32Array(
                records.flatMap((record) => embedText(`${record.title} ${record.text} ${record.meta}`))
              );
              instance = WasmSearchEngine.from_vectors(flat, dims, 16, 200, 64);
              if (!cancelled) setEngine(instance);
            })();

            return () => {
              cancelled = true;
              instance?.free();
            };
          }, []);

          useEffect(() => {
            if (!engine || query.trim().length < 2) {
              setResults([]);
              return;
            }

            const hits = JSON.parse(engine.search(new Float32Array(embedText(query)), 5));
            setResults(
              hits.map(([id, distance]) => ({
                ...records[id],
                similarity: Number((1 - distance).toFixed(3)),
              }))
            );
          }, [engine, query]);

          return (
            <section>
              <input
                value={query}
                onChange={(event) => setQuery(event.target.value)}
                placeholder="Find semantically similar content"
              />
              <ul>
                {results.map((result) => (
                  <li key={result.title}>
                    <strong>{result.title}</strong> — {result.meta} (score {result.similarity})
                  </li>
                ))}
              </ul>
            </section>
          );
        }

How this example works

The pattern has three moving parts. First, you choose what text represents each record: title, description, metadata, or a chunk of content. Second, you turn that text into vectors and flatten them into one Float32Array. Third, you build the HNSW graph and query it with a vector created from the user input. The library returns nearest-neighbor IDs and distances, and your app decides how to display or post-process them.

Because the retrieval step is approximate nearest-neighbor search, it stays fast even as the dataset grows beyond trivial linear scans. The most important quality lever is still the embedding model. Better vectors usually matter more than micro-optimizing ANN parameters, so teams should benchmark retrieval quality on real user phrasing before shipping the experience widely.

When to use this pattern

This is a practical fit when the search corpus is small to medium, shipped with the app, and searched frequently enough that backend latency would be noticeable. Common examples include docs portals, embedded support widgets, local-first assistants, and curated catalogs.

• CMS cleanup
• Editorial QA
• Dataset curation
• Knowledge-base migrations

Limitations

A nearest-neighbor match is not proof that two documents are safe to collapse. Legal wording, version history, and ownership still matter, so keep a human review step for consequential content decisions.

Be especially careful about corpus size, update frequency, and data sensitivity. Browser vector search is excellent when those three constraints are favorable, but it is not the right answer when the dataset is huge, private, or changing constantly for every user.

FAQ

Is vector search enough to deduplicate content automatically?

Usually not. It is excellent for surfacing likely duplicates, but you still need review logic before deleting or merging records.

What similarity threshold should I use?

There is no universal threshold. Benchmark it on your own corpus because writing style and chunk length change the right cutoff.

Can this help before a CMS migration?

Yes. Candidate duplicate lists are a common way to shrink and clean a corpus before you move it.

Get started: npm install altor-vec · GitHub