Tutorial

This tutorial walks through building, querying, persisting, and customizing text search indexes with TextSearch.jl and SimilaritySearch.jl. Every code block on this page is a real, executed Julia session (not hand-typed transcripts), so the output you see is always in sync with the current code.

A few sections use packages beyond TextSearch/SimilaritySearch:

] add JLD2 WordTokenizers

• JLD2.jl — saving/loading indexes to disk.
• WordTokenizers.jl — an alternative, general-purpose English tokenizer, used both to feed TextSearch's own pipeline (sentence splitting) and to replace it entirely.

A small corpus from Project Gutenberg

As a running example we use Edgar Allan Poe's short story The Cask of Amontillado (1846), split into its 54 paragraphs — public domain, small enough to read in one sitting, and long enough to make search results meaningful. The text comes from Project Gutenberg; each paragraph below is one "document".

length(CASK_OF_AMONTILLADO), CASK_OF_AMONTILLADO[1]

(54, "The thousand injuries of Fortunato I had borne as I best could, but when he ventured upon insult, I vowed revenge. You, who so well know the nature of my soul, will not suppose, however, that I gave utterance to a threat. _At length_ I would be avenged; this was a point definitely settled--but the very definitiveness with which it was resolved, precluded the idea of risk. I must not only punish, but punish with impunity. A wrong is unredressed when retribution overtakes its redresser. It is equally unredressed when the avenger fails to make himself felt as such to him who has done the wrong.")

Building a vocabulary and a vector model

Vocabulary parses the corpus once and accumulates per-token statistics (TextConfig() defaults to word unigrams). A VectorModel then turns that vocabulary into a weighting scheme — here, classic TF-IDF — and vectorize_corpus applies it to every paragraph, producing one sparse SVEC per document.

voc = Vocabulary(TextConfig(), CASK_OF_AMONTILLADO; verbose=false)
vocsize(voc), trainsize(voc)

(798, 54)

model = VectorModel(IdfWeighting(), TfWeighting(), voc)
vecs = vectorize_corpus(model, CASK_OF_AMONTILLADO)
vecs[1]

798-element SparseArrays.SparseVector{Float32, Int32} with 82 stored entries:
  [1 ]  =  0.0531555
  [2 ]  =  0.1168
  [3 ]  =  0.1168
  [4 ]  =  0.0598652
  [5 ]  =  0.050581
  [6 ]  =  0.0781976
  [7 ]  =  0.0478702
        ⋮
  [75]  =  0.1168
  [76]  =  0.0892545
  [77]  =  0.0892545
  [78]  =  0.100193
  [79]  =  0.1168
  [80]  =  0.0478702
  [81]  =  0.0892545
  [82]  =  0.100193

Pruning the vocabulary by minimum frequency

798 tokens out of a ~4000-word story is a lot — most of them are one-off words (hapax legomena) that add noise more than signal. Each entry of a Vocabulary carries its own occs (total occurrence count across the corpus) and ndocs (number of documents it appears in), so filter_tokens can prune by either:

hapax_count = count(t -> t.occs == 1, voc[i] for i in eachindex(voc))
vocsize(voc), hapax_count

(798, 540)

pruned_voc = filter_tokens(t -> t.occs >= 3, voc)
vocsize(pruned_voc)

148

filter_tokens returns a brand new Vocabulary — voc itself is untouched, so the rest of this tutorial keeps using the original, unpruned vocabulary. To actually build a model on the pruned vocabulary instead, just use pruned_voc in place of voc from here on (e.g. VectorModel(IdfWeighting(), TfWeighting(), pruned_voc)); words below the frequency cutoff are treated as out-of-vocabulary from then on, the same as any other unseen word.

Filtering by document frequency (t.ndocs) instead of raw occurrence count is often a better cutoff for longer/multi-document corpora — it discards words that are common within a single document but never recur elsewhere, which raw frequency alone wouldn't catch:

pruned_by_docfreq = filter_tokens(t -> t.ndocs >= 3, voc)
vocsize(pruned_by_docfreq)

136

Searching with a raw inverted file (vector-space ranking)

WeightedInvertedFile indexes the weight vectors directly and ranks by a distance over them — cosine here, via NormCosine (SimilaritySearch's cosine distance, re-exported by TextSearch). This is the same kind of index you'd use for any sparse vector search, not just text.

wif = WeightedInvertedFile(vocsize(voc))
wctx = quietctx()
append_items!(wif, wctx, VectorDatabase(vecs))

res = knnqueue(KnnSorted, 5)
search(wif, wctx, vecs[1], res)
collect(IdView(res))

5-element Vector{UInt32}:
 0x00000001
 0x00000030
 0x00000003
 0x00000031
 0x00000032

The first hit is paragraph 1 itself (distance 0 — a document is always its own nearest neighbor); the rest are the paragraphs whose TF-IDF vectors are closest to it. Querying with free text instead of an existing document's vector works the same way, through vectorize:

qvec = vectorize(model, "vector search library")
res = knnqueue(KnnSorted, 5)
search(wif, wctx, qvec, res)
[(id, first(CASK_OF_AMONTILLADO[id], 60)) for id in collect(IdView(res))]

1-element Vector{Tuple{UInt32, String}}:
 (0x00000028, "\"Be it so,\" I said, replacing the tool beneath the cloak and")

Only one hit came back even though we asked for 5 — an inverted file can only rank documents that share at least one token with the query, and here just one paragraph happens to contain "search". Unsurprisingly, a 19th-century short story has nothing to do with vector search libraries anyway; every score here is essentially noise. Try a query drawn from the story itself, like "amontillado nitre" or "trowel wall", to see closer, more meaningful matches.

Vector arithmetic: dot products, centroids, and normalization

Since paragraph and query vectors are just SVECs (sparse Dicts), ordinary LinearAlgebra operations work on them directly — +, -, dot, norm, normalize! are all defined for SVEC/BOW. Two things make this useful: comparing documents/queries directly via dot, and building a query that represents more than one idea at once.

using LinearAlgebra

q_wine = vectorize(model, "amontillado wine")
q_damp = vectorize(model, "nitre damp catacombs")
norm(q_wine), norm(q_damp)

(1.0f0, 1.0f0)

vectorize normalizes its output to unit length by default (normalize=true) — this is what makes dot directly meaningful as a similarity score: for unit ("spherical") vectors, the dot product is the cosine similarity, bounded the same way cosine similarity is. It also matches what WeightedInvertedFile itself assumes — its distance is a cheap running dot-product sum (see Dist.NormCosine's docstring), valid only when the vectors being compared are already unit length.

dot(q_wine, q_damp)

0.0f0

Zero — these two queries share no vocabulary at all, so as far as the dot product is concerned they're unrelated (not literally "opposite", just orthogonal). Compare that to two paragraphs that are actually about the same scene:

dot(vecs[51], vecs[52])  # two consecutive paragraphs of Fortunato's manic "ha! ha!" laughter

0.5051559f0

To search for both ideas at once — say, a query that's part "the wine", part "the damp vaults" — combine the two query vectors into their centroid and search with that, instead of running two separate queries and merging results by hand:

q_both = centroid([q_wine, q_damp])
norm(q_both)

1.0f0

res = knnqueue(KnnSorted, 5)
search(wif, wctx, q_both, res)
[(id, first(CASK_OF_AMONTILLADO[id], 60)) for id in collect(IdView(res))]

5-element Vector{Tuple{UInt32, String}}:
 (0x0000001b, "\"Drink,\" I said, presenting him the wine.")
 (0x0000000c, "\"My friend, no. It is not the engagement, but the severe col")
 (0x00000014, "\"Nitre,\" I replied. \"How long have you had that cough?\"")
 (0x00000021, "The wine sparkled in his eyes and the bells jingled. My own ")
 (0x0000002d, "\"Pass your hand,\" I said, \"over the wall; you cannot help fe")

The results blend both themes, rather than only ever matching one or the other.

Why normalize! matters here

centroid normalizes its output, but that alone doesn't make a fair blend — it fixes the final vector's length, not the direction that a plain sum already baked in. If the inputs going in aren't themselves unit vectors, whichever one happens to have the larger magnitude dominates the sum, and the final normalize! just rescales that already-skewed direction to length 1. A minimal example makes this concrete:

a = SVEC(1 => 1.0f0)  # a unit vector, pointing along "axis 1"
b = SVEC(2 => 1.0f0)  # a unit vector, pointing along "axis 2"
centroid([a, b])       # evenly split between both directions, as expected

b_scaled = b * 5.0f0   # same direction as `b`, but 5x the magnitude
norm(b_scaled)

centroid([a, b_scaled])  # direction is pulled almost entirely toward b, not an even blend

vectorize's default normalize=true is precisely what saves you from this: every vector TextSearch itself produces already lies on the unit sphere, so summing any number of them and normalizing the result gives a genuinely even blend, as in q_both above. The failure mode above only bites when vectors come from somewhere vectorize didn't touch — built with normalize=false, assembled by hand, or imported from a different pipeline entirely. The fix is always the same: call normalize! on each vector individually before combining them, so every input to a centroid is on equal footing before it's summed.

Searching with BM25 (probabilistic ranking)

BM25InvertedFile is a different index entirely: instead of building explicit weight vectors, it indexes the corpus's bags of words and a BM25Scorer directly, ranking by the Okapi BM25 formula. There's no separate VectorModel/ vectorize_corpus step — append_items! takes raw text (or TokenizedText, or a pre-computed BOW) and computes everything it needs from voc.

bm25idx = BM25InvertedFile(voc)
bctx = quietctx()
append_items!(bm25idx, bctx, CASK_OF_AMONTILLADO)

res = knnqueue(KnnSorted, 5)
search(bm25idx, bctx, "amontillado nitre", res)
[(id, first(CASK_OF_AMONTILLADO[id], 60)) for id in collect(IdView(res))]

5-element Vector{Tuple{UInt32, String}}:
 (0x00000014, "\"Nitre,\" I replied. \"How long have you had that cough?\"")
 (0x0000000c, "\"My friend, no. It is not the engagement, but the severe col")
 (0x00000022, "\"The nitre!\" I said; \"see, it increases. It hangs like moss ")
 (0x0000002d, "\"Pass your hand,\" I said, \"over the wall; you cannot help fe")
 (0x00000009, "\"Luchesi cannot tell Amontillado from Sherry.\"")

Both index types answer top-k queries the same way (append_items!/push_item! to build, search to query), so switching between them is mostly a matter of which one matches your ranking needs: WeightedInvertedFile for vector-space similarity over any weighting scheme you've built, BM25InvertedFile when you want BM25's document-length normalization and term-saturation behavior without hand-building vectors first.

Saving and loading indexes with JLD2

Every type used above — Vocabulary, VectorModel, BM25InvertedFile, WeightedInvertedFile — is a plain Julia struct, so JLD2.jl can save and load them directly with no special glue code.

using JLD2

path = tempname() * ".jld2"
jldsave(path; voc, model, bm25idx)

loaded = load(path)
voc2, model2, bm25idx2 = loaded["voc"], loaded["model"], loaded["bm25idx"]
vectorize(model2, CASK_OF_AMONTILLADO[1]) == vecs[1]

true

res = knnqueue(KnnSorted, 5)
search(bm25idx2, quietctx(), "amontillado nitre", res)
collect(IdView(res))

5-element Vector{UInt32}:
 0x00000014
 0x0000000c
 0x00000022
 0x0000002d
 0x00000009

The reloaded BM25InvertedFile answers the same query with the same ranking as the original — nothing needs to be rebuilt or refit.

Working with WordTokenizers.jl

TextSearch's own tokenizer (TextConfig/tokenize) is tuned for short, noisy, informal text (tweets, chat messages) and is deliberately dependency-free. For general-purpose English text you may prefer a more linguistically-aware tokenizer — WordTokenizers.jl is a common choice. There are two ways to bring it in: compose it with TextSearch's pipeline, or replace TextSearch's tokenizer entirely.

Composing: sentence splitting as a preprocessing step

TextSearch has no sentence segmenter of its own — it tokenizes whatever "documents" you give it. Nothing stops you from making the documents finer-grained first. Here we split each paragraph into sentences with WordTokenizers.split_sentences, then hand the resulting sentence list to Vocabulary/vectorize_corpus exactly as before — TextConfig's own tokenizer still does the actual word-level tokenization.

using WordTokenizers

sentences = String[]
for paragraph in CASK_OF_AMONTILLADO
    for s in split_sentences(paragraph)
        push!(sentences, String(s))
    end
end

length(sentences), sentences[1]

(195, "The thousand injuries of Fortunato I had borne as I best could, but when he ventured upon insult, I vowed revenge.")

sentence_voc = Vocabulary(TextConfig(), sentences; verbose=false)
vocsize(sentence_voc), trainsize(sentence_voc)

(798, 195)

195 sentences from 54 paragraphs — a finer search granularity, built with one extra preprocessing step and no changes to TextSearch itself.

Replacing: bypassing TextSearch's tokenizer entirely

If you'd rather use WordTokenizers' own word splitting instead of TextSearch's, wrap its output in a TokenizedText — the same type tokenize itself returns. TokenizedText is TextSearch's universal "already tokenized" contract: Vocabulary/bagofwords/etc. all recognize it and skip their own normalization and tokenization step entirely, using your tokens as-is.

wt_docs = [TokenizedText(String.(WordTokenizers.tokenize(lowercase(p)))) for p in CASK_OF_AMONTILLADO]
collect(wt_docs[1])[1:8]

8-element Vector{String}:
 "the"
 "thousand"
 "injuries"
 "of"
 "fortunato"
 "i"
 "had"
 "borne"

wt_voc = Vocabulary(TextConfig(), wt_docs; verbose=false)
vocsize(wt_voc)

838

TextConfig() is still passed here — Vocabulary keeps it around for later use (e.g. tokenizing a raw-text query at search time) — but since every document already arrives as a TokenizedText, none of TextConfig's own tokenization settings (nlist, qlist, del_diac, ...) have any effect on how these documents were split; that happened entirely inside WordTokenizers.tokenize.

A small tweet-like corpus

TextConfig has several options aimed specifically at short, informal, social-media text: grouping @mentions, URLs, and emoji into single normalized tokens instead of leaving them as noisy character soup. The messages below are a small illustrative set written to exercise these options (not scraped from a live feed, so the example needs no network access and no data-license considerations).

using TextSearch, SimilaritySearch

quietctx() = InvertedFileContext(logger=SimilaritySearch.LogList(SimilaritySearch.AbstractLog[]))

tweets = [
    "Just landed in Mexico City!! 🎉 cant wait to try the tacos @VisitMexico #travel",
    "Ugh, stuck in traffic again on the highway :( #mondayblues",
    "New paper on approximate similarity search is out! check it out https://example.org/paper",
    "@juli_ai loved your talk on vector databases today, so insightful #ai #ml",
    "Rainy day, perfect for reading a good book ☕📚",
    "Why does @united keep cancelling flights?? this is the third time this month #travelfail",
    "Excited to announce our new open source vector search release! https://github.com/example/repo #julialang",
    "lol this meme is too real 😂😂😂 #mood",
    "Can anyone recommend a good vector search library for Julia? asking for a friend @julialang",
    "Beautiful sunset over the bay tonight 🌅 #nofilter",
]

cfg = TextConfig(group_usr=true, group_url=true, group_emo=true, del_punc=false)
collect(TextSearch.tokenize(cfg, tweets[1]))

15-element Vector{String}:
 "just"
 "landed"
 "in"
 "mexico"
 "city"
 "!!"
 "👾"
 "cant"
 "wait"
 "to"
 "try"
 "the"
 "tacos"
 "_usr"
 "#travel"

@VisitMexico collapsed to _usr, and the 🎉 emoji collapsed to 👾 — with group_emo=true, every emoji character is replaced by this single placeholder glyph before tokenization, so any emoji becomes the same token instead of each distinct emoji being its own rare, one-off token. #travel stayed intact — hashtags are treated as regular content, not stripped, since they usually carry meaning.

voc = Vocabulary(cfg, tweets; verbose=false)
bm25idx = BM25InvertedFile(voc)
ctx = quietctx()
append_items!(bm25idx, ctx, tweets)

res = knnqueue(KnnSorted, 3)
search(bm25idx, ctx, "vector search library", res)
[(id, tweets[id]) for id in collect(IdView(res))]

3-element Vector{Tuple{UInt32, String}}:
 (0x00000009, "Can anyone recommend a good vector search library for Julia? asking for a friend @julialang")
 (0x00000007, "Excited to announce our new open source vector search release! https://github.com/example/repo #julialang")
 (0x00000003, "New paper on approximate similarity search is out! check it out https://example.org/paper")

The top matches are exactly the three tweets that actually mention vector search — BM25 ranks them by how much of the query they cover and how rare/salient those terms are across the small corpus.

Next steps

See the TextSearch API page for the full reference — every function and type used above (and many more, including the lower-level building blocks in TextSearch.Intersections and TextSearch.InvertedFiles) is documented there with its own runnable example.