Tutorial

This tutorial provides a comprehensive guide to text preprocessing, vector representation, inverted indexing, semantic dimensionality reduction, and profile management in TextSearch.jl, in integration with SimilaritySearch.jl.

All code blocks in this tutorial are executed during documentation generation to guarantee consistency between explanations and output.

The following optional packages are used in specific sections:

] add JLD2 WordTokenizers

• JLD2.jl: For serializing and deserializing indexes and models to disk.
• WordTokenizers.jl: An external NLP tokenization library used to demonstrate pipeline extensibility.

The Reference Corpus: The Cask of Amontillado

To illustrate text retrieval workflows on real text, we use Edgar Allan Poe's short story The Cask of Amontillado (1846) from Project Gutenberg, partitioned into its 54 constituent paragraphs. Each paragraph represents an individual document.

length(CASK_OF_AMONTILLADO), CASK_OF_AMONTILLADO[1]

(54, "The thousand injuries of Fortunato I had borne as I best could, but when he ventured upon insult, I vowed revenge. You, who so well know the nature of my soul, will not suppose, however, that I gave utterance to a threat. _At length_ I would be avenged; this was a point definitely settled--but the very definitiveness with which it was resolved, precluded the idea of risk. I must not only punish, but punish with impunity. A wrong is unredressed when retribution overtakes its redresser. It is equally unredressed when the avenger fails to make himself felt as such to him who has done the wrong.")

Vocabulary Building and Vector Models

1. Vocabulary Extraction

A Vocabulary processes a text corpus according to a TextConfig specification (which defaults to character normalization and word unigram extraction) and accumulates global token statistics:

• t.occs: Total frequency of occurrences across the entire corpus.
• t.ndocs: Number of distinct documents containing the token (document frequency).

voc = Vocabulary(TextConfig(), CASK_OF_AMONTILLADO; verbose=false)
vocsize(voc), gettrainsize(voc)

(798, 54)

2. Term Weighting and Vector Models

A VectorModel maps bag-of-words token counts into numeric weight vectors using term weighting schemes. Here, we instantiate standard Term Frequency - Inverse Document Frequency (TF-IDF) weighting using TfWeighting() and IdfWeighting():

\[\text{TF-IDF}(t, d) = \text{TF}(t, d) \times \log\left( 1 + \frac{|D|}{\text{DF}(t)} \right)\]

Using vectorize_corpus, the entire corpus is transformed into a collection of sparse vectors (SparseVector{Float32, Int32}):

model = VectorModel(IdfWeighting(), TfWeighting(), voc)
vecs = vectorize_corpus(model, CASK_OF_AMONTILLADO)
vecs[1]

798-element SparseArrays.SparseVector{Float32, Int32} with 82 stored entries:
  [1 ]  =  0.0531555
  [2 ]  =  0.1168
  [3 ]  =  0.1168
  [4 ]  =  0.0598652
  [5 ]  =  0.050581
  [6 ]  =  0.0781976
  [7 ]  =  0.0478702
        ⋮
  [75]  =  0.1168
  [76]  =  0.0892545
  [77]  =  0.0892545
  [78]  =  0.100193
  [79]  =  0.1168
  [80]  =  0.0478702
  [81]  =  0.0892545
  [82]  =  0.100193

Vocabulary Pruning

In natural language corpora, a significant fraction of terms appear only once (hapax legomena). These low-frequency terms increase vocabulary dimensionality without contributing generalizable discriminative information.

The filter_tokens function produces a pruned Vocabulary using predicate functions evaluated on token occurrences:

hapax_count = count(t -> t.occs == 1, voc[i] for i in eachindex(voc))
vocsize(voc), hapax_count

(798, 540)

# Prune terms with fewer than 3 total occurrences across the corpus
pruned_voc = filter_tokens(t -> t.occs >= 3, voc)
vocsize(pruned_voc)

148

filter_tokens returns a new Vocabulary without modifying the original instance. In multi-document collections, filtering by document frequency (t.ndocs) is often preferable to discard words that are frequent within a single document but absent elsewhere:

# Prune terms appearing in fewer than 3 distinct documents
pruned_by_docfreq = filter_tokens(t -> t.ndocs >= 3, voc)
vocsize(pruned_by_docfreq)

136

Vector-Space Information Retrieval: WeightedInvertedFile

WeightedInvertedFile indexes sparse weight vectors and evaluates similarity under a specified distance metric, defaulting to cosine distance via Dist.NormCosine from SimilaritySearch.jl:

wif = WeightedInvertedFile(vocsize(voc))
wctx = quietctx()
append_items!(wif, wctx, VectorDatabase(vecs))

res = knnqueue(KnnSorted, 5)
search(wif, wctx, vecs[1], res)
collect(IdView(res))

In the output, the first match is Document 1 at distance 0.0 (self-match), followed by the documents whose TF-IDF profiles are closest in the cosine space.

Free-Text Querying

To query the index with arbitrary unstructured text, transform the string into a sparse vector using vectorize:

qvec = vectorize(model, "vector search library")
res = knnqueue(KnnSorted, 5)
search(wif, wctx, qvec, res)
[(id, first(CASK_OF_AMONTILLADO[id], 60)) for id in collect(IdView(res))]

An inverted index generates candidate documents sharing at least one token with the query vector. If a query shares minimal vocabulary with the indexed corpus, the candidate set is appropriately restricted.

Vector Algebra: Dot Products, Centroids, and Normalization

Because document vectors are represented as SparseVector instances from SparseArrays.jl, standard linear algebra operations (+, -, dot, norm, normalize!) apply directly.

using LinearAlgebra, SparseArrays

q_wine = vectorize(model, "amontillado wine")
q_damp = vectorize(model, "nitre damp catacombs")
norm(q_wine), norm(q_damp)

(1.0f0, 1.0f0)

By default, vectorize scales vectors to unit Euclidean length ($\|v\|_2 = 1$). For unit-normalized vectors, the inner product equals the cosine similarity:

\[\langle u, v \rangle = \cos(\theta) = 1 - d_{\text{Cosine}}(u, v)\]

dot(q_wine, q_damp)  # Evaluates to 0.0 because the two queries have disjoint term supports

0.0f0

dot(vecs[51], vecs[52])  # Non-zero similarity between consecutive related paragraphs

0.5051559f0

Composite Query Formulation via Centroids

To construct a composite query representing multiple thematic aspects simultaneously, compute their spherical centroid:

q_both = centroid([q_wine, q_damp])
norm(q_both)

1.0f0

res = knnqueue(KnnSorted, 5)
search(wif, wctx, q_both, res)
[(id, first(CASK_OF_AMONTILLADO[id], 60)) for id in collect(IdView(res))]

Mathematical Requirement for Input Normalization

The centroid function computes the sum of input vectors and normalizes the resultant vector to unit length:

\[c = \frac{\sum_{i=1}^m v_i}{\left\| \sum_{i=1}^m v_i \right\|_2}\]

If input vectors do not possess identical $L_2$ norms, the resultant direction is biased toward the vector with larger magnitude:

a = sparsevec([1], [1.0f0], 2)  # Unit vector along coordinate 1
b = sparsevec([2], [1.0f0], 2)  # Unit vector along coordinate 2
centroid([a, b])                # Balanced combination with equal weights

2-element SparseArrays.SparseVector{Float32, Int64} with 2 stored entries:
  [1]  =  0.707107
  [2]  =  0.707107

b_scaled = b * 5.0f0            # Vector along coordinate 2 with 5x magnitude
centroid([a, b_scaled])         # Direction is skewed predominantly toward coordinate 2

2-element SparseArrays.SparseVector{Float32, Int64} with 2 stored entries:
  [1]  =  0.196116
  [2]  =  0.980581

Because vectorize produces unit-normalized vectors by default, queries combined via centroid maintain equal weighting. When combining vectors generated externally, ensure normalize!(v) is called prior to centroid computation.

Probabilistic Information Retrieval: BM25InvertedFile

BM25InvertedFile implements the Okapi BM25 ranking function. Unlike vector space models that evaluate geometric cosine distance over pre-computed sparse vectors, BM25 models term saturation and document length normalization directly:

\[\text{Score}(D, Q) = \sum_{q \in Q} \text{IDF}(q) \cdot \frac{f(q, D) \cdot (k_1 + 1)}{f(q, D) + k_1 \cdot \left(1 - b + b \cdot \frac{|D|}{\text{avgdl}}\right)}\]

BM25InvertedFile ingests raw strings, pre-tokenized structures, or bag-of-words representations directly without requiring an intermediate VectorModel:

bm25idx = BM25InvertedFile(voc)
bctx = quietctx()
append_items!(bm25idx, bctx, CASK_OF_AMONTILLADO)

res = knnqueue(KnnSorted, 5)
search(bm25idx, bctx, "amontillado nitre", res)
[(id, first(CASK_OF_AMONTILLADO[id], 60)) for id in collect(IdView(res))]

Selection Summary: Vector Space vs. BM25

• WeightedInvertedFile: Use when ranking under customized term weighting schemes (TF, IDF, TF-IDF) or when operating on general sparse feature vectors.
• BM25InvertedFile: Use for standard full-text document retrieval tasks benefiting from non-linear term saturation ($k_1$) and document-length penalization ($b$).

Index Persistence with JLD2

All primary structures (Vocabulary, VectorModel, BM25InvertedFile, WeightedInvertedFile) are concrete Julia types compatible with JLD2.jl serialization:

using JLD2

path = tempname() * ".jld2"
jldsave(path; voc, model, bm25idx)

loaded = load(path)
voc2, model2, bm25idx2 = loaded["voc"], loaded["model"], loaded["bm25idx"]
vectorize(model2, CASK_OF_AMONTILLADO[1]) == vecs[1]

true

res = knnqueue(KnnSorted, 5)
search(bm25idx2, quietctx(), "amontillado nitre", res)
collect(IdView(res))

Deserializing an index restores its posting lists and scoring parameters, enabling immediate query execution without retraining.

Granular Segmentation and External Tokenizers

1. Paragraph and Sentence Segmentation

TextSearch.jl includes utility functions tokenize_paragraphs and tokenize_sentences to segment long documents into fine-grained passages before index creation:

all_sentences = tokenize_sentences(CASK_OF_AMONTILLADO)
length(all_sentences), all_sentences[1]

(195, "The thousand injuries of Fortunato I had borne as I best could, but when he ventured upon insult, I vowed revenge.")

sentence_voc = Vocabulary(TextConfig(), all_sentences; verbose=false)
vocsize(sentence_voc), gettrainsize(sentence_voc)

(798, 195)

External sentence splitters (such as WordTokenizers.split_sentences) can also be integrated into preprocessing pipelines:

using WordTokenizers

sentences = String[]
for paragraph in CASK_OF_AMONTILLADO
    for s in split_sentences(paragraph)
        push!(sentences, String(s))
    end
end

length(sentences), sentences[1]

(195, "The thousand injuries of Fortunato I had borne as I best could, but when he ventured upon insult, I vowed revenge.")

2. Bypassing Redundant Normalization with isnormalized

When text has already undergone character normalization (e.g., lowercasing, punctuation stripping), passing isnormalized=true skips redundant transformation passes during vocabulary building and vectorization:

cfg = TextConfig(normalization=NormalizationConfig(lc=true, del_punc=true))
norm_sentences = tokenize_sentences(cfg, CASK_OF_AMONTILLADO)

norm_voc = Vocabulary(cfg, norm_sentences; isnormalized=true, verbose=false)
vocsize(norm_voc)

780

3. Integrating External Tokenizers with TokenizedText

To integrate external subword tokenizers (such as BPE, WordPiece, SentencePiece, or WordTokenizers.jl), wrap pre-tokenized string arrays in a TokenizedText container.

When functions such as Vocabulary, bagofwords, vectorize, and append_items! receive a TokenizedText, they consume the supplied tokens directly and bypass internal tokenization:

# Tokenize documents using an external tokenizer
wt_docs = [TokenizedText(String.(WordTokenizers.tokenize(lowercase(p)))) for p in CASK_OF_AMONTILLADO]
collect(wt_docs[1])[1:8]

8-element Vector{String}:
 "the"
 "thousand"
 "injuries"
 "of"
 "fortunato"
 "i"
 "had"
 "borne"

# Construct Vocabulary directly from pre-tokenized documents
wt_voc = Vocabulary(TextConfig(), wt_docs; verbose=false)
vocsize(wt_voc)

790

Dense Semantic Representations and Dimensionality Reduction

While inverted files provide exact retrieval for sparse representations, dense vector embeddings map semantically related terms and documents to continuous vector spaces.

TextSearch.jl implements two dimensionality reduction paradigms:

1. Latent Semantic Indexing (LSI): Low-rank matrix approximation via truncated Singular Value Decomposition (SVD).
2. Random Indexing (RI): Randomized projections governed by the Johnson-Lindenstrauss lemma, supporting scalar quantization and binary bit sketches.

Latent Semantic Indexing (LSI)

Theoretical Formulation

Let $X \in \mathbb{R}^{v \times n}$ denote the term-document matrix. Truncated SVD computes the rank-$k$ approximation:

\[X \approx U_k \Sigma_k V_k^T\]

The projection matrix $P = \Sigma_k^{-1} U_k^T$ maps sparse document vectors $d \in \mathbb{R}^v$ into dense semantic coordinates $z = P d \in \mathbb{R}^k$.

Training and Indexing with LSI

# Fit an LSI model with k = 16 latent dimensions
lsi = LatentSemanticIndexing(CASK_OF_AMONTILLADO; maxoutdim=16, verbose=false)
lsi

LatentSemanticIndexing: (indim=798 -> outdim=16)
  scaling: :none
  top singular values: Float32[1.9744779, 1.4556473, 1.3319054, 1.2267796, 1.193729]
  model: VectorModel:
  global_weighting: IdfWeighting()
  local_weighting: TfWeighting()
  maxoccs: 159
  Vocabulary:
    vocsize: 798
    trainsize: 54
    numtokens: 2635
    avgdoclen: 48.7962962962963
    TextConfig:
      NormalizationConfig:
          del_diac: true
          del_dup: false
          del_punc: false
          group_num: true
          group_url: true
          group_usr: false
          group_emo: false
          lc: true
          re_user: r"@[^;:,.@#&\\\-\"'/:\*\(\)\[\]\¿\?\¡\!\{\}~\<\>\|\s]+"
          re_url: r"(http|ftp|https)://\S+"
          re_num: r"[-+]?(\d+\.?\d*)|(\.\d+)"
          emojis: 3923 emoji chars
      TokenizationConfig:
          nlist: Int8[1]
          mark_token_type: true
          generators: AbstractTokenGenerator[]
      pipeline: TokenPipeline(lemmas=nothing, stopwords=nothing)
      language: unknown

# Project query string into a dense 16-dimensional vector
q_vec = vectorize(lsi, "wine vaults and connoisseur")
(length(q_vec), typeof(q_vec))

(16, Vector{Float32})

# Vectorize entire corpus into a dense MatrixDatabase
lsi_db = vectorize_corpus(lsi, CASK_OF_AMONTILLADO; verbose=false)
size(lsi_db.matrix)

(16, 54)

Dense LSI databases can be indexed using graph-based approximate indexes such as SearchGraph:

sctx = SearchGraphContext()
lsi_index = SearchGraph(Dist.NormCosine(), lsi_db)
index!(lsi_index, sctx)

res = knnqueue(KnnSorted, 3)
search(lsi_index, sctx, q_vec, res)
[(id, first(CASK_OF_AMONTILLADO[id], 60) * "...") for id in collect(IdView(res))]

3-element Vector{Tuple{UInt32, String}}:
 (0x0000001e, "\"These vaults,\" he said, \"are extensive.\"...")
 (0x0000001b, "\"Drink,\" I said, presenting him the wine....")
 (0x0000001d, "\"I drink,\" he said, \"to the buried that repose around us.\"...")

Word Embeddings and Query Expansion Networks

The column vectors of the projection matrix correspond to dense word embeddings. The function wordvectors extracts these embeddings into a MatrixDatabase:

W = wordvectors(lsi)
size(W.matrix)

(16, 798)

The function query_expansion computes an all-pairs nearest-neighbor graph over the vocabulary embeddings to produce a semantic term-expansion network:

net = query_expansion(lsi, 5; verbose=false)
net.query_expansion["wine"]

5-element Vector{String}:
 "presenting"
 "drink"
 "around"
 "buried"
 "repose"

Random Indexing and Quantization Pipelines

Properties of Random Indexing

Random Indexing assigns a fixed, pseudo-orthogonal random vector $r_t \in \mathbb{R}^k$ ($k \ll v$) to each vocabulary token $t$. A document vector is constructed incrementally as the linear combination of its constituent token vectors:

\[z = \sum_{t \in D} w(t, D) r_t\]

Advantages include:

• Streaming Computation: Incremental projection of new documents without requiring full-matrix SVD refactoring.
• Distance Preservation: Bounds metric distortion in accordance with the Johnson-Lindenstrauss lemma.
• Compression Compatibility: Direct integration with 8-bit scalar quantization and binary bit sketches.

1. Dense Random Indexing (Float32)

ri = RandomIndexing(CASK_OF_AMONTILLADO; maxoutdim=64, method=:gaussian, verbose=false)
ri_db = vectorize_corpus(ri, CASK_OF_AMONTILLADO; verbose=false)
size(ri_db.matrix)

(64, 54)

2. 8-Bit Scalar Quantization (SQu8 / SQgu8)

Scalar quantization compresses 32-bit floating point dimensions into 8-bit unsigned integers (UInt8), reducing memory requirements by 4$\times$:

using SimilaritySearch.ScalarQuant: SQu8, SQgu8

# Quantize entire corpus representation
squ8_db = vectorize_corpus(SQu8, ri, CASK_OF_AMONTILLADO; verbose=false)

# Quantize single query vector
q_squ8 = vectorize(SQu8, ri, "damp vaults and catacombs")

# Search over quantized representations using SQu8.NormCosine()
squ8_index = ExhaustiveSearch(SQu8.NormCosine(), squ8_db)
res_squ8 = knnqueue(KnnSorted, 2)
search(squ8_index, GenericContext(), q_squ8, res_squ8)
[(id, first(CASK_OF_AMONTILLADO[id], 60) * "...") for id in collect(IdView(res_squ8))]

2-element Vector{Tuple{UInt32, String}}:
 (0x0000001e, "\"These vaults,\" he said, \"are extensive.\"...")
 (0x0000001c, "He raised it to his lips with a leer. He paused and nodded t...")

3. Binary Bit Sketches and Hamming Search (BitSketch)

BitSketch applies random hyperplane projections, packing projection signs into 64-bit unsigned integers (UInt64). Distance evaluation is computed via hardware-accelerated bitwise Hamming distance:

# Project corpus into 512-bit binary signatures (8 × UInt64 words per document)
ri_bits = RandomIndexing(CASK_OF_AMONTILLADO; maxoutdim=512, verbose=false)
bits_db = bitsketch(ri_bits, CASK_OF_AMONTILLADO; verbose=false)
(typeof(bits_db), size(bits_db.matrix))

(MatrixDatabase{Matrix{UInt64}}, (8, 54))

# Query bit sketch generation
q_bits = bitsketch(ri_bits, "damp vaults and catacombs")

# Exact Hamming distance search
bit_index = ExhaustiveSearch(Dist.Bits.Hamming(), bits_db)
res_bits = knnqueue(KnnSorted, 2)
search(bit_index, GenericContext(), q_bits, res_bits)
[(id, first(CASK_OF_AMONTILLADO[id], 60) * "...") for id in collect(IdView(res_bits))]

2-element Vector{Tuple{UInt32, String}}:
 (0x0000001e, "\"These vaults,\" he said, \"are extensive.\"...")
 (0x0000000c, "\"My friend, no. It is not the engagement, but the severe col...")

Portable Text Profiles (TextProfile)

A TextProfile encapsulates the statistical and linguistic artifacts estimated from a corpus (vocabulary frequencies, term weightings, stopword sets, lemma mappings, and expansion networks) into a portable, inspectable specification.

Architecture: Policy vs. Artifacts

• Policy (TextConfig): Declarative rules governing text normalization and token extraction (independent of corpus statistics).
• Artifacts (TextProfile): Empirical models estimated from data. The profile derives its active tokenizer configuration directly from its declared policy and active artifacts.

stop = Set(stopword_candidates(voc, 0.5))
lemmas = lemma_clusters(voc, W)

profile = TextProfile(model;
                      stopwords=stop, lemmas,
                      query_expansion=net.query_expansion,
                      query_expansion_distances=net.distances,
                      applied=AppliedArtifacts(stopwords=true),
                      lineage=[LineageStep(:fit; trainsize=length(CASK_OF_AMONTILLADO), outdim=16)])

(stopwords=length(profile.stopwords), lemmas=length(profile.lemmas),
 expansion=length(profile.query_expansion), base=isbase(profile))

(stopwords = 8, lemmas = 13, expansion = 798, base = true)

The function fit_profile provides a single-call pipeline to estimate all profile components:

oneshot = fit_profile(TextConfig(), CASK_OF_AMONTILLADO;
                      stopwords=(; doc_freq_threshold=0.5),
                      encoder=(; outdim=16), expansion=(; k=5), verbose=false)
(vocsize=vocsize(oneshot.model.voc), stopwords=length(oneshot.stopwords),
 expansion=length(oneshot.query_expansion), base=isbase(oneshot))

(vocsize = 790, stopwords = 8, expansion = 790, base = true)

JSON Serialization and Inspection

Profiles serialize to standard JSON files via save_profile and zip_profile, enabling cross-platform inspection and version control without executing arbitrary serialized code:

dir = mktempdir()
save_profile(dir, profile)
sort(readdir(dir))

7-element Vector{String}:
 "lemmas.json"
 "manifest.json"
 "query_expansion.json"
 "query_expansion_distances.json"
 "stopwords.json"
 "vocabulary.json"
 "weights.json"

Applied vs. Carried Artifacts

Artifacts can be stored in a profile as reference data without being activated in the tokenization pipeline. The function with_applied dynamically activates or deactivates artifacts:

p = load_profile(dir)
gettextconfig(p).pipeline, p.applied

(TokenPipeline(lemmas=nothing, stopwords=8 tokens), applied(stopwords))

# Activate lemmatization in the profile's tokenization pipeline
q = with_applied(p; lemmas=true)
gettextconfig(q).pipeline.lemmas !== nothing, q.applied

(true, applied(stopwords, lemmas))

Merging Partitioned Profiles: merge_profiles

When processing large-scale corpora in distributed partitions, merge_profiles combines independent batch profiles into a single unified profile. Vocabulary counts across disjoint subsets are additive, guaranteeing mathematically exact inverse document frequencies:

half = length(CASK_OF_AMONTILLADO) ÷ 2
onebatch(docs) = TextProfile(VectorModel(IdfWeighting(), TfWeighting(),
                                         Vocabulary(TextConfig(), docs; verbose=false)))
a = onebatch(CASK_OF_AMONTILLADO[1:half])
b = onebatch(CASK_OF_AMONTILLADO[half+1:end])
merged = merge_profiles([a, b])

(a=gettrainsize(a.model.voc), b=gettrainsize(b.model.voc),
 merged=gettrainsize(merged.model.voc), lineage=lineage_summary(merged))

(a = 27, b = 27, merged = 54, lineage = "merge(n_sources=2, trainsize=54)")

Profile Adaptation: refit_profile

refit_profile adapts an existing base profile to a specialized target domain using Bayesian-style updating. The base profile serves as a prior weighted by parameter kappa relative to empirical observations in the adaptation sample:

tuned = refit_profile(p, CASK_OF_AMONTILLADO[1:6]; verbose=false)
(tuned=istuned(tuned), vocsize=vocsize(tuned.model.voc), lineage=lineage_summary(tuned))

(tuned = true, vocsize = 232, lineage = "fit(outdim=16, trainsize=54) -> refit(kappa=6.0, lemmas_applied=true, sample_trainsize=6, trainsize=12)")

The predicates isbase and istuned verify model provenance directly from recorded lineage history.

Pre-trained Language Profiles and CLI Management

TextSearch.jl distributes pre-computed linguistic profiles for major languages (e.g., English en, Spanish es, Basque eu, French fr, Italian it, and Portuguese pt) fitted on large Wikipedia paragraph corpora. These profiles provide production-ready vocabularies, IDF weights, lemma maps, and query expansion networks out of the box.

Julia API

You can discover, download, and load pre-trained profiles directly in Julia:

using TextSearch

# List available remote profiles published on GitHub releases
remotes = list_remote_profiles()
for r in remotes
    println(r.name, " (", round(r.size / 1024^2; digits=1), " MB) -> ", r.url)
end

# Download and install a pre-trained profile locally (~/.textsearch/profiles/es.zip)
path = download_profile("es")

# Load the downloaded profile into memory
p = load_profile(path)

Command-Line Interface (textsearch)

The integrated CLI tool textsearch (located in apps/textsearch/) allows managing the complete lifecycle of profiles from the terminal:

# Discover available pre-computed profiles on GitHub releases (or custom --url)
textsearch list --remote

# Download and install profiles locally
textsearch download es en pt

# Download from an arbitrary URL or custom release tag
textsearch download https://example.com/profiles/custom_model.zip --as custom
textsearch download es --tag v1.1.0 --force

# List locally installed profiles
textsearch list

# Inspect detailed vocabulary, lineage, and artifact statistics
textsearch info es

# Search a document collection using the profile's pipeline and expansions
textsearch search es "aprendizaje automático" --collection dataset.jsonl

# Uninstall an installed profile
textsearch uninstall custom --force

Query Token Resolution and Spelling Normalization

When a corpus preserves case and diacritics, queries entered in un-normalized form may fail to match exact vocabulary entries:

cased = TextConfig(normalization=NormalizationConfig(lc=false))
cvoc = Vocabulary(cased, CASK_OF_AMONTILLADO; verbose=false)
token2id(cvoc, "amontillado"), token2id(cvoc, "Amontillado")  # ID 0 indicates out-of-vocabulary

(0x00000000, 0x000000dd)

The function derive_variants computes an auxiliary dictionary of non-derivable surface variants:

variants = derive_variants(cvoc)

Dict{String, Vector{String}} with 2 entries:
  "_at" => ["_At"]
  "_in" => ["_In"]

The function resolve_query_tokens maps query tokens to their most probable vocabulary forms:

r = resolve_query_tokens(cvoc, ["amontillado", "wine"], variants)
r.tokens, explain(r)

(["Amontillado", "wine"], ["amontillado not found, searched as Amontillado (derived) instead"])

The QueryPolicy structure controls query resolution behavior, permitting manual disabling of spelling substitution or query expansion:

resolve_query_tokens(cvoc, ["amontillado"], variants, QueryPolicy(correction=:off)).tokens

1-element Vector{String}:
 "amontillado"

expansion_sources(r)

2-element Vector{String}:
 "Amontillado"
 "wine"

Social Media and Informal Text Processing

TextConfig includes dedicated normalization options for informal and social media text:

• group_usr: Maps user handles (@username) to the canonical placeholder token _usr.
• group_url: Maps web links (https://...) to _url.
• group_emo: Maps emoji characters to a unified symbol (👾), preventing vocabulary fragmentation across sparse emojis.
• Hashtags (#topic) are preserved as informative semantic tokens.

using TextSearch, SimilaritySearch

quietctx() = InvertedFileContext(logger=SimilaritySearch.LogList(SimilaritySearch.AbstractLog[]))

tweets = [
    "Just landed in Mexico City!! 🎉 cant wait to try the tacos @VisitMexico #travel",
    "Ugh, stuck in traffic again on the highway :( #mondayblues",
    "New paper on approximate similarity search is out! check it out https://example.org/paper",
    "@juli_ai loved your talk on vector databases today, so insightful #ai #ml",
    "Rainy day, perfect for reading a good book ☕📚",
    "Why does @united keep cancelling flights?? this is the third time this month #travelfail",
    "Excited to announce our new open source vector search release! https://github.com/example/repo #julialang",
    "lol this meme is too real 😂😂😂 #mood",
    "Can anyone recommend a good vector search library for Julia? asking for a friend @julialang",
    "Beautiful sunset over the bay tonight 🌅 #nofilter",
]

cfg = TextConfig(normalization=NormalizationConfig(group_usr=true, group_url=true, group_emo=true, del_punc=false))
collect(TextSearch.tokenize(cfg, tweets[1]))

15-element Vector{String}:
 "just"
 "landed"
 "in"
 "mexico"
 "city"
 "!!"
 "👾"
 "cant"
 "wait"
 "to"
 "try"
 "the"
 "tacos"
 "_usr"
 "#travel"

voc = Vocabulary(cfg, tweets; verbose=false)
bm25idx = BM25InvertedFile(voc)
ctx = quietctx()
append_items!(bm25idx, ctx, tweets)

res = knnqueue(KnnSorted, 3)
search(bm25idx, ctx, "vector search library", res)
[(id, tweets[id]) for id in collect(IdView(res))]

BM25 scores reflect query term coverage and document-level term salience across the informal collection.

Summary and API Reference

For complete function signatures and algorithmic details, consult the TextSearch API reference.