Tutorial

This tutorial builds up SimilaritySearch.jl from first principles, using small, self-contained synthetic datasets instead of large embedding collections (no downloads, nothing random-looking-but-meaningless – every example dataset here has a concrete story you can check by hand). The running example across most pages is numbers seen through different lenses: as sets of prime factors, as sequences of prime factors, as divisibility bit patterns, and as prime-gap vectors. The same handful of integers keep reappearing so you can compare how differently each distance function treats them.

Pages, in the order we recommend reading them:

1. This page – installation and a five-minute quickstart.
2. Databases: why not just a Matrix? – the AbstractDatabase abstraction and why the library is built around it instead of raw arrays.
3. A gallery of distances – worked synthetic examples for vector, angular, set, sequence, and bit distances, and why SearchGraph should not be used with discrete/combinatorial distances.
4. SearchGraph, in depth – the approximate index: how it works, when to reach for it, and (again, because it matters) when not to.
5. Radius queries: RadiusSorted, RadiusHeap – "every point within distance r", instead of a fixed k, on both ExhaustiveSearch and SearchGraph.
6. Beyond search: fft, allknn, closestpair, neardup – other things you can do with an index besides answering k-NN queries.
7. Two datasets, one distance: bichromatic closest pairs and joins – closestpair's generalization to two distinct datasets, its k-pairs version, and a metric join between them.
8. Parallelism: what to expect, what not to do – the @BATCHES-based threading model, its context objects, and concrete anti-patterns to avoid.
9. Saving and loading indexes with JLD2 – indexes are plain structs; here's the DIY save/load pattern now that this package doesn't ship its own.
10. Logs, and capturing neighbors as they're built – the logging system, custom loggers, and how (and how not) to use them to capture a SearchGraph's neighbor lists incrementally during construction.
11. Inverted files and posting list intersections – the InvertedFile index for sparse vector, MIPS, and set search, and posting list intersection algorithms (Intersections).
12. Quantization and Bit Sketches – techniques for compressing vectors into smaller memory footprints and accelerating search using ScalarQuant and bitsketch with ExhaustiveSearch.

Installation

] add SimilaritySearch

Everything in this tutorial only needs SimilaritySearch itself plus Distances (for the evaluate function used to call distance objects directly) – no extra packages, no downloaded datasets.

using SimilaritySearch, Distances

Five-minute quickstart

Throughout the tutorial we lean on ExhaustiveSearch for almost every worked example: it is exact (no recall/approximation questions to worry about while you're learning the API) and, on the small datasets used here, essentially instant. SearchGraph gets its own dedicated page once the underlying concepts (databases, distances) are in place.

Let's index a tiny, fully synthetic "space of numbers": we represent each integer by the set of its distinct prime factors, and compare integers by how much those sets overlap (a Dice distance – two numbers are "close" if they share many small prime factors).

using SimilaritySearch, Distances

"""
Distinct prime factors of `n`, sorted -- e.g. `factors(60) == Int32[2, 3, 5]`
(60 = 2²·3·5). `Dist.Sets` distances expect their inputs as sorted vectors like this one.
"""
function factors(n::Integer)
    f = Int32[]
    m = n
    d = Int32(2)
    while d * d <= m
        if m % d == 0
            push!(f, d)
            while m % d == 0
                m ÷= d
            end
        end
        d += 1
    end
    m > 1 && push!(f, m)
    isempty(f) ? Int32[1] : f  # 1 has no prime factors; give it its own placeholder "factor"
end

n = 1000
X = VectorDatabase([factors(i) for i in 1:n])   # X[i] holds the prime factors of i
dist = Dist.Sets.Dice()

idx = ExhaustiveSearch(dist, X)
ctx = GenericContext()

res = knnqueue(ctx, 5)                 # a reusable result buffer for k=5
search(idx, ctx, factors(1000), res)   # which numbers "look like" 1000 = 2³·5³?

for p in IdDistView(res)  # (id, dist) pairs, in storage order
    println(p.id, " => ", factors(p.id), "  dist=", p.dist)
end

Running this prints five numbers whose set of prime factors overlaps heavily with {2, 5} (1000's factors) – e.g. other numbers of the form 2^a · 5^b. Note that a result's dist == 0.0 doesn't mean "the same number", it means "the same set of distinct prime factors" – e.g. 1000 and 500 both factor as {2, 5}, so they tie at distance 0 despite being different integers. That's expected: Dice/Jaccard-style set distances only see which primes divide a number, not how many times or what the number itself is.

To search many queries at once instead of one at a time, use searchbatch:

queries = VectorDatabase([factors(i) for i in (7, 60, 97, 360, 999)])
knns = searchbatch(idx, ctx, queries, 5)   # a (5, 5) matrix of IdDist

knns[:, j] holds the 5 nearest neighbors of queries[j], sorted or not depending on the sorted keyword (see searchbatch).

From here: why does the library wrap X in a VectorDatabase instead of just using the Vector directly? – next page.