using SimilaritySearch, SimSearchManifoldLearning, Plots, StatsBase, LinearAlgebra, Markdown, Random, PrintfVisualizing MNIST database
by: Eric S. Téllez
This demonstration shows in a 2D example the functionality of SearchGraph.
n = 100_000
M = randn(Float16, 2, n)
db = MatrixDatabase(M)
dist = SqL2_asf32()
size(M)Now we can create the index
1G = SearchGraph(; dist, db)
ctx = SearchGraphContext(hyperparameters_callback=OptimizeParameters(MinRecall(0.99)))
2index!(G, ctx)
3optimize_index!(G, ctx, MinRecall(0.9))- 1
-
Defines the index and the search context (caches and hyperparameters); particularly, we use a very high quality build
MinRecall(0.99); high quality constructions yield to faster queries due to the underlying graph structure. - 2
- Actual indexing procedure using the given search context.
- 3
- Optimizing the index to trade quality and speed.
The set of queries
We define a small set of queries being close to the border of the dataset and also in the most dense regions of the dataset.
Q = [Float32[-2, -2], Float32[2, -2], Float32[-2, 0], Float32[-0, 2], Float32[0, 0], Float32[-3, 3], Float32[4, 4], Float32[1, 0.5]]
knns = searchbatch(G, ctx, VectorDatabase(Q), 30)Please note how queries in low and high dense regions are located.
scatter(view(M, 1, :), view(M, 2, :), fmt=:png, c=:cyan, ma=0.3, a=0.3, ms=1, msw=0)
scatter!(getindex.(Q, 1), getindex.(Q, 2), c=:red, ma=0.7, a=0.7, ms=6, msw=0)
for c in eachcol(knns)
X = M[:, sort!(collect(IdView(c)))]
scatter!(view(X, 1, :), view(X, 2, :), c=:blue, ma=0.5, a=0.5, ms=2, msw=0)
#scatter!( c=:auto, ms=2)
end
display(plot!(legend=nothing))GKS: cannot open display - headless operation mode active
Since points are distributed in several regions with disparate density, their radii are also quite diverse. The next list illustrates the distribution of distances for the set of queries
| query ID | x | y | radius |
|---|---|---|---|
| 1 | -2.0 | -2.0 | 0.4191 |
| 2 | 2.0 | -2.0 | 0.2382 |
| 3 | -2.0 | 0.0 | 0.0643 |
| 4 | 0.0 | 2.0 | 0.0759 |
| 5 | 0.0 | 0.0 | 0.0262 |
| 6 | -3.0 | 3.0 | 1.1678 |
| 7 | 4.0 | 4.0 | 2.5621 |
| 8 | 1.0 | 0.5 | 0.0371 |
Note how the central radius are quite dense.
Environment and dependencies
Julia Version 1.10.10
Commit 95f30e51f41 (2025-06-27 09:51 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Linux (x86_64-linux-gnu)
CPU: 64 × Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-15.0.7 (ORCJIT, cascadelake)
Threads: 64 default, 0 interactive, 32 GC (on 64 virtual cores)
Environment:
JULIA_PROJECT = .
JULIA_NUM_THREADS = auto
JULIA_LOAD_PATH = @:@stdlib
Status `~/Research/SimilaritySearchDemos/Project.toml`
[aaaa29a8] Clustering v0.15.8
[944b1d66] CodecZlib v0.7.8
[a93c6f00] DataFrames v1.8.0
[c5bfea45] Embeddings v0.4.6
[f67ccb44] HDF5 v0.17.2
[b20bd276] InvertedFiles v0.8.1
[682c06a0] JSON v0.21.4
[23fbe1c1] Latexify v0.16.10
[eb30cadb] MLDatasets v0.7.18
[06eb3307] ManifoldLearning v0.9.0
⌃ [ca7969ec] PlotlyLight v0.11.0
⌃ [91a5bcdd] Plots v1.40.20
[27ebfcd6] Primes v0.5.7
[ca7ab67e] SimSearchManifoldLearning v0.3.2
[053f045d] SimilaritySearch v0.13.0
⌅ [2913bbd2] StatsBase v0.33.21
⌃ [f3b207a7] StatsPlots v0.15.7
[7f6f6c8a] TextSearch v0.19.6
Info Packages marked with ⌃ and ⌅ have new versions available. Those with ⌃ may be upgradable, but those with ⌅ are restricted by compatibility constraints from upgrading. To see why use `status --outdated`