Parallel construction and parallel search

by: Eric S. Téllez

using SimilaritySearch

Similarity search on very large datasets and high-dimensional datasets require high computational resources. In this example we show how to arallelize both the construction and search to be able to handle this kind of databases.

dim = 16
1db = MatrixDatabase(randn(Float32, dim, 10^5))
2queries = MatrixDatabase(randn(Float32, dim, 30))
3dist = Dist.SqL2()
4ctx = SearchGraphContext(parallel_block=512)
5G = SearchGraph(; dist, db)
6index!(G, ctx)

1

A synthetic database of dimension 16 and \(10^5\) vectors.

2

A synthetic query set of \(30\) points.

3

The distance function.

4

The search context working with SearchGraph; a set of hyperparameters for the index.

5

The index definition.

6

The index construction.

The SearchGraph construction algorithm is incremental:

• If the index is empty, an element is inserted just inserting it into the index.
• If the index is not empty, the element is inserted and connected to its nearest neighbors (looking into the current index).

The parallel construction is made with index! or append_items!; for this matter these functions accept a parallel_block argument via the ctx context, that controls how many elements are inserted at once, i.e., looking for its nearest neighbors in parallel and connected also in parallel.

The parallel_block argument should be set to at least the number of available threads, and perhaps multiplying it by a small constant is also a good approach.

Note that you must not call push_item!, append_items!, or index! from several threads. The default algorithm will takes advantage of the available threads using a single call.

Searching

Once the index is constructed, you can solve batches in parallel and also single queries. In contrast with append, these functions can be called in multithreading algorithms. However, you must pause the searching requests while perform insertions (parallel or sequential); mixing insertions and search produces an undefined behavior for search results.

knns = searchbatch(G, ctx, queries, 10)

Threads.@threads for i in eachindex(queries)
    res = search(G, ctx, queries[i], knnqueue(ctx, 10))
    
    print("=== $i -- nearest neighbor:")
    println(nearest(res))
    print("=== $i -- result set:")
    println(collect(viewitems(res)))
    print("=== $i -- identifiers:")
    println(collect(IdView(res))) # do something with `res`
    print("=== $i -- distances:")
    println(collect(DistView(res))) # do something with `res`
end

=== 5 -- nearest neighbor:=== 9 -- nearest neighbor:=== 1 -- nearest neighbor:=== 28 -- nearest neighbor:=== 21 -- nearest neighbor:=== 13 -- nearest neighbor:IdDist(=== 17 -- nearest neighbor:=== 25 -- nearest neighbor:0x000002e5, 4.3059063f0)
=== 21 -- result set:IdDist(0x00016bc3, 5.4525123f0)
IdDist=== 5 -- result set:(0x00016290, 5.8617473f0)
IdDist=== 1 -- result set:(0x0000db18, 6.104665f0)
=== 28 -- result set:IdDist(0x000107ef, 4.8560977f0)
IdDist=== 13 -- result set:(0x0000f54b, 6.845487f0)
IdDist=== 9 -- result set:(0x0000e709, 6.609312f0)
=== 25 -- result set:IdDist(0x00012310, 5.0925045f0)
=== 17 -- result set:IdDist[IdDist(0x000002e5, 4.3059063f0), IdDist(0x0000f3b2, 4.482109f0), IdDist(0x00017420, 5.1111937f0), IdDist(0x00013954, 5.1871443f0), IdDist(0x00013670, 5.2748733f0), IdDist(0x00015742, 5.312173f0), IdDist(0x00005f6c, 5.707f0), IdDist(0x0000c6ab, 6.194291f0), IdDist(0x0000f5c0, 6.2215834f0), IdDist(0x00005c46, 6.2245893f0)]
=== 21 -- identifiers:IdDist[IdDist(0x00012310, 5.0925045f0), IdDist(0x00003274, 5.4198375f0), IdDist(0x0000de55, 6.102253f0), IdDist(0x00010b3e, 6.432191f0), IdDist(0x0001809f, 6.7323294f0), IdDist(0x0000b8bc, 6.893191f0), IdDist(0x0000cbec, 7.225857f0), IdDist(0x0000cf4e, 7.2660074f0), IdDist(0x0000be12, 7.7268887f0), IdDist(0x00002a09, 7.953128f0)]
=== 17 -- identifiers:IdDist[IdDist(0x00016bc3, 5.4525123f0), IdDist(0x000155e4, 5.7611504f0), IdDist(0x00000823, 5.7633977f0), IdDist(0x0001145a, 5.832895f0), IdDist(0x0000c2ba, 5.888486f0), IdDist(0x00000f7f, 5.893327f0), IdDist(0x0000cc8b, 6.0296946f0), IdDist(0x000154cc, 6.0736747f0), IdDist(0x0000ed44, 6.5637093f0), IdDist(0x0000d7a7, 6.7250385f0)]
=== 5 -- identifiers:IdDist[IdDist(0x0000f54b, 6.845487f0), IdDist(0x00000127, 7.2372f0), IdDist(0x0001529b, 7.6255383f0), IdDist(0x00005829, 7.6853895f0), IdDist(0x0000da6a, 7.7175717f0), IdDist(0x0000de88, 7.7305503f0), IdDist(0x00010209, 7.987199f0), IdDist(0x0000dbf4, 8.392647f0), IdDist(0x000143bc, 8.524873f0), IdDist(0x000171c4, 8.713152f0)]
=== 9 -- identifiers:IdDist[IdDist(0x0000e709, 6.609312f0), IdDist(0x000036a7, 7.8590126f0), IdDist(0x0000a5df, 8.166045f0), IdDist(0x00005ecb, 8.210051f0), IdDist(0x000027cd, 8.374267f0), IdDist(0x00003599, 8.521223f0), IdDist(0x00004e76, 8.596121f0), IdDist(0x000183f3, 8.667982f0), IdDist(0x0000e8b9, 8.759136f0), IdDist(0x00007803, 8.767553f0)]
IdDist=== 25 -- identifiers:[IdDist(0x00016290, 5.8617473f0), IdDist(0x00010263, 6.007363f0), IdDist(0x0000afd7, 6.0218587f0), IdDist(0x0000efa6, 6.4703975f0), IdDist(0x00011971, 6.8498907f0), IdDist(0x00000b77, 7.266276f0), IdDist(0x00005ad4, 7.736638f0), IdDist(0x00015f13, 7.9257145f0), IdDist(0x00006297, 7.9318075f0), IdDist(0x00010b38, 8.000961f0)]
IdDist=== 1 -- identifiers:[IdDist(0x0000db18, 6.104665f0), IdDist(0x000017a7, 7.1883287f0), IdDist(0x00005983, 7.749861f0), IdDist(0x0000bc15, 8.438281f0), IdDist(0x00010fe5, 8.454905f0), IdDist(0x0000aa3e, 8.5403f0), IdDist(0x0000c5c3, 8.879941f0), IdDist(0x0000db59, 9.382742f0), IdDist(0x000113bd, 9.581569f0), IdDist(0x00015afd, 9.9981365f0)]
=== 28 -- identifiers:IdDist[IdDist(0x000107ef, 4.8560977f0), IdDist(0x00017700, 4.8576574f0), IdDist(0x0000e89e, 4.9859977f0), IdDist(0x00017690, 5.146263f0), IdDist(0x00005991, 5.1891136f0), IdDist(0x000141c9, 5.2270436f0), IdDist(0x00015482, 5.7881804f0), IdDist(0x0000057a, 5.80683f0), IdDist(0x000155e9, 5.8250713f0), IdDist(0x00002baa, 5.8350725f0)]
=== 13 -- identifiers:UInt32[0x000002e5, 0x0000f3b2, 0x00017420, 0x00013954, 0x00013670, 0x00015742, 0x00005f6c, 0x0000c6ab, 0x0000f5c0, 0x00005c46]
UInt32=== 21 -- distances:[0x00016290, 0x00010263, 0x0000afd7, 0x0000efa6, 0x00011971, 0x00000b77, 0x00005ad4, 0x00015f13, 0x00006297, 0x00010b38]
=== 1 -- distances:UInt32[0x0000e709, 0x000036a7, 0x0000a5df, 0x00005ecb, 0x000027cd, 0x00003599, 0x00004e76, 0x000183f3, 0x0000e8b9, 0x00007803]
UInt32=== 25 -- distances:[0x0000f54b, 0x00000127, 0x0001529b, 0x00005829, 0x0000da6a, 0x0000de88, 0x00010209, 0x0000dbf4, 0x000143bc, 0x000171c4]
=== 9 -- distances:UInt32[0x000107ef, 0x00017700, 0x0000e89e, 0x00017690, 0x00005991, 0x000141c9, 0x00015482, 0x0000057a, 0x000155e9, 0x00002baa]
UInt32=== 13 -- distances:[0x0000db18, 0x000017a7, 0x00005983, 0x0000bc15, 0x00010fe5, 0x0000aa3e, 0x0000c5c3, 0x0000db59, 0x000113bd, 0x00015afd]
=== 28 -- distances:UInt32[0x00012310, 0x00003274, 0x0000de55, 0x00010b3e, 0x0001809f, 0x0000b8bc, 0x0000cbec, 0x0000cf4e, 0x0000be12, 0x00002a09]
=== 17 -- distances:UInt32[0x00016bc3, 0x000155e4, 0x00000823, 0x0001145a, 0x0000c2ba, 0x00000f7f, 0x0000cc8b, 0x000154cc, 0x0000ed44, 0x0000d7a7]
=== 5 -- distances:Float32[4.3059063, 4.482109, 5.1111937, 5.1871443, 5.2748733, 5.312173, 5.707, 6.194291, 6.2215834, 6.2245893]
Float32[6.609312, 7.8590126,=== 22 -- nearest neighbor: 8.166045, 8.210051, 8.374267, 8.521223, 8.596121, 8.667982, 8.759136, 8.767553]
Float32=== 26 -- nearest neighbor:[5.0925045, 5.4198375, 6.102253, 6.432191, 6.7323294, 6.893191, 7.225857, 7.2660074, 7.7268887, 7.953128]
Float32[5.4525123=== 18 -- nearest neighbor:, 5.7611504, 5.7633977, 5.832895, 5.888486, 5.893327, 6.0296946, 6.0736747, 6.5637093, 6.7250385]
Float32[5.8617473, === 6 -- nearest neighbor:6.007363, 6.0218587, 6.4703975, 6.8498907, 7.266276, 7.736638, 7.9257145, 7.9318075, 8.000961]
IdDist(0x000046b3, === 2 -- nearest neighbor:4.237101f0)
=== 22 -- result set:IdDist(0x00007e2b, 3.675293f0)
=== 18 -- result set:Float32[6.104665, 7.1883287, 7.749861, 8.438281, 8.454905, 8.5403, 8.879941, 9.382742, 9.581569, 9.9981365]
IdDist(0x00015b9c, 6.8040795f0)=== 29 -- nearest neighbor:
=== 26 -- result set:IdDist[IdDist(0x00007e2b, 3.675293f0), IdDist(0x00005f2a, 4.157851f0), IdDist(0x00000b10, 4.617572f0), IdDist(0x0000a2ac, 4.772978f0), IdDist(0x0000284d, 4.9854174f0), IdDist(0x00014477, 4.9955883f0), IdDist(0x0000578e, 5.1739335f0), IdDist(0x00005fd6, 5.2381277f0), IdDist(0x000077fc, 5.25712f0), IdDist(0x00009550, 5.258646f0)]
=== 18 -- identifiers:IdDist(0x00017338, 5.7050834f0)
IdDist=== 29 -- result set:[IdDist(0x000046b3, 4.237101f0), IdDist(0x0000cddc, 4.995167f0), IdDist(0x0000ccba, 5.519479f0), IdDist(0x00010722, 5.5932918f0), IdDist(0x00013864, 5.612316f0), IdDist(0x0000d5a6, 5.8138113f0), IdDist(0x0000e077, 5.9479756f0), IdDist(0x0000e9db, 6.055376f0), IdDist(0x00002960, 6.1627603f0), IdDist(0x00001ddf, 6.1689744f0)]
=== 22 -- identifiers:Float32[6.845487, 7.2372, 7.6255383, 7.6853895, 7.7175717, 7.7305503, 7.987199, 8.392647, 8.524873, 8.713152]
UInt32[0x00007e2b, 0x00005f2a=== 10 -- nearest neighbor:, 0x00000b10, 0x0000a2ac, 0x0000284d, 0x00014477, 0x0000578e, 0x00005fd6, 0x000077fc, 0x00009550]
=== 18 -- distances:Float32[4.8560977, 4.8576574, 4.9859977, 5.146263, 5.1891136, 5.2270436, 5.7881804, 5.80683, 5.8250713, 5.8350725]
IdDist(0x0000d3da, 3.3228424f0)=== 14 -- nearest neighbor:
=== 2 -- result set:IdDist[IdDist(0x00017338, 5.7050834f0), IdDist(0x00010633, 5.813157f0), IdDist(0x0000c55b, 6.1480823f0), IdDist(0x00014949, 6.224143f0), IdDist(0x00003b78, 6.2271614f0), IdDist(0x000169b2, 6.4017773f0), IdDist(0x000022c1, 6.4356084f0), IdDist(0x00016192, 6.7351327f0), IdDist(0x000024c8, 6.979508f0), IdDist(0x00006dc9, 7.1391606f0)]
IdDist=== 29 -- identifiers:(0x00015477, 3.6226947f0)
=== 6 -- result set:IdDist(0x00011394, 4.1133566f0)
=== 14 -- result set:IdDist[IdDist(0x0000d3da, 3.3228424f0), IdDist(0x00017e8f, 5.1148386f0), IdDist(0x00000876, 5.256465f0), IdDist(0x00010411, 5.2807727f0), IdDist(0x00003c3e, 5.4705386f0), IdDist(0x00005d55, 5.478687f0), IdDist(0x0000111b, 5.510663f0), IdDist(0x000168a8, 5.7863193f0), IdDist(0x00003093, 5.8510666f0), IdDist(0x0000005b, 5.886432f0)]
=== 2 -- identifiers:IdDist[IdDist(0x00015b9c, 6.8040795f0), IdDist(0x00000800, 7.447742f0), IdDist(0x00003863, 7.673588f0), IdDist(0x000100fb, 8.493005f0), IdDist(0x00005590, 8.503584f0), IdDist(0x00008d50, 8.529927f0), IdDist(0x00008223, 8.96452f0), IdDist(0x00012a3d, 9.277115f0), IdDist(0x00017670, 9.328961f0), IdDist(0x00005ff8, 9.668932f0)]
UInt32=== 26 -- identifiers:[0x00017338, 0x00010633, 0x0000c55b, 0x00014949, 0x00003b78, 0x000169b2, 0x000022c1, 0x00016192, 0x000024c8, 0x00006dc9]
IdDist=== 29 -- distances:(0x000016c0, 5.7069054f0)
Float32=== 10 -- result set:[3.675293, 4.157851, 4.617572, 4.772978, 4.9854174, 4.9955883, 5.1739335, 5.2381277, 5.25712, 5.258646]
Float32[5.7050834, 5.813157, 6.1480823=== 19 -- nearest neighbor:, 6.224143, 6.2271614, 6.4017773, 6.4356084, 6.7351327, 6.979508, 7.1391606]
IdDist[IdDist(0x000016c0, === 30 -- nearest neighbor:5.7069054f0), IdDist(0x0000b16b, 6.3375754f0), IdDist(0x00000de8, 6.685611f0), IdDist(0x00016592, 7.232927f0), IdDist(0x00012c6c, 7.2425156f0), IdDist(0x0000b6d6, 7.315733f0), IdDist(0x0000030c, 7.538448f0), IdDist(0x000064d5, 7.556486f0), IdDist(0x0000a1ea, 7.5782905f0), IdDist(0x00005ead, 7.843588f0)]
=== 10 -- identifiers:IdDist[IdDist(0x00015477, 3.6226947f0), IdDist(0x00003bb3, 4.074771f0), IdDist(0x0000b72f, 4.822839f0), IdDist(0x0000c4fc, 4.9219384f0), IdDist(0x00017d71, 4.934731f0), IdDist(0x00011e14, 5.320342f0), IdDist(0x00000366, 5.6002398f0), IdDist(0x00007e6f, 5.652965f0), IdDist(0x00016a1f, 6.0879908f0), IdDist(0x000101bb, 6.0961123f0)]
=== 6 -- identifiers:UInt32[0x00015b9c, 0x00000800, 0x00003863, 0x000100fb, 0x00005590, 0x00008d50, 0x00008223, 0x00012a3d, 0x00017670, 0x00005ff8]
=== 26 -- distances:UInt32[0x000016c0, 0x0000b16b, 0x00000de8, 0x00016592, 0x00012c6c, 0x0000b6d6, 0x0000030c, 0x000064d5, 0x0000a1ea, 0x00005ead]
IdDist=== 10 -- distances:(0x00003b79, 4.133578f0)
UInt32=== 19 -- result set:[0x000046b3, 0x0000cddc, 0x0000ccba, 0x00010722, 0x00013864, 0x0000d5a6, 0x0000e077, 0x0000e9db, 0x00002960, 0x00001ddf]
=== 22 -- distances:IdDist[IdDist(0x00011394, 4.1133566f0), IdDist(0x0000553d, 4.8949213f0), IdDist(0x00003bd5, 4.9883957f0), IdDist(0x00006f36, 5.1679077f0), IdDist(0x0000a8ea, 5.3847394f0), IdDist(0x00000e8f, 5.660733f0), IdDist(0x00001a7f, 5.7420354f0), IdDist(0x0000ba08, 5.7641015f0), IdDist(0x0001367b, 5.8484545f0), IdDist(0x00008a0c, 5.858608f0)]
UInt32=== 14 -- identifiers:[0x00015477, 0x00003bb3, 0x0000b72f, 0x0000c4fc, 0x00017d71, 0x00011e14, 0x00000366, 0x00007e6f, 0x00016a1f, 0x000101bb]
Float32=== 6 -- distances:[4.237101, 4.995167, 5.519479, 5.5932918, 5.612316, 5.8138113, 5.9479756, 6.055376, 6.1627603, 6.1689744]
IdDist[IdDist(0x00003b79, === 23 -- nearest neighbor:4.133578f0), IdDist(0x0000de4f, 4.741813f0), IdDist(0x0000018e, 5.4380856f0), IdDist(0x0000edc9, 5.46094f0), IdDist(0x00006f5b, 5.727246f0), IdDist(0x00006d88, 5.764778f0), IdDist(0x0000e78b, 5.9772415f0), IdDist(0x0000afab, 5.996315f0), IdDist(0x0000246c, 6.129073f0), IdDist(0x0000b085, 6.14421f0)]
=== 19 -- identifiers:UInt32[0x0000d3da, 0x00017e8f, 0x00000876, 0x00010411, 0x00003c3e, 0x00005d55, 0x0000111b, 0x000168a8, 0x00003093, 0x0000005b]
=== 2 -- distances:IdDist(0x00008755, 5.0209303f0)
=== 23 -- result set:Float32[6.8040795, 7.447742, 7.673588, 8.493005, 8.503584, 8.529927, 8.96452, 9.277115, 9.328961, 9.668932]
UInt32[0x00011394=== 27 -- nearest neighbor:, 0x0000553d, 0x00003bd5, 0x00006f36, 0x0000a8ea, 0x00000e8f, 0x00001a7f, 0x0000ba08, 0x0001367b, 0x00008a0c]
Float32=== 14 -- distances:[3.3228424, 5.1148386, 5.256465, 5.2807727, 5.4705386, 5.478687, 5.510663, 5.7863193, 5.8510666, 5.886432]
IdDist[IdDist(0x=== 3 -- nearest neighbor:00008755, 5.0209303f0), IdDist(0x00011b14, 5.3232765f0), IdDist(0x00010b2c, 5.622811f0), IdDist(0x0001292e, 5.8693914f0), IdDist(0x0000d660, 5.96799f0), IdDist(0x0001347c, 5.9942255f0), IdDist(0x00005a85, 6.1475f0), IdDist(0x000156b8, 6.4276695f0), IdDist(0x0000f23a, 6.5021176f0), IdDist(0x0000f747, 6.5391407f0)]
=== 23 -- identifiers:Float32[4.1133566, 4.8949213, 4.9883957, 5.1679077, 5.3847394, 5.660733, 5.7420354, 5.7641015, 5.8484545, 5.858608]
UInt32[0x00003b79, 0x0000de4f=== 15 -- nearest neighbor:, 0x0000018e, 0x0000edc9, 0x00006f5b, 0x00006d88, 0x0000e78b, 0x0000afab, 0x0000246c, 0x0000b085]
UInt32=== 19 -- distances:[0x00008755, 0x00011b14, 0x00010b2c, 0x0001292e, 0x0000d660, 0x0001347c, 0x00005a85, 0x000156b8, 0x0000f23a, 0x0000f747]
IdDist=== 23 -- distances:(0x00012538, 3.4767742f0)
Float32=== 30 -- result set:[5.7069054, 6.3375754, 6.685611, 7.232927, 7.2425156, 7.315733, 7.538448, 7.556486, 7.5782905, 7.843588]
IdDist(0x0000f549, === 11 -- nearest neighbor:7.284191f0)
=== 27 -- result set:Float32[5.0209303, 5.3232765, 5.622811, 5.8693914, 5.96799, 5.9942255, 6.1475, 6.4276695, 6.5021176, 6.5391407]
Float32[=== 24 -- nearest neighbor:4.133578, 4.741813, 5.4380856, 5.46094, 5.727246, 5.764778, 5.9772415, 5.996315, 6.129073, 6.14421]
IdDist(0x0000f13c, === 20 -- nearest neighbor:5.393652f0)
=== 3 -- result set:IdDist[IdDist(0x0000f549, 7.284191f0), IdDist(0x000117a1, 8.011044f0), IdDist(0x00003ddb, 8.031088f0), IdDist(0x000115bd, 8.722832f0), IdDist(0x0000b424, 8.914991f0), IdDist(0x00004a1a, 9.359341f0), IdDist(0x00003914, 9.364273f0), IdDist(0x00007564, 9.376333f0), IdDist(0x00005093, 9.381775f0), IdDist(0x00013405, 9.902296f0)]
=== 27 -- identifiers:IdDist(0x00008413, 5.1414504f0)
=== 15 -- result set:Float32[3.6226947, 4.074771, 4.822839, 4.9219384, 4.934731, 5.320342, 5.6002398, 5.652965, 6.0879908, 6.0961123]
IdDist[=== 7 -- nearest neighbor:IdDist(0x0000f13c, 5.393652f0), IdDist(0x00010845, 5.9110575f0), IdDist(0x000021d4, 6.3453403f0), IdDist(0x000101cb, 6.3455286f0), IdDist(0x00002f90, 6.898633f0), IdDist(0x00013858, 6.9106183f0), IdDist(0x0000cc98, 7.010584f0), IdDist(0x0000530f, 7.1093607f0), IdDist(0x00000a96, 7.191145f0), IdDist(0x00013373, 7.30588f0)]
IdDist=== 3 -- identifiers:(0x0000caf8, 4.9733167f0)
IdDist=== 20 -- result set:(0x00005258, 3.4473765f0)
=== 7 -- result set:IdDist(0x0001185a, 3.992005f0)
=== 24 -- result set:IdDist(0x00017d7c, 9.868795f0)
=== 11 -- result set:UInt32[0x0000f549, 0x000117a1, 0x00003ddb, 0x000115bd, 0x0000b424, 0x00004a1a, 0x00003914, 0x00007564, 0x00005093, 0x00013405]
IdDist=== 27 -- distances:[IdDist(0x0001185a, 3.992005f0), IdDist(0x00001337, 4.9291162f0), IdDist(0x000050e8, 5.355009f0), IdDist(0x00015f44, 5.384289f0), IdDist(0x00004b1c, 5.4277697f0), IdDist(0x00007086, 5.5263424f0), IdDist(0x0000b363, 5.7034106f0), IdDist(0x000146d7, 5.738125f0), IdDist(0x00000710, 5.7491794f0), IdDist(0x0000e414, 5.7983756f0)]
=== 24 -- identifiers:IdDist[IdDist(0x00008413, 5.1414504f0), IdDist(0x000180ba, 6.3001485f0), IdDist(0x0000ca38, 6.512064f0), IdDist(0x00005c63, 7.0336237f0), IdDist(0x00003497, 7.2288895f0), IdDist(0x00003ac2, 7.26844f0), IdDist(0x000058fe, 7.309383f0), IdDist(0x00004dd9, 7.351431f0), IdDist(0x0000cbe1, 7.558257f0), IdDist(0x00016790, 7.634893f0)]
IdDist=== 15 -- identifiers:[IdDist(0x00005258, 3.4473765f0), IdDist(0x00013f59, 4.095377f0), IdDist(0x00008881, 4.2692914f0), IdDist(0x00010e45, 4.4451303f0), IdDist(0x0000019a, 4.6292806f0), IdDist(0x00004df9, 4.6982117f0), IdDist(0x000145ee, 4.839198f0), IdDist(0x0000cf64, 4.865141f0), IdDist(0x0000653a, 4.9991274f0), IdDist(0x00005e52, 5.028851f0)]
=== 7 -- identifiers:IdDist[IdDist(0x0000caf8, 4.9733167f0), IdDist(0x000176b2, 5.1393404f0), IdDist(0x00010906, 5.262912f0), IdDist(0x000171c5, 5.5710855f0), IdDist(0x0001061f, 5.7874975f0), IdDist(0x0001227e, 5.793222f0), IdDist(0x00005ff2, 6.3865533f0), IdDist(0x00001794, 6.4906406f0), IdDist(0x0000adbc, 6.9201994f0), IdDist(0x00012786, 6.9248285f0)]
=== 20 -- identifiers:IdDist[IdDist(0x00012538, 3.4767742f0), IdDist(0x0000f305, 4.296953f0), IdDist(0x000068c6, 4.4126143f0), IdDist(0x00016758, 4.445423f0), IdDist(0x00003c7f, 4.4598117f0), IdDist(0x0000cc9f, 4.472585f0), IdDist(0x0000ed84, 4.7729397f0), IdDist(0x000111a2, 4.9134693f0), IdDist(0x0000b74e, 4.9283032f0), IdDist(0x00004657, 4.969076f0)]
=== 30 -- identifiers:IdDist[IdDist(0x00017d7c, 9.868795f0), IdDist(0x0000586a, 9.921723f0), IdDist(0x00001f29, 10.037449f0), IdDist(0x00008c9a, 10.806721f0), IdDist(0x00011265, 12.553823f0), IdDist(0x0000f4ff, 12.558981f0), IdDist(0x00003d31, 12.677736f0), IdDist(0x00001e8a, 12.775283f0), IdDist(0x00008a7e, 13.053132f0), IdDist(0x00014f4f, 13.2244005f0)]
=== 11 -- identifiers:UInt32[0x00005258, 0x00013f59, 0x00008881, 0x00010e45, 0x0000019a, 0x00004df9, 0x000145ee, 0x0000cf64, 0x0000653a, 0x00005e52]
=== 7 -- distances:Float32[7.284191, 8.011044, 8.031088, 8.722832, 8.914991, 9.359341, 9.364273, 9.376333, 9.381775, 9.902296]
UInt32[0x00008413, 0x000180ba, 0x0000ca38, 0x00005c63, 0x00003497, 0x00003ac2, 0x000058fe, 0x00004dd9, 0x0000cbe1, 0x00016790]
UInt32=== 15 -- distances:[0x0000f13c, 0x00010845, 0x000021d4, 0x000101cb, 0x00002f90, 0x00013858, 0x0000cc98, 0x0000530f, 0x00000a96, 0x00013373]
=== 3 -- distances:UInt32[0x0001185a, 0x00001337, 0x000050e8, 0x00015f44, 0x00004b1c, 0x00007086, 0x0000b363, 0x000146d7, 0x00000710, 0x0000e414]
Float32=== 24 -- distances:[5.1414504, 6.3001485, 6.512064, 7.0336237, 7.2288895, 7.26844, 7.309383, 7.351431, 7.558257, 7.634893]
Float32[3.4473765, 4.095377,=== 16 -- nearest neighbor: 4.2692914, 4.4451303, 4.6292806, 4.6982117, 4.839198, 4.865141, 4.9991274, 5.028851]
UInt32[0x00017d7c, 0x0000586a, 0x=== 8 -- nearest neighbor:00001f29, 0x00008c9a, 0x00011265, 0x0000f4ff, 0x00003d31, 0x00001e8a, 0x00008a7e, 0x00014f4f]
=== 11 -- distances:UInt32[0x00012538, 0x0000f305, 0x000068c6, 0x00016758, 0x00003c7f, 0x0000cc9f, 0x0000ed84, 0x000111a2, 0x0000b74e, 0x00004657]
=== 30 -- distances:Float32[5.393652, 5.9110575, 6.3453403, 6.3455286, 6.898633, 6.9106183, 7.010584, 7.1093607, 7.191145, 7.30588]
UInt32[0x0000caf8, 0x000176b2=== 4 -- nearest neighbor:, 0x00010906, 0x000171c5, 0x0001061f, 0x0001227e, 0x00005ff2, 0x00001794, 0x0000adbc, 0x00012786]
IdDist=== 20 -- distances:(0x000029ab, 3.558713f0)
=== 8 -- result set:IdDist(0x00013229, 3.9677713f0)
=== 4 -- result set:Float32[3.4767742, 4.296953, 4.4126143, 4.445423, 4.4598117, 4.472585, 4.7729397, 4.9134693, 4.9283032, 4.969076]
IdDist(0x00017b21, 4.402828f0)
=== 16 -- result set:Float32[3.992005, 4.9291162, 5.355009, 5.384289, 5.4277697, 5.5263424, 5.7034106, 5.738125, 5.7491794, 5.7983756]
IdDist[IdDist(0x000029ab, 3.558713f0), IdDist(0x00005277, 3.8482122f0), IdDist(0x000171ef, 3.9154968f0), IdDist(0x0001768b, 3.919181f0), IdDist(0x00008aad, 3.9737177f0), IdDist(0x000178a5, 4.1200695f0), IdDist(0x0000061b, 4.1365585f0), IdDist(0x0001078c, 4.247946f0), IdDist(0x0000943f, 4.3681407f0), IdDist(0x0000bba1, 4.4379377f0)]
=== 8 -- identifiers:Float32[9.868795, 9.921723, 10.037449, 10.806721, 12.553823, 12.558981, 12.677736, 12.775283, 13.053132, 13.2244005]
IdDist=== 12 -- nearest neighbor:[IdDist(0x00013229, 3.9677713f0), IdDist(0x00009bd0, 4.9281144f0), IdDist(0x00013dde, 5.330806f0), IdDist(0x0000e3c9, 5.4604096f0), IdDist(0x00005631, 5.646955f0), IdDist(0x0000a997, 5.8043404f0), IdDist(0x0000eedc, 5.868377f0), IdDist(0x0000d344, 5.9340653f0), IdDist(0x00008c93, 6.088916f0), IdDist(0x00004b3f, 6.280229f0)]
=== 4 -- identifiers:IdDist(0x0000c288, 4.567849f0)
=== 12 -- result set:Float32[4.9733167, 5.1393404, 5.262912, 5.5710855, 5.7874975, 5.793222, 6.3865533, 6.4906406, 6.9201994, 6.9248285]
UInt32[0x000029ab, 0x00005277, 0x000171ef, 0x0001768b, 0x00008aad, 0x000178a5, 0x0000061b, 0x0001078c, 0x0000943f, 0x0000bba1]
=== 8 -- distances:IdDist[IdDist(0x00017b21, 4.402828f0), IdDist(0x00009c76, 5.3538895f0), IdDist(0x00003ea0, 5.486088f0), IdDist(0x00008e68, 5.5713105f0), IdDist(0x00009a43, 5.579166f0), IdDist(0x00003371, 5.818046f0), IdDist(0x000047d9, 6.1070213f0), IdDist(0x00014bfd, 6.1310916f0), IdDist(0x00007ef8, 6.172512f0), IdDist(0x00002323, 6.268899f0)]
=== 16 -- identifiers:UInt32[0x00013229, 0x00009bd0, 0x00013dde, 0x0000e3c9, 0x00005631, 0x0000a997, 0x0000eedc, 0x0000d344, 0x00008c93, 0x00004b3f]
UInt32=== 4 -- distances:[0x00017b21, 0x00009c76, 0x00003ea0, 0x00008e68, 0x00009a43, 0x00003371, 0x000047d9, 0x00014bfd, 0x00007ef8, 0x00002323]
=== 16 -- distances:IdDist[IdDist(0x0000c288, 4.567849f0), IdDist(0x000070d4, 5.999233f0), IdDist(0x00008279, 6.2647986f0), IdDist(0x00000a5e, 6.785245f0), IdDist(0x00007a77, 7.0595245f0), IdDist(0x000159e1, 7.402501f0), IdDist(0x00008690, 7.5159426f0), IdDist(0x0000ba55, 7.7649183f0), IdDist(0x00013b91, 8.000528f0), IdDist(0x00016597, 8.135601f0)]
=== 12 -- identifiers:Float32[3.558713, 3.8482122, 3.9154968, 3.919181, 3.9737177, 4.1200695, 4.1365585, 4.247946, 4.3681407, 4.4379377]
Float32[3.9677713, 4.9281144, 5.330806, 5.4604096, 5.646955, 5.8043404, 5.868377, 5.9340653, 6.088916, 6.280229]
UInt32[0x0000c288, 0x000070d4, 0x00008279, 0x00000a5e, 0x00007a77, 0x000159e1, 0x00008690, 0x0000ba55, 0x00013b91, 0x00016597]
=== 12 -- distances:Float32[4.402828, 5.3538895, 5.486088, 5.5713105, 5.579166, 5.818046, 6.1070213, 6.1310916, 6.172512, 6.268899]
Float32[4.567849, 5.999233, 6.2647986, 6.785245, 7.0595245, 7.402501, 7.5159426, 7.7649183, 8.000528, 8.135601]

Environment and dependencies

Julia Version 1.10.11
Commit a2b11907d7b (2026-03-09 14:59 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (x86_64-apple-darwin24.0.0)
  CPU: 8 × Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, skylake)
Threads: 8 default, 0 interactive, 4 GC (on 8 virtual cores)
Environment:
  JULIA_NUM_THREADS = auto
  JULIA_PROJECT = @.
  JULIA_LOAD_PATH = @:@stdlib
Status `~/Research/SimilaritySearchDemos/Project.toml`
  [aaaa29a8] Clustering v0.15.8
  [944b1d66] CodecZlib v0.7.8
  [5ae59095] Colors v0.13.1
  [a93c6f00] DataFrames v1.8.1
  [c5bfea45] Embeddings v0.4.6
  [f67ccb44] HDF5 v0.17.2
  [916415d5] Images v0.26.2
  [b20bd276] InvertedFiles v0.9.2
⌅ [682c06a0] JSON v0.21.4
  [23fbe1c1] Latexify v0.16.10
  [eb30cadb] MLDatasets v0.7.21
  [06eb3307] ManifoldLearning v0.9.0
⌃ [ca7969ec] PlotlyLight v0.11.0
  [27ebfcd6] Primes v0.5.7
  [ca7ab67e] SimSearchManifoldLearning v0.4.0
  [053f045d] SimilaritySearch v0.14.3
⌅ [2913bbd2] StatsBase v0.33.21
  [7f6f6c8a] TextSearch v0.20.0
Info Packages marked with ⌃ and ⌅ have new versions available. Those with ⌃ may be upgradable, but those with ⌅ are restricted by compatibility constraints from upgrading. To see why use `status --outdated`