Wikipedia Word Embeddings with LSI

by: Eric S. Téllez

This demonstration shows how to use Latent Semantic Indexing (LSI) to generate dense word embeddings from Spanish and English Wikipedia, and how to use those embeddings for synonym search, arithmetic analogies, and UMAP visualisations.

Embeddings are generated with TextSearch.jl from Wikipedia’s introductory corpus (the first 3 paragraphs per article). The vocabulary profile and global TF-IDF weights were trained on the full corpus (~14M paragraphs in ES, ~46M in EN), but the SVD is computed on the introductory subset—giving more weight to the definitional content of each article.

Note: This demo requires having run generate_wiki_embeddings.jl to produce the embedding files in demos/data/.

using SimilaritySearch, SimSearchManifoldLearning
using TextSearch, TextSearch.LSI
using Plots, StatsBase, LinearAlgebra, Markdown, Random, Printf, JSON3

Loading the embeddings

The embeddings are stored in a GloVe-like text format: one line per token, with the token followed by whitespace-separated vector components.

"""
Loads embeddings in GloVe text format.
Returns (matrix Float16 dim×n, vocab Vector{String}).
We use Float16 to reduce memory usage; distance functions convert to Float32 dynamically during computation.
"""
function load_glove_format(path::String; dtype=Float16, max_words=100_000)
    @info "Loading embeddings from $path (max $max_words words)"
    lines = readlines(path)
    n     = min(length(lines), max_words)
    dim   = length(split(lines[1])) - 1
    M     = Matrix{dtype}(undef, dim, n)
    vocab = Vector{String}(undef, n)
    for i in 1:n
        parts    = split(lines[i])
        vocab[i] = parts[1]
        for (j, v) in enumerate(@view parts[2:end])
            M[j, i] = parse(Float32, v)
        end
    end
    M, vocab
end

# Load Spanish embeddings (top 100k tokens)
emb_es, vocab_es = load_glove_format(joinpath(@__DIR__, "data", "wiki-es-lsi.txt"); max_words=100_000)

# Inverse mapping: token → index
vocab2id_es = Dict(w => i for (i, w) in enumerate(vocab_es))

@info "ES: $(length(vocab_es)) tokens, dim=$(size(emb_es, 1))"

# Load English embeddings (top 100k tokens)
emb_en, vocab_en = load_glove_format(joinpath(@__DIR__, "data", "wiki-en-lsi.txt"); max_words=100_000)
vocab2id_en = Dict(w => i for (i, w) in enumerate(vocab_en))
@info "EN: $(length(vocab_en)) tokens, dim=$(size(emb_en, 1))"

Building the search index

We use SearchGraph with normalized cosine distance to perform fast approximate nearest neighbor search. Since all LSI vectors are unit-normalized (\(\|v\|_2 = 1\)), Dist.NormCosine() computes cosine distance without recalculating vector norms.

dist = Dist.NormCosine()

# Spanish index
db_es  = MatrixDatabase(emb_es)
G_es   = SearchGraph(dist, db_es)
ctx_es = SearchGraphContext(hyperparameters_callback=OptimizeParameters(MinRecall(0.99)))
index!(G_es, ctx_es)
optimize_index!(G_es, ctx_es, MinRecall(0.9))

# English index
db_en  = MatrixDatabase(emb_en)
G_en   = SearchGraph(dist, db_en)
ctx_en = SearchGraphContext(hyperparameters_callback=OptimizeParameters(MinRecall(0.99)))
index!(G_en, ctx_en)
optimize_index!(G_en, ctx_en, MinRecall(0.9))

Synonym and related word search

Nearest neighbors in the LSI vector space identify semantically related words and synonyms:

function search_similar(G, ctx, vocab, vocab2id, word::String, k::Int=10)
    haskey(vocab2id, word) || return Markdown.parse("> ⚠️ Word '$word' not found in vocabulary")
    qid = vocab2id[word]
    res = knnqueue(ctx, k)
    search(G, ctx, G[qid], res)

    L = [
        """## Similar to _$(word)_""",
        """| # | Token | Cosine distance |""",
        """|---|-------|-----------------|"""
    ]
    for (j, p) in enumerate(IdDistView(res))
        w = vocab[p.id]
        push!(L, """| $j | **$w** | $(@sprintf "%.4f" p.dist) |""")
    end
    Markdown.parse(join(L, "\n"))
end

# Spanish examples
for word in ["física", "historia", "computación", "música", "planeta"]
    search_similar(G_es, ctx_es, vocab_es, vocab2id_es, word, 8) |> display
end

Similar to física

#	Token	Cosine distance
1	física	0.0020
2	matemática	0.1357
3	teórica	0.1538
4	fisiología	0.1597
5	patología	0.1611
6	matemáticas	0.1699
7	biología	0.1758
8	farmacología	0.1777

Similar to historia

#	Token	Cosine distance
1	historia	0.0005
2	contada	0.1440
3	mentalidades	0.2031
4	relatada	0.2539
5	narra	0.2603
6	providencialista	0.2622
7	relata	0.2676
8	Relata	0.2939

Similar to computación

#	Token	Cosine distance
1	computación	0.0005
2	computacional	0.0454
3	informática	0.0479
4	computacionales	0.0630
5	informáticas	0.0903
6	optimización	0.1064
7	informáticos	0.1128
8	Procesamiento	0.1196

Similar to música

#	Token	Cosine distance
1	música	0.0020
2	folclórica	0.0254
3	electroacústica	0.0293
4	músicas	0.0371
5	sacra	0.0396
6	incidental	0.0503
7	techno	0.0508
8	electrónica	0.0586

Similar to planeta

#	Token	Cosine distance
1	planeta	0.0020
2	b	0.0122
3	sonda	0.0132
4	planetas	0.0142
5	extrasolar	0.0156
6	planetario	0.0161
7	tránsito	0.0166
8	astronómico	0.0171

# English examples
for word in ["physics", "history", "computer", "music", "planet"]
    search_similar(G_en, ctx_en, vocab_en, vocab2id_en, word, 8) |> display
end

Similar to physics

#	Token	Cosine distance
1	physics	0.0000
2	chemistry	0.0249
3	biophysics	0.0503
4	astrophysics	0.0610
5	biochemistry	0.0645
6	geophysics	0.0684
7	biology	0.0723
8	microbiology	0.0835

Similar to history

#	Token	Cosine distance
1	history	0.0015
2	details	0.0645
3	Pomerania	0.0674
4	see	0.0742
5	Postage	0.1196
6	region	0.1440
7	mentalities	0.1841
8	Wolgast	0.1948

Similar to computer

#	Token	Cosine distance
1	computer	0.0010
2	computing	0.0654
3	simulations	0.0820
4	haptic	0.0889
5	optimization	0.0923
6	visualization	0.0923
7	robotics	0.0942
8	Adaptive	0.0947

Similar to music

#	Token	Cosine distance
1	music	0.0039
2	electroacoustic	0.0483
3	incidental	0.0830
4	musics	0.0908
5	composes	0.0918
6	ethnomusicology	0.0962
7	choral	0.0967
8	polyphony	0.1055

Similar to planet

#	Token	Cosine distance
1	planet	0.0000
2	planets	0.1831
3	moons	0.2192
4	interstellar	0.2251
5	Pluto	0.2319
6	cometary	0.2451
7	astrometry	0.2495
8	orbiting	0.2500

Query expansion and synonym network

The LSI embeddings allow precomputing a query expansion network: for each token, its \(k\) nearest neighbors in semantic space form an expansion set.

# Load pre-computed query expansion network
qexp_obj = JSON3.read(read(joinpath(@__DIR__, "data", "wiki-es-qexp.json"), String))
qexp_es  = qexp_obj.query_expansion

function show_expansion(qexp, word)
    sym = Symbol(word)
    haskey(qexp, sym) || return Markdown.parse("> ⚠️ '$word' not found in expansion network")
    neighbors = qexp[sym]
    isempty(neighbors) && return Markdown.parse("> '$word' has no expansion (high-frequency token)")
    Markdown.parse("""
    **$word** → $(join(["_$(v)_" for v in neighbors], ", "))
    """)
end

for p in ["ciencia", "guerra", "amor", "tecnología", "ciudad", "arte", "economía"]
    show_expansion(qexp_es, p) |> display
end

ciencia → ficción, especulativa, fantasía, postciberpunk, Asimov, fandom, fantástica, distópica

guerra → bando, Thanagarianos, lucharon, combatieron, bélico, conflicto, luchó, librada

amor → amar, amorosa, sueños, Kāmadeva, soledad, amorosas, triste, loca

tecnología → tecnologías, tecnológicas, tecnológicos, informáticas, automatización, tecnológica, desarrollo, robótica

ciudad → Algenrodt, Resurgam, Caleo, אֶת, Mikros, Bulero, Mahrouk, Ftouh

arte → contemporáneo, exponer, exposiciones, invisual, Arte, exposición, pictóricas, coleccionismo

economía → económicas, económica, económico, macroeconómico, macroeconomía, economistas, productivos, economías

Do arithmetic analogies work?

Analogies of the form “king is to man as queen is to woman” are well known in word2vec models. In LSI, the semantic space is linear and derived from document-level co-occurrence. Linear analogy arithmetic works when concepts share similar document distributions:

\[\vec{v}_{\text{target}} \approx \vec{v}_a - \vec{v}_b + \vec{v}_d \quad \text{where } a:b :: c:d\]

function analogy(G, ctx, vocab, vocab2id, a, b, d, k=10)
    for w in (a, b, d)
        haskey(vocab2id, w) || return Markdown.parse("> ⚠️ Word '$w' not found in vocabulary")
    end
    va = Float32.(G[vocab2id[a]])
    vb = Float32.(G[vocab2id[b]])
    vd = Float32.(G[vocab2id[d]])
    vc = va - vb + vd
    normalize!(vc)

    res = knnqueue(ctx, k + 3)  # request extra candidates to filter out inputs
    search(G, ctx, Float16.(vc), res)

    exclude = Set([a, b, d])
    L = [
        """### Analogy: _$(a)_ is to _$(b)_ as _?_ is to _$(d)_""",
        """| # | Token | Distance |""",
        """|---|-------|----------|"""
    ]
    j = 0
    for p in IdDistView(res)
        w = vocab[p.id]
        w in exclude && continue
        j += 1
        j > k && break
        push!(L, """| $j | **$w** | $(@sprintf "%.4f" p.dist) |""")
    end
    Markdown.parse(join(L, "\n"))
end

# Classic analogies in Spanish
analogy(G_es, ctx_es, vocab_es, vocab2id_es, "rey", "hombre", "mujer") |> display
analogy(G_es, ctx_es, vocab_es, vocab2id_es, "París", "Francia", "España") |> display
analogy(G_es, ctx_es, vocab_es, vocab2id_es, "Madrid", "España", "Francia") |> display
analogy(G_es, ctx_es, vocab_es, vocab2id_es, "perro", "animal", "planta") |> display

Analogy: rey is to hombre as ? is to mujer

#	Token	Distance
1	consorte	0.1914
2	reina	0.2124
3	III	0.2197
4	IV	0.2231
5	II	0.2271
6	duque	0.2314
7	conde	0.2319
8	regente	0.2349
9	Hohenstaufen	0.2402
10	Balduino	0.2422

Analogy: París is to Francia as ? is to España

#	Token	Distance
1	Gourmont	0.4688
2	Carpeaux	0.4888
3	Théodore	0.5059
4	Corot	0.5068
5	Gleizes	0.5122
6	Bichat	0.5127
7	Puvis	0.5132
8	Lévy	0.5146
9	Musset	0.5249
10	simbolista	0.5298

Analogy: Madrid is to España as ? is to Francia

#	Token	Distance
1	Dykinson	0.4155
2	madrid	0.4385
3	Humanes	0.4497
4	Torrelodones	0.4561
5	Carmena	0.4663
6	comisionada	0.4702
7	Griñón	0.4727
8	Leguina	0.4756
9	RUBIO	0.4795
10	BESCAM	0.4810

Analogy: perro is to animal as ? is to planta

#	Token	Distance
1	disfraza	0.6113
2	soborna	0.6152
3	atormentado	0.6211
4	enamora	0.6230
5	Trama	0.6255
6	Argumento	0.6260
7	novia	0.6270
8	roba	0.6270
9	chica	0.6309
10	obsesiona	0.6309

# Analogies in English
analogy(G_en, ctx_en, vocab_en, vocab2id_en, "king", "man", "woman") |> display
analogy(G_en, ctx_en, vocab_en, vocab2id_en, "Paris", "France", "Spain") |> display
analogy(G_en, ctx_en, vocab_en, vocab2id_en, "Madrid", "Spain", "France") |> display

Analogy: king is to man as ? is to woman

#	Token	Distance
1	regnant	0.1680
2	consort	0.1709
3	queen	0.1860
4	regent	0.1948
5	princess	0.2129
6	dowager	0.2144
7	throne	0.2280
8	Amalaric	0.2363
9	Laodice	0.2378
10	duchess	0.2402

Analogy: Paris is to France as ? is to Spain

#	Token	Distance
1	Ribera	0.3403
2	Carles	0.3730
3	Seville	0.3735
4	Pujol	0.3740
5	Valladolid	0.3750
6	Zaragoza	0.3755
7	Aragonese	0.3760
8	Cervera	0.3809
9	Catalonia	0.3848
10	Vallès	0.3867

Analogy: Madrid is to Spain as ? is to France

#	Token	Distance
1	Belfort	0.1543
2	Hauts	0.1602
3	Essonne	0.1602
4	Oise	0.1709
5	Aisne	0.1758
6	Calais	0.1787
7	Provence	0.1904
8	Pas	0.1914
9	Marne	0.1934
10	Île	0.1938

UMAP visualization

We project a sample of the most frequent vocabulary tokens into 2D using UMAP, using SearchGraph for the kNN neighborhood graph. We color the points using a 3D UMAP embedding normalized to RGB channels.

# Sample the most frequent vocabulary tokens for visualization
sample_size = min(15_000, length(vocab_es))
sample_idx  = 1:sample_size

db_sample  = SubDatabase(db_es, collect(sample_idx))
G_sample   = SearchGraph(dist, db_sample)
ctx_sample = SearchGraphContext(hyperparameters_callback=OptimizeParameters(MinRecall(0.95)))
index!(G_sample, ctx_sample)

e2, e3 = let k=15, n_epochs=75, neg_sample_rate=3, tol=1e-3, min_dist=0.3f0,
              layout=SpectralLayout()
    @time "UMAP 2D" U2 = fit(UMAP, G_sample; k, neg_sample_rate, layout, n_epochs, tol, min_dist)
    @time "UMAP 3D" U3 = fit(U2, 3; neg_sample_rate, n_epochs, tol)
    @time "predict 2D" e2 = clamp.(predict(U2), -10f0, 10f0)
    @time "predict 3D" e3 = clamp.(predict(U3), -10f0, 10f0)
    e2, e3
end

function normcolors(V)
    mn, mx = extrema(V)
    @. V = clamp((V - mn) / (mx - mn), 0, 1)
end

normcolors(@view e3[1, :])
normcolors(@view e3[2, :])
normcolors(@view e3[3, :])

C = [RGB(c[1], c[2], c[3]) for c in eachcol(e3)]
X, Y = view(e2, 1, :), view(e2, 2, :)

scatter(X, Y; color=C, fmt=:png, alpha=0.15, size=(700, 700),
        ma=0.2, ms=2, msw=0, label="",
        yticks=nothing, xticks=nothing, xaxis=false, yaxis=false,
        title="Wikipedia ES — 256d LSI projected with UMAP")

# Annotate ~80 random tokens
for _ in 1:80
    j = rand(1:sample_size)
    annotate!(X[j], Y[j], text(vocab_es[sample_idx[j]], :blue, :right, 5, "sans-serif"))
end
plot!()

Concluding remarks

This demonstration shows how LSI embeddings derived from Wikipedia capture semantic structure: words associated through document co-occurrence naturally cluster in vector space.

Key differences from GloVe/word2vec: - LSI captures full-document co-occurrence rather than local sliding windows, making it particularly strong for broad topical affinities. - Analogies of the form king - man + woman ≈ queen work well when concepts are topically distinct in Wikipedia (countries, capitals, scientific domains). - Morphological or syntactic relations (e.g., plurals, verb tenses) are less emphasized than topical semantics.

Environment and dependencies

Julia Version 1.12.7
Commit 6d172b025e4 (2026-08-15 08:05 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 64 × Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, cascadelake)
  GC: Built with stock GC
Threads: 64 default, 1 interactive, 64 GC (on 64 virtual cores)
Environment:
  JULIA_PROJECT = @.
  JULIA_NUM_THREADS = auto
  JULIA_LOAD_PATH = @:@stdlib
Status `~/Research/SimilaritySearchDemos/Project.toml`
  [aaaa29a8] Clustering v0.15.8
  [944b1d66] CodecZlib v0.7.9
⌅ [5ae59095] Colors v0.12.11
  [a93c6f00] DataFrames v1.8.2
  [f67ccb44] HDF5 v0.17.3
  [0f8b85d8] JSON3 v1.14.3
  [23fbe1c1] Latexify v0.16.12
  [eb30cadb] MLDatasets v0.7.21
  [06eb3307] ManifoldLearning v0.9.0
⌅ [ca7969ec] PlotlyLight v0.11.1
  [91a5bcdd] Plots v1.41.7
  [27ebfcd6] Primes v0.5.7
  [92933f4c] ProgressMeter v1.11.0
  [ca7ab67e] SimSearchManifoldLearning v0.4.0 `~/Research/SimSearchManifoldLearning.jl`
  [053f045d] SimilaritySearch v1.2.0 `~/Research/SimilaritySearch.jl`
⌅ [2913bbd2] StatsBase v0.33.21
  [7f6f6c8a] TextSearch v1.1.1 `~/Research/TextSearch.jl`
Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated`