TextSearch API

Base.sum(cluster::AbstractVector{<:SparseVector})

SparseVector counterpart of sum(::AbstractVector{<:Dict}): concatenate every (index, value) pair from every input vector, sort once by index, then combine consecutive equal indices in a single linear pass. Beat every alternative tried (naive +-folding, pairwise tree merging, a k-way heap merge) at every cluster size benchmarked, and — unlike a dense-accumulator approach — its cost does not depend on the vectors' dimension, only on their total number of stored entries. See sadit/TextSearch.jl#25.

All vectors in cluster must have the same dimension (length).

evaluate(::NormCosine, a::SparseVector, b::SparseVector)::Float64
evaluate(::Cosine, a::SparseVector, b::SparseVector)::Float64
evaluate(::NormAngle, a::SparseVector, b::SparseVector)::Float64
evaluate(::Angle, a::SparseVector, b::SparseVector)::Float64

SparseVector counterparts of the Dict-based distance functions above, using sparsedot instead of the plain-merge dot for the underlying inner product. NormCosine/NormAngle assume a/b are already normalized; Cosine/Angle do not.

Example

julia> using SparseArrays

julia> evaluate(NormCosine(), sparsevec(UInt32[1, 2], Float32[0.6, 0.8], 10), sparsevec(UInt32[2], Float32[1.0], 10))
0.19999998807907104

sparsevec(vec::Dict{Ti,Tv}, m=0) where {Ti<:Integer,Tv<:Number}

Creates a sparse vector from a Dict-based sparse vector

Example

julia> sparsevec(Dict{UInt32,Float32}(1 => 0.5, 3 => 0.2))
  [1]  =  0.5
  [3]  =  0.2

_bow_sizehint(voc::Vocabulary)

Estimated final size (number of unique tokens) of a BOW computed under voc, used to sizehint! it up front and avoid rehashing while it's filled. Uses voc's own avgdoclen (average tokens per document across its training corpus) as the estimate, falling back to a small default before voc has seen any training documents.

avgdoclen(voc::Vocabulary)

Average document length in tokens (numtokens(voc) / trainsize(voc)), used by BM25Scorer.

bagofwords!(bow::BOW, voc::Vocabulary, messages)

Computes a bag of words from a multi-field document (a list of texts), accumulating the result into bow. See bagofwords for the non-mutating version.

Example

julia> voc = Vocabulary(TextConfig(), ["hello world"]; verbose=false);

julia> bow = BOW();

julia> TextSearch.bagofwords!(bow, voc, "hello hello world");

julia> bow
Dict{UInt32, Int32}(0x00000002 => 1, 0x00000001 => 2)

bagofwords!(bow::BOW, voc::Vocabulary, tokenlist::TokenizedText)

Accumulates the tokens in tokenlist into bow (a BOW), looking up each token's id in voc; out-of-vocabulary tokens are skipped. Returns bow.

Example

julia> voc = Vocabulary(TextConfig(), ["hello world"]; verbose=false);

julia> TextSearch.bagofwords!(BOW(), voc, tokenize(TextConfig(), "hello hello"))
Dict{UInt32, Int32}(0x00000001 => 2)

bagofwords(voc::Vocabulary, messages)

Tokenizes messages (a string or a list of strings) under voc's TextConfig and returns its bag of words (BOW): a token id => occurrence count mapping. An already-computed BOW is returned unchanged.

Example

julia> voc = Vocabulary(TextConfig(), ["hello world"]; verbose=false);

julia> bagofwords(voc, "hello hello world")
Dict{UInt32, Int32}(0x00000002 => 1, 0x00000001 => 2)

bagofwords_corpus(voc::Vocabulary, corpus::AbstractVector; verbose=true)

Computes a list of bag of words (BOWs) from a corpus, one per document, in parallel across threads.

Example

julia> voc = Vocabulary(TextConfig(), ["hello world", "hello there"]; verbose=false);

julia> bagofwords_corpus(voc, ["hello world", "hello there"]; verbose=false)[1]
Dict{UInt32, Int32}(0x00000002 => 1, 0x00000001 => 1)

centroid(cluster::AbstractVector{<:SparseVector})

Centroid (normalized sum) of a cluster of SparseVectors. See sum(::AbstractVector{<:SparseVector}) for the algorithm.

decode(voc::Vocabulary, bow::Dict)

Converts a Dict sparse vector indexed by token id (e.g., a BOW) into a Dict indexed by the corresponding token string.

Example

julia> voc = Vocabulary(TextConfig(), ["hello world"]; verbose=false);

julia> decode(voc, bagofwords(voc, "hello hello"))
Dict{String, Int32}("hello" => 2)

dvec(x::AbstractSparseVector)

Converts an sparse vector into a dict-based sparse vector

Example

julia> dvec(sparsevec(Dict{UInt32,Float32}(1 => 0.5, 3 => 0.2)))
Dict{UInt32, Float32}(0x00000003 => 0.2, 0x00000001 => 0.5)

encode(voc::Vocabulary, bow::Dict)

Converts a Dict sparse vector indexed by token string into a Dict indexed by token id, the inverse of decode.

Example

julia> voc = Vocabulary(TextConfig(), ["hello world"]; verbose=false);

julia> encode(voc, Dict("hello" => 2))
Dict{UInt32, Int64}(0x00000001 => 2)

filter_tokens!(voc::Vocabulary, arr::AbstractVector{TokenizedText})

Applies filter_tokens!(voc, text) to every tokenized document in arr.

filter_tokens!(voc::Vocabulary, text::TokenizedText)

Removes tokens from a given tokenized text based using the valid vocabulary

Example

julia> voc = Vocabulary(TextConfig(), ["hello world"]; verbose=false);

julia> tks = tokenize(TextConfig(), "hello unknownword world");

julia> filter_tokens!(voc, tks); collect(tks)
["hello", "world"]

filter_tokens(pred::Function, model::VectorModel)

Returns a copy of model reduced to the tokens for which pred(t) is true, where t is a (; id, occs, ndocs, weight, token) named tuple (see also filter_tokens(pred, voc::Vocabulary)).

Example

julia> corpus = ["hello world", "hello there", "the cat sat"];

julia> voc = Vocabulary(TextConfig(), corpus; verbose=false);

julia> model = VectorModel(IdfWeighting(), TfWeighting(), voc);

julia> vocsize(filter_tokens(t -> t.ndocs >= 2, model))
1

filter_tokens(pred::Function, voc::Vocabulary)

Returns a copy of reduced vocabulary based on evaluating pred function for each entry in voc

Example

julia> voc = Vocabulary(TextConfig(), ["hello world", "hello there"]; verbose=false);

julia> voc2 = filter_tokens(t -> t.ndocs >= 2, voc);

julia> vocsize(voc2)
1

merge_voc(voc1::Vocabulary, voc2::Vocabulary[, ...])
merge_voc(pred::Function, voc1::Vocabulary, voc2::Vocabulary[, ...])

Merges two or more vocabularies into a new one. A predicate function can be used to filter token entries.

Note: All vocabularies should had been created with a compatible TextConfig to be able to work on them.

Example

julia> cfg = TextConfig();

julia> voc1 = Vocabulary(cfg, ["hello world"]; verbose=false);

julia> voc2 = Vocabulary(cfg, ["hello there"]; verbose=false);

julia> vocsize(merge_voc(voc1, voc2))
3

ndocs(voc::Vocabulary, tokenID::Integer)
ndocs(voc::Vocabulary)

Number of documents containing the token tokenID (0 is out-of-vocabulary and yields 0 instead of erroring), or the whole per-token vector when called without a tokenID.

Example

julia> voc = Vocabulary(TextConfig(), ["hello world", "hello there"]; verbose=false);

julia> ndocs(voc, token2id(voc, "hello"))
2

numtokens(voc::Vocabulary)

Total number of (non-unique) tokens seen while building voc.

occs(voc::Vocabulary, tokenID::Integer)
occs(voc::Vocabulary)

Total occurrences of the token tokenID across the corpus (0 is out-of-vocabulary and yields 0 instead of erroring), or the whole per-token vector when called without a tokenID.

Example

julia> voc = Vocabulary(TextConfig(), ["hello world", "hello there"]; verbose=false);

julia> occs(voc, token2id(voc, "hello"))
2

push_token!(voc::Vocabulary, token, occs::Integer, ndocs::Integer)
push_token!(voc::Vocabulary, token; occs::Integer=0, ndocs::Integer=0)

Registers token in voc if not already present (assigning it a new id), or accumulates occs/ndocs into its existing entry otherwise. Returns the token's id.

Example

julia> voc = Vocabulary(TextConfig(), 0, 0);

julia> TextSearch.push_token!(voc, "cat"; occs=1, ndocs=1)
0x00000001

sparse(cols::AbstractVector{<:Dict}, m=0; minweight=1e-9) 
sparse_coo(cols::AbstractVector{<:Dict}, minweight=1e-9)

Creates a sparse matrix from an array of Dict sparse vectors.

Example

julia> cols = [Dict{UInt32,Float32}(1 => 0.5), Dict{UInt32,Float32}(2 => 0.8)];

julia> Matrix(sparse(cols))
2×2 Matrix{Float32}:
 0.5  0.0
 0.0  0.8

sparsedot(a::SparseVector, b::SparseVector; small_threshold::Int=30, ratio_threshold::Float64=3.0)

Adaptive dot product between two SparseVectors:

• both sides have fewer than small_threshold stored entries, or their sizes are within ratio_threshold of each other: a plain linear merge (the same algorithm LinearAlgebra.dot already uses for SparseVector).
• otherwise (one side much larger than the other — e.g. a short query against a long document): a Hwang-Lin/galloping merge — for each stored entry of the smaller side, an exponential ("galloping") search with memory of the last found position locates its match in the larger side in O(log gap) instead of a full linear scan.

This is deliberately not a method of LinearAlgebra.dot(::SparseVector,::SparseVector): SparseArrays already owns that method (a plain merge), and shadowing it package-wide would be a much more aggressive form of type piracy than TextSearch's existing Dict overloads of dot/normalize! (neither SparseVector nor dot belong to TextSearch, whereas extending dot for Dict doesn't collide with any other package's definitions). evaluate for SparseVector uses sparsedot internally.

Example

julia> using SparseArrays

julia> sparsedot(sparsevec(UInt32[1, 2], Float32[0.6, 0.8], 10), sparsevec(UInt32[2], Float32[1.0], 10))
0.8f0

table(model::VectorModel, TableConstructor)

Builds a Tables.jl-compatible table (e.g., a DataFrame) with one row per token, using TableConstructor (e.g. DataFrame) as the row-table constructor. Columns are token, ndocs, occs, and weight.

Example

julia> using DataFrames

julia> corpus = ["hello world", "hello there", "the cat sat"];

julia> voc = Vocabulary(TextConfig(), corpus; verbose=false);

julia> model = VectorModel(IdfWeighting(), TfWeighting(), voc);

julia> table(model, DataFrame)
6×4 DataFrame
 Row │ token   ndocs  occs   weight
     │ String  Int32  Int32  Float32
─────┼────────────────────────────────
   1 │ hello       2      2  0.485427
   2 │ world       1      1  1.22239
   3 │ there       1      1  1.22239
   4 │ the         1      1  1.22239
   5 │ cat         1      1  1.22239
   6 │ sat         1      1  1.22239

table(voc::Vocabulary, TableConstructor)

Builds a Tables.jl-compatible table (e.g., a DataFrame) with one row per token, using TableConstructor (e.g. DataFrame) as the row-table constructor. Columns are token, ndocs, and occs.

Example

julia> using DataFrames

julia> corpus = ["hello world", "hello there", "the cat sat"];

julia> voc = Vocabulary(TextConfig(), corpus; verbose=false);

julia> table(voc, DataFrame)
6×3 DataFrame
 Row │ token   ndocs  occs
     │ String  Int32  Int32
─────┼──────────────────────
   1 │ hello       2      2
   2 │ world       1      1
   3 │ there       1      1
   4 │ the         1      1
   5 │ cat         1      1
   6 │ sat         1      1

token(voc::Vocabulary, tokenID::Integer)
token(voc::Vocabulary)

The token string for tokenID (0 is out-of-vocabulary and yields "" instead of erroring), or the whole token vector when called without a tokenID.

Example

julia> voc = Vocabulary(TextConfig(), ["hello world", "hello there"]; verbose=false);

julia> token(voc, token2id(voc, "hello"))
"hello"

token2id(voc::Vocabulary, tok::AbstractString)::UInt32

Looks up the id of tok in voc; returns 0 when tok is out of vocabulary.

Example

julia> voc = Vocabulary(TextConfig(), ["hello world"]; verbose=false);

julia> token2id(voc, "hello")
0x00000001

julia> token2id(voc, "unknown")
0x00000000

tokenize_and_append!(voc::Vocabulary, corpus)

Parse each document in the given corpus and appends each token to the vocabulary.

Example

julia> voc = Vocabulary(TextConfig(), 0, 0);

julia> tokenize_and_append!(voc, ["hello world", "hello there"]);

julia> vocsize(voc)
3

trainsize(voc::Vocabulary)

Number of documents used to build voc.

update_voc!(voc::Vocabulary, another::Vocabulary)
update_voc!(pred::Function, voc::Vocabulary, another::Vocabulary)

Update voc vocabulary using another vocabulary. Optionally a predicate can be given to filter vocabularies.

Note 1: corpuslen remains unchanged (the structure is immutable and a new Vocabulary should be created to update this field). Note 2: Both voc and another vocabularies should had been created with a compatible TextConfig to be able to work on them.

Example

julia> cfg = TextConfig();

julia> voc = Vocabulary(cfg, 0, 0);

julia> update_voc!(voc, Vocabulary(cfg, ["hello world"]; verbose=false));

julia> vocsize(voc)
2

vectorize!(buff::VectorizeBuffer, model::VectorModel, text; normalize=true, minweight=1e-6)

Tokenizes text and weights it using model's local/global weighting scheme, returning the result as a SparseVector{Float32,Int32}; entries with a weight below minweight are dropped, and the result is L2-normalized unless normalize=false. buff is used as scratch space (see VectorizeBuffer; tokenization scratch space is borrowed separately via tokenizerbuffer). See vectorize for a version that manages the scratch buffer for you.

Example

julia> corpus = ["hello world", "hello there", "the cat sat"];

julia> voc = Vocabulary(TextConfig(), corpus; verbose=false);

julia> model = VectorModel(IdfWeighting(), TfWeighting(), voc);

julia> buff = TextSearch.VectorizeBuffer();

julia> TextSearch.vectorize!(buff, model, "hello world")
6-element SparseArrays.SparseVector{Float32, Int32} with 2 stored entries:
  [1]  =  0.369076
  [2]  =  0.929399

vectorize(model::VectorModel, text; normalize=true, minweight=1e-6)

Computes the weighted sparse vector (a SparseVector{Float32,Int32}) representation of text under model. text can be a string or a list of strings (a multi-field document).

Example

julia> corpus = ["hello world", "hello there", "the cat sat"];

julia> voc = Vocabulary(TextConfig(), corpus; verbose=false);

julia> model = VectorModel(IdfWeighting(), TfWeighting(), voc);

julia> vectorize(model, "hello world")
6-element SparseArrays.SparseVector{Float32, Int32} with 2 stored entries:
  [1]  =  0.369076
  [2]  =  0.929399

vectorize_corpus(model::VectorModel, corpus; normalize=true, minweight=1e-6, verbose=true)

Computes the vectorize representation of every document in corpus, processed in parallel across threads (the batch size is picked automatically from the size of corpus, as in SimilaritySearch.getminbatch).

Example

julia> corpus = ["hello world", "hello there", "the cat sat"];

julia> voc = Vocabulary(TextConfig(), corpus; verbose=false);

julia> model = VectorModel(IdfWeighting(), TfWeighting(), voc);

julia> vectorize_corpus(model, corpus; verbose=false)[1]
6-element SparseArrays.SparseVector{Float32, Int32} with 2 stored entries:
  [1]  =  0.369076
  [2]  =  0.929399

vocabulary_from_thesaurus(textconfig::TextConfig, tokens::AbstractVector)

Creates a Vocabulary directly from a list of tokens (a thesaurus), instead of tokenizing a corpus; every token is registered with occs=1 and ndocs=1.

Example

julia> voc = vocabulary_from_thesaurus(TextConfig(), ["cat", "dog", "bird"]);

julia> vocsize(voc)
3

julia> token2id(voc, "cat")
0x00000001

vocsize(voc::Vocabulary)

Number of unique tokens in voc.

BOW = Dict{UInt32,Int32}

A bag of words: a sparse token id => occurrence count mapping for a single document, as produced by bagofwords/bagofwords!.

Example

julia> BOW(0x00000001 => 2, 0x00000002 => 1)
Dict{UInt32, Int32}(0x00000002 => 1, 0x00000001 => 2)

BinaryGlobalWeighting()

The weight is 1 for known tokens, 0 for out of vocabulary tokens

BinaryLocalWeighting()

The weight is 1 for known tokens, 0 for out of vocabulary tokens

CombineWeighting

Abstract type for the strategies that turn a per-token, per-class distribution and its empirical entropy into a single global weight (via the internal combine_weight function), used by EntropyWeighting. Available strategies: NormalizedEntropy and SigmoidPenalizeFewSamples.

EntropyWeighting()

A GlobalWeighting that scores each token by the empirical entropy of its occurrences across document classes/labels, instead of the plain document frequency used by IdfWeighting — tokens whose occurrences concentrate on few classes get a higher weight than tokens spread uniformly across all classes. Since it needs document labels, it is not built via the generic VectorModel(gw, lw, voc) constructor; use VectorModel(::EntropyWeighting, lw, voc, corpus, labels) instead.

Example

julia> corpus = ["me gusta", "me encanta", "no me gusta", "odio esto"];

julia> labels = ["pos", "pos", "neg", "neg"];

julia> voc = Vocabulary(TextConfig(), corpus; verbose=false);

julia> model = VectorModel(EntropyWeighting(), TfWeighting(), voc, corpus, labels; verbose=false);

julia> vectorize(model, "me gusta")
6-element SparseArrays.SparseVector{Float32, Int32} with 1 stored entry:
  [1]  =  1.0

FreqWeighting()

Frequency weighting

GlobalWeighting

Abstract type for global weighting

IdfWeighting()

Inverse document frequency weighting

LocalWeighting

Abstract type for local weighting

NormalizedEntropy()

A CombineWeighting that weights a token as 1 - entropy / maxent: tokens that discriminate well between classes (low entropy) get a weight close to 1, tokens spread uniformly across classes (entropy close to maxent) get a weight close to 0.

Example

julia> corpus = ["me gusta", "me encanta", "no me gusta", "odio esto"];

julia> labels = ["pos", "pos", "neg", "neg"];

julia> voc = Vocabulary(TextConfig(), corpus; verbose=false);

julia> model = VectorModel(EntropyWeighting(), TfWeighting(), voc, corpus, labels; mindocs=1, comb=NormalizedEntropy(), verbose=false);

julia> vectorize(model, "me gusta")
6-element SparseArrays.SparseVector{Float32, Int32} with 2 stored entries:
  [1]  =  0.9996127
  [2]  =  0.0278291

SVEC = Dict{UInt32,Float32}

A weighted sparse vector: a token id => weight mapping, as produced by vectorize/vectorize!. Arithmetic (+, -, *, /), norms, and distances for SVEC/BOW-like Dicts are defined in dvec.jl (see normalize!, dot, norm, centroid).

Example

julia> SVEC(0x00000001 => 0.5f0)
Dict{UInt32, Float32}(0x00000001 => 0.5)

SigmoidPenalizeFewSamples()

Like NormalizedEntropy, but additionally down-weights tokens seen in very few documents (low ndocs) via a sigmoid-like penalty on log2(ndocs), so that rare tokens don't get an unduly high weight just because they appear in a single class.

Example

julia> corpus = ["me gusta", "me encanta", "no me gusta", "odio esto"];

julia> labels = ["pos", "pos", "neg", "neg"];

julia> voc = Vocabulary(TextConfig(), corpus; verbose=false);

julia> model = VectorModel(EntropyWeighting(), TfWeighting(), voc, corpus, labels; mindocs=1, comb=SigmoidPenalizeFewSamples(), verbose=false);

julia> vectorize(model, "me gusta")
6-element SparseArrays.SparseVector{Float32, Int32} with 2 stored entries:
  [1]  =  0.99974245
  [2]  =  0.02269659

TextModel

Abstract type for text-to-vector weighting models (see VectorModel).

TfWeighting()

Term frequency weighting

TpWeighting()

Term probability weighting

VectorModel{_G<:GlobalWeighting, _L<:LocalWeighting}

Combines a Vocabulary with a local/global term-weighting scheme (e.g. TfWeighting+IdfWeighting for classical TF-IDF) to turn bags of words into weighted sparse vectors (SparseVector{Float32,Int32}) via vectorize/vectorize!. Build one with VectorModel(gw, lw, voc).

Fields

• global_weighting: the GlobalWeighting scheme (e.g. IDF), applied per-token, corpus-wide.
• local_weighting: the LocalWeighting scheme (e.g. TF), applied per-token, per-document.
• voc: the underlying Vocabulary.
• maxoccs: maximum per-token occurrence count in voc, used by some local weightings.
• weight: precomputed per-token global weight (weight[tokenID]).

Example

julia> corpus = ["hello world", "hello there", "the cat sat"];

julia> voc = Vocabulary(TextConfig(), corpus; verbose=false);

julia> model = VectorModel(IdfWeighting(), TfWeighting(), voc);

julia> vectorize(model, "hello world")
6-element SparseArrays.SparseVector{Float32, Int32} with 2 stored entries:
  [1]  =  0.369076
  [2]  =  0.929399

VectorModel(ent::EntropyWeighting, lw::LocalWeighting, voc::Vocabulary, corpus::AbstractVector, labels::AbstractVector;
    mindocs=3,
    smooth=3,
    weights=:balance,
    comb::CombineWeighting=NormalizedEntropy(),
    verbose=true
)

Creates a VectorModel with EntropyWeighting as its global weighting scheme. Unlike the generic VectorModel(gw, lw, voc) constructor, this one needs the actual corpus and a matching labels vector (one label per document) to compute, for each token, its occurrence distribution across classes and the resulting entropy-based weight.

• mindocs: tokens occurring in fewer than mindocs documents get weight 0.
• smooth: additive (Laplace-like) smoothing applied to the per-class occurrence counts before computing entropy, to avoid zero counts.
• weights: how to reweight classes before computing entropy — :balance compensates for class-size imbalance, :none (or nothing) leaves classes unweighted, or pass an AbstractVector of per-class weights directly.
• comb: the CombineWeighting strategy combining entropy and evidence into the final per-token weight.

Example

julia> corpus = ["me gusta", "me encanta", "no me gusta", "odio esto"];

julia> labels = ["pos", "pos", "neg", "neg"];

julia> voc = Vocabulary(TextConfig(), corpus; verbose=false);

julia> model = VectorModel(EntropyWeighting(), TfWeighting(), voc, corpus, labels; mindocs=1, verbose=false);

julia> vectorize(model, "me gusta")
6-element SparseArrays.SparseVector{Float32, Int32} with 2 stored entries:
  [1]  =  0.9996127
  [2]  =  0.0278291

VectorModel(gw::GlobalWeighting, lw::LocalWeighting, voc::Vocabulary; weight=nothing)

Creates a VectorModel for the given vocabulary voc using the local weighting lw (e.g. TfWeighting) and global weighting gw (e.g. IdfWeighting). The per-token global weight vector is computed from voc unless weight is given explicitly (e.g. when reusing weights computed elsewhere, such as EntropyWeighting).

Example

julia> corpus = ["hello world", "hello there", "the cat sat"];

julia> voc = Vocabulary(TextConfig(), corpus; verbose=false);

julia> model = VectorModel(IdfWeighting(), TfWeighting(), voc);

julia> length(model.weight)
6

VectorizeBuffer(n=128)

Pooled per-thread scratch space for vectorize!: ids accumulates every in-vocabulary token id seen in a document (with repeats), which is then sorted and run-length-encoded to recover per-token occurrence counts — the same merge-based strategy sum(::AbstractVector{<:SparseVector}) uses, avoiding the Dict allocation/hashing a BOW would need for this per-call, performance-sensitive path.

Vocabulary

Holds the token ⇄ id mapping produced while parsing a corpus, along with per-token occurrence and document-frequency counters. A Vocabulary is the entry point of the processing pipeline: it is built from a TextConfig and a corpus, and is later consumed by VectorModel, BM25Scorer, and bagofwords.

Fields

• textconfig: the TextConfig used to tokenize the corpus that produced this vocabulary.
• token: id -> token string table.
• occs: id -> total number of occurrences of the token across the corpus.
• ndocs: id -> number of documents containing the token.
• token2id: token -> id reverse mapping (0 means "unknown token").
• trainsize: number of documents used to build the vocabulary.
• numtokens: total number of (non-unique) tokens seen while building the vocabulary.

Example

julia> voc = Vocabulary(TextConfig(), ["hello world", "hello there"]; verbose=false);

julia> vocsize(voc)
3

julia> token2id(voc, "hello")
0x00000001

Vocabulary(textconfig::TextConfig, corpus; buffsize=2^16, verbose=true)

Tokenizes corpus under textconfig and builds the resulting Vocabulary. corpus can be any vector of documents (each document a string or a list of strings) or an iterable/generator of documents (useful for corpora too large to fit in memory); in the generator case, documents are consumed and tokenized in batches of buffsize.

Example

julia> voc = Vocabulary(TextConfig(), ["hello world", "hello there"]; verbose=false);

julia> vocsize(voc), trainsize(voc)
(3, 2)

Vocabulary(textconfig::TextConfig, trainsize::Int, numtokens::Int)

Creates an empty Vocabulary (no tokens registered yet) preallocated with capacity hints based on trainsize (following Heaps' law). trainsize and numtokens may be 0 when unknown ahead of time; use push_token! or tokenize_and_append! to fill it, or use Vocabulary(textconfig, corpus) to build it directly from a corpus.

Example

julia> voc = Vocabulary(TextConfig(), 0, 0);

julia> TextSearch.push_token!(voc, "cat"; occs=1, ndocs=1)
0x00000001

julia> vocsize(voc)
1

_remove_empty!(L, P)

Inplace removal of empty lists

_sort!(L, P)

Adaptive bubble sort, efficient than other approaches because we expect a few sets and almost sorted

binarysearch(A, x, sp=1, ep=length(A))

Finds the insertion position of x in A in the range sp:ep

bk!(output, L, P, findpos::Function=doublingsearch) -> int. size

Computes the intersection of a list of posting lists using the Barbay and Kenyon algorithm.

See umerge! if you need to change how matches are captured (onmatch! function).

bkt!(output, L, [P,], [findpos::Function=doublingsearch]; t::Int=length(L)) -> num. matches

Barybay & Kenyon t-thresholds

See umerge! if you need to change how matches are captured (onmatch! function).

doublingsearch(A, x, sp=1, ep=length(A))

Finds the insertion position of x in A, starting at sp

doublingsearchrev(A, x, sp=1, ep=length(A))

Finds the insertion position of x in A, starting at the end

imerge2!(output, A, B) -> num. matches

Computes the intersection of A and B (sorted arrays), using a merge-like algorithm, and stores it in output or calls onmatch(A, i, B, j) on every match. See bk! for how change the behaviour of matches.

searchrev(A, x, sp=1, ep=length(A))

Sequential search, i.e., it starts from sp to ep

seqsearchrev(A, x, sp=1, ep=length(A))

Reverse sequential search, i.e., it starts from ep to sp

svs(postinglists, intersect2=baezayates!) -> output

Computes the intersection of the ordered lists in postinglists using a small vs small strategy. Accepts an intersection algorithm of two sets.

This method does not give explicit support for onmatch2!

 umerge!(output, L, P=ones(Int32, length(L_)); t::Int=1) -> num. matches

Merges posting lists in L and saves the union in output. The merge result is stored into output array. You can customize how to do this specializing the onmatch!(output, L, P, t::Int) function.

Arguments:

• L: The array of posting lists, the array can be destroyed in the process.

• P: The array of current positions in posting lists, i.e., initial state as an array of ones of size $|L|$.

• t: Computes t-thresholds, i.e., t from 1 (union) to |L| (intersection) of posting lists in L using findpos storing the result set in output.

About the callback function

output, L and P are the arguments same than the input, while t is the actual number of lists having the match. Note 1: you should access L[i][P[i]] to get the entry of the ith list, i.e., $1 \leq t \leq |P|$. Note 2: L and P as container lists will be also modified, the contained lists remain untouched.

xmerge!(output, L, P=ones(Int32, length(L_)); t::Int=1) -> num. matches

Solves t-threshold set operation using other algorithms choosing among them by given t

Arguments:

• output: vector like to store the t-threshold set
• L: the list of posting lists to be merged. The posting lists are left untouched but the container is modified.
• P: indices of the current merging-state (idem to L)
• t: the threshold, i.e., t=1 (union) ... t=|L| performs intersection)

Simple wrapper around other specific operations depending on t value

See umerge! if you need to modify the output behaviour.

length(idx::AbstractInvertedFile)

Number of indexed elements

append_items!(idx, ctx, items; tol=1e-6)

Appends all items elements into the index idx. It work in parallel using all available threads.

Arguments:

• idx: The inverted index
• items: The database of sparse objects, it can be only indices if each object is a list of integers or a set of integers (useful for BinaryInvertedFile), sparse matrices, dense matrices, among other combinations.
• n: The number of items to insert (defaults to all)

Keyword arguments:

• tol: controls what is a zero (i.e., weights < tol will be ignored).

push_item!(idx::AbstractInvertedFile, ctx::InvertedFileContext, obj; tol=1e-6)

Inserts a single element into the index. This operation is not thread-safe.

Arguments

• idx: The inverted index
• ctx: the index's context
• obj: The object to be indexed

Keyword arguments

• tol: controls what is a zero (i.e., weight < tol will be ignored)

search(idx::AbstractInvertedFile, ctx::InvertedFileContext, q, res::AbstractKnn; tol=1e-6, t=1)

Searches q in idx using the cosine dissimilarity, it computes the full operation on idx. res specify the query

convertpair(u)

Converts an element of an sparseiterator into an usable pair.

searchinvfile(acceptposting_list::Function, idx::BinaryInvertedFile, ctx::InvertedFileContext, Q, res::AbstractKnn, t)

Find candidates for solving query Q using idx. It calls callback on each candidate (objID, dist)

Arguments

• accept_posting_list: predicate to accept or reject a posting list
• idx: inverted index
• Q: the set of involved posting lists, see select_posting_lists
• t: threshold (t=1 union, t > 1 solves the t-threshold problem)

searchinvfile(acceptposting_list::Function, idx::WeightedInvertedFile, ctx::InvertedFileContext, q, res::AbstractKnn, t)

Find candidates for solving query Q using idx. It calls callback on each candidate (objID, dist)

Arguments:

• accept_posting_list: predicate to accept or reject a posting list
• idx: inverted index
• Q: the set of involved posting lists, see select_posting_lists

select_posting_lists(idx::AbstractInvertedFile, ctx::InvertedFileContext, q, tol)

Fetches and prepares the involved posting lists to solve q

set_distance_evaluate(dist::SemiMetric, intersection::Integer, size1::Integer, size2::Integer)

Computes the distance function dist on a BinaryInvertedFile.

sparseiterator(obj)

(id, weight) iterator for obj for generic databases.

sparseiterator(db, i)

Creates an iterator for indices and values of the i-th db's element (e.g., column). Several specializations are provided.

abstract type AbstractInvertedFile <: AbstractSearchIndex end

Abstract inverted file, actual data structures are WeightedInvertedFile and BinaryInvertedFile

BinaryInvertedFile(vocsize::Integer, dist=Dist.Sets.Jaccard())

Creates an BinaryInvertedFile with the given vocabulary size and for the given distance function dist:

Arguments:

• vocsize: the vocabulary size of the index
• dist: the distance function to be used in searches

struct BinaryInvertedFile <: AbstractInvertedFile

Creates a binary weighted inverted index. An inverted index is an sparse matrix representation optimized for computing k nn elements (columns) under some distance.

Properties:

• dist: Distance function to be applied, valid values are: Dist.Sets.Intersection(), Dist.Sets.Dice(), Dist.Sets.Jaccard(), and `Dist.Sets.CosineSet()
• lists: posting lists (non-zero values of the rows in the matrix)
• sizes: number of non-zero values per object (number of non-zero values per column)
• locks: Per row locks for multithreaded construction

IdIntWeight(id, weight)

Stores a pair of objects to be accessed. Similar to IdWeight but it stores an integer weight

IdWeight(id, weight)

Stores a pair of entries of the posting lists

struct PostingList

A paired list of identifiers and weights

struct WeightedInvertedFile <: AbstractInvertedFile

An inverted index is a sparse matrix representation of with floating point weights, it supports only positive non-zero values. This index is optimized to efficiently solve k nearest neighbors (cosine distance, using previously normalized vectors).

Parameters

• lists: posting lists (non-zero id-elements in rows)
• weights: non-zero weights (in rows)
• sizes: number of non-zero values in each element (non-zero values in columns)

WeightedInvertedFile(vocsize::Integer)

Convenient function to create an empty WeightedInvertedFile with the given vocabulary size.

append_items!(idx::BM25InvertedFile, ctx::InvertedFileContext, corpus; kwargs...)

Adds every document in corpus to idx, computing each one's bag of words under idx.voc first. corpus can hold raw text (AbstractString), already-tokenized TokenizedText, or pre-tokenized string vectors; a corpus of already-computed BOWs is accepted directly by the generic SimilaritySearch.append_items! method without going through this conversion. See also push_item!.

Example

julia> voc = Vocabulary(TextConfig(), ["hello world", "hello there"]; verbose=false);

julia> invfile = BM25InvertedFile(voc);

julia> ctx = InvertedFileContext();

julia> append_items!(invfile, ctx, ["hello world", "hello there"]);

julia> length(invfile)
2

push_item!(idx::BM25InvertedFile, ctx::InvertedFileContext, doc)

Adds a single document doc to idx, computing its bag of words under idx.voc first. doc can be raw text (AbstractString), already-tokenized TokenizedText, or a pre-tokenized string vector; an already-computed BOW is accepted directly by the generic SimilaritySearch.push_item! method without going through this conversion. See also append_items!.

Example

julia> corpus = ["hello world", "hello there"];

julia> voc = Vocabulary(TextConfig(), corpus; verbose=false);

julia> invfile = BM25InvertedFile(voc);

julia> ctx = InvertedFileContext();

julia> append_items!(invfile, ctx, corpus);

julia> push_item!(invfile, ctx, "hello again");

julia> length(invfile)
3

search(idx::BM25InvertedFile, ctx::InvertedFileContext, qtext, res::AbstractKnn)
search(accept_posting_list::Function, idx::BM25InvertedFile, ctx::InvertedFileContext, qtext, res::AbstractKnn; t::Int=1)

Solves a top-k query over idx for qtext (raw text, TokenizedText, or an already-computed bag of words), accumulating matches into res (an AbstractKnn, e.g. KnnResult/KnnSorted). Documents are ranked by BM25 score (stored internally as a negative value in res, so lower "distance" still means better match, consistent with SimilaritySearch.jl's convention).

The accept_posting_list variant additionally receives a predicate called with each query token's posting list before scanning it, letting the caller skip lists (e.g. to implement stopword-like filtering at query time); it defaults to accepting every list. Returns res.

Example

julia> corpus = ["hello world", "hello there", "the cat sat"];

julia> voc = Vocabulary(TextConfig(), corpus; verbose=false);

julia> invfile = BM25InvertedFile(voc);

julia> ctx = InvertedFileContext();

julia> append_items!(invfile, ctx, corpus);

julia> res = knnqueue(KnnSorted, 2);

julia> search(invfile, ctx, "hello", res) do lst
           true
       end;

julia> collect(IdView(res))
UInt32[0x00000001, 0x00000002]

bm25score(bm25::BM25Scorer, voc::Vocabulary, query::Dict, doc::Dict)::Float32

Computes the BM25 relevance score of doc (a bag of words) for query (a bag of words), summing tokenscore over every query token present in doc. Higher is more relevant.

Example

julia> corpus = ["hello world", "hello there", "the cat sat"];

julia> voc = Vocabulary(TextConfig(), corpus; verbose=false);

julia> bm25 = BM25Scorer(voc);

julia> bm25score(bm25, voc, bagofwords(voc, "hello"), bagofwords(voc, "hello world"))
0.96917987f0

filter_lists!(
    idx::BM25InvertedFile;
    list_min_length_for_checking::Int=96,
    list_max_allowed_length::Int=1024,
    doc_min_freq::Int=1,
    doc_max_freq::Int=128,
    always_sort::Bool=false
)

Prunes each posting list of idx in place, once it is already populated. Lists shorter than list_min_length_for_checking are left untouched (optionally sorted by document id when always_sort=true). Longer lists are filtered to entries whose term frequency lies in [doc_min_freq, doc_max_freq], then truncated to the list_max_allowed_length highest-frequency entries — this both discards overly rare/common (likely noisy) postings and bounds the cost of scanning very long lists at query time. Returns idx.

Example

julia> corpus = ["hello world", "hello there", "the cat sat"];

julia> voc = Vocabulary(TextConfig(), corpus; verbose=false);

julia> invfile = BM25InvertedFile(voc);

julia> ctx = InvertedFileContext();

julia> append_items!(invfile, ctx, corpus);

julia> filter_lists!(invfile) === invfile
true

tokenscore(bm25::BM25Scorer, toknumdocs, doclen, tokfreqindoc)

Computes the BM25 contribution of a single token to a document's score, given the number of documents containing the token (toknumdocs, i.e. its document frequency), the document's length in tokens (doclen), and the token's frequency in the document (tokfreqindoc). Used internally by bm25score and BM25InvertedFile search.

Example

julia> corpus = ["hello world", "hello there", "the cat sat"];

julia> voc = Vocabulary(TextConfig(), corpus; verbose=false);

julia> tokenscore(BM25Scorer(voc), 2, 3, 1)
0.8908208f0

BM25InvertedFile{AdjType<:AbstractAdjList} <: AbstractInvertedFile

An inverted-file index (built on top of InvertedFiles.jl) that answers approximate/exact top-k queries ranked by BM25 relevance. Build it with BM25InvertedFile(voc), populate it with append_items!/push_item!, and query it with search (from SimilaritySearch.jl).

Fields

• voc: the Vocabulary shared by every indexed document (also used to tokenize/encode query text).
• bm25: the BM25Scorer used to rank matches.
• adj: the adjacency list of posting lists (one per token id), mapping each token to the documents containing it and their term frequency.
• doclens: number of tokens per indexed document.

Example

julia> using SimilaritySearch

julia> corpus = ["hello world", "hello there", "the cat sat"];

julia> voc = Vocabulary(TextConfig(), corpus; verbose=false);

julia> invfile = BM25InvertedFile(voc);

julia> ctx = InvertedFileContext();

julia> append_items!(invfile, ctx, corpus);

julia> length(invfile)
3

julia> res = knnqueue(KnnSorted, 2);

julia> search(invfile, ctx, "hello", res);

julia> collect(IdView(res))
UInt32[0x00000001, 0x00000002]

BM25InvertedFile(voc::Vocabulary; k1=1.2f0, b=0.75f0, δ=1f0)

Creates an empty BM25InvertedFile, fitting its BM25Scorer from voc (see BM25Scorer(voc) for k1/b/δ). Populate it with append_items!/push_item!.

Example

julia> voc = Vocabulary(TextConfig(), ["hello world"]; verbose=false);

julia> invfile = BM25InvertedFile(voc);

julia> length(invfile)
0

BM25Scorer

Precomputed coefficients for the Okapi BM25 (BM25+) scoring function, used to rank documents against a query given per-token document frequencies and document lengths. Build one with BM25Scorer(voc) or BM25Scorer(trainsize, avgdoclen); score individual (token-frequency, document-length) pairs with tokenscore, or whole query/document bags of words with bm25score. BM25InvertedFile uses a BM25Scorer internally to answer top-k queries efficiently.

Fields

k1_plus_1, k1_mult_1_min_b, and k1_mult_b_div_avg_doc_len are combinations of the BM25 k1/b hyperparameters and the corpus' average document length, precomputed for faster scoring; δ is the BM25+ lower-bound correction term; trainsize is the number of documents the corpus statistics were computed from.

Example

julia> corpus = ["hello world", "hello there", "the cat sat"];

julia> voc = Vocabulary(TextConfig(), corpus; verbose=false);

julia> bm25 = BM25Scorer(voc);

julia> bm25.trainsize
3

BM25Scorer(trainsize::Integer, avgdoclen::AbstractFloat; k1=1.2f0, b=0.75f0, δ=1f0)

Creates a BM25Scorer for a corpus of trainsize documents with average document length avgdoclen. k1 controls term-frequency saturation, b controls document-length normalization, and δ is the BM25+ lower-bound correction.

Example

julia> bm25 = BM25Scorer(3, 3.5);

julia> bm25.trainsize
3

BM25Scorer(voc::Vocabulary; k1=1.2f0, b=0.75f0, δ=1f0)

Creates a BM25Scorer using the training size and average document length already computed in voc (see trainsize and avgdoclen).

Example

julia> corpus = ["hello world", "hello there", "the cat sat"];

julia> voc = Vocabulary(TextConfig(), corpus; verbose=false);

julia> BM25Scorer(voc).trainsize
3

empty!(buff::TokenizerBuffer; normtext=true, tokens=true, unigrams=true)

Clears the requested scratch fields of buff in place. Returns buff.

alltokengenerators(cfg::TextConfig)::Vector{AbstractTokenGenerator}

Builds the full, ordered list of AbstractTokenGenerators cfg runs: the built-in ones implied by cfg.qlist/cfg.nlist/cfg.slist/cfg.collocations, followed by cfg.generators (any extra/custom generators). Called once per tokenize invocation.

Example

julia> alltokengenerators(TextConfig(nlist=[1, 2]))
2-element Vector{AbstractTokenGenerator}:
 UnigramGenerator()
 NWordGenerator(2)

collocations(gen::CollocationGenerator, buff::TokenizerBuffer, tt::AbstractTokenTransformation, mark_token_type)

Computes a kind of collocations of the given text

flush_token!(buff::TokenizerBuffer, tt::AbstractTokenTransformation, gen::AbstractTokenGenerator, mark_token_type::Bool)

Pushes the token accumulated in buff.io to the token list, applying gen's tokentag (when mark_token_type) and transform hook; discards empty strings and tokens the transformation drops (returns nothing for).

generate!(gen::AbstractTokenGenerator, buff::TokenizerBuffer, tt::AbstractTokenTransformation, mark_token_type::Bool)

Runs gen over buff, appending its produced tokens to buff.tokens. Called by tokenize for every generator in alltokengenerators; a new AbstractTokenGenerator subtype implements this method to define what it does. UnigramGenerator's tokens are emitted as a side effect of the shared unigrams pass, so its own generate! is a no-op.

isemoji(c::Char, emojis::Set{Char}=DEFAULT_EMOJIS)::Bool

Tests whether c is one of the emoji characters in emojis (by default, DEFAULT_EMOJIS, the set known to TextSearch, loaded from emojis.txt). Used by normalize_text when TextConfig's group_emo option is set.

Example

julia> isemoji('😀')
true

julia> isemoji('a')
false

needs_unigrams(gen::AbstractTokenGenerator)::Bool

Whether gen needs the shared word-level unigrams basis (see unigrams) computed before it runs. Defaults to false; NWordGenerator, SkipgramGenerator, CollocationGenerator, and UnigramGenerator override it to true. QGramGenerator operates directly on the normalized text and does not need it.

normalize_text(config::TextConfig, text::AbstractString, output::Vector{Char}; limits::Bool=true)

Normalizes a given text using the specified transformations of config

Example

julia> buff = Char[];

julia> normalize_text(TextConfig(), "Café", buff);

julia> String(buff)
" cafe "

normalize_text(textconfig::TextConfig, text; limits::Bool=false)

Convenience method that normalizes text under textconfig (see normalize_text(config, text, output; limits)) and returns the result as a String instead of writing into a caller-provided buffer.

Example

julia> normalize_text(TextConfig(), "Café!!")
"cafe!!"

nwords(gen::NWordGenerator, buff::TokenizerBuffer, tt::AbstractTokenTransformation, mark_token_type)

qgrams(gen::QGramGenerator, buff::TokenizerBuffer, tt::AbstractTokenTransformation, mark_token_type)

Computes character q-grams for the given input

skipgrams(gen::SkipgramGenerator, buff::TokenizerBuffer, tt::AbstractTokenTransformation, mark_token_type)

Tokenizes using skipgrams

tokenize(textconfig::TextConfig, text)
tokenize(copy_::Function, textconfig::TextConfig, text)

tokenize(textconfig::TextConfig, text, buff)
tokenize(copy_::Function, textconfig::TextConfig, text, buff)

Tokenizes text using the given configuration. The tokenize makes heavy usage of buffers, and when these buffers are shared it is mandatory to create a copy of the result (buff.tokens).

Change the default copy function to make an additional filtering of the tokens. You can also pass the identity function to avoid copying.

Example

julia> collect(tokenize(TextConfig(), "Hello world!!"))
["hello", "world", "!!"]

tokenize_corpus(textconfig::TextConfig, arr; verbose=true)
tokenize_corpus(copy_::Function, textconfig::TextConfig, arr; verbose=true)

Tokenize a list of texts. The copy_ function is passed to tokenize as first argument.

Example

julia> corpus = ["hello world", "the cat sat"];

julia> toks = tokenize_corpus(TextConfig(), corpus; verbose=false);

julia> collect(toks[1])
["hello", "world"]

tokenizerbuffer(f)

Borrows a TokenizerBuffer from Tokenizer's own pool, passes it to f, and returns it to the pool afterwards. Unlike the buffer-less tokenize methods (which release their borrowed buffer before returning), the buffer stays borrowed for the whole extent of f, so it is safe to alias its contents (e.g. via borrowtokenizedtext) as long as they are consumed inside f.

Example

julia> TextSearch.Tokenizer.tokenizerbuffer() do buff
           tokenize(borrowtokenizedtext, TextConfig(), "hello world", buff) |> collect
       end
["hello", "world"]

tokentag(gen::AbstractTokenGenerator)::Union{Char,Nothing}

The single-character tag appended (as \ttag) to every token gen produces when mark_token_type=true. Defaults to nothing (untagged).

transform(tt::AbstractTokenTransformation, gen::AbstractTokenGenerator, tok)

Hook applied in the tokenization stage to change the input token tok, produced by generator gen (e.g. a QGramGenerator or NWordGenerator), if needed. For instance, it can be used to apply stemming or any other kind of normalization. Return nothing to ignore the tok occurrence (e.g., stop words).

The default falls through to identity for any gen a custom AbstractTokenTransformation doesn't specialize, so adding a new AbstractTokenGenerator kind never requires touching existing transformations. The built-in generators dispatch to the legacy transform_unigram/transform_nword/transform_qgram/transform_skipgram/ transform_collocation names for backward compatibility with transformations written against those.

Example

julia> transform(IdentityTokenTransformation(), UnigramGenerator(), "cat")
"cat"

transform_collocation(::AbstractTokenTransformation, tok)

Legacy per-kind hook kept for backward compatibility; prefer specializing transform on AbstractTokenGenerator subtypes in new code. Called by the default transform method for CollocationGenerator tokens. Return nothing to ignore the tok occurence (e.g., stop words).

transform_nword(::AbstractTokenTransformation, tok)

Legacy per-kind hook kept for backward compatibility; prefer specializing transform on AbstractTokenGenerator subtypes in new code. Called by the default transform method for NWordGenerator tokens. Return nothing to ignore the tok occurence (e.g., stop words).

transform_qgram(::AbstractTokenTransformation, tok)

Legacy per-kind hook kept for backward compatibility; prefer specializing transform on AbstractTokenGenerator subtypes in new code. Called by the default transform method for QGramGenerator tokens. Return nothing to ignore the tok occurence (e.g., stop words).

transform_skipgram(::AbstractTokenTransformation, tok)

Legacy per-kind hook kept for backward compatibility; prefer specializing transform on AbstractTokenGenerator subtypes in new code. Called by the default transform method for SkipgramGenerator tokens. Return nothing to ignore the tok occurence (e.g., stop words).

transform_unigram(::AbstractTokenTransformation, tok)

Legacy per-kind hook kept for backward compatibility; prefer specializing transform on AbstractTokenGenerator subtypes in new code. Called by the default transform method for UnigramGenerator tokens. Return nothing to ignore the tok occurence (e.g., stop words).

unigrams(buff::TokenizerBuffer, tt::AbstractTokenTransformation)

Performs the word tokenization

AbstractTokenGenerator

Abstract type for a single token-producing strategy inside a TextConfig's generators list. TextConfig's qlist/nlist/slist/collocations keyword arguments are convenience sugar that build the built-in generators below; passing generators directly (or mixing in your own AbstractTokenGenerator subtype) is how new kinds of tokens can be added without touching TextConfig or the tokenizer's dispatch logic.

Implementing a new generator kind requires:

• a struct <: AbstractTokenGenerator holding whatever parameters it needs;
• needs_unigrams (defaults to false) if it needs the shared word-level unigrams basis computed first;
• TextSearch.Tokenizer.generate! performing the actual token production;
• optionally tokentag (defaults to nothing, i.e. untagged) for the single-character tag appended to each token when mark_token_type=true.

The transform hook already dispatches on AbstractTokenGenerator with an identity default, so a new generator kind is usable with any existing AbstractTokenTransformation without further changes.

AbstractTokenTransformation

Abstract type for token transformation hooks applied during tokenization (see transform). A TextConfig holds one such transformation in its tt field; it is applied to every generated token before it is pushed to the token list, and can be used to implement stemming, casing rules, or stopword removal (by returning nothing).

ChainTransformation(list::AbstractVector{<:AbstractTokenTransformation})

Holds an ordered sequence of AbstractTokenTransformations, applied one after the other over each token via transform; if any step returns nothing the token is dropped and the remaining steps are skipped.

Example

julia> ct = ChainTransformation([IdentityTokenTransformation(), IgnoreStopwords(Set(["the"]))]);

julia> collect(tokenize(TextConfig(nlist=[1], tt=ct), "the cat sat"))
["cat", "sat"]

CollocationGenerator(window)

Produces word collocations within window from the shared unigram basis (tagged 'c'). Built from TextConfig's collocations keyword argument.

IdentityTokenTransformation()

The default, no-op AbstractTokenTransformation: every token is kept unchanged.

Example

julia> collect(tokenize(TextConfig(tt=IdentityTokenTransformation()), "the cat sat"))
["the", "cat", "sat"]

IgnoreStopwords(stopwords::Set{String})

An AbstractTokenTransformation that discards unigrams found in stopwords (returns nothing for them, causing the tokenizer to drop the token) and passes every other token through unchanged.

Example

julia> cfg = TextConfig(nlist=[1], tt=IgnoreStopwords(Set(["the", "a"])));

julia> collect(tokenize(cfg, "the cat sat"))
["cat", "sat"]

NWordGenerator(q)

Produces word q-grams (q > 1) from the shared unigram basis (tagged 'n'). Built from TextConfig's nlist keyword argument for every entry other than 1.

QGramGenerator(q)

Produces character q-grams from the normalized text (tagged 'q'). Built from TextConfig's qlist keyword argument.

Skipgram(qsize, skip)

A skipgram is a kind of tokenization where qsize words having skip separation are used as a single token.

Example

julia> collect(tokenize(TextConfig(slist=[Skipgram(2, 1)]), "the cat sat down"))
["the sat	s", "cat down	s"]

SkipgramGenerator(skipgram::Skipgram)

Produces skip-grams from the shared unigram basis (tagged 's'). Built from TextConfig's slist keyword argument.

TextConfig(;
    del_diac::Bool=true,
    del_dup::Bool=false,
    del_punc::Bool=false,
    group_num::Bool=true,
    group_url::Bool=true,
    group_usr::Bool=false,
    group_emo::Bool=false,
    lc::Bool=true,
    collocations::Int8=0,
    qlist::Vector=Int8[],
    nlist::Vector=Int8[],
    slist::Vector{Skipgram}=Skipgram[],
    mark_token_type::Bool = true
    re_user::Regex=DEFAULT_RE_USER,
    re_url::Regex=DEFAULT_RE_URL,
    re_num::Regex=DEFAULT_RE_NUM,
    emojis::Set{Char}=DEFAULT_EMOJIS,
    generators::Vector{<:AbstractTokenGenerator}=AbstractTokenGenerator[],
    tt=IdentityTokenTransformation()
)

Defines a preprocessing and tokenization pipeline

• del_diac: indicates if diacritic symbols should be removed
• del_dup: indicates if duplicate contiguous symbols must be replaced for a single symbol
• del_punc: indicates if punctuaction symbols must be removed
• group_num: indicates if numbers should be grouped _num
• group_url: indicates if urls should be grouped as _url
• group_usr: indicates if users (@usr) should be grouped as _usr
• group_emo: indicates if emojis should be grouped as _emo
• lc: indicates if the text should be normalized to lower case
• collocations: window to expand collocations as tokens, please take into account that:
  • 0 => disables collocations
  • 1 => will compute words (ignored in favor of use typical unigrams)
  • 2 => will compute bigrams (don't use this, but not disabled)
  • 3 <= typical values
• qlist: a list of character q-grams to use
• nlist: a list of words n-grams to use
• slist: a list of skip-grams tokenizers to use
• mark_token_type: each token is marked with its type (qgram, skipgram, nword) when is true.
• re_user, re_url, re_num: the regexes used to detect @user mentions, URLs, and numbers when their corresponding group_* flag is set (see normalize_text). Override them to customize detection (e.g. for a different language or domain).
• emojis: the set of emoji characters grouped when group_emo is set (see isemoji).
• generators: extra AbstractTokenGenerators to run in addition to the ones qlist/nlist/slist/collocations build; this is the extension point for adding new kinds of tokens without needing a new TextConfig keyword argument (see alltokengenerators).
• tt: An AbstractTokenTransformation struct

Note: If qlist, nlist, slist, and generators are all empty, then it defaults to nlist=[1]

Example

julia> cfg = TextConfig(nlist=[1], qlist=[3]);

julia> collect(tokenize(cfg, "cats"))
[" ca	q", "cat	q", "ats	q", "ts 	q", "cats"]

TokenizedText(tokens::AbstractVector{String})

Wraps the token list produced by tokenize for a single document. Behaves like an AbstractVector{String} (it supports indexing, iteration, push!, append!, etc.) and is the type consumed by bagofwords/bagofwords! and by Vocabulary-building functions.

Example

julia> collect(tokenize(TextConfig(), "Hello world!!"))
["hello", "world", "!!"]

TokenizerBuffer(n=128)

Self-contained scratch space reused across tokenization calls to avoid reallocating on every call: normtext holds the normalized text, tokens accumulates the produced tokens, unigrams holds the word-level basis used by n-word/skip-gram/collocation generators, and io is scratch space for building individual token strings.

Tokenizer pools these internally for its own buffer-less convenience API (see tokenize); callers that need to hold a buffer across several calls (e.g. to safely alias its contents via borrowtokenizedtext) should borrow one from the same pool via tokenizerbuffer instead of constructing their own.

UnigramGenerator()

Emits the word-level unigrams themselves as output tokens (untagged). Built from TextConfig's nlist keyword argument when it contains 1. Every other generator that needs the word-level basis (see needs_unigrams) triggers the same underlying computation regardless of whether UnigramGenerator is present — this generator only controls whether the plain words also appear in the output.

DEFAULT_EMOJIS::Set{Char}

The built-in emoji set (loaded from emojis.txt), used as TextConfig's default emojis field. Pass a different Set{Char} to TextConfig(; emojis=...) to override it.

DEFAULT_RE_NUM::Regex

The built-in number regex, used as TextConfig's default re_num field. Pass a different Regex to TextConfig(; re_num=...) to override it.

DEFAULT_RE_URL::Regex

The built-in URL regex, used as TextConfig's default re_url field. Pass a different Regex to TextConfig(; re_url=...) to override it.

DEFAULT_RE_USER::Regex

The built-in @user-mention regex, used as TextConfig's default re_user field. Pass a different Regex to TextConfig(; re_user=...) to override it.