Module table 

The reduced table: the spam filter as every device runs it, a megabyte or two, with no word of your mail in it: each word under a 64-bit hash of it (word_hash), never the word itself, and only words seen five times or more.

Two kinds (Kind), both linear, so that they fold (docs/spam-filter.md, “The model, and how it folds”):

• The classifier’s (Kind::Rows, what a training makes): fastText’s supervised model, two labels, reads a message as its words, then its header facts as words (features::header_words), then the end of the line. Each word brings its input rows (its own, when fastText knows it, and those of its character n-grams’ buckets), and the model’s score is the mean of every row’s scalar s_r = (w_spam − w_ham) · row_r, the softmax of two labels being the sigmoid of that. The table keeps, for each word fastText knows, the sum of its rows’ scalars; for each bucket, its scalar; for the end of the line (bias), its own. A message’s score: f = (bias + Σ its words' sums) / (1 + Σ their rows), a word having 1 + its n-grams’ count of rows when the table knows it, else its n-grams’ (ngram_buckets), each counted as often as it comes. No header weights: the facts are words.
• The centroid’s (Kind::Centroid, format 1’s only kind, still made on request): fastText learns a vector for each word (the language model, private, never shared), and a linear SVM weighs the mean of a message’s word vectors with its header features (features.rs). With u = w / σ the SVM’s embedding weights over the standardization’s deviations, a word scores s_w = u · v_w, an n-gram bucket s_b = u · row_b, and the mean of a message’s word scores is the SVM’s score of the mean of its vectors. f = bias + (mean of its words' scores, or 0 without words) − text_mean + Σ weightₕ·(xₕ − meanₕ), where a word missing from the table scores the mean of its character n-grams’ buckets, as fastText builds the vector of a word it never saw. The standardization’s mean terms stay beside the weights rather than folded into the bias, so that the explanation can say what departs from the usual; the score is the same.

Its probability of being spam, either kind: Platt’s p = 1 / (1 + exp(A·f + B)).

The file (little-endian), $XDG_DATA_HOME/sioul/spam/table.bin:

bytes	what
8	magic SIOULSPM
4 × 12	format version (2), tokenizer version, features version, embedding dimension, minn, maxn, bucket count B, word count W, header feature count H (0: the classifier’s kind; features::N: the centroid’s), metadata length L, two zeros (so that the hashes start on 8 bytes)
4 × 4	bias (the classifier’s: the end of the line’s scalar), text_mean (the classifier’s: 0), Platt A, Platt B (f32)
8 × W	the words’ hashes, ascending (u64)
4 × W	their scores (f32): the classifier’s, the sum of each word’s rows’ scalars
4 × B	the buckets’ scores (f32)
4 × H, 4 × H	the header features’ weights, then their means (f32), the centroid’s alone
L	metadata, JSON (Meta)
8	FNV-1a 64 of every byte before it

Structs

Meta

What the settings say of the training that made the table.

Score

A message’s score, and what made it.

Table

The spam filter, folded (see the module’s documentation).

Enums

Kind

How a table scores a message (see the module’s documentation).

Constants

HEADER 🔒

Bytes before the words: the magic, twelve counts, four numbers.

MAGIC

The file’s first bytes.

VERSION

The file format’s version: 2 since the classifier’s kind (Kind); a Sioul that reads 1 refuses it, rather than reading its sums as scores.

Functions

fasttext_hash 🔒

fastText’s hash: 32-bit FNV-1a, each byte sign-extended first, as the C++ casts it (uint32_t(int8_t(c))): bytes above 127 differ from plain FNV-1a.

fnv64 🔒

keep_previous

Before another table takes path’s place (a training here, another device’s table through the sharing): the one there, when this Sioul can use it, copied beside as table.prev.bin (written beside, then moved). A table this Sioul refuses then leaves the last good one in use (Table::cached), and never takes its place there. Best effort: a copy that fails leaves the one before as it was.

ngram_buckets

The buckets of a word’s character n-grams, as fastText 0.8.0 computes them (https://github.com/messense/fasttext-rs, tag v0.8.0: src/dictionary.rs, for_each_ngram and get_subwords_for_string; src/utils.rs, hash): the word between < and >; from each character (not each byte: UTF-8 continuation bytes start nothing), every n-gram of minn to maxn characters (a single character never at either end); each hashed (fasttext_hash) modulo bucket; in that order, repeats kept. fastText’s input row for one is nwords + bucket, and the vector of a word it never saw is the mean of those rows (get_word_vector_into); of a word it knows, the mean of its own row and those.

platt

Platt’s probability for a decision value.

previous

Where the table replaced by Table::write, or by another device’s through the sharing, is kept.

sorted 🔒

Words sorted by their hash; two words of one hash (never seen yet: 64 bits) share their mean score.

word_hash

A word’s key in the table: FNV-1a, 64 bits, over its UTF-8 bytes.