Module supervised 

fastText’s classifier, the filter’s model (train::Model::Supervised, the default; docs/spam-filter.md, “The model, and how it folds”): each message one line, its words, then its header facts as words (features::header_words), two labels, softmax. Word pairs only when asked (word_ngrams above 1), in a trial: the table reads none.

It folds per row: with two labels, softmax’s P(spam) is the sigmoid of f = (w_spam − w_ham) · h, where h is the mean of every input row the line reads (each known word’s own, each of its character n-grams’ buckets, the end of the line’s), so f is the mean of the rows’ scalars (w_spam − w_ham) · row (row_scalars). The table keeps each known word’s rows’ sum, each bucket’s scalar and the end of the line’s (table::Kind::Rows); it must give fastText’s own prediction within 1e-4 on the test messages, or nothing is exported.

Calibrated by Platt on scores of mail the model never learned from: in each account’s and each outside source’s mail of each label, the newest fifth of the messages learned from, scored by a model of the rest (as the test is split: train::newest_fifths), Platt fitted on your own mail’s part, the mail the filter will judge. In time order alone, that newest fifth would be your own mail only, scored by a model of outside material only, which has no header facts: its probabilities came out far too sure of spam.

After the tokens (train::train_with’s first steps, shared): the same parts (train::parts); calibration, then the model of the oldest 80 %; its table, checked; the numbers and the comparison with the table in place, as the centroid’s; kept, calibrated and learned again from every message, the newest month included, folded, checked and written, the model kept as the language model’s file (private, never shared).

Structs

LearnFile 🔒

The file a model learns from, removed whatever happens.

Constants

FOLD_TOLERANCE 🔒

The largest difference allowed between the table’s probability and fastText’s.

HAM_LABEL 🔒

LABEL 🔒

fastText’s label prefix here: capitals, which no word of a message has (the tokenizer lowercases them), so that no sender’s text reads as a label.

SPAM_LABEL 🔒

Functions

calibrate 🔒

Platt’s sigmoid for a model of ids: in each account’s and each outside source’s mail of each label, the newest fifth of ids scored by a model of the rest; fitted on your own mail’s part (all of it when your own lacks a kind). With how many scores it was fitted on, and the AUC of all.

checked_fold 🔒

The largest difference between the table’s probability, Platt aside, and fastText’s own prediction, on messages ids (500 at most); over 1e-4, the fold is broken and nothing is exported.

decision 🔒

A text’s decision value, as predict reads it: its rows (words, their n-grams, word pairs), the end of the line’s appended, their scalars’ mean.

fold 🔒

The model folded into the classifier’s table (table::Kind::Rows): each known word’s rows’ sum (its own and its n-grams’, as fastText lists them), each bucket’s scalar, the end of the line’s as the bias.

header_weights 🔒

What the classifier learned of each header fact: its word’s rows’ sum in the table (toward spam above 0), by the fact’s name (“dkim_signed”, “links_0”…); a fact never seen often enough, none.

learn 🔒

fastText’s classifier learned from the messages ids.

row_scalars 🔒

Each input row’s scalar in a model of two labels: (w_spam − w_ham) · row.

text_of 🔒

A message as the classifier reads it: its words, then its header facts’ words; outside material, which has no headers, its words alone.

train 🔒

The training, from the messages read (train::train_with, which hands them here for this model): see the module.

write_learning 🔒

Writes the learning file of some messages, one line each, shuffled (the same seed, the same order).