Module train 

Training on demand: “Train now” in the mail settings, and sioul spam train (docs/spam-filter.md, “Training on demand”).

The model learns in one of two ways (Model). The filter’s, by default, is fastText’s classifier (supervised, which says its own steps after the second): a message’s words and its header facts as words, two labels; it ranked a real mailbox’s mail better than the centroid did, and gave the same numbers from one training to the next. The centroid’s, on request, is the one below, the filter’s before features version 3:

1. The corpus is brought up to date (corpus::update, by run).
2. Each message gets one label (labels), and its words and header features from sioul-core’s spam module, read as the Porch reads mail.
3. fastText learns your mail’s words: skip-gram, 100 dimensions, character n-grams of 3 to 6, 200 000 buckets, words seen 5 times.
4. A message’s vector is the plain mean of its words’ vectors (fastText’s get_word_vector), never normalized, so that the model folds; the header features follow it; each dimension is standardized with the training messages’ mean and deviation.
5. Oldest 80 % to learn, newest 20 % to test, by INTERNALDATE. The SVM’s cost is chosen among 0.01, 0.1, 1 and 10 on the newest fifth of the learning part; ham weighs 5. Outside material (external), when imported, is split alike, source by source, by its own dates: its oldest 80 % joins the learning part at its dates, with its header features at the learning part’s mean (0 once standardized: only its words weigh); its newest 20 % is held out, the baseline.
6. Platt’s A and B are fitted on scores each message got from an SVM trained on older mail only (forward folds).
7. The numbers, on the newest 20 % of your own mail: aggregates only (eval); the same numbers on each outside source’s held-out part, apart: the baseline.
8. The model is folded into the reduced table (one scalar per word and per bucket, the header weights, the bias); the table must give the model’s score within 1e-4 on every test message, or nothing is exported. It replaces table.bin only if it loses no more ham on the newest 20 %, at your spam threshold, than the table in place, on those of them the table in place never learned from (it learned from all mail before its training: on that mail it would be judged on what it was taught); the one before is kept as table.prev.bin. trained.toml says what happened, for the settings.
9. Kept, the model is learned again, with the same cost, from every message, the newest month included (spam changes, and the filter that runs must have seen this month’s): standardized anew, calibrated by Platt on forward folds over all of it (each fifth scored by an SVM that learned the fifths before it), folded and tested the same way. That is the table written; the numbers said are those of steps 5 to 7.

A trial (Options::replace false: sioul spam train --no-replace, the MCP’s spam_train unless asked) stops after step 8’s comparison: it says what the table in place would become, and changes nothing of the filter, the table, the one before, the language model nor trained.toml. train_with also says what the test said message by message (detail).

Structs

Compared

The replacement’s comparison: the test messages the table in place never learned from, and the ham each table sets aside among them.

Counts

Evaluation

The table in place, tested on the newest 20 % of the corpus as it is now.

Message 🔒

One message on its way to the classifier.

ModelSummary

The model learned from the oldest 80 %, whose numbers test gives, and how it was asked to learn (Options: the command line’s overrides too).

Options

How training goes; Default holds the values of docs/spam-filter.md.

Outside

An outside source in a training: its oldest 80 % learned from, its newest 20 % held out, and the numbers there (the baseline).

Parts 🔒

The messages’ parts: every message in time order; the learning part (time order); your own mail’s test part; each outside source’s held-out part (newest_fifths).

Read 🔒

What the corpus and the outside material gave, read once, for the classifier’s own steps (supervised).

Refit

The table written: the model learned again, with the same cost, from every labelled message, the newest included.

Rows 🔒

The classifier’s rows: each message’s vector, then its header features.

Split

Summary

The last training, as trained.toml keeps it for the settings: numbers only.

TokenFile 🔒

The tokenized corpus, removed whatever happens.

Unseen

The numbers on the test messages the table in place never learned from.

Why

What weighed in one message’s verdict. It names words of the message: for you, in your terminal (sioul spam why), never through MCP.

Enums

Model

Which model learns: fastText’s classifier (the filter’s), or the language model’s centroid and the SVM (the filter’s before features version 3, on request).

Constants

LINE_WORDS 🔒

fastText reads lines of at most 1024 words: longer messages are cut in lines of this many.

Functions

against_current 🔒

The table in place, on the same test messages: those of them it never learned from (it learned from every message before its training), so that both are judged on mail they never saw; then whether the new table takes its place (replacement), and why. test_words and new_scored are in test_ids’ order, read with this training’s words (Options::lexicon). Each table reads with its own words: when those of the table in place (Table::lexicon: its own, or the built-in French and English ones when it keeps none) are not this training’s, its test messages are read again from the corpus with them, and one that can no longer be read is left out for both tables; a change of words never makes the table in place look worse than it is.

counted 🔒

What was learned from and tested on, in all and per account: your own mail’s.

evaluate

Tests the table in place on the newest fifth of the corpus as it is now, without training: aggregate numbers only (sioul spam eval). New mail since the training makes it a test of how the table holds up.

evaluate_with

evaluate, and what the test said message by message (detail), with up to errors errors of each kind (sioul spam eval --errors).

features_of 🔒

A card’s words and header features; own, Sioul’s checks kept apart from it (a corpus record’s), else those its headers carry (Sioul’s stamp first, your provider’s second: features::auth_results); its words read with lexicon.

fit_prefix 🔒

The SVM learned from the first end rows of x (time order), cost c.

fit_with_platt 🔒

The SVM learned from every row (time order), cost c, and Platt’s sigmoid from forward folds over them: each fifth scored by an SVM learned from the fifths before it. When the folds give none (too few of a kind early on), the model’s own scores, which make it surer than it is. step is told after each fit; the scores the sigmoid was fitted on are counted.

fold 🔒

The model folded into the reduced table (spam-core-api.md): with u = w/σ over the vector’s dimensions, each vocabulary word’s scalar is u · get_word_vector(word), each bucket’s u · its input row; the standardization’s means become text_mean = u · μ and the header means.

fold_error 🔒

The largest difference between a table’s score and its model’s on some messages (x, their standardized rows); over 1e-4, the fold is broken.

held_out 🔒

An outside source’s held-out part, scored: its counts and its numbers, the baseline.

hinge_loss 🔒

The mean weighted squared hinge loss of scores (lower is better).

is_zero 🔒

is_zero_f 🔒

kept 🔒

The words fastText can learn: it splits on whitespace, so a word never holds any.

lacking 🔒

What outside material, which has no headers, is scored with: every header feature missing, which weighs nothing in either kind of table (the classifier reads no word for it, the centroid stands it at its mean).

last

The last training’s summary; none before the first.

message_vectors 🔒

Every message’s vector: the plain mean of its words’ fastText vectors, computed on threads threads, in f64, from the tokenized corpus.

metrics 🔒

The test’s numbers the table carries, for the settings of every device.

newest_fifths 🔒

The oldest 80 % and the newest 20 % of each group of ids (in time order): an account’s ham, its spam; an outside source’s. Every group is on both sides, so that a group whose mail all came late (a provider that keeps spam a month) is learned from too, and none is told apart by when it came.

parts 🔒

Your own mail: each account’s ham, and its spam, oldest 80 % to learn, newest 20 % to test; each outside source alike, its ham and its spam, by their own dates, the newest 20 % held out (the baseline).

read_message 🔒

A message’s words and header features, from its corpus record, as the Porch reads the message itself: Sioul’s own checks of it first (made on the whole message when it was fetched, as the Porch reads the stamp the sync writes on it; beside a signature, their failures unknown when made long after it came: corpus::Checked::reading), your provider’s second. The words are read with lexicon (a training’s, or the table’s own).

refit_all 🔒

The model learned again, cost c, from every message (order, time order), the newest included: standardized anew, Platt on forward folds over all of it, folded; the fold tested on the test messages. The table to write (its metadata counts what it learned from; its metrics are the caller’s), and what to say of it.

replacement 🔒

Whether the new table replaces the one in place, and why, from the ham each sets aside among the compared test messages the one in place never learned from. None of them (nothing came since it was trained): nothing says it is better; the new one learned from the same mail, and replaces it.

run

“Train now”: the corpus brought up to date from every account (each password as the sync uses it), then train with your thresholds. For the window: run it on a thread of its own; progress says each step, cancel stops it at the next.

standardization 🔒

The mean and the deviation (population) of width columns over n rows, in two passes, rows made one at a time; 0 for a column that never changes. The columns from dim on (the header features) only over the rows that have them (known); without any, 0. A missing value (NaN) counts in neither.

tested_of 🔒

A test message as detail takes it: where it is, its label, how likely spam.

train

Trains on the corpus as it is (see the module’s steps). The table is replaced only when no worse; trained.toml says what happened.

train_with

train, and what its test said message by message (detail): the test messages counted by account and folder, the grid of thresholds, and up to errors errors of each kind, your own mail’s and each outside source’s, as the model learned from the oldest 80 % judged them.

trusted_ids

The authserv-ids each account trusts, as the Porch reads them (Sioul’s own first, then the provider’s: Config::mail_sources).

why

The table’s verdict on a whole message (a file’s bytes), and why. trusted are its account’s authserv-ids; internal_date when it reached the server (a Maildir file’s modification time).

words_of 🔒

The words of some messages, read back from the tokenized corpus.

words_of_record 🔒

A record’s words alone, read with lexicon as read_message reads them: what the table in place reads of a test message when its words are not the training’s (against_current).