Skip to main content

Crate sioul_learn

Crate sioul_learn 

Source
Expand description

Sioul’s spam filter, learned on the desktop from your own mail.

Nothing here runs on the phone: sioul-cli and sioul-app depend on this crate on desktops only. Nothing here shows a message to an AI agent: the numbers it gives are aggregates; sioul spam why is for you, in your terminal.

What every device runs (the tokenizer, the header features, the reduced table that scores both, the label log) is sioul-core’s sioul_core::spam. This crate trains on your mail, then writes that table. It reads mail through sioul-sync; the command line (sioul spam) and the window (“Train now”) run it. The design, its numbers and its reasons: docs/spam-filter.md, also on the website: https://aurelienpierre.github.io/sioul/dev/spam-filter.html.

Long jobs report where they stand through a callback (Progress) and stop when asked (Cancel), so the window can run them on a thread.

§Where to start reading

§Modules

  • corpus: what training reads, downloaded from every folder of every account and kept here, junk included after your provider purges it, with Sioul’s own checks of each message (its signatures, its sender) made on the whole message.
  • external: outside training material, mail labelled elsewhere, imported once (sioul spam import): more words to learn, and a baseline.
  • labels: spam or ham for each message, from its folder, its keywords, your own filter’s word and your own actions (every device’s logs).
  • auto: training again by itself, once a week, on the computer that made the table in use, when it is plugged in and idle: the rule, and what it knows.
  • train: training on demand: fastText on your mail, the message vectors, the SVM, its calibration, the evaluation, and the reduced table every device reads, replaced only when no worse.
  • svm: the linear SVM, solved as liblinear solves it.
  • platt: the SVM’s scores turned into probabilities (Platt scaling).
  • eval: the filter’s numbers, aggregates only, each with its interval.
  • detail: what a test said message by message: counts by account and folder, a grid of thresholds, the worst errors (sioul spam eval --errors).
  • diagnose: each header feature’s mean by account and label (sioul spam features): which ones tell accounts apart rather than spam from ham.
  • spamcore: the seam to sioul-core’s spam module: a corpus record read as the Porch reads the message.
  • synthetic, for tests only, this crate’s and the command line’s (so not shown here): invented mail on reserved domains.

Modules§

auto
Training again by itself (docs/spam-filter.md, “Training by itself”): on the computer that made the table in use, once a week at most after the last training (by hand or by itself), when it is on mains power, not saving power, idle for 15 minutes or locked, and no focus session runs; on a metered connection, without downloading first. A job run apart (sioul spam train --by-itself), at the lowest priority, on half the cores, with the usual rule for replacing the table; stopped through its stop file when the computer is unplugged or starts saving power, to try again later. Never a notification: the settings say when it last trained by itself, and whether its table replaced the one in use.
corpus
The training corpus: every message of every folder of every account, since forever, kept small enough for any disk.
detail
What a test said message by message (sioul spam eval --errors, sioul spam train --errors, and the MCP’s spam_eval and spam_train): the test messages counted by account and by folder, their label (spam or ham, as decided) against the filter’s class (probably spam, maybe spam, probably not spam); the numbers at a grid of thresholds, so that one can be chosen; and the worst errors: ham the filter called spam or maybe spam, the surest first, then spam it did not call spam, the least sure first.
diagnose
What the header features say of each account’s mail (sioul spam features): each feature’s mean over the corpus, by account and label, and how often it is known. A feature whose mean differs from one account to another more than from ham to spam tells the classifier where mail was collected rather than what it is: it learns each account’s share of spam instead of spam. Optionally, only the mail of one sender’s domain: what the features carry of a bank’s or a provider’s real messages. Numbers only, never a message.
eval
The filter’s numbers, aggregate only: how much ham it would call spam and how much spam it would catch at your two thresholds, each with its 95 % exact binomial interval (Clopper–Pearson), the share it calls unsure, and the area under the ROC curve. Nothing here names a message: these numbers may be read by an AI agent, so no subject, sender or word ever enters them.
external
Outside training material: mail labelled elsewhere (an older filter’s dump, another mail client’s archive, a public corpus), imported once to give the training more to learn from, and a baseline to measure it against (docs/spam-filter.md, “Outside material”). sioul spam import <file> reads it; sioul spam import --remove <source> takes it away.
labels
Spam or ham for each message of the corpus, from what is already there (docs/spam-filter.md, “The labels”): what you said, your mail apps’ keywords, the Junk folders, your own filter’s flags, the other folders.
platt
Platt scaling: an SVM’s decision values turned into probabilities, P(spam | f) = 1 / (1 + exp(A·f + B)), fitted on scores the SVM gave to messages it had not learned from (Platt 1999). Fitted by Newton’s method with a backtracking line search, as Lin, Lin & Weng (“A note on Platt’s probabilistic outputs for support vector machines”, Machine Learning 68, 2007) wrote it: their targets, their stopping rule, a stable log-loss.
spamcore 🔒
The seam to sioul-core’s spam module: the tokenizer, the header features and the reduced table, the same code the phone runs.
supervised 🔒
fastText’s classifier, the filter’s model (train::Model::Supervised, the default; docs/spam-filter.md, “The model, and how it folds”): each message one line, its words, then its header facts as words (features::header_words), two labels, softmax. Word pairs only when asked (word_ngrams above 1), in a trial: the table reads none.
svm
A linear SVM: L2-regularized, squared hinge loss, solved in the dual by coordinate descent with shrinking (Hsieh, Chang, Lin, Keerthi & Sundararajan, “A dual coordinate descent method for large-scale linear SVM”, ICML 2008: liblinear’s solver for -s 1).
train
Training on demand: “Train now” in the mail settings, and sioul spam train (docs/spam-filter.md, “Training on demand”).

Structs§

Cancel
Asks a long job to stop at its next step; shared between threads.
Dirs
Where the filter keeps its files. standard() is Sioul’s own folders; tests give their own.
Progress
Where a long job stands: done of total in this stage (total 0 when unknown), and what it is on (an account and a folder; never a message).

Enums§

LearnError
What went wrong, said plainly; never a message’s content.
Stage
The steps of a long job, in order.

Functions§

io_error 🔒
now 🔒
Now, in Unix seconds.