Module external 

Outside training material: mail labelled elsewhere (an older filter’s dump, another mail client’s archive, a public corpus), imported once to give the training more to learn from, and a baseline to measure it against (docs/spam-filter.md, “Outside material”). sioul spam import <file> reads it; sioul spam import --remove <source> takes it away.

The format: JSON lines, one message each, a plain file or gzipped:

{"label":"spam","date":"2023-06-20T16:23:00+02:00","subject":"You won","text":"Claim your prize…","source":"old-filter"}
{"label":"ham","date":1687271000,"subject":"Lunch","text":"Tomorrow?","from":"Jane <jane@example.org>"}

• label: “spam” or “ham” (required);
• date: when it arrived, RFC 3339 or Unix seconds (required): the training’s split is by time;
• subject, text: its subject and its text part (required; one of them, or html, not empty);
• html: its HTML, read as text as the Porch reads it when there is no text, or only a stand-in (“view it in your browser”);
• from: its sender; headers: its raw header block, of which only the Message-ID is read now (a message also in your own mail is learned once, from yours); both are kept for a later version;
• source: a name for where it comes from; else the one given to the import, else the file’s name.

Kept apart from your own corpus, never shared; an AI agent sees of it only the sender and the subject, masked, of a test’s worst errors, when you let it use the spam filter’s tools ([mcp] spam, docs/mcp.md): $XDG_DATA_HOME/sioul/spam/external/<source>.jsonl.gz, each message checked and cut to what the Porch reads (6 000 characters of text, 64 KB of HTML, 32 KB of headers), with <source>.toml beside it (counts only). An import replaces what was imported before under the same name.

In training (train::train): its words join the language model’s corpus, and its messages the classifier’s learning set with their header features at the training’s mean (0 once standardized): they pull no header weight, only the words learn from them. Each source is split in time as your own mail is (its oldest 80 % learned from, its newest 20 % held out); the held-out part gives “the baseline”, the same numbers as for your own mail, said apart.

Structs

Counts

What a source holds: counts and dates only.

Imported

What one import did: counts only, never a message.

Line 🔒

One line as written by whoever made the file, before it is checked.

Message

One message as imported: checked, its date in Unix seconds, its texts cut to what the Porch reads. Its source is its file’s name.

Source

A source as sioul spam status and the settings say it.

Writing 🔒

A source being written: its file beside its place, its counts, what it holds already.

Constants

REFUSED

Why a line was left out: said by these words, counted.

SUBJECT_CHARS 🔒

The subject kept, in characters.

TEXT_CHARS 🔒

The text kept of a message, in characters: what the Porch reads.

UNNAMED 🔒

A source’s name when none is given anywhere.

Functions

check 🔒

A line checked: its source’s name and the message; or why it is left out.

counts_of 🔒

cut_bytes 🔒

The first bytes bytes of a text, cut on a character’s edge.

cut_chars 🔒

The first chars characters of a text.

date_of 🔒

A date as the format allows it: Unix seconds (a number, or digits), or RFC 3339.

file_of 🔒

import

Imports a file of outside material (JSON lines, gzipped or not): each source it names replaces what was kept before under that name. A line’s own source first, else source, else the file’s name.

import_from

Imports lines from reader (see import).

key_of 🔒

What tells one message from another within a source: its Message-ID, else its date, subject and text.

make_private 🔒

The folder made, yours alone (Unix).

private_file 🔒

A new file, yours alone (Unix).

read_all

Every message of every source, source after source (by name), in the order imported.

remove

Takes a source away: its file and its counts. False when there was none.

source_name

A source’s name, safe as a file’s: lowercase letters, digits and dashes.

sources

The sources kept, by name: what each holds, from the counts written at its import.