Module corpus 

The training corpus: every message of every folder of every account, since forever, kept small enough for any disk.

Sioul’s sync already brings all your mail, but never into the disk’s reserve, and whole messages since forever rarely fit. Training needs much less: per message, the header block (at most 32 KB), its MIME structure from BODYSTRUCTURE (the parts’ types and names, attachments included), and the text the Porch reads of it: the first 6 000 characters of its text part, and its HTML (up to 64 KB) when it has no text part or only a stand-in (“view it in your browser”); which part is which, mail-parser says, as for the Porch (spamcore::bodies). A part is read whole up to 256 KB and cut here (so it decodes as the Porch decodes it), only its start beyond. With where it sits and its flags and keywords ($Junk, $NotJunk). Records are JSON lines, gzipped a batch at a time and appended to one file per account and folder, under $XDG_DATA_HOME/sioul/spam/corpus/<account>/.

Downloading uses Sioul’s own IMAP session (the sync’s TLS, login and Google sign-in) and changes nothing on the server: folders are opened with EXAMINE, messages read with BODY.PEEK. Trash, Drafts and Sent are left out. Gmail’s All Mail is read last, for what Gmail keeps only there (archived mail): your own messages (from one of your addresses, or labelled \Sent) and those kept already from another folder (by Message-ID) are left out of it. Another server’s “all mail” folder is left out: what it holds is its own (the trash, junk?). It is incremental: where each folder stopped is kept after every batch (state.toml), so it resumes there; a folder the server renumbered (a new UIDVALIDITY) is read again, its older records kept. A record is never deleted when the server deletes the message: that is how junk outlives the provider’s purge (Gmail keeps Junk a month).

It never fills the disk: it stops before the free space falls under 1 GB, and says so (status), for the window and the command line. A message that breaks the connection (one the IMAP parser cannot read) is taken alone on a new one and, failing twice, set aside (skipped), so that it never holds back its folder.

Each message is checked as Sioul checks the mail it stores (Checked, sioul_sync::verify: DKIM on the whole message, SPF from the server that handed it to your provider, DMARC, ARC, the reverse DNS), so that training reads the same checks as the sorting does: fetched whole (up to CHECK_WHOLE; a larger one by its header block alone), in memory only, checked, and dropped; only the checks’ results are kept, in the record. Records kept before, or while the DNS did not answer, are checked later (verify, and every download after its new mail): each message fetched once by its UID, the results appended beside the records (checks.jsonl.gz), so a run stopped goes on there; a message its server no longer has stays unchecked for good (Checked::gone). The DNS: one resolver per run, so each answer is asked once; a few checks a second, a few at a time, each within CHECK_LIMIT; when no DNS server answers (not even that a name made up under the mail server’s domain does not exist), the checks stop, and what is left waits, unknown, for the next run.

The corpus is private: never shared, never shown to an AI agent.

Structs

AccountState 🔒

AccountStatus

AllMail 🔒

In Gmail’s All Mail, what is left out: your own messages (from one of own, or labelled \Sent when labels), and copies of messages kept already from the account’s other folders (seen, by Message-ID).

CheckCounts

What Sioul’s checks say in the corpus, per account, as the training reads them (Checked::reading: old signed mail’s failures unknown): totals only, never a message.

CheckLine 🔒

A record’s checks made after it was kept (verify), by its place in its account: one line of the account’s checks.jsonl.gz.

Checked

Sioul’s own checks of a message (SPF, DKIM, DMARC, ARC, the reverse DNS of the server that handed it to your provider: sioul_sync::verify), made as the Maildir fetch makes them as mail arrives, on the whole message, when the corpus fetched it (or later: verify). The message itself is never kept: only what the checks found.

Checker 🔒

Sioul’s checks for a whole run: one resolver, so that each DNS answer is asked once in the run (its cache); at most CHECKS_PER_SECOND started each second, CHECKS_AT_ONCE at a time, each within CHECK_LIMIT. Offline (no DNS answer, not even that a name made up does not exist), it stops for the rest of the run.

FolderState 🔒

One folder’s download, by its server name.

FolderStatus

Head 🔒

What the first FETCH says of a message, before its text is asked for.

Held

Leaf

Place

A message’s place on its server, which names it for good (RFC 9051 §2.3.1.1).

Record

One message as training reads it.

Run 🔒

One folder’s checks under way: those kept, and those of the last messages that got no DNS answer at all, held until it is known whether the DNS is there (settle).

Status

The download’s state, as the window and the command line say it.

Tally

One check’s results over many messages, as the training reads them.

Text

A text kept, and the part it comes from.

Update

What one download did.

Verdicts

What the corpus’s headers say of your providers’ own spam filters: totals only, never a message.

Enums

Node

A message’s MIME structure, as BODYSTRUCTURE gives it.

Stop 🔒

Why an account’s download ended early.

Constants

BATCH 🔒

Messages per FETCH and per appended batch (a few in tests, to cross batches).

BATCH_BYTES 🔒

What one batch may take on disk, at most, before compression.

CHECKS_AT_ONCE 🔒

CHECKS_FILE 🔒

The account’s checks kept apart from its records (verify).

CHECKS_PER_SECOND 🔒

Checks started each second, at most, and at once.

CHECK_GROUP 🔒

Whole messages asked in one FETCH: at most this many bytes in all.

CHECK_LIMIT 🔒

One message’s checks, at most; each DNS question, asked once, at most DNS_TIMEOUT.

CHECK_WHOLE

A message up to this size is fetched whole for Sioul’s checks, in memory only; a larger one is checked by its header block alone (SPF, the reverse DNS; DMARC when SPF passes it), what needs the body left unsaid.

DNS_TIMEOUT 🔒

EXCERPT_CHARS 🔒

What the Porch reads of the text part (card.rs EXCERPT_LIMIT), and under how many visible characters it reads the HTML after it (card.rs STUB).

FETCH 🔒

HEADER_BYTES

The header block kept, at most.

HTML_BYTES

The HTML kept, decoded, when the Porch reads it: a mail’s first kilobytes are often only its styles, and the Porch reads the text after them.

KEEP_FREE

The download stops before the disk’s free space would fall under this.

MOST_PARTS 🔒

Parts listed per message, at most.

MOST_RECONNECTS 🔒

New connections an account’s download may make after a broken batch, in one run.

OFFLINE_STREAK 🔒

So many messages in a row without any DNS answer, and the network is asked whether it is there.

STUB_CHARS 🔒

TEXT_BYTES

The start of a larger text part read, in bytes as the server stores it: enough for the Porch’s 6 000 characters, in base64 or quoted-printable too.

WHOLE_PART 🔒

A text or HTML part up to this size is read whole, and cut here; a larger one, only its start (a partial FETCH, which some servers write wrongly).

Functions

account_host 🔒

The mail server’s name, under which the DNS is asked whether it answers at all.

answered 🔒

Whether a message’s checks got any answer from the DNS: not when SPF failed for want of one, nor DKIM when no server was there to check SPF on.

append 🔒

Appends one batch to a folder’s file (or the checks’ file) as a gzip member, after cutting off whatever a crash left past committed; the file’s new committed length.

check_kept 🔒

The messages of an account kept without Sioul’s checks (verify’s work, and every download’s after its new mail), on a session of their own: folder after folder, the newest first, each group’s checks appended to checks.jsonl.gz as they come, so a run stopped goes on there. Their folder gone or renumbered, they are gone: their UIDs name nothing any more.

check_records 🔒

New records checked as they come (Checked): each message fetched whole (or its header block, above CHECK_WHOLE) from the folder open in session, checked, the checks put in its record. Those left unchecked (the DNS silent, a group cut short) wait for check_kept; a connection lost is said, the checks made until then kept.

checks

What Sioul’s checks say in the corpus, per account (CheckCounts).

copies_of

Where a message is kept in the corpus, by its Message-ID (bare, as the label log writes it): each copy’s place and its folder’s role, account after account. For a label said of a message no longer stored here (sioul spam label mid:…): its place on the server is what the label log names it by.

cut 🔒

A text cut to at most bytes, on a character’s edge.

decode_text 🔒

The start of a text part, decoded as Sioul reads mail: its transfer encoding and charset undone by mail-parser, which forgives a part cut short. HTML stays HTML: what is visible is read from it at training.

decode_words 🔒

RFC 2047 encoded words, read as mail-parser reads a subject.

fetch_batch 🔒

One batch: its records, and the UIDs of All Mail’s messages left out (AllMail).

fetch_raw 🔒

Messages of the folder open in session, by UID: whole, or their header block (BODY.PEEK: nothing changes on the server), in memory only; a UID answered without them, None; one the server does not have, absent. A FETCH refused whole is asked again one message at a time; one refused alone counts as answered without it.

fetch_raw_once 🔒

fetch_records 🔒

One batch’s records, and the UIDs not kept: left out (All Mail), or given nothing for alone, so that they are not asked for again and again. A batch the server refuses whole is asked again one message at a time.

fetch_texts 🔒

Each part asked for: whole up to WHOLE_PART, else its first limit bytes; one FETCH per part number (“1”, “1.1”, “2”…) and way. The bytes as stored, with the leaf that says how they are encoded.

file_name 🔒

The file of a folder’s records, in its account’s directory: “INBOX.jsonl.gz”, “_Junk.jsonl.gz” (the folder’s Maildir++ name, made safe for any disk).

flag_name 🔒

“\Seen”, “$Junk”: a flag as the server writes it.

free_space

The free space on the disk holding a path; None when the system does not say.

groups 🔒

Messages to fetch together, from their (UID, size): the whole ones in groups of at most CHECK_GROUP bytes and BATCH messages, then the larger ones by their header blocks.

head_of 🔒

header_text 🔒

The header block as text: UTF-8 as it is; anything else byte for character (Latin-1), which gives the bytes back.

is_gmail 🔒

Whether the server is Gmail’s: it says X-GM-EXT-1 (its labels, its All Mail).

isolate 🔒

A broken batch’s messages, one at a time, each on a working connection: one that breaks it alone, twice, is set aside (skipped), so that it never holds back the rest of its folder. When none of them comes, the fault is the server’s or the line’s, not a message’s: nothing is set aside, and the error is said. The connections it made.

keep 🔒

One batch kept: its records appended to the folder’s file, their UIDs marked done; those not kept (the server could not give them, or All Mail leaves them out) marked skipped; where the folder stopped saved.

keep_checks 🔒

Checks made after their records were kept: appended to the account’s checks.jsonl.gz, then committed in its state.

kept

Whether a folder’s mail enters the corpus: not the trash, drafts or what you sent; Gmail’s All Mail (gmail: the server is Gmail’s, X-GM-EXT-1), for the mail Gmail keeps only there; never another server’s \All folder.

kept_ids 🔒

The Message-IDs of what an account’s folders but except hold in the corpus.

name_param 🔒

A name parameter (filename, name), decoded: RFC 2231’s key* (a charset and percent-encoding, possibly in numbered pieces), else the plain one (RFC 2047’s encoded words read as mail-parser reads a subject).

node_of 🔒

A message’s structure from BODYSTRUCTURE: its multiparts and their parts, each leaf’s type, names, charset and encoding; an attached message is one leaf. At most MOST_PARTS leaves (count counts them).

own_addresses 🔒

Your addresses, lowercase: every account’s address, and its login when that is one.

percent_decode 🔒

ranges 🔒

“1:5,7,9:12” for a set of UIDs; read back by read_ranges.

read_all

Every record of the corpus, account after account, folder after folder, each with Sioul’s checks of it, made as it was fetched or later (verify).

read_checks 🔒

An account’s checks kept apart from its records, by (folder, UIDVALIDITY, UID), up to committed bytes; the newest of a place wins.

read_file

Reads a folder’s records, up to committed bytes (all when None); a line that cannot be read is skipped, a member cut short ends the reading.

read_ranges 🔒

reading_order 🔒

The folders read, in the order read: the server’s, All Mail last, so that what the other folders hold is kept from them and left out of it.

run 🔒

update (download), or verify.

save_status 🔒

status

What the corpus holds and how its last download went.

status_path 🔒

to_come 🔒

About how many messages of an account’s kept folders are not recorded yet: each one’s MESSAGES (STATUS), less what is recorded under the same UIDVALIDITY.

update

Brings the corpus up to date: every account given, every folder kept (kept), what is new since the last time. password gives an account’s password as the sync uses it (sioul_sync::secret::password); room the free space on a disk (free_space; tests say less).

update_account 🔒

An account’s download (download), its new messages checked as they come, then its messages kept without checks checked (check_kept).

update_folders 🔒

verdicts

The providers’ verdicts in the corpus, per account.

verify

Sioul’s own checks of the messages kept without them (Checked), and nothing else: nothing new is downloaded (sioul spam fetch --verify). Each message is fetched once by its UID, whole (or its header block, above CHECK_WHOLE), in memory only; its checks are appended beside the records after each group, so a run stopped goes on there. A message its server no longer has (its folder gone or renumbered, its UID gone) is gone: unknown for good. Progress: Verify, account after account.

whole_base64 🔒

Base64 up to its last whole group of four characters.

write_whole 🔒

Writes a file beside its place, then moves it there.