Module tokenize 

The spam filter’s tokenizer, version 2: what a message says, as the words the language model learns on the desktop and the table scores on every device.

A port of Virtual Secretary’s text pipeline (its src/core/nlp.py: Tokenizer.prefilter, normalize_token, lemmatize; patterns.py; language.py; utils.py: typography_undo), rewritten for the regex crate, which has no look-around: a look-behind or a look-ahead of one character becomes a character matched before or after the part replaced, and the search goes on from the end of that part, so that the character after one match can stand before the next (Pass::apply). Training and scoring share this one function: any change to a step changes the words, so it raises TOKENIZER, and a table made by another version is refused.

The steps, in order:

1. The text: the subject, a blank line, the body (HTML mail as its visible text). A mark your provider put at the start of the subject (***Potentiel-SPAM***, [SPAM], {Spam?}) is taken off first (without_provider_tags): it is the provider’s verdict, not the sender’s words, and a model that read it would learn to copy the provider (version 2: a provider may tag nearly every message it sets aside).
2. Normalization (normalize): invisible characters out (zero-width spaces and joiners would otherwise split or hide words); NFKC; letters borrowed from other scripts folded into the Latin ones they mimic, in words that mix them (“Vіаgrа” with Cyrillic і and а); lowercase (after NFKC, so that its capitals, “™” → “TM”, “𝐕” → “V”, come out lowercase); Virtual Secretary’s table (apostrophes, spaces, dashes, quotes, ligatures, fractions, arrows; accents stripped, every accent, not only French ones); the symbols its patterns read written in ASCII (€ → “eur”, £ → “gbp”, ° → “deg”, ⌘ → “cmd”); everything else not ASCII dropped (Virtual Secretary’s choice); whitespace collapsed.
3. Cleanup (clean): runs of dots, dashes and question marks made one, a character repeated ten times or more, BB code and base64 blocks removed, markup brackets dropped (their text kept).
4. Placeholders (placeholders): Virtual Secretary’s 36 meta-tokens, in its order, with _IBAN_ and _PHONE_ after the addresses; each link also gives its domain (Tokens::links).
5. Splitting (split): words on whitespace and punctuation, a dot or a comma kept between letters or digits (“2.5”, “e.g”), an apostrophe between letters (“don’t”, “aujourd’hui”), French elisions split off and dropped (“l’information” → “information”).
6. Per word (word): placeholders kept; number words → _NUMBER_; five digits or more, or a hexadecimal hash → _HASH_; numbers → _NUMBER_; single characters and Virtual Secretary’s stop words out; its French and English suffix stemmer.

The words these steps look for (a provider’s marks, month and key names, elisions, number words, stop words) are a Lexicon: the spam lists of the language packs in use, and your changes (words::SpamWords). A table keeps the lexicon it was trained with (table::Meta::lexicon) and every device reads each message with it (tokens_with), so a change of words counts from the next training, on every device at once; a table made before 8 October 2026 keeps none and reads with the packs built in, French and English, which are the words that made it. A change of words does not raise TOKENIZER; a change of a step still does. The modifier keys of a shortcut, the ordinal suffixes, the units and the stemmer stay in code.

Where this port departs from Virtual Secretary on purpose, the line says so (“Virtual Secretary …”). Tests: tests below, on Virtual Secretary’s own inputs (src/tests/test-patterns.py).

Structs

Lexicon

The words the tokenizer looks for (words::SpamWords) and the patterns made from them (see the module).

Pass 🔒

One substitution: the part named m is replaced, what is matched around it stays, and the search goes on from the end of m.

Tokens

What the tokenizer read in a message.

Enums

Kind 🔒

Constants

AFTER 🔒

What may stand after it, end of markup (regex_stopper).

AMOUNT 🔒

A number with its separators, as Virtual Secretary’s units read it.

BEFORE 🔒

What may stand before a placeholder: the start, a space, an opening bracket or quote (Virtual Secretary’s regex_starter).

END 🔒

What may stand after it, end of a word (end_of_word).

FILE_END 🔒

After a file name: the end, a space, or one character then the end or a space (Virtual Secretary’s (?![\.\S]\S)).

NAME 🔒

What a file name is made of, and what stands around one.

NOT_NAME 🔒

PLACEHOLDERS

Every placeholder a word can be.

PREFIXES 🔒

SIGN 🔒

A sign before a number (regex_algebra; its ≠ and ± are gone with what is not ASCII).

SIGN_ONLY 🔒

SIGN_SPACE 🔒

A sign, then maybe a space, before a number ((regex_algebra)? ?); a sign alone.

TOKENIZER

The tokenizer’s version, stamped in every table (table.rs).

Statics

BASE_64 🔒

BB_IMG 🔒

BB_QUOTE 🔒

DASHES_RUN 🔒

DOTS 🔒

MARKUP 🔒

NUMBER 🔒

QUESTIONS 🔒

Functions

alternatives 🔒

An alternation of words, each escaped, the longest first; one that never matches when there is none (an empty alternation would match anywhere).

ascii 🔒

One character in ASCII, as Virtual Secretary’s table writes it (utils.py, _unicode_to_ascii), else its letter without its accent; nothing when it has no ASCII form.

clean 🔒

Runs of dots, dashes and question marks made one; a character repeated ten times or more, BB code and base64 blocks removed; markup brackets dropped, their text kept.

elided 🔒

A word without its elided articles and pronouns, before a vowel or a mute h (the lexicon’s elisions): “l’information” → “information”, “qu’aujourd’hui” → “aujourd’hui”.

file 🔒

fold_lookalikes 🔒

Lookalikes folded where they hide a Latin word: in a word that holds a Latin letter (“Pаypal”), or made of nothing but lookalikes (“РАУ”). A word of another script stays itself, to be dropped with the rest of it (step 2).

hash 🔒

Virtual Secretary’s HASH_PATTERN_FAST, ^[0-9a-f]{6,}$, took words made of the letters a to f (“decade”, “facade”) for hashes: a digit is asked here.

iban_valid 🔒

Whether an IBAN’s check digits are right (ISO 13616: the first four characters moved to the end, letters as 10 to 35, the number modulo 97 is 1).

invisible 🔒

Characters that show nothing: dropped before anything reads words, so that one put inside a word (“V\u{200b}iagra”) neither splits nor hides it. Sioul’s own (text::readable), with the marks that turn text around and the tag and variation characters.

longest_first 🔒

Alternatives longest first: Virtual Secretary’s “tif|tiff” took “x.tiff” for “x.tif” and an “f”.

lookalike 🔒

The Latin letter a Cyrillic, Greek or Armenian one mimics, case kept (the families of lookalike.rs, by the letter they pass for).

normalize

A text as the placeholders read it: lowercase ASCII, one space between words.

normalized 🔒

A list’s words as the steps write the text (normalize: lowercase, no accents, ASCII), without empty ones.

passes 🔒

The placeholders’ passes, in Virtual Secretary’s order; months and keys, a lexicon’s month and key names as alternations.

placeholders 🔒

The placeholders, in Virtual Secretary’s order; then its last cleanups: a colon is a space (C++ members), a backslash nothing (LaTeX).

provider_tag 🔒

A provider’s mark, at the start of a subject: one of its words (marks, an alternation: spam, junk, pourriel, indésirable) between stars, brackets, braces or parentheses, with what goes with it (“Potentiel-SPAM”, “[SPAM?]”, “{Spam}”, “(Indésirable)”), or “SPAM:”.

registrable

The domain a host belongs to, as registered: its last two labels, or three under a country’s own second level (“co.uk”, “gouv.fr”, “com.au”). An approximation of the Public Suffix List, which Sioul does not carry.

repeated 🔒

A character repeated ten times or more (“==========”, “xxxxxxxxxx”) made a space: Virtual Secretary’s (.)\1{9,}, whose back-reference the regex crate lacks.

single_consonants 🔒

Double consonants made single after the first two letters: commission → comision.

split 🔒

The words of the text: runs of letters, digits and underscores; a dot or a comma between two of them kept (“2.5”, “1,000”, “e.g”), an apostrophe between two letters (“don’t”, “aujourd’hui”); French elisions split off and dropped (“l’information”, “qu’il”: Lexicon::elisions).

stem

Virtual Secretary’s French and English suffix stemmer (lemmatize), on each run of letters and digits of a word, the dots and apostrophes between them kept: its patterns ended at those as at the word’s end. Its first rule, the elisions, is the splitting’s (step 5).

stem_run 🔒

tokens

The words of a message, and the domains its links point to, read with the packs built in (Lexicon::builtin).

tokens_with

The words of a message, and the domains its links point to, read with lexicon (a table’s own: table::Table::lexicon).

unaccented 🔒

A Latin letter without its accents (“é” → ‘e’, “ñ” → ‘n’, “ä” → ‘a’): the letter its canonical decomposition starts with, when the rest are accents. Virtual Secretary stripped the French accents only and dropped the other letters (“señor” → “seor”).

unit 🔒

A number then a unit, as Virtual Secretary’s physical quantities: head before the number (a sign, a space), unit after it.

without_provider_tags

The subject without the marks a provider put at its start, read with the packs built in (Lexicon::without_provider_tags).

word 🔒

One word as the model learns it, or none (normalize_token).

word_byte 🔒