api/Mail/Sieve: tokenise filter rule values to align with EGroupware …

[CActor]

Summary

Extends the IMAP-search tokenisation introduced in #240 to the Mail filter rules (Sieve scripts generated by EGroupware), gated behind a per-rule opt-in checkbox. Existing rules are unaffected at deploy time — zero regression.

Depends on / pairs with #240.

Discussion: https://help.egroupware.org/t/79137

Why a second patch

#240 makes the Mail search box accept the EGroupware-standard +token, -token, and, or, "..." syntax (same as Addressbook / Calendar / InfoLog). But the same user-facing limitation also applies to Mail filter rules: a filter "Subject contains invoice overdue" still produces a Sieve header :contains "subject" "invoice overdue" test, which only matches the literal contiguous substring.

From the end-user point of view it is unintuitive that the search box and the filter rules of the same module follow two different syntaxes. This PR closes the loop so users get one mental model across the whole Mail module.

Design — opt-in per rule

Following the request in https://help.egroupware.org/t/79137 to "not change existing rules silently", the new behaviour is fully gated.

New per-rule checkbox in mail/templates/default/sieve.edit.xet:

☐ Use tokenised search syntax (+word required, -word forbidden, "..." literal phrase; same as the EGroupware search box)

Persistence: stored as the next free bit 256 in the existing flg integer column of the rule — no schema migration, no new column, no DB touch.

Generator gating in api/src/Mail/Sieve/Script.php: the tokenised branch only fires when ($rule['flg'] & 256) AND !$rule['regexp'] AND the value has no */? wildcards. All other modes (regex, wildcards, plain unchecked) are emitted exactly as before.

Small UX touch in mail/inc/class.mail_sieve.inc.php: the last state of the checkbox is remembered as a per-user preference (mail/sieve_last_tokenized) and used as the default for the next new rule that user creates. Existing rules are unaffected by the preference — only the default for new ones.

User-facing syntax (only when the checkbox is ticked)

Input in any filter value field	Meaning
invoice	substring "invoice" anywhere
invoice overdue	"invoice" OR "overdue" (default for whitespace)
invoice and overdue	"invoice" AND "overdue"
invoice +overdue	"invoice" AND "overdue"
invoice -spam	"invoice" AND NOT "spam"
"invoice overdue"	literal phrase as one token (= legacy behaviour)

Affected condition rows (in plain :contains mode only): From, To, Subject, custom header, body (:text / :raw). Numeric comparators, size check, attachment-type filter are untouched.

Generated Sieve

For a rule with the checkbox ticked, Subject contains invoice +overdue:

if allof (
    allof (
        header :contains "subject" "invoice",
        header :contains "subject" "overdue"
    )
) {
    fileinto "Reminders";
}

For an unticked rule (default), output is byte-identical to the unpatched generator — no regression risk for existing filters.

Files changed (3)

1. api/src/Mail/Sieve/Script.php — adds two static helpers (buildTokenizedSieveTest, parseSieveTokens), the $tokenizedbit = 256 constant, the tokenised branch in 5 case blocks (FROM / TO / SUBJECT / custom header / body).
2. mail/templates/default/sieve.edit.xet — adds the new <et2-checkbox id="tokenized"> after the existing regexp checkbox.
3. mail/inc/class.mail_sieve.inc.php — load/save of flg & 256, plus the per-user sieve_last_tokenized preference.

~204 lines effective delta, no new dependencies, no schema migration.

Backward compatibility — zero deploy-time regression

Because the patch is gated on flg & 256, no existing rule changes behaviour at deploy time. Rules saved before the patch have flg & 256 == 0, so the generator emits the historical contiguous :contains Sieve, byte-identical to pre-patch. The editor renders existing rules with the new checkbox unchecked, which is the correct default for legacy rules.

The first time a user explicitly opens an existing rule, ticks the checkbox, and saves, that single rule is regenerated under the patched generator with flg += 256. From that moment on the rule uses tokenised matching. No data migration, no admin action required.

For instances that prefer to bulk-normalise the stored Sieve scripts at deploy time anyway (e.g. to validate the patched parser end-to-end across all users), an optional CLI helper resave-sieve-rules.php is provided in the companion gist below. It iterates all active users and round-trips their rules through retrieveRules() → setRules(). Pure bookkeeping, no semantic effect.

Test plan

• Manual UI testing on EGroupware 26.1 (Docker image), Dovecot with fts_flatcurve enabled. Patched files bind-mounted source-side to survive Watchtower image updates.
• 34 functional test cases via SMTP from a separate Gmail account:
  • 8 tokenised positives (checkbox ticked: AND / OR / +token / -token / quoted phrase / case-insensitive / cross-position / substring-of-larger-word) — all matched as expected
  • 4 tokenised negatives (missing required token, forbidden token present, spaced-out non-substring) — correctly stayed in INBOX
  • 22 legacy rules (checkbox unticked) — routed identically to pre-patch behaviour for every message, zero regression
• Re-save migration script (resave-sieve-rules.php) tested live on 48 existing user rules; produces no diff vs. baseline for legacy-mode rules.
• Unit tests for buildTokenizedSieveTest() and parseSieveTokens() — happy to add as part of review.

Code-sharing note (for review-time discussion)

The tokeniser in this PR (buildTokenizedSieveTest / parseSieveTokens) is structurally identical to the one in #240 (buildTokenizedSearch / parseSearchTokens). I deliberately left both as their own static methods on each class to keep this PR minimal-impact. If you prefer a shared EGroupware\Api\Mail\SearchTokeniserTrait factored out before either PR is merged, I am happy to do the refactor on the search PR first, then this one will reuse the trait. Just let me know on either thread.

Related

• Companion PR (prerequisite, IMAP search side): api/Mail: tokenise search input to align with EGroupware syntax #240
• Forum discussion: https://help.egroupware.org/t/79137
• Code gist with full diff + migration helper: https://gist.github.com/CActor/82621b8df78662e6117cb3beeedb94e0

This PR is marked as Draft because #240 is its functional prerequisite (both should be reviewed together; this one converted to ready-for-review once #240 is mergeable).

[ralfbecker]

Thx for your pull request :)

Ralf