zig-maildir#

Maildirs for Zig 0.16: delivering messages into one, reading them back out, changing their flags, and the Maildir++ tree of folders and quota that the rest of the world built on top.

$ zig-maildir create Maildir
$ zig-maildir deliver Maildir message.eml
1757700000.M492817R3f0a1c2b4d5e6f70Q1.mail.example.com
$ zig-maildir list Maildir
new	INBOX		4211	1757700000.M492817R3f0a1c2b4d5e6f70Q1.mail.example.com
$ zig-maildir flag Maildir 1757700000.M492817R3f0a1c2b4d5e6f70Q1.mail.example.com +S +R
RS

Three things, and you can stop at any of them:

The message content is zig-mime's: this library files messages and does not read them, and Message.parse hands the bytes straight over. The API documentation is generated from the doc comments, which carry most of the explanation of why a maildir is the shape it is.

Where this lives#

The repository lives in three places that carry the same history. The Forgejo instance at https://git.jcollie.dev/jeff/zig-maildir is the web-visible one, and is where the continuous integration and the published documentation are:

$ git clone https://git.jcollie.dev/jeff/zig-maildir.git

It is mirrored on Tangled at https://tangled.org/jcollie.dev/zig-maildir, a forge built on the AT Protocol, where a repository is addressed by its owner's identity rather than by a server name.

It is also on Radicle, a peer-to-peer forge that needs no account on anything. A Radicle repository is findable only by its repository ID, so this is that ID:

rad:z3zvKaC531F7uLpN5YdTXHXYTY7Mk

and rad clone rad:z3zvKaC531F7uLpN5YdTXHXYTY7Mk fetches it from any node that seeds it. rad clone finds seeds through your local node's routing table, so the node has to be running first, and cloning seeds the repository in turn, which helps keep it available:

$ rad node start
$ rad clone rad:z3zvKaC531F7uLpN5YdTXHXYTY7Mk

If you already have the repository and only want to help host it:

$ rad seed rad:z3zvKaC531F7uLpN5YdTXHXYTY7Mk

Any of the three is the whole project.

Adding it to a project#

$ zig fetch --save git+https://git.jcollie.dev/jeff/zig-maildir.git

const maildir = b.dependency("zig_maildir", .{ .target = target }).module("maildir");
exe.root_module.addImport("maildir", maildir);

Why a maildir is shaped like this#

A maildir is a directory holding three others — tmp, new and cur — and one idea: a message is a file whose name carries everything mutable about it. Everything else follows from that.

• Delivery takes no lock. A message is written into tmp under a name nobody else will invent, and then renamed into new. A reader never sees a partial message, because a message only appears in new once it is whole and rename within a filesystem is atomic. Two mail servers, an IMAP daemon and a procmail can deliver at the same moment, over NFS, with nothing arbitrating between them.
• Reading is a directory listing. There is no index to corrupt, no lock to hold while a slow client reads its mail, and no way for one crashed process to leave the mailbox unusable.
• Changing a flag is a rename. Nothing is rewritten, so a message file is written exactly once and never modified — which is what lets the size recorded in its name be trusted, and what makes a maildir safe to back up while it is in use.

The cost is that tmp accumulates the wreckage of interrupted deliveries, which is why cleanTemp exists and why the specification says to delete anything in there older than thirty-six hours.

One maildir#

const maildir = @import("maildir");

var mailbox: maildir.Maildir = try .create(.cwd(), io, "Maildir", .{});
defer mailbox.close(io);

// Delivery. The message lands in `new` with no flags, which is what "new"
// means, and its name records how big it is.
const delivered = try mailbox.deliver(io, bytes, .{});

// Reading. An iterator yields a `Message`, which is a name and nothing else
// until you ask it for something.
var it = mailbox.iterate(.new);
while (try it.next(io)) |message| {
    var m = message;
    std.debug.print("{s} {d} bytes\n", .{ m.id(), try m.size(&mailbox, io) });
    try m.setFlags(&mailbox, io, .{ .seen = true });
}

setFlags moves the message from new into cur, and that is not a convenience: a name in new has no info field, so there is nowhere in new for a flag to be written. "Read but still new" is not a thing a maildir can express.

Delivery comes in two shapes. deliver takes a slice; beginDelivery gives back a writer, which is what to use when the message is arriving from a socket or being written by something else:

var buffer: [4096]u8 = undefined;
var delivery = try mailbox.beginDelivery(io, 10);
errdefer delivery.abort(io);
try built_message.write(delivery.writer(io, &buffer));   // zig-mime writes
const delivered = try delivery.commit(io, .{});

Either way the delivery is all or nothing: if anything fails before the rename, the partial file in tmp is removed and nothing ever appears in new.

deliver also takes a destination, because not every message is new. An IMAP APPEND of an already-read message, or a client saving a draft, goes straight into cur with its flags on:

_ = try mailbox.deliver(io, bytes, .{ .to = .{ .cur = .{ .seen = true, .draft = true } } });

Names and flags#

const name: maildir.Name = .parse("1757700000.M1R2Q3.host,S=4211:2,RS", ':');
name.base();          // "1757700000.M1R2Q3.host" -- the identity
name.size();          // 4211, without touching the filesystem
name.flags().seen;    // true

Three things about the name are worth knowing before writing anything that touches one.

The identity is base, and it must never change. Two programs sharing a maildir agree about which message is which by that string and nothing else, so changing it loses every IMAP UID, every read/unread record and every synchronisation state keyed to it. setFlags and addFlags rewrite the info field and leave it alone; moveTo deliberately does not, and the section below says why.

Flags are a set of letters written in ASCII order, and six of them are defined: D draft, F flagged, P passed, R replied, S seen, T trashed. Those get named fields. Every other letter is kept in other —

var flags = try maildir.Flags.parse("Sb");   // Dovecot wrote this
flags.replied = true;
// "RSb": the keyword `b` is still there.

— and keeping it is the whole point. Dovecot stores IMAP keywords, which are arbitrary user-applied labels, as the letters a to z with a dovecot-keywords file mapping them to names. A library that read :2,Sb, marked the message replied and wrote back :2,RS would have silently deleted a label the user applied. Reading flags, changing one and writing them back is the most common thing anybody does to a maildir, and it has to be lossless.

The separator is a parameter. A colon is what the specification says and what every Unix mail program expects, and it is also illegal in a FAT, exFAT or NTFS filename — so a maildir on a memory stick or a Windows share is written with ! or ; instead, and a reader that insists on a colon sees every message in it as new and flagless.

var mailbox: maildir.Maildir = try .open(.cwd(), io, "Maildir", .{ .separator = '!' });

Maildir++ folders#

A maildir has no room for a second mailbox in it, so Maildir++ puts the folders beside the messages, as hidden directories in the top-level maildir, each a complete maildir of its own:

Maildir/            the top-level maildir, which IMAP calls INBOX
  tmp/ new/ cur/    its own messages
  maildirsize       the quota ledger, covering everything below
  .Work/            the folder "Work"
    tmp/ new/ cur/
    maildirfolder   the marker that says this is a folder
  .Work.Reports/    the folder "Work/Reports"

The hierarchy is flat on disk and nested in the name: .Work.Reports is a sibling of .Work, not a child, which is what lets a whole folder tree be listed with one readdir.

var store: maildir.Store = try .create(.cwd(), io, "Maildir", .{});
defer store.close(io);

var inbox = try store.inbox(io);
defer inbox.close(io);

var reports = try store.createFolder(io, &.{ "Work", "Reports" });   // and ".Work"
defer reports.close(io);

var names = try store.folders(gpa, io);   // ".Work", ".Work.Reports", sorted
defer names.deinit(gpa);

A folder name cannot contain a dot. The dot is the hierarchy delimiter and Maildir++ never defined an escape for one, so a mailbox called example.com is either the folder com inside the folder example or it is not representable. This library returns error.InvalidComponent rather than silently creating the wrong thing.

Moving a message between folders changes its name, and has to:

try message.moveTo(&inbox, &archive, io);

Two maildirs are two directories and nothing coordinates the names in them, so a message carrying its name into a folder that already had one like it would overwrite a message — and rename would do it silently. The flags travel; the identity does not. This is what IMAP's MOVE does, and it is why an IMAP server cannot promise a moved message keeps its UID.

A keyword, though, travels only half way, and the NixOS test pins it down. The letter survives the move because it is part of the flags; what the letter means does not, because that is recorded in a dovecot-keywords file inside each mailbox, so a message labelled "Important" in the inbox arrives in the archive carrying a letter that mailbox has never assigned. Dovecot's own MOVE updates the destination's mapping. Nothing outside Dovecot can, because the mapping is Dovecot's rather than the maildir's — the six standard flags are the only ones that mean the same thing in every directory.

Quota#

Totalling a mail store means listing every folder and adding up every message, which is fine once and ruinous on every delivery. Courier's answer is maildirsize, a small file in the top-level maildir written like a ledger rather than a balance:

10485760S,1000C
4211 1
8320 1
-4211 -1

The first line is the quota and every line after it is a change, appended by whoever made it. The usage is their sum.

try store.setQuota(io, .{ .bytes = 10 * 1024 * 1024, .messages = 1000 }, .zero);
try store.recordUsage(io, .{ .bytes = @intCast(bytes.len), .messages = 1 });

const state = (try store.quotaState(gpa, io)).?;
if (state.exceeded()) return error.OverQuota;
if (state.isStale(io, .{})) _ = try store.recalculateQuota(gpa, io);

Treat a number from there as "what the store believed last time somebody checked". The ledger is advisory and self-healing rather than authoritative: it drifts, because a message deleted by something that does not know about the file is never subtracted, and it is meant to — isStale eventually says so and recalculateQuota walks the store and writes a fresh total. Note that isStale only applies its age test when the store is over quota, which is deliberate: being wrongly under quota costs a little unfairness, while being wrongly over it bounces mail, so the expensive check is spent only on the answer that would refuse a delivery.

recalculateQuota counts a message from the ,S= in its name where there is one, so keeping record_size on at delivery is what makes a recalculation a directory walk rather than a stat of every message in the store.

Reading the message#

var parsed = try message.parse(&mailbox, gpa, io, .unlimited, .{});
defer parsed.deinit();

std.debug.print("{s}\n", .{(try parsed.root.subject()) orelse "(no subject)"});

That is zig-mime from there on: RFC 5322 and the MIME documents, addresses, dates, encoded words, the whole tree of parts. It parses from a slice, so the message is read into memory first and the mime.Message owns that copy.

The division of labour is the point. A maildir is a naming convention over a directory and has no opinion about what is in the files; a message parser has no opinion about where the message came from. Keeping them apart is what lets this library deliver a message it cannot parse, which is exactly what a mail store must do — a message that has already arrived has to be filed whatever is in its headers.

The command-line tool#

$ zig-maildir create   Maildir              # a maildir, or a whole store
$ zig-maildir deliver  Maildir message.eml  # `-` reads standard input
$ zig-maildir list     Maildir              # every message, with its flags
$ zig-maildir headers  Maildir <id>         # decoded, through zig-mime
$ zig-maildir flag     Maildir <id> +S -F
$ zig-maildir move     Maildir <id> Archive
$ zig-maildir folders  Maildir
$ zig-maildir mkfolder Maildir Work/Reports
$ zig-maildir quota    Maildir 10485760 1000
$ zig-maildir recalc   Maildir
$ zig-maildir clean    Maildir              # old wreckage out of tmp

It exists to show what the library looks like from outside, and to give the NixOS test a second program to point at a maildir Dovecot is also looking at.

What is deliberately not here#

• No message parsing. That is zig-mime, and Message.parse is the whole of the connection between them.
• No IMAP, POP or SMTP. This library is about the store, not about serving it or filling it. It does provide the pieces an IMAP server needs — stable identities, keyword-preserving flags, Maildir++ folders, quota — but the protocol is somebody else's.
• No subscriptions. Which folders a client has subscribed to is IMAP's business and every server keeps it differently: Courier in courierimapsubscribed, Dovecot in subscriptions. A file with either name is left alone rather than guessed at.
• No index, and no caching of one. Every listing is a readdir. That is what a maildir is, and a program that needs an index over one should keep it itself, keyed by Message.id.
• No locking, anywhere except the quota ledger. The design does not need any, which is its whole appeal. quota.record takes an advisory lock because Zig 0.16 cannot open a file O_APPEND and the position therefore has to be read before it is written; that is the one race the original design closed with a flag this library cannot ask for.

Interoperability#

nix flake check boots a NixOS guest with a real Dovecot serving a maildir this library wrote, and drives both. It asserts what would otherwise be a comfortable assumption:

• Dovecot finds and indexes a message zig-maildir delivered, its RFC822.SIZE is the number in the name, and it keeps that ,S= field when it renames the file itself.
• Dovecot leaves a flagged message in new, renaming it in place to something like new/…,S=121:2,S, which the specification says should only ever happen in cur. A reader that took the rule literally would show that message as unread forever, which is why Name.parse cannot fail.
• A flag set by either is seen by the other, in both directions.
• An IMAP keyword Dovecot applied survives this library changing a flag, which is the one that would silently lose a user's labels.
• A folder made by either is listed by the other, and is flat on disk with a maildirfolder in it.
• A message moved between folders keeps its standard flags and gets a new name — and its keyword arrives as a letter the destination mailbox cannot name, which is a limit of the format rather than of this library.

It needs KVM, so it runs on the ephemeral runner tiers rather than the always-on one.

Fuzzing#

tests/fuzz.zig holds four targets, each a property rather than an example: whatever arrives, the parser terminates, stays inside its buffers, and — if it claims to have understood the input — writing it back out and reading it again gives the same answer.

$ zig build test                    # the properties, over the checked-in seeds
$ zig build fuzz-run                # a minute of each, with generated input
$ zig build fuzz-run -- --seconds 300 --target name
$ zig build fuzz-run -- --input fuzz-findings/x.bin --target name

Stability is the property that earns its keep, and it is a stronger claim than it sounds. A maildir library rewrites names constantly, since every flag change is a parse, a change and a write. If that round trip is not a fixed point — if writing a parsed name can produce something that parses differently — then a message's name drifts a little every time anybody touches it, and since the name is the message's identity, the message eventually becomes a different message. So the name target asks for the second write to equal the first rather than for the output to equal the input: a name may legitimately be repaired on the way in (unsorted flags get sorted, a duplicate letter is dropped), but repairing it twice must change nothing, and the repair must never touch base.

tools/fuzz.zig is that loop and says at the top why it exists: Zig 0.16.0 cannot build a test executable in fuzz mode without a patched standard library — flake.nix carries the patch — and leaves the fuzzer's coverage table empty even then. A fuzzer with no coverage is a random number generator, so this is one written down honestly, with a corpus of real names to mutate instead.

The API documentation#

https://jeff.jcollie.page/zig-maildir/, rebuilt from main on every push. Zig writes it out of the doc comments, which in this project carry most of the explanation — why delivery needs no lock, why a flag change is a rename, what ,S= is for and why it may be trusted.

$ zig build docs-serve            # then open http://127.0.0.1:8000/
$ zig build docs                  # or just build it, into zig-out/docs

It has to be served rather than opened. What Zig emits is not a page but a program: a WebAssembly viewer that fetches the source of everything it shows out of a tar file beside it, and a browser refuses to fetch anything from a file:// page. That is the same reason zig std runs a server rather than opening a file. tools/docs_server.zig is that server — one directory, one person, the loopback interface, and nothing else.

Licence#

MIT. See LICENSES/MIT.txt.