# zig-maildir Maildirs for Zig 0.16: delivering messages into one, reading them back out, changing their flags, and the Maildir++ tree of folders and quota that the rest of the world built on top. ```console $ zig-maildir create Maildir $ zig-maildir deliver Maildir message.eml 1757700000.M492817R3f0a1c2b4d5e6f70Q1.mail.example.com $ zig-maildir list Maildir new INBOX 4211 1757700000.M492817R3f0a1c2b4d5e6f70Q1.mail.example.com $ zig-maildir flag Maildir 1757700000.M492817R3f0a1c2b4d5e6f70Q1.mail.example.com +S +R RS ``` Three things, and you can stop at any of them: | | | | --- | --- | | `Name` and `Flags` | [the naming convention](#names-and-flags) — pure functions over bytes, no I/O and no allocation | | `Maildir` | [one mailbox](#one-maildir): delivery, iteration, flags, and the `tmp`/`new`/`cur` dance that makes it safe | | `Store` | [a Maildir++ tree](#maildir-folders): folders, and [the quota ledger](#quota) covering all of them | The message content is [zig-mime][zig-mime]'s: this library files messages and does not read them, and `Message.parse` hands the bytes straight over. The [API documentation](https://jeff.jcollie.page/zig-maildir/) is generated from the doc comments, which carry most of the explanation of why a maildir is the shape it is. [zig-mime]: https://git.jcollie.dev/jeff/zig-mime ## Where this lives The repository lives on the Forgejo instance at , which is where the continuous integration and the published documentation are: ```console $ git clone https://git.jcollie.dev/jeff/zig-maildir.git ``` ## Adding it to a project ```console $ zig fetch --save git+https://git.jcollie.dev/jeff/zig-maildir.git ``` ```zig const maildir = b.dependency("zig_maildir", .{ .target = target }).module("maildir"); exe.root_module.addImport("maildir", maildir); ``` ## Why a maildir is shaped like this A maildir is a directory holding three others — `tmp`, `new` and `cur` — and one idea: **a message is a file whose name carries everything mutable about it.** Everything else follows from that. * **Delivery takes no lock.** A message is written into `tmp` under a name nobody else will invent, and then renamed into `new`. A reader never sees a partial message, because a message only appears in `new` once it is whole and `rename` within a filesystem is atomic. Two mail servers, an IMAP daemon and a `procmail` can deliver at the same moment, over NFS, with nothing arbitrating between them. * **Reading is a directory listing.** There is no index to corrupt, no lock to hold while a slow client reads its mail, and no way for one crashed process to leave the mailbox unusable. * **Changing a flag is a `rename`.** Nothing is rewritten, so a message file is written exactly once and never modified — which is what lets the size recorded in its name be trusted, and what makes a maildir safe to back up while it is in use. The cost is that `tmp` accumulates the wreckage of interrupted deliveries, which is why `cleanTemp` exists and why the specification says to delete anything in there older than thirty-six hours. ## One maildir ```zig const maildir = @import("maildir"); var mailbox: maildir.Maildir = try .create(.cwd(), io, "Maildir", .{}); defer mailbox.close(io); // Delivery. The message lands in `new` with no flags, which is what "new" // means, and its name records how big it is. const delivered = try mailbox.deliver(io, bytes, .{}); // Reading. An iterator yields a `Message`, which is a name and nothing else // until you ask it for something. var it = mailbox.iterate(.new); while (try it.next(io)) |message| { var m = message; std.debug.print("{s} {d} bytes\n", .{ m.id(), try m.size(&mailbox, io) }); try m.setFlags(&mailbox, io, .{ .seen = true }); } ``` `setFlags` moves the message from `new` into `cur`, and that is not a convenience: a name in `new` has no info field, so there is nowhere in `new` for a flag to be written. "Read but still new" is not a thing a maildir can express. Delivery comes in two shapes. `deliver` takes a slice; `beginDelivery` gives back a writer, which is what to use when the message is arriving from a socket or being written by something else: ```zig var buffer: [4096]u8 = undefined; var delivery = try mailbox.beginDelivery(io, 10); errdefer delivery.abort(io); try built_message.write(delivery.writer(io, &buffer)); // zig-mime writes const delivered = try delivery.commit(io, .{}); ``` Either way the delivery is all or nothing: if anything fails before the rename, the partial file in `tmp` is removed and nothing ever appears in `new`. `deliver` also takes a destination, because not every message is new. An IMAP `APPEND` of an already-read message, or a client saving a draft, goes straight into `cur` with its flags on: ```zig _ = try mailbox.deliver(io, bytes, .{ .to = .{ .cur = .{ .seen = true, .draft = true } } }); ``` ## Names and flags ```zig const name: maildir.Name = .parse("1757700000.M1R2Q3.host,S=4211:2,RS", ':'); name.base(); // "1757700000.M1R2Q3.host" -- the identity name.size(); // 4211, without touching the filesystem name.flags().seen; // true ``` Three things about the name are worth knowing before writing anything that touches one. **The identity is `base`, and it must never change.** Two programs sharing a maildir agree about which message is which by that string and nothing else, so changing it loses every IMAP UID, every read/unread record and every synchronisation state keyed to it. `setFlags` and `addFlags` rewrite the info field and leave it alone; `moveTo` deliberately does not, and the section below says why. **Flags are a set of letters written in ASCII order**, and six of them are defined: `D` draft, `F` flagged, `P` passed, `R` replied, `S` seen, `T` trashed. Those get named fields. Every other letter is kept in `other` — ```zig var flags = try maildir.Flags.parse("Sb"); // Dovecot wrote this flags.replied = true; // "RSb": the keyword `b` is still there. ``` — and keeping it is the whole point. Dovecot stores IMAP keywords, which are arbitrary user-applied labels, as the letters `a` to `z` with a `dovecot-keywords` file mapping them to names. A library that read `:2,Sb`, marked the message replied and wrote back `:2,RS` would have silently deleted a label the user applied. Reading flags, changing one and writing them back is the most common thing anybody does to a maildir, and it has to be lossless. **The separator is a parameter.** A colon is what the specification says and what every Unix mail program expects, and it is also illegal in a FAT, exFAT or NTFS filename — so a maildir on a memory stick or a Windows share is written with `!` or `;` instead, and a reader that insists on a colon sees every message in it as new and flagless. ```zig var mailbox: maildir.Maildir = try .open(.cwd(), io, "Maildir", .{ .separator = '!' }); ``` ## Maildir++ folders A maildir has no room for a second mailbox in it, so Maildir++ puts the folders *beside* the messages, as hidden directories in the top-level maildir, each a complete maildir of its own: ```text Maildir/ the top-level maildir, which IMAP calls INBOX tmp/ new/ cur/ its own messages maildirsize the quota ledger, covering everything below .Work/ the folder "Work" tmp/ new/ cur/ maildirfolder the marker that says this is a folder .Work.Reports/ the folder "Work/Reports" ``` The hierarchy is **flat on disk and nested in the name**: `.Work.Reports` is a sibling of `.Work`, not a child, which is what lets a whole folder tree be listed with one `readdir`. ```zig var store: maildir.Store = try .create(.cwd(), io, "Maildir", .{}); defer store.close(io); var inbox = try store.inbox(io); defer inbox.close(io); var reports = try store.createFolder(io, &.{ "Work", "Reports" }); // and ".Work" defer reports.close(io); var names = try store.folders(gpa, io); // ".Work", ".Work.Reports", sorted defer names.deinit(gpa); ``` A folder name **cannot contain a dot**. The dot is the hierarchy delimiter and Maildir++ never defined an escape for one, so a mailbox called `example.com` is either the folder `com` inside the folder `example` or it is not representable. This library returns `error.InvalidComponent` rather than silently creating the wrong thing. Moving a message between folders changes its name, and has to: ```zig try message.moveTo(&inbox, &archive, io); ``` Two maildirs are two directories and nothing coordinates the names in them, so a message carrying its name into a folder that already had one like it would overwrite a message — and `rename` would do it silently. The flags travel; the identity does not. This is what IMAP's `MOVE` does, and it is why an IMAP server cannot promise a moved message keeps its UID. A keyword, though, travels only half way, and the NixOS test pins it down. The letter survives the move because it is part of the flags; what the letter *means* does not, because that is recorded in a `dovecot-keywords` file inside each mailbox, so a message labelled "Important" in the inbox arrives in the archive carrying a letter that mailbox has never assigned. Dovecot's own `MOVE` updates the destination's mapping. Nothing outside Dovecot can, because the mapping is Dovecot's rather than the maildir's — the six standard flags are the only ones that mean the same thing in every directory. ## Quota Totalling a mail store means listing every folder and adding up every message, which is fine once and ruinous on every delivery. Courier's answer is `maildirsize`, a small file in the top-level maildir written like a ledger rather than a balance: ```text 10485760S,1000C 4211 1 8320 1 -4211 -1 ``` The first line is the quota and every line after it is a *change*, appended by whoever made it. The usage is their sum. ```zig try store.setQuota(io, .{ .bytes = 10 * 1024 * 1024, .messages = 1000 }, .zero); try store.recordUsage(io, .{ .bytes = @intCast(bytes.len), .messages = 1 }); const state = (try store.quotaState(gpa, io)).?; if (state.exceeded()) return error.OverQuota; if (state.isStale(io, .{})) _ = try store.recalculateQuota(gpa, io); ``` Treat a number from there as "what the store believed last time somebody checked". The ledger is **advisory and self-healing rather than authoritative**: it drifts, because a message deleted by something that does not know about the file is never subtracted, and it is meant to — `isStale` eventually says so and `recalculateQuota` walks the store and writes a fresh total. Note that `isStale` only applies its age test when the store is *over* quota, which is deliberate: being wrongly under quota costs a little unfairness, while being wrongly over it bounces mail, so the expensive check is spent only on the answer that would refuse a delivery. `recalculateQuota` counts a message from the `,S=` in its name where there is one, so keeping `record_size` on at delivery is what makes a recalculation a directory walk rather than a `stat` of every message in the store. ## Reading the message ```zig var parsed = try message.parse(&mailbox, gpa, io, .unlimited, .{}); defer parsed.deinit(); std.debug.print("{s}\n", .{(try parsed.root.subject()) orelse "(no subject)"}); ``` That is [zig-mime][zig-mime] from there on: RFC 5322 and the MIME documents, addresses, dates, encoded words, the whole tree of parts. It parses from a slice, so the message is read into memory first and the `mime.Message` owns that copy. The division of labour is the point. A maildir is a naming convention over a directory and has no opinion about what is in the files; a message parser has no opinion about where the message came from. Keeping them apart is what lets this library deliver a message it cannot parse, which is exactly what a mail store must do — a message that has already arrived has to be filed whatever is in its headers. ## The command-line tool ```console $ zig-maildir create Maildir # a maildir, or a whole store $ zig-maildir deliver Maildir message.eml # `-` reads standard input $ zig-maildir list Maildir # every message, with its flags $ zig-maildir headers Maildir # decoded, through zig-mime $ zig-maildir flag Maildir +S -F $ zig-maildir move Maildir Archive $ zig-maildir folders Maildir $ zig-maildir mkfolder Maildir Work/Reports $ zig-maildir quota Maildir 10485760 1000 $ zig-maildir recalc Maildir $ zig-maildir clean Maildir # old wreckage out of tmp ``` It exists to show what the library looks like from outside, and to give the NixOS test a second program to point at a maildir Dovecot is also looking at. ## What is deliberately not here - **No message parsing.** That is [zig-mime][zig-mime], and `Message.parse` is the whole of the connection between them. - **No IMAP, POP or SMTP.** This library is about the store, not about serving it or filling it. It does provide the pieces an IMAP server needs — stable identities, keyword-preserving flags, Maildir++ folders, quota — but the protocol is somebody else's. - **No subscriptions.** Which folders a client has subscribed to is IMAP's business and every server keeps it differently: Courier in `courierimapsubscribed`, Dovecot in `subscriptions`. A file with either name is left alone rather than guessed at. - **No index, and no caching of one.** Every listing is a `readdir`. That is what a maildir is, and a program that needs an index over one should keep it itself, keyed by `Message.id`. - **No locking, anywhere except the quota ledger.** The design does not need any, which is its whole appeal. `quota.record` takes an advisory lock because Zig 0.16 cannot open a file `O_APPEND` and the position therefore has to be read before it is written; that is the one race the original design closed with a flag this library cannot ask for. ## Interoperability `nix flake check` boots a NixOS guest with a real Dovecot serving a maildir this library wrote, and drives both. It asserts what would otherwise be a comfortable assumption: - Dovecot finds and indexes a message `zig-maildir` delivered, its `RFC822.SIZE` is the number in the name, and it keeps that `,S=` field when it renames the file itself. - **Dovecot leaves a flagged message in `new`**, renaming it in place to something like `new/…,S=121:2,S`, which the specification says should only ever happen in `cur`. A reader that took the rule literally would show that message as unread forever, which is why `Name.parse` cannot fail. - A flag set by either is seen by the other, in both directions. - **An IMAP keyword Dovecot applied survives this library changing a flag**, which is the one that would silently lose a user's labels. - A folder made by either is listed by the other, and is flat on disk with a `maildirfolder` in it. - A message moved between folders keeps its standard flags and gets a new name — and its keyword arrives as a letter the destination mailbox cannot name, which is a limit of the format rather than of this library. It needs KVM, so it runs on the ephemeral runner tiers rather than the always-on one. ## Fuzzing `tests/fuzz.zig` holds four targets, each a property rather than an example: whatever arrives, the parser terminates, stays inside its buffers, and — if it claims to have understood the input — writing it back out and reading it again gives the same answer. ```console $ zig build test # the properties, over the checked-in seeds $ zig build fuzz-run # a minute of each, with generated input $ zig build fuzz-run -- --seconds 300 --target name $ zig build fuzz-run -- --input fuzz-findings/x.bin --target name ``` Stability is the property that earns its keep, and it is a stronger claim than it sounds. A maildir library rewrites names constantly, since every flag change is a parse, a change and a write. If that round trip is not a fixed point — if writing a parsed name can produce something that parses differently — then a message's name drifts a little every time anybody touches it, and since the name *is* the message's identity, the message eventually becomes a different message. So the name target asks for the second write to equal the first rather than for the output to equal the input: a name may legitimately be repaired on the way in (unsorted flags get sorted, a duplicate letter is dropped), but repairing it twice must change nothing, and the repair must never touch `base`. `tools/fuzz.zig` is that loop and says at the top why it exists: Zig 0.16.0 cannot build a test executable in fuzz mode without a patched standard library — `flake.nix` carries the patch — and leaves the fuzzer's coverage table empty even then. A fuzzer with no coverage is a random number generator, so this is one written down honestly, with a corpus of real names to mutate instead. ## The API documentation , rebuilt from `main` on every push. Zig writes it out of the doc comments, which in this project carry most of the explanation — why delivery needs no lock, why a flag change is a rename, what `,S=` is for and why it may be trusted. ```console $ zig build docs-serve # then open http://127.0.0.1:8000/ $ zig build docs # or just build it, into zig-out/docs ``` It has to be served rather than opened. What Zig emits is not a page but a program: a WebAssembly viewer that fetches the source of everything it shows out of a tar file beside it, and a browser refuses to fetch anything from a `file://` page. That is the same reason `zig std` runs a server rather than opening a file. `tools/docs_server.zig` is that server — one directory, one person, the loopback interface, and nothing else. ## Licence MIT. See `LICENSES/MIT.txt`.