A maildir and Maildir++ library for Zig 0.16: delivery, flags, folders and quota.
19 kB
450 lines
1<!--
2SPDX-FileCopyrightText: © 2026 Jeffrey C. Ollie <jeff@ocjtech.us>
3SPDX-License-Identifier: MIT
4-->
5
6# zig-maildir
7
8Maildirs for Zig 0.16: delivering messages into one, reading them back out,
9changing their flags, and the Maildir++ tree of folders and quota that the rest
10of the world built on top.
11
12```console
13$ zig-maildir create Maildir
14$ zig-maildir deliver Maildir message.eml
151757700000.M492817R3f0a1c2b4d5e6f70Q1.mail.example.com
16$ zig-maildir list Maildir
17new INBOX 4211 1757700000.M492817R3f0a1c2b4d5e6f70Q1.mail.example.com
18$ zig-maildir flag Maildir 1757700000.M492817R3f0a1c2b4d5e6f70Q1.mail.example.com +S +R
19RS
20```
21
22Three things, and you can stop at any of them:
23
24| | |
25| --- | --- |
26| `Name` and `Flags` | [the naming convention](#names-and-flags) — pure functions over bytes, no I/O and no allocation |
27| `Maildir` | [one mailbox](#one-maildir): delivery, iteration, flags, and the `tmp`/`new`/`cur` dance that makes it safe |
28| `Store` | [a Maildir++ tree](#maildir-folders): folders, and [the quota ledger](#quota) covering all of them |
29
30The message content is [zig-mime][zig-mime]'s: this library files messages and
31does not read them, and `Message.parse` hands the bytes straight over. The
32[API documentation](https://jeff.jcollie.page/zig-maildir/) is generated from
33the doc comments, which carry most of the explanation of why a maildir is the
34shape it is.
35
36[zig-mime]: https://git.jcollie.dev/jeff/zig-mime
37
38## Where this lives
39
40The repository lives in three places that carry the same history. The Forgejo
41instance at <https://git.jcollie.dev/jeff/zig-maildir> is the web-visible one,
42and is where the continuous integration and the published documentation are:
43
44```console
45$ git clone https://git.jcollie.dev/jeff/zig-maildir.git
46```
47
48It is mirrored on Tangled at <https://tangled.org/jcollie.dev/zig-maildir>, a
49forge built on the AT Protocol, where a repository is addressed by its owner's
50identity rather than by a server name.
51
52It is also on [Radicle][radicle], a peer-to-peer forge that needs no account on
53anything. A Radicle repository is findable only by its repository ID, so this
54is that ID:
55
56```
57rad:z3zvKaC531F7uLpN5YdTXHXYTY7Mk
58```
59
60and `rad clone rad:z3zvKaC531F7uLpN5YdTXHXYTY7Mk` fetches it from any node that
61seeds it. `rad clone` finds seeds through your local node's routing table, so
62the node has to be running first, and cloning seeds the repository in turn,
63which helps keep it available:
64
65```console
66$ rad node start
67$ rad clone rad:z3zvKaC531F7uLpN5YdTXHXYTY7Mk
68```
69
70If you already have the repository and only want to help host it:
71
72```console
73$ rad seed rad:z3zvKaC531F7uLpN5YdTXHXYTY7Mk
74```
75
76Any of the three is the whole project.
77
78[radicle]: https://radicle.xyz
79
80## Adding it to a project
81
82```console
83$ zig fetch --save git+https://git.jcollie.dev/jeff/zig-maildir.git
84```
85
86```zig
87const maildir = b.dependency("zig_maildir", .{ .target = target }).module("maildir");
88exe.root_module.addImport("maildir", maildir);
89```
90
91## Why a maildir is shaped like this
92
93A maildir is a directory holding three others — `tmp`, `new` and `cur` — and
94one idea: **a message is a file whose name carries everything mutable about
95it.** Everything else follows from that.
96
97* **Delivery takes no lock.** A message is written into `tmp` under a name
98 nobody else will invent, and then renamed into `new`. A reader never sees a
99 partial message, because a message only appears in `new` once it is whole and
100 `rename` within a filesystem is atomic. Two mail servers, an IMAP daemon and
101 a `procmail` can deliver at the same moment, over NFS, with nothing
102 arbitrating between them.
103* **Reading is a directory listing.** There is no index to corrupt, no lock to
104 hold while a slow client reads its mail, and no way for one crashed process
105 to leave the mailbox unusable.
106* **Changing a flag is a `rename`.** Nothing is rewritten, so a message file is
107 written exactly once and never modified — which is what lets the size
108 recorded in its name be trusted, and what makes a maildir safe to back up
109 while it is in use.
110
111The cost is that `tmp` accumulates the wreckage of interrupted deliveries,
112which is why `cleanTemp` exists and why the specification says to delete
113anything in there older than thirty-six hours.
114
115## One maildir
116
117```zig
118const maildir = @import("maildir");
119
120var mailbox: maildir.Maildir = try .create(.cwd(), io, "Maildir", .{});
121defer mailbox.close(io);
122
123// Delivery. The message lands in `new` with no flags, which is what "new"
124// means, and its name records how big it is.
125const delivered = try mailbox.deliver(io, bytes, .{});
126
127// Reading. An iterator yields a `Message`, which is a name and nothing else
128// until you ask it for something.
129var it = mailbox.iterate(.new);
130while (try it.next(io)) |message| {
131 var m = message;
132 std.debug.print("{s} {d} bytes\n", .{ m.id(), try m.size(&mailbox, io) });
133 try m.setFlags(&mailbox, io, .{ .seen = true });
134}
135```
136
137`setFlags` moves the message from `new` into `cur`, and that is not a
138convenience: a name in `new` has no info field, so there is nowhere in `new`
139for a flag to be written. "Read but still new" is not a thing a maildir can
140express.
141
142Delivery comes in two shapes. `deliver` takes a slice; `beginDelivery` gives
143back a writer, which is what to use when the message is arriving from a socket
144or being written by something else:
145
146```zig
147var buffer: [4096]u8 = undefined;
148var delivery = try mailbox.beginDelivery(io, 10);
149errdefer delivery.abort(io);
150try built_message.write(delivery.writer(io, &buffer)); // zig-mime writes
151const delivered = try delivery.commit(io, .{});
152```
153
154Either way the delivery is all or nothing: if anything fails before the rename,
155the partial file in `tmp` is removed and nothing ever appears in `new`.
156
157`deliver` also takes a destination, because not every message is new. An IMAP
158`APPEND` of an already-read message, or a client saving a draft, goes straight
159into `cur` with its flags on:
160
161```zig
162_ = try mailbox.deliver(io, bytes, .{ .to = .{ .cur = .{ .seen = true, .draft = true } } });
163```
164
165## Names and flags
166
167```zig
168const name: maildir.Name = .parse("1757700000.M1R2Q3.host,S=4211:2,RS", ':');
169name.base(); // "1757700000.M1R2Q3.host" -- the identity
170name.size(); // 4211, without touching the filesystem
171name.flags().seen; // true
172```
173
174Three things about the name are worth knowing before writing anything that
175touches one.
176
177**The identity is `base`, and it must never change.** Two programs sharing a
178maildir agree about which message is which by that string and nothing else, so
179changing it loses every IMAP UID, every read/unread record and every
180synchronisation state keyed to it. `setFlags` and `addFlags` rewrite the info
181field and leave it alone; `moveTo` deliberately does not, and the section below
182says why.
183
184**Flags are a set of letters written in ASCII order**, and six of them are
185defined: `D` draft, `F` flagged, `P` passed, `R` replied, `S` seen, `T`
186trashed. Those get named fields. Every other letter is kept in `other` —
187
188```zig
189var flags = try maildir.Flags.parse("Sb"); // Dovecot wrote this
190flags.replied = true;
191// "RSb": the keyword `b` is still there.
192```
193
194— and keeping it is the whole point. Dovecot stores IMAP keywords, which are
195arbitrary user-applied labels, as the letters `a` to `z` with a
196`dovecot-keywords` file mapping them to names. A library that read `:2,Sb`,
197marked the message replied and wrote back `:2,RS` would have silently deleted a
198label the user applied. Reading flags, changing one and writing them back is
199the most common thing anybody does to a maildir, and it has to be lossless.
200
201**The separator is a parameter.** A colon is what the specification says and
202what every Unix mail program expects, and it is also illegal in a FAT, exFAT or
203NTFS filename — so a maildir on a memory stick or a Windows share is written
204with `!` or `;` instead, and a reader that insists on a colon sees every
205message in it as new and flagless.
206
207```zig
208var mailbox: maildir.Maildir = try .open(.cwd(), io, "Maildir", .{ .separator = '!' });
209```
210
211## Maildir++ folders
212
213A maildir has no room for a second mailbox in it, so Maildir++ puts the folders
214*beside* the messages, as hidden directories in the top-level maildir, each a
215complete maildir of its own:
216
217```text
218Maildir/ the top-level maildir, which IMAP calls INBOX
219 tmp/ new/ cur/ its own messages
220 maildirsize the quota ledger, covering everything below
221 .Work/ the folder "Work"
222 tmp/ new/ cur/
223 maildirfolder the marker that says this is a folder
224 .Work.Reports/ the folder "Work/Reports"
225```
226
227The hierarchy is **flat on disk and nested in the name**: `.Work.Reports` is a
228sibling of `.Work`, not a child, which is what lets a whole folder tree be
229listed with one `readdir`.
230
231```zig
232var store: maildir.Store = try .create(.cwd(), io, "Maildir", .{});
233defer store.close(io);
234
235var inbox = try store.inbox(io);
236defer inbox.close(io);
237
238var reports = try store.createFolder(io, &.{ "Work", "Reports" }); // and ".Work"
239defer reports.close(io);
240
241var names = try store.folders(gpa, io); // ".Work", ".Work.Reports", sorted
242defer names.deinit(gpa);
243```
244
245A folder name **cannot contain a dot**. The dot is the hierarchy delimiter and
246Maildir++ never defined an escape for one, so a mailbox called `example.com` is
247either the folder `com` inside the folder `example` or it is not representable.
248This library returns `error.InvalidComponent` rather than silently creating the
249wrong thing.
250
251Moving a message between folders changes its name, and has to:
252
253```zig
254try message.moveTo(&inbox, &archive, io);
255```
256
257Two maildirs are two directories and nothing coordinates the names in them, so
258a message carrying its name into a folder that already had one like it would
259overwrite a message — and `rename` would do it silently. The flags travel; the
260identity does not. This is what IMAP's `MOVE` does, and it is why an IMAP
261server cannot promise a moved message keeps its UID.
262
263A keyword, though, travels only half way, and the NixOS test pins it down. The
264letter survives the move because it is part of the flags; what the letter
265*means* does not, because that is recorded in a `dovecot-keywords` file inside
266each mailbox, so a message labelled "Important" in the inbox arrives in the
267archive carrying a letter that mailbox has never assigned. Dovecot's own `MOVE`
268updates the destination's mapping. Nothing outside Dovecot can, because the
269mapping is Dovecot's rather than the maildir's — the six standard flags are the
270only ones that mean the same thing in every directory.
271
272## Quota
273
274Totalling a mail store means listing every folder and adding up every message,
275which is fine once and ruinous on every delivery. Courier's answer is
276`maildirsize`, a small file in the top-level maildir written like a ledger
277rather than a balance:
278
279```text
28010485760S,1000C
2814211 1
2828320 1
283-4211 -1
284```
285
286The first line is the quota and every line after it is a *change*, appended by
287whoever made it. The usage is their sum.
288
289```zig
290try store.setQuota(io, .{ .bytes = 10 * 1024 * 1024, .messages = 1000 }, .zero);
291try store.recordUsage(io, .{ .bytes = @intCast(bytes.len), .messages = 1 });
292
293const state = (try store.quotaState(gpa, io)).?;
294if (state.exceeded()) return error.OverQuota;
295if (state.isStale(io, .{})) _ = try store.recalculateQuota(gpa, io);
296```
297
298Treat a number from there as "what the store believed last time somebody
299checked". The ledger is **advisory and self-healing rather than
300authoritative**: it drifts, because a message deleted by something that does
301not know about the file is never subtracted, and it is meant to — `isStale`
302eventually says so and `recalculateQuota` walks the store and writes a fresh
303total. Note that `isStale` only applies its age test when the store is *over*
304quota, which is deliberate: being wrongly under quota costs a little
305unfairness, while being wrongly over it bounces mail, so the expensive check is
306spent only on the answer that would refuse a delivery.
307
308`recalculateQuota` counts a message from the `,S=` in its name where there is
309one, so keeping `record_size` on at delivery is what makes a recalculation a
310directory walk rather than a `stat` of every message in the store.
311
312## Reading the message
313
314```zig
315var parsed = try message.parse(&mailbox, gpa, io, .unlimited, .{});
316defer parsed.deinit();
317
318std.debug.print("{s}\n", .{(try parsed.root.subject()) orelse "(no subject)"});
319```
320
321That is [zig-mime][zig-mime] from there on: RFC 5322 and the MIME documents,
322addresses, dates, encoded words, the whole tree of parts. It parses from a
323slice, so the message is read into memory first and the `mime.Message` owns
324that copy.
325
326The division of labour is the point. A maildir is a naming convention over a
327directory and has no opinion about what is in the files; a message parser has
328no opinion about where the message came from. Keeping them apart is what lets
329this library deliver a message it cannot parse, which is exactly what a mail
330store must do — a message that has already arrived has to be filed whatever is
331in its headers.
332
333## The command-line tool
334
335```console
336$ zig-maildir create Maildir # a maildir, or a whole store
337$ zig-maildir deliver Maildir message.eml # `-` reads standard input
338$ zig-maildir list Maildir # every message, with its flags
339$ zig-maildir headers Maildir <id> # decoded, through zig-mime
340$ zig-maildir flag Maildir <id> +S -F
341$ zig-maildir move Maildir <id> Archive
342$ zig-maildir folders Maildir
343$ zig-maildir mkfolder Maildir Work/Reports
344$ zig-maildir quota Maildir 10485760 1000
345$ zig-maildir recalc Maildir
346$ zig-maildir clean Maildir # old wreckage out of tmp
347```
348
349It exists to show what the library looks like from outside, and to give the
350NixOS test a second program to point at a maildir Dovecot is also looking at.
351
352## What is deliberately not here
353
354- **No message parsing.** That is [zig-mime][zig-mime], and `Message.parse` is
355 the whole of the connection between them.
356- **No IMAP, POP or SMTP.** This library is about the store, not about serving
357 it or filling it. It does provide the pieces an IMAP server needs — stable
358 identities, keyword-preserving flags, Maildir++ folders, quota — but the
359 protocol is somebody else's.
360- **No subscriptions.** Which folders a client has subscribed to is IMAP's
361 business and every server keeps it differently: Courier in
362 `courierimapsubscribed`, Dovecot in `subscriptions`. A file with either name
363 is left alone rather than guessed at.
364- **No index, and no caching of one.** Every listing is a `readdir`. That is
365 what a maildir is, and a program that needs an index over one should keep it
366 itself, keyed by `Message.id`.
367- **No locking, anywhere except the quota ledger.** The design does not need
368 any, which is its whole appeal. `quota.record` takes an advisory lock because
369 Zig 0.16 cannot open a file `O_APPEND` and the position therefore has to be
370 read before it is written; that is the one race the original design closed
371 with a flag this library cannot ask for.
372
373## Interoperability
374
375`nix flake check` boots a NixOS guest with a real Dovecot serving a maildir
376this library wrote, and drives both. It asserts what would otherwise be a
377comfortable assumption:
378
379- Dovecot finds and indexes a message `zig-maildir` delivered, its
380 `RFC822.SIZE` is the number in the name, and it keeps that `,S=` field when
381 it renames the file itself.
382- **Dovecot leaves a flagged message in `new`**, renaming it in place to
383 something like `new/…,S=121:2,S`, which the specification says should only
384 ever happen in `cur`. A reader that took the rule literally would show that
385 message as unread forever, which is why `Name.parse` cannot fail.
386- A flag set by either is seen by the other, in both directions.
387- **An IMAP keyword Dovecot applied survives this library changing a flag**,
388 which is the one that would silently lose a user's labels.
389- A folder made by either is listed by the other, and is flat on disk with a
390 `maildirfolder` in it.
391- A message moved between folders keeps its standard flags and gets a new name
392 — and its keyword arrives as a letter the destination mailbox cannot name,
393 which is a limit of the format rather than of this library.
394
395It needs KVM, so it runs on the ephemeral runner tiers rather than the
396always-on one.
397
398## Fuzzing
399
400`tests/fuzz.zig` holds four targets, each a property rather than an example:
401whatever arrives, the parser terminates, stays inside its buffers, and — if it
402claims to have understood the input — writing it back out and reading it again
403gives the same answer.
404
405```console
406$ zig build test # the properties, over the checked-in seeds
407$ zig build fuzz-run # a minute of each, with generated input
408$ zig build fuzz-run -- --seconds 300 --target name
409$ zig build fuzz-run -- --input fuzz-findings/x.bin --target name
410```
411
412Stability is the property that earns its keep, and it is a stronger claim than
413it sounds. A maildir library rewrites names constantly, since every flag change
414is a parse, a change and a write. If that round trip is not a fixed point — if
415writing a parsed name can produce something that parses differently — then a
416message's name drifts a little every time anybody touches it, and since the
417name *is* the message's identity, the message eventually becomes a different
418message. So the name target asks for the second write to equal the first rather
419than for the output to equal the input: a name may legitimately be repaired on
420the way in (unsorted flags get sorted, a duplicate letter is dropped), but
421repairing it twice must change nothing, and the repair must never touch `base`.
422
423`tools/fuzz.zig` is that loop and says at the top why it exists: Zig 0.16.0
424cannot build a test executable in fuzz mode without a patched standard library
425— `flake.nix` carries the patch — and leaves the fuzzer's coverage table empty
426even then. A fuzzer with no coverage is a random number generator, so this is
427one written down honestly, with a corpus of real names to mutate instead.
428
429## The API documentation
430
431<https://jeff.jcollie.page/zig-maildir/>, rebuilt from `main` on every push.
432Zig writes it out of the doc comments, which in this project carry most of the
433explanation — why delivery needs no lock, why a flag change is a rename, what
434`,S=` is for and why it may be trusted.
435
436```console
437$ zig build docs-serve # then open http://127.0.0.1:8000/
438$ zig build docs # or just build it, into zig-out/docs
439```
440
441It has to be served rather than opened. What Zig emits is not a page but a
442program: a WebAssembly viewer that fetches the source of everything it shows
443out of a tar file beside it, and a browser refuses to fetch anything from a
444`file://` page. That is the same reason `zig std` runs a server rather than
445opening a file. `tools/docs_server.zig` is that server — one directory, one
446person, the loopback interface, and nothing else.
447
448## Licence
449
450MIT. See `LICENSES/MIT.txt`.