A maildir and Maildir++ library for Zig 0.16: delivery, flags, folders and quota.
18 kB
418 lines
1<!--
2SPDX-FileCopyrightText: © 2026 Jeffrey C. Ollie <jeff@ocjtech.us>
3SPDX-License-Identifier: MIT
4-->
5
6# zig-maildir
7
8Maildirs for Zig 0.16: delivering messages into one, reading them back out,
9changing their flags, and the Maildir++ tree of folders and quota that the rest
10of the world built on top.
11
12```console
13$ zig-maildir create Maildir
14$ zig-maildir deliver Maildir message.eml
151757700000.M492817R3f0a1c2b4d5e6f70Q1.mail.example.com
16$ zig-maildir list Maildir
17new INBOX 4211 1757700000.M492817R3f0a1c2b4d5e6f70Q1.mail.example.com
18$ zig-maildir flag Maildir 1757700000.M492817R3f0a1c2b4d5e6f70Q1.mail.example.com +S +R
19RS
20```
21
22Three things, and you can stop at any of them:
23
24| | |
25| --- | --- |
26| `Name` and `Flags` | [the naming convention](#names-and-flags) — pure functions over bytes, no I/O and no allocation |
27| `Maildir` | [one mailbox](#one-maildir): delivery, iteration, flags, and the `tmp`/`new`/`cur` dance that makes it safe |
28| `Store` | [a Maildir++ tree](#maildir-folders): folders, and [the quota ledger](#quota) covering all of them |
29
30The message content is [zig-mime][zig-mime]'s: this library files messages and
31does not read them, and `Message.parse` hands the bytes straight over. The
32[API documentation](https://jeff.jcollie.page/zig-maildir/) is generated from
33the doc comments, which carry most of the explanation of why a maildir is the
34shape it is.
35
36[zig-mime]: https://git.jcollie.dev/jeff/zig-mime
37
38## Where this lives
39
40The repository lives on the Forgejo instance at
41<https://git.jcollie.dev/jeff/zig-maildir>, which is where the continuous
42integration and the published documentation are:
43
44```console
45$ git clone https://git.jcollie.dev/jeff/zig-maildir.git
46```
47
48## Adding it to a project
49
50```console
51$ zig fetch --save git+https://git.jcollie.dev/jeff/zig-maildir.git
52```
53
54```zig
55const maildir = b.dependency("zig_maildir", .{ .target = target }).module("maildir");
56exe.root_module.addImport("maildir", maildir);
57```
58
59## Why a maildir is shaped like this
60
61A maildir is a directory holding three others — `tmp`, `new` and `cur` — and
62one idea: **a message is a file whose name carries everything mutable about
63it.** Everything else follows from that.
64
65* **Delivery takes no lock.** A message is written into `tmp` under a name
66 nobody else will invent, and then renamed into `new`. A reader never sees a
67 partial message, because a message only appears in `new` once it is whole and
68 `rename` within a filesystem is atomic. Two mail servers, an IMAP daemon and
69 a `procmail` can deliver at the same moment, over NFS, with nothing
70 arbitrating between them.
71* **Reading is a directory listing.** There is no index to corrupt, no lock to
72 hold while a slow client reads its mail, and no way for one crashed process
73 to leave the mailbox unusable.
74* **Changing a flag is a `rename`.** Nothing is rewritten, so a message file is
75 written exactly once and never modified — which is what lets the size
76 recorded in its name be trusted, and what makes a maildir safe to back up
77 while it is in use.
78
79The cost is that `tmp` accumulates the wreckage of interrupted deliveries,
80which is why `cleanTemp` exists and why the specification says to delete
81anything in there older than thirty-six hours.
82
83## One maildir
84
85```zig
86const maildir = @import("maildir");
87
88var mailbox: maildir.Maildir = try .create(.cwd(), io, "Maildir", .{});
89defer mailbox.close(io);
90
91// Delivery. The message lands in `new` with no flags, which is what "new"
92// means, and its name records how big it is.
93const delivered = try mailbox.deliver(io, bytes, .{});
94
95// Reading. An iterator yields a `Message`, which is a name and nothing else
96// until you ask it for something.
97var it = mailbox.iterate(.new);
98while (try it.next(io)) |message| {
99 var m = message;
100 std.debug.print("{s} {d} bytes\n", .{ m.id(), try m.size(&mailbox, io) });
101 try m.setFlags(&mailbox, io, .{ .seen = true });
102}
103```
104
105`setFlags` moves the message from `new` into `cur`, and that is not a
106convenience: a name in `new` has no info field, so there is nowhere in `new`
107for a flag to be written. "Read but still new" is not a thing a maildir can
108express.
109
110Delivery comes in two shapes. `deliver` takes a slice; `beginDelivery` gives
111back a writer, which is what to use when the message is arriving from a socket
112or being written by something else:
113
114```zig
115var buffer: [4096]u8 = undefined;
116var delivery = try mailbox.beginDelivery(io, 10);
117errdefer delivery.abort(io);
118try built_message.write(delivery.writer(io, &buffer)); // zig-mime writes
119const delivered = try delivery.commit(io, .{});
120```
121
122Either way the delivery is all or nothing: if anything fails before the rename,
123the partial file in `tmp` is removed and nothing ever appears in `new`.
124
125`deliver` also takes a destination, because not every message is new. An IMAP
126`APPEND` of an already-read message, or a client saving a draft, goes straight
127into `cur` with its flags on:
128
129```zig
130_ = try mailbox.deliver(io, bytes, .{ .to = .{ .cur = .{ .seen = true, .draft = true } } });
131```
132
133## Names and flags
134
135```zig
136const name: maildir.Name = .parse("1757700000.M1R2Q3.host,S=4211:2,RS", ':');
137name.base(); // "1757700000.M1R2Q3.host" -- the identity
138name.size(); // 4211, without touching the filesystem
139name.flags().seen; // true
140```
141
142Three things about the name are worth knowing before writing anything that
143touches one.
144
145**The identity is `base`, and it must never change.** Two programs sharing a
146maildir agree about which message is which by that string and nothing else, so
147changing it loses every IMAP UID, every read/unread record and every
148synchronisation state keyed to it. `setFlags` and `addFlags` rewrite the info
149field and leave it alone; `moveTo` deliberately does not, and the section below
150says why.
151
152**Flags are a set of letters written in ASCII order**, and six of them are
153defined: `D` draft, `F` flagged, `P` passed, `R` replied, `S` seen, `T`
154trashed. Those get named fields. Every other letter is kept in `other` —
155
156```zig
157var flags = try maildir.Flags.parse("Sb"); // Dovecot wrote this
158flags.replied = true;
159// "RSb": the keyword `b` is still there.
160```
161
162— and keeping it is the whole point. Dovecot stores IMAP keywords, which are
163arbitrary user-applied labels, as the letters `a` to `z` with a
164`dovecot-keywords` file mapping them to names. A library that read `:2,Sb`,
165marked the message replied and wrote back `:2,RS` would have silently deleted a
166label the user applied. Reading flags, changing one and writing them back is
167the most common thing anybody does to a maildir, and it has to be lossless.
168
169**The separator is a parameter.** A colon is what the specification says and
170what every Unix mail program expects, and it is also illegal in a FAT, exFAT or
171NTFS filename — so a maildir on a memory stick or a Windows share is written
172with `!` or `;` instead, and a reader that insists on a colon sees every
173message in it as new and flagless.
174
175```zig
176var mailbox: maildir.Maildir = try .open(.cwd(), io, "Maildir", .{ .separator = '!' });
177```
178
179## Maildir++ folders
180
181A maildir has no room for a second mailbox in it, so Maildir++ puts the folders
182*beside* the messages, as hidden directories in the top-level maildir, each a
183complete maildir of its own:
184
185```text
186Maildir/ the top-level maildir, which IMAP calls INBOX
187 tmp/ new/ cur/ its own messages
188 maildirsize the quota ledger, covering everything below
189 .Work/ the folder "Work"
190 tmp/ new/ cur/
191 maildirfolder the marker that says this is a folder
192 .Work.Reports/ the folder "Work/Reports"
193```
194
195The hierarchy is **flat on disk and nested in the name**: `.Work.Reports` is a
196sibling of `.Work`, not a child, which is what lets a whole folder tree be
197listed with one `readdir`.
198
199```zig
200var store: maildir.Store = try .create(.cwd(), io, "Maildir", .{});
201defer store.close(io);
202
203var inbox = try store.inbox(io);
204defer inbox.close(io);
205
206var reports = try store.createFolder(io, &.{ "Work", "Reports" }); // and ".Work"
207defer reports.close(io);
208
209var names = try store.folders(gpa, io); // ".Work", ".Work.Reports", sorted
210defer names.deinit(gpa);
211```
212
213A folder name **cannot contain a dot**. The dot is the hierarchy delimiter and
214Maildir++ never defined an escape for one, so a mailbox called `example.com` is
215either the folder `com` inside the folder `example` or it is not representable.
216This library returns `error.InvalidComponent` rather than silently creating the
217wrong thing.
218
219Moving a message between folders changes its name, and has to:
220
221```zig
222try message.moveTo(&inbox, &archive, io);
223```
224
225Two maildirs are two directories and nothing coordinates the names in them, so
226a message carrying its name into a folder that already had one like it would
227overwrite a message — and `rename` would do it silently. The flags travel; the
228identity does not. This is what IMAP's `MOVE` does, and it is why an IMAP
229server cannot promise a moved message keeps its UID.
230
231A keyword, though, travels only half way, and the NixOS test pins it down. The
232letter survives the move because it is part of the flags; what the letter
233*means* does not, because that is recorded in a `dovecot-keywords` file inside
234each mailbox, so a message labelled "Important" in the inbox arrives in the
235archive carrying a letter that mailbox has never assigned. Dovecot's own `MOVE`
236updates the destination's mapping. Nothing outside Dovecot can, because the
237mapping is Dovecot's rather than the maildir's — the six standard flags are the
238only ones that mean the same thing in every directory.
239
240## Quota
241
242Totalling a mail store means listing every folder and adding up every message,
243which is fine once and ruinous on every delivery. Courier's answer is
244`maildirsize`, a small file in the top-level maildir written like a ledger
245rather than a balance:
246
247```text
24810485760S,1000C
2494211 1
2508320 1
251-4211 -1
252```
253
254The first line is the quota and every line after it is a *change*, appended by
255whoever made it. The usage is their sum.
256
257```zig
258try store.setQuota(io, .{ .bytes = 10 * 1024 * 1024, .messages = 1000 }, .zero);
259try store.recordUsage(io, .{ .bytes = @intCast(bytes.len), .messages = 1 });
260
261const state = (try store.quotaState(gpa, io)).?;
262if (state.exceeded()) return error.OverQuota;
263if (state.isStale(io, .{})) _ = try store.recalculateQuota(gpa, io);
264```
265
266Treat a number from there as "what the store believed last time somebody
267checked". The ledger is **advisory and self-healing rather than
268authoritative**: it drifts, because a message deleted by something that does
269not know about the file is never subtracted, and it is meant to — `isStale`
270eventually says so and `recalculateQuota` walks the store and writes a fresh
271total. Note that `isStale` only applies its age test when the store is *over*
272quota, which is deliberate: being wrongly under quota costs a little
273unfairness, while being wrongly over it bounces mail, so the expensive check is
274spent only on the answer that would refuse a delivery.
275
276`recalculateQuota` counts a message from the `,S=` in its name where there is
277one, so keeping `record_size` on at delivery is what makes a recalculation a
278directory walk rather than a `stat` of every message in the store.
279
280## Reading the message
281
282```zig
283var parsed = try message.parse(&mailbox, gpa, io, .unlimited, .{});
284defer parsed.deinit();
285
286std.debug.print("{s}\n", .{(try parsed.root.subject()) orelse "(no subject)"});
287```
288
289That is [zig-mime][zig-mime] from there on: RFC 5322 and the MIME documents,
290addresses, dates, encoded words, the whole tree of parts. It parses from a
291slice, so the message is read into memory first and the `mime.Message` owns
292that copy.
293
294The division of labour is the point. A maildir is a naming convention over a
295directory and has no opinion about what is in the files; a message parser has
296no opinion about where the message came from. Keeping them apart is what lets
297this library deliver a message it cannot parse, which is exactly what a mail
298store must do — a message that has already arrived has to be filed whatever is
299in its headers.
300
301## The command-line tool
302
303```console
304$ zig-maildir create Maildir # a maildir, or a whole store
305$ zig-maildir deliver Maildir message.eml # `-` reads standard input
306$ zig-maildir list Maildir # every message, with its flags
307$ zig-maildir headers Maildir <id> # decoded, through zig-mime
308$ zig-maildir flag Maildir <id> +S -F
309$ zig-maildir move Maildir <id> Archive
310$ zig-maildir folders Maildir
311$ zig-maildir mkfolder Maildir Work/Reports
312$ zig-maildir quota Maildir 10485760 1000
313$ zig-maildir recalc Maildir
314$ zig-maildir clean Maildir # old wreckage out of tmp
315```
316
317It exists to show what the library looks like from outside, and to give the
318NixOS test a second program to point at a maildir Dovecot is also looking at.
319
320## What is deliberately not here
321
322- **No message parsing.** That is [zig-mime][zig-mime], and `Message.parse` is
323 the whole of the connection between them.
324- **No IMAP, POP or SMTP.** This library is about the store, not about serving
325 it or filling it. It does provide the pieces an IMAP server needs — stable
326 identities, keyword-preserving flags, Maildir++ folders, quota — but the
327 protocol is somebody else's.
328- **No subscriptions.** Which folders a client has subscribed to is IMAP's
329 business and every server keeps it differently: Courier in
330 `courierimapsubscribed`, Dovecot in `subscriptions`. A file with either name
331 is left alone rather than guessed at.
332- **No index, and no caching of one.** Every listing is a `readdir`. That is
333 what a maildir is, and a program that needs an index over one should keep it
334 itself, keyed by `Message.id`.
335- **No locking, anywhere except the quota ledger.** The design does not need
336 any, which is its whole appeal. `quota.record` takes an advisory lock because
337 Zig 0.16 cannot open a file `O_APPEND` and the position therefore has to be
338 read before it is written; that is the one race the original design closed
339 with a flag this library cannot ask for.
340
341## Interoperability
342
343`nix flake check` boots a NixOS guest with a real Dovecot serving a maildir
344this library wrote, and drives both. It asserts what would otherwise be a
345comfortable assumption:
346
347- Dovecot finds and indexes a message `zig-maildir` delivered, its
348 `RFC822.SIZE` is the number in the name, and it keeps that `,S=` field when
349 it renames the file itself.
350- **Dovecot leaves a flagged message in `new`**, renaming it in place to
351 something like `new/…,S=121:2,S`, which the specification says should only
352 ever happen in `cur`. A reader that took the rule literally would show that
353 message as unread forever, which is why `Name.parse` cannot fail.
354- A flag set by either is seen by the other, in both directions.
355- **An IMAP keyword Dovecot applied survives this library changing a flag**,
356 which is the one that would silently lose a user's labels.
357- A folder made by either is listed by the other, and is flat on disk with a
358 `maildirfolder` in it.
359- A message moved between folders keeps its standard flags and gets a new name
360 — and its keyword arrives as a letter the destination mailbox cannot name,
361 which is a limit of the format rather than of this library.
362
363It needs KVM, so it runs on the ephemeral runner tiers rather than the
364always-on one.
365
366## Fuzzing
367
368`tests/fuzz.zig` holds four targets, each a property rather than an example:
369whatever arrives, the parser terminates, stays inside its buffers, and — if it
370claims to have understood the input — writing it back out and reading it again
371gives the same answer.
372
373```console
374$ zig build test # the properties, over the checked-in seeds
375$ zig build fuzz-run # a minute of each, with generated input
376$ zig build fuzz-run -- --seconds 300 --target name
377$ zig build fuzz-run -- --input fuzz-findings/x.bin --target name
378```
379
380Stability is the property that earns its keep, and it is a stronger claim than
381it sounds. A maildir library rewrites names constantly, since every flag change
382is a parse, a change and a write. If that round trip is not a fixed point — if
383writing a parsed name can produce something that parses differently — then a
384message's name drifts a little every time anybody touches it, and since the
385name *is* the message's identity, the message eventually becomes a different
386message. So the name target asks for the second write to equal the first rather
387than for the output to equal the input: a name may legitimately be repaired on
388the way in (unsorted flags get sorted, a duplicate letter is dropped), but
389repairing it twice must change nothing, and the repair must never touch `base`.
390
391`tools/fuzz.zig` is that loop and says at the top why it exists: Zig 0.16.0
392cannot build a test executable in fuzz mode without a patched standard library
393— `flake.nix` carries the patch — and leaves the fuzzer's coverage table empty
394even then. A fuzzer with no coverage is a random number generator, so this is
395one written down honestly, with a corpus of real names to mutate instead.
396
397## The API documentation
398
399<https://jeff.jcollie.page/zig-maildir/>, rebuilt from `main` on every push.
400Zig writes it out of the doc comments, which in this project carry most of the
401explanation — why delivery needs no lock, why a flag change is a rename, what
402`,S=` is for and why it may be trusted.
403
404```console
405$ zig build docs-serve # then open http://127.0.0.1:8000/
406$ zig build docs # or just build it, into zig-out/docs
407```
408
409It has to be served rather than opened. What Zig emits is not a page but a
410program: a WebAssembly viewer that fetches the source of everything it shows
411out of a tar file beside it, and a browser refuses to fetch anything from a
412`file://` page. That is the same reason `zig std` runs a server rather than
413opening a file. `tools/docs_server.zig` is that server — one directory, one
414person, the loopback interface, and nothing else.
415
416## Licence
417
418MIT. See `LICENSES/MIT.txt`.