Build status:

Maildir

A Crystal library for reading and writing files in the "Maildir" file and directory structure. Even though this format is mainly used for email messages, the Maildir structure and the implementation of this module are general - they do not require the file contents to be related to email.

What's so Great About the Maildir Format

See http://cr.yp.to/proto/maildir.html and http://en.wikipedia.org/wiki/Maildir

"Two words: no locks." -- Daniel J. Bernstein

The maildir format allows multiple processes to read and write arbitrary messages without file locks.

New messages are initially written to a "tmp/" directory with an automatically-generated unique filename. Once they are written, they are atomically moved to the "new/" directory where other processes can see and use them.

While the maildir format was created for email, it works well for arbitrary data. This library can read and write any contents using the Maildir file and directory structure. And if you want the contents to be automatically serialized/deserialized objects, pluggable serializers are supported as well.

Installation

Add the following to your application's shard.yml:

dependencies:
  maildir:
    github: crystallabs/maildir.cr
    version: ~> 6.0

And run shards install.

Usage

Initialize a Maildir and create a Maildir directory structure in /tmp/maildir_test:

require "maildir"
maildir = Maildir.new("/tmp/maildir_test") # creates tmp, new, and cur dirs
# To skip directory creation, call Maildir.new("/tmp/maildir_test", false)

Add a new message. This will create a new file with the contents "Hello, Crystal!" and return the message. As mentioned, messages are written to the "tmp/" directory and then moved to "new/".

message = maildir.add("Hello, Crystal!")

List new messages:

maildir.list("new") # => [message]

Move the message from "new" to "cur" to indicate that some process has retrieved and/or processed the message.

message.process

Indeed, the message is now in "cur/", not "new/".

maildir.list("new") # => []
maildir.list("cur") # => [message]

Add some flags to the message to indicate state. See "What can I put in info" at http://cr.yp.to/proto/maildir.html for flag conventions. The library has convenience methods like seen! and seen? for all of the 6 standard flags, but arbitrary flags can be set.

message.add_flag("S")    # Mark the message as "seen"
message.add_flag("F")    # Mark the message as "flagged"
message.remove_flag("F") # Unflag the message
message.add_flag("DPR")  # Mark the message as "draft", "passed" and "replied"
message.remove_flag("DPR") # Remove the three flags
message.add_flag("T")    # Mark the message as "trashed"
message.add_flag("X")    # Mark with arbitrary-letter flag

List "cur/" messages based on flags. Flags must be specified in ascending ASCII order ("ST" and not "TS").

maildir.list("cur", {:flags => ""})   # => lists all messages without any flags
maildir.list("cur", {:flags => "F"})  # => lists all messages with flag "F"
maildir.list("cur", {:flags => "FS"}) # => lists all messages with flags "F" and "S"
maildir.list("cur", {:flags => "ST"}) # => lists all messages with flags "S" and "T"

Retrieve the key that uniquely identifies the message:

key = message.key

Read/load the contents of the message:

data = message.data

Find the message based on key:

message_copy = maildir.get(key)
message == message_copy # => true

Delete the message from disk:

message.destroy
maildir.list("cur") # => []

Cleaning up Orphaned Messages

An expected (though rare) behavior is for partially-written messages to be orphaned in the "tmp/" folder (when clients fail before fully writing a message).

Find messages in "tmp/" that haven't been changed in 36 hours:

maildir.get_stale_tmp

Clean them up:

maildir.get_stale_tmp.each { |msg| msg.destroy }

For more usage examples, please see files in the library's spec/ folder.

Durability

The maildir format promises that a message which is visible in "new/" is complete. For that to hold across a crash or a power loss, the message has to reach the disk before it is moved out of "tmp/", so by default every message is fsynced on write.

If your data is cheap to reproduce and you care more about delivery throughput than about surviving a crash, you can turn this off globally:

Maildir.fsync = false

A custom serializer that manages durability itself can override Maildir::Serializer::Base#sync instead.

Folders

Maildirs can contain folders, in the Maildir++ layout used by Courier and Dovecot. A folder is a dot-prefixed directory inside the root maildir (.a), nesting is expressed by joining the name components with a dot (.a.x), and each folder carries an empty maildirfolder file so that other tools recognize it as a folder. All of that is handled for you:

maildir = Maildir.new("/tmp/maildir_test")

folder = maildir.folder("a")        # => the maildir at /tmp/maildir_test/.a/
nested = maildir.folder("a.x")      # => the maildir at /tmp/maildir_test/.a.x/
nested == folder.folder("x")        # => true; naming by dot and by chaining agree
maildir.folder("b", false)          # => a folder object without creating it on disk

Folders are maildirs like any other, so they add, list and process messages the same way:

folder.add("Hello from a folder!")
folder.list("new") # => [message]

Navigate the tree with #folders (immediate subfolders, sorted), #parent and #root:

maildir.folders.map(&.folder_name) # => ["a", "b"]
folder.folders.map(&.folder_name)  # => ["a.x"]
nested.parent == folder            # => true
nested.root == maildir             # => true
maildir.parent                     # => nil

#folder? and #folder_name tell a folder from a root maildir. A maildir which merely lives at a dot-path, such as ~/.maildir, is correctly treated as a root and not as a folder of its parent directory.

maildir.folder? # => false
nested.folder?  # => true
nested.folder_name # => "a.x"

Pluggable Serializers

By default, message data are written and read from disk as a string. However, it may be desirable to automatically process strings into useful objects. This library supports configurable serializers to convert objects to strings and back.

The following serializers are included:

• Maildir::Serializer::Base (default - no serialization, writes and reads contents as string)
• Maildir::Serializer::JSON (uses #to_json and JSON#parse)
• Maildir::Serializer::YAML (uses #to_yaml and YAML#parse)

Maildir.serializer and Maildir.serializer= let you set the default serializer.

Maildir.serializer # => Maildir::Serializer::Base.new (default serializer - strings)
message = maildir.add("Hello, Crystal!") # writes "Hello, Crystal!" to disk
message.data # => "Hello, Crystal!"

You can also set the serializer per individual maildir:

maildir = Maildir.new("Maildir")
maildir.serializer = Maildir::Serializer::JSON.new

The JSON and YAML serializers work similarly, e.g.:

maildir.serializer = Maildir::Serializer::JSON.new
my_data = {"foo" => nil, "my_array" => [1, 2, 3]}
message = maildir.add(my_data) # writes {"foo":null,"my_array":[1,2,3]}
message.data == my_data # => true

It is trivial to create a custom serializer. Just implement the following two methods:

load(path)
dump(data, path)

Similar projects

• https://github.com/ktheory/maildir - Ruby implementation