A Replacement Protocol

IMAP has been extended by very many RFCs, of which servers implement random parts, and clients either have to special case all of them, or just use the most simplistic protocol. The "IMAP5" mailing list didn't get anywhere.

I think it's time to get serious about an IMAP replacement.

The eventual goal here is "be compelling enough that the effort required to change is worthwhile". This includes server implementors, client implementors, and end users. If we're going to make an open protocol to replace IMAP, it had better give everyone a reason to switch.

I would like to use this page to store our discussion.

Requirements:

• client and server should be able to present existing interfaces/talk to existing storage with minimal changes and without breaking existing semantics. You switch to this protocol and you get benefits, yay. Not downsides/caveats.
• A few implementations - more than one client / server so we don't get too specific to how one system does things.
• Decent test suite for server implementations, and test plan for clients - "X happens on server, you must present world-state Y on action Z" - so implementors can be sure of the quality of their implementations. Not just a wordy spec.

Some areas of interest:

LIST/LSUB/Special-Use:

• Initial setup - detect the correct folders in clients
• Changes - create, delete, "currently selected folder is deleted". Inform the client or allow them to find out about changes cheaply.
• Wider question: folders vs "tags". "Select" vs filters. * Namespaces? * LIST syntax - major cause of bugs in both Cyrus and Dovecot - complex code. The problem isn't that complicated, we must be doing something wrong.
• "special folders" containing extra data like Calendar entries, etc - not just normal emails.

Message management:

• MOVE vs "COPY + expunge".
• Deletion - in-place set "\Deleted" flag vs copy to trash.
• Undo.
• Efficiently detect actions taken by another client. If another client copies 100,000 messages from Inbox to Archive, don't have to re-download. GUID or similar.
• Batching / pipelines. SEARCH + MARK FLAGGED + MOVE TO ANOTHER MAILBOX - basically, "lego blocks" vs "pre-defined" - leads on to;

Transactions:

• tell the server all the things you want: pipeline vs combined command.
• combined command: can posix_fadvise interest in all the necessary files, combine locks.
• Question: do we want full transactional semantics? Great for clients, hard for servers. What sort of changes can you make part of a transactions? Folder rename plus expunge plus ???

Implicit vs Explicit:

• FETCH BODY - may set seen flag - need to check for side effects.
• SELECT "UNSEEN X" response - have to calculate even if client doesn't care. A lot of the worst parts of IMAP come from implicit magic that helped a use-case nobody wants any more.
• My analogy - "a star-fighter lego kit" rather than "lego blocks". If you want a star fighter, great. If not, you wind up building the ugliest looking set of mis-shapen blocks, because they're not as generic as they should be.

Stateless Operation:

• Phones / poorly connected devices
• Power usage considerations.
• Dropped connections/changing IPs
• High availability/failover.

• With HTTP (at FastMail) we mark a server down and wait 15 seconds, then watch for TCP disconnections. Frontend has moved traffic elsewhere, ZERO user-visible outage. With IMAP, connections dropped, visible. Silent reconnection and "stateless" simplifies many things.

• Idempotent changes - replay is always safe.

Notifications:

• Able to easily receive notifications about ALL changes of interest, emails / folders / whatever.
• Notifications still work if connection disconnected (see above) - can recover state by e.g. modseq.
• Compatible with out-of-band notification to do cheap resync (use OS remote notification system in case of phone, etc) - if present. Even SMS.

Bandwidth-wise:

• Don't waste bandwidth stupidly.

Consistent Synax:

• It should be possible to easily build a proxy which undersands the traffic flow and can inspect/modify individual known items in the traffic without needing to understand the contents of everything.
• This means regular syntax, at least at the top level.
• Potential to be more "transport agnostic" - direct translations to/from XML/JSON/whatever.

• UTF8 throughout, obviously

A lot of these is the CISC vs RISC debate. I believe it's better to compose your messages from client to server and server to client out of groups of small "lego bricks" each of which expresses one thing succinctly rather than pre-formed "fighter wing" shapes. The biggest lack I see in the current email landscape is that that IMAP clients wind up doing convoluted things to support all the possible combinations of multiple RFCs out there, or just giving up and supporting a very simple profile, because that way they don't need multiple codepaths.

If we can agree on a more expressive communication protocol between the clients and servers, I don't think we need to change the model at either end very far.

The goal, again, is to provide a complelling _experience_ for users of this protocol - that accessing your email via this protocol is a better, more reliable, more predictable experience. That's how we win hearts and minds, and make others want to implement the protocl

Thanks,

Bron.