4058
Comment:
|
6978
|
Deletions are marked like this. | Additions are marked like this. |
Line 1: | Line 1: |
= IMAP Server Compliance = |
= IMAP Server Compliancy Status = |
Line 4: | Line 3: |
* It would actually be better to be consistent across all IMAP commands, but especially when NOOP or IDLE is used. Some servers have inconsistent IMAP session states because the client connections end up in e.g. different servers that don't synchronize immediately. This can cause problems with clients that rely on the different sessions seeing the same state. | |
Line 5: | Line 5: |
* Atomic flags: Flags and keywords can be added/removed in multiple sessions without one session dropping changes made by other (own_flags test) | * imaptest prints an error if it notices multiple sessions having the same \Recent flag ("Message seq=.. UID=.. has \Recent flag in multiple sessions"). This happens automatically when running imaptest with multiple clients for the same user, e.g. `imaptest user=testuser clients=10` * But missing \Recent flags isn't detected with any tests * Atomic flags: Flags and keywords can be added/removed in multiple sessions without one session dropping changes made by other * Test by running stress test with own_flags parameter and multiple clients for the same user, e.g. `imaptest user=testuser clients=10 own_flags` - If no errors are printed then it's likely ok. |
Line 10: | Line 13: |
* This needs to be tested manually to see how it behaves - there is no ready imaptest test for this. Manual testing with IMAP: * Open two IMAP sessions and `SELECT INBOX` in both * Session 1: `STORE 1 +FLAGS \Deleted`, `EXPUNGE` * Session 2: `FETCH 1 BODYSTRUCTURE` -> Success = Cached * Session 2: `FETCH 1 BODY.PEEK[]` -> Success = Yes |
|
Line 16: | Line 24: |
* Failures: Number of failures using [:ImapTest/Tests:command/reply tests] (failures / number of tests). These numbers may not be exact all the time, because the tests are still changing. | * This needs to be tested manually to see how it behaves - there is no ready imaptest test for this. Manual testing with IMAP: * Open three IMAP sessions and `SELECT INBOX` in all * Session 1: `STORE 1 +FLAGS \Deleted`, `EXPUNGE` * Session 2: `STORE 1,2 +FLAGS \Seen` * Session 3: `FETCH 1,2 FLAGS` * If neither mail shows \Seen flag, run in Session 3 `NOOP` and `FETCH 1 FLAGS`. This should result in EXPUNGE notification and mail 2 showing \Seen flag. This result is `Delayed`. * Failures (failures/total): Number of failures using [[ImapTest/ScriptedTests|scripted tests]]. These numbers may not be exact all the time, because the tests are still changing. * Failure groups: Each test belongs to a wider group of tests, typically testing a command or part of a command. If this count is low but individual command failure count is high, it probably means that the server has failed to implement wrong only a couple of commands. * Base fails: Number of individual base IMAP4rev1 protocol commands that failed. * Ext fails: Number of individual IMAP extension commands that failed. Extensions not supported aren't included in the numbers. * Test using e.g. `imaptest user=testuser test=tests/` Fully compliant servers: ||'''Server''' ||'''Checkpoint''' ||'''\Recent''' ||'''Atomic flags''' ||'''Expunge fetch''' ||'''Expunge store''' ||'''Failure groups''' ||'''Base fails''' ||'''Ext fails''' || ||[[http://www.dovecot.org/|Dovecot]] ||Yes ||Yes ||Yes ||Yes / Cached (depends on storage) ||Yes ||0/40 ||0/403 ||0/100 || ||[[https://github.com/jonabbey/panda-imap/|Panda IMAP]] 2008, mix format ||Yes ||Yes ||Yes ||Yes ||Session ||0/34 ||0/328 ||0/97 || ||[[http://netwinsite.com/surgemail/|SurgeMail]] 5.0h3 ||Yes ||Yes ||Yes ||Yes ||Yes ||0/35 ||0/342 ||0/26 || |
Line 18: | Line 41: |
Fully compliant servers: | |
Line 20: | Line 42: |
|| '''Server''' || '''Checkpoint''' || '''\Recent''' || '''Atomic flags''' || '''Expunge fetch''' || '''Expunge store''' || '''Failures''' || || [http://www.dovecot.org/ Dovecot] 1.1.0+ || Yes || Yes || Yes || Cached || Yes || 0/34 || || [http://panda.com/imap/ Panda IMAP] 2008, mix format || Yes || Yes || Yes || Yes || Session || 0/34 || |
|
Line 25: | Line 45: |
||'''Server''' ||'''Checkpoint''' ||'''\Recent''' ||'''Atomic flags''' ||'''Expunge fetch''' ||'''Expunge store''' ||'''Failure groups''' ||'''Base fails''' ||'''Ext fails''' || ||[[http://www.washington.edu/imap/|UW-IMAP]] 2007b, mix format ||Yes ||Yes ||Yes ||Yes ||Session ||2/34 ||0/328 ||6/53 || ||[[http://www.isode.com/products/m-box.html|Isode M-Box]] 14.3a0 ||No ||Unreliable ||Yes ||Cached ||Ignored ||4/40 ||1/408 ||8/112 || ||[[http://www.communigate.com/community/|CommuniGate Pro]] 5.2.1 ||Yes ||Yes ||Yes ||Cached (some) ||No ||8/34 ||8/328 ||0/0 || ||[[https://www.cyrusimap.org/|Cyrus]] 3.0.10 ||No ||Unreliable ||Bugs ||Cached ||Delayed ||0/35 ||0/366 ||0/100 || ||[[http://www.sun.com/software/products/messaging_srvr/index.xml|Sun Java Messenging Server]] 6.3-0.15 ||No ||Unreliable ||Bugs ||Yes ||Delayed ||9/34 ||17/328 ||9/21 || ||[[http://www.archiveopteryx.org/|Archiveopteryx]] 3.0.3 ||No ||Unreliable ||No ||No ||No ||13/38 ||25/346 ||6/26 || ||[[http://www.courier-mta.org/imap/|Courier]] 4.3.1 ||Yes ||Unreliable ||Yes ||No ||No ||18/34 ||33/328 ||20/53 || ||[[http://www.gmail.com/|GMail]] 2012-08-12 ||No ||Not implemented ||Bugs ||Cached (some) ||Ignored ||12/49 ||66/376 ||0/0 || ||[[http://www.zarafa.com/|Zarafa Collaboration Platform]] 7.1.10 ||No ||Unreliable ||Yes ||No ||No ||6/35 ||14/340 ||0/0 || ||[[http://www.zimbra.com/|Zimbra]] 5.0.5 ||Yes ||Yes ||Yes ||Cached ||No ||8/34 ||50/328 ||2/33 || ||[[http://www.microsoft.com/exchange/default.mspx|MS Exchange]] ||No ||Unreliable ||? ||No ||No ||16/40 ||52/287 ||0/0 || ||[[http://www.citadel.org/|Citadel]] 7.36 ||No ||Unreliable ||Yes ||Yes ||Session (flag changes are never seen by other sessions) ||19/34 ||98/328 ||0/0 || ||[[http://www.hmailserver.com/|hMailServer]] 5.3.3 b1879 ||? ||? ||? ||? ||? ||||||<style="text-align: center;">Fails hardcoded OK response format test. Other tests don't work, if OK response test fails. || |
|
Line 26: | Line 60: |
|| '''Server''' || '''Checkpoint''' || '''\Recent''' || '''Atomic flags''' || '''Expunge fetch''' || '''Expunge store''' || '''Failures''' || || [http://www.dovecot.org/ Dovecot] 1.0.15 || Yes || Unreliable || Bugs || Cached || Yes || 1/34 || || [http://www.washington.edu/imap/ UW-IMAP] 2007b, mix format || Yes || Yes || Yes || Yes || Session || 2/34 || || [http://www.isode.com/products/m-box.html Isode M-Box] 14.3a0 || No || Unreliable || Yes || Cached || Ignored || 6/34 || || [http://www.communigate.com/community/ CommuniGate Pro] 5.2.1 || Yes || Yes || Yes || Cached (some) || No || 8/34 || || [http://www.zimbra.com/ Zimbra] 5.0.5 || Yes || Yes || Yes || Cached || No || 8/34 || || [http://cyrusimap.web.cmu.edu/ Cyrus] 2.3.12p2 || No || Unreliable || Bugs || Cached || Delayed || 9/34 || || [http://www.sun.com/software/products/messaging_srvr/index.xml Sun Java Messenging Server] 6.3-0.15 || No || Unreliable || Bugs || Yes || Delayed || 9/34 || || [http://www.gmail.com/ GMail] 2008-06-24 || No || Not implemented || Bugs || Yes || No (\Deleted flag expunges) || 15/34 || || [http://www.courier-mta.org/imap/ Courier] 4.3.1 || Yes || Unreliable || Yes || No || No || 18/34 || || [http://www.citadel.org/ Citadel] 7.36 || No || Unreliable || Yes || Yes || Session (flag changes are never seen by other sessions) || 19/34 || || [http://www.dbmail.org/ dbmail] 2.3.3 |||||| Crashes too early to test. APPEND missing EXISTS replies. || Yes || Delayed (always) || ? || |
|
Line 40: | Line 64: |
|| '''Server''' || '''Checkpoint''' || '''\Recent''' || '''Atomic flags''' || '''Expunge fetch''' || '''Expunge store''' || '''Failures''' || || [http://www.archiveopteryx.org/ Archiveopteryx] 3.0.0 || ? || ? || ? || No || No || EXPUNGEs are sent wrong || || [http://www.ibm.com/software/lotus/products/notes/ IBM Domino] 8.0 || No || Unreliable || ? || ? || No || 16/34 - Too many EXPUNGEs are sent, EXISTS is dropped before sending EXPUNGEs, FETCHing with valid messagesets produce errors || || [http://www.kerio.eu/kms_home.html Kerio Mail Server] 6.5.1 || ? || ? || ? || ? || ? || 18/34 - EXPUNGEs are sent wrong || || [http://drupal.zarafa.com/download-community Zarafa] 6.20 beta 3 || - || - || - || - || - || 19/34 - No EXPUNGEs are sent, EXISTS is used to drop message count || |
||'''Server''' ||'''Checkpoint''' ||'''\Recent''' ||'''Atomic flags''' ||'''Expunge fetch''' ||'''Expunge store''' ||'''Failures''' || ||[[http://www.dbmail.org/|dbmail]] 3.0.2 ||? ||Unreliable ||? ||Yes ||Yes || 9/361 - UID/sequence mapping becomes wrong || ||[[http://www.ibm.com/software/lotus/products/notes/|IBM Domino]] 8.0 ||No ||Unreliable ||? ||? ||No ||16/34 - Too many EXPUNGEs are sent, EXISTS is dropped before sending EXPUNGEs, FETCHing with valid messagesets produce errors || ||[[http://www.kerio.eu/kms_home.html|Kerio Mail Server]] 6.5.1 ||? ||? ||? ||? ||? ||18/34 - EXPUNGEs are sent wrong || ||[[http://www.axigen.com/|Axigen]] 7.1.2 ||? ||Unreliable ||Yes ||Broken ||Broken ||19/312 - FETCH/STORE sends EXPUNGEs immediately || |
IMAP Server Compliancy Status
- Checkpoint: checkpoint parameter works. When issuing a CHECK command in all sessions, their state looks identical.
- It would actually be better to be consistent across all IMAP commands, but especially when NOOP or IDLE is used. Some servers have inconsistent IMAP session states because the client connections end up in e.g. different servers that don't synchronize immediately. This can cause problems with clients that rely on the different sessions seeing the same state.
- \Recent: Exactly one session sees a new message as \Recent - no more and no less.
imaptest prints an error if it notices multiple sessions having the same \Recent flag ("Message seq=.. UID=.. has \Recent flag in multiple sessions"). This happens automatically when running imaptest with multiple clients for the same user, e.g. imaptest user=testuser clients=10
- But missing \Recent flags isn't detected with any tests
- Atomic flags: Flags and keywords can be added/removed in multiple sessions without one session dropping changes made by other
Test by running stress test with own_flags parameter and multiple clients for the same user, e.g. imaptest user=testuser clients=10 own_flags - If no errors are printed then it's likely ok.
- Expunge fetch: If message is expunged in one session, can another session that still sees the message fetch its contents?
- Yes: Yes, everything can be fetched
- Cached: Only cached data can be fetched. Message header or body can't be.
- No: Nothing but flags can be fetched.
- This needs to be tested manually to see how it behaves - there is no ready imaptest test for this. Manual testing with IMAP:
Open two IMAP sessions and SELECT INBOX in both
Session 1: STORE 1 +FLAGS \Deleted, EXPUNGE
Session 2: FETCH 1 BODYSTRUCTURE -> Success = Cached
Session 2: FETCH 1 BODY.PEEK[] -> Success = Yes
- Expunge store: If message is expunged in one session, can another session update its flags?
- Yes: Yes, and a 3rd session also sees the change
- Session: Yes, but a 3rd session won't see the change
Delayed: Yes, but a 3rd session won't see any changes until EXPUNGEs are reported
- Ignored: STORE command replies OK, but no change is made
- No: STORE command replies NO
- This needs to be tested manually to see how it behaves - there is no ready imaptest test for this. Manual testing with IMAP:
Open three IMAP sessions and SELECT INBOX in all
Session 1: STORE 1 +FLAGS \Deleted, EXPUNGE
Session 2: STORE 1,2 +FLAGS \Seen
Session 3: FETCH 1,2 FLAGS
If neither mail shows \Seen flag, run in Session 3 NOOP and FETCH 1 FLAGS. This should result in EXPUNGE notification and mail 2 showing \Seen flag. This result is Delayed.
Failures (failures/total): Number of failures using scripted tests. These numbers may not be exact all the time, because the tests are still changing.
- Failure groups: Each test belongs to a wider group of tests, typically testing a command or part of a command. If this count is low but individual command failure count is high, it probably means that the server has failed to implement wrong only a couple of commands.
- Base fails: Number of individual base IMAP4rev1 protocol commands that failed.
- Ext fails: Number of individual IMAP extension commands that failed. Extensions not supported aren't included in the numbers.
Test using e.g. imaptest user=testuser test=tests/
Fully compliant servers:
Server |
Checkpoint |
\Recent |
Atomic flags |
Expunge fetch |
Expunge store |
Failure groups |
Base fails |
Ext fails |
Yes |
Yes |
Yes |
Yes / Cached (depends on storage) |
Yes |
0/40 |
0/403 |
0/100 |
|
Panda IMAP 2008, mix format |
Yes |
Yes |
Yes |
Yes |
Session |
0/34 |
0/328 |
0/97 |
SurgeMail 5.0h3 |
Yes |
Yes |
Yes |
Yes |
Yes |
0/35 |
0/342 |
0/26 |
Non-compliant servers:
Server |
Checkpoint |
\Recent |
Atomic flags |
Expunge fetch |
Expunge store |
Failure groups |
Base fails |
Ext fails |
UW-IMAP 2007b, mix format |
Yes |
Yes |
Yes |
Yes |
Session |
2/34 |
0/328 |
6/53 |
Isode M-Box 14.3a0 |
No |
Unreliable |
Yes |
Cached |
Ignored |
4/40 |
1/408 |
8/112 |
CommuniGate Pro 5.2.1 |
Yes |
Yes |
Yes |
Cached (some) |
No |
8/34 |
8/328 |
0/0 |
Cyrus 3.0.10 |
No |
Unreliable |
Bugs |
Cached |
Delayed |
0/35 |
0/366 |
0/100 |
Sun Java Messenging Server 6.3-0.15 |
No |
Unreliable |
Bugs |
Yes |
Delayed |
9/34 |
17/328 |
9/21 |
Archiveopteryx 3.0.3 |
No |
Unreliable |
No |
No |
No |
13/38 |
25/346 |
6/26 |
Courier 4.3.1 |
Yes |
Unreliable |
Yes |
No |
No |
18/34 |
33/328 |
20/53 |
GMail 2012-08-12 |
No |
Not implemented |
Bugs |
Cached (some) |
Ignored |
12/49 |
66/376 |
0/0 |
No |
Unreliable |
Yes |
No |
No |
6/35 |
14/340 |
0/0 |
|
Zimbra 5.0.5 |
Yes |
Yes |
Yes |
Cached |
No |
8/34 |
50/328 |
2/33 |
No |
Unreliable |
? |
No |
No |
16/40 |
52/287 |
0/0 |
|
Citadel 7.36 |
No |
Unreliable |
Yes |
Yes |
Session (flag changes are never seen by other sessions) |
19/34 |
98/328 |
0/0 |
hMailServer 5.3.3 b1879 |
? |
? |
? |
? |
? |
Fails hardcoded OK response format test. Other tests don't work, if OK response test fails. |
Major problems with multiple connections (makes further testing difficult and MAY CAUSE ACCIDENTAL MAIL LOSS!):
Server |
Checkpoint |
\Recent |
Atomic flags |
Expunge fetch |
Expunge store |
Failures |
dbmail 3.0.2 |
? |
Unreliable |
? |
Yes |
Yes |
9/361 - UID/sequence mapping becomes wrong |
IBM Domino 8.0 |
No |
Unreliable |
? |
? |
No |
16/34 - Too many EXPUNGEs are sent, EXISTS is dropped before sending EXPUNGEs, FETCHing with valid messagesets produce errors |
Kerio Mail Server 6.5.1 |
? |
? |
? |
? |
? |
18/34 - EXPUNGEs are sent wrong |
Axigen 7.1.2 |
? |
Unreliable |
Yes |
Broken |
Broken |
19/312 - FETCH/STORE sends EXPUNGEs immediately |