Differences between revisions 1 and 60 (spanning 59 versions)
Revision 1 as of 2007-12-01 13:19:45
Size: 2732
Editor: TimoSirainen
Comment:
Revision 60 as of 2008-02-16 21:21:09
Size: 11958
Editor: TimoSirainen
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
= IMAP Tester =

imaptest was originally written for stress testing [http://www.dovecot.org/ Dovecot] IMAP server. Its primary testing method is still stress testing, but now it also parses the incoming IMAP replies and complains if anything looks broken.
= IMAP Server Tester =

[[TableOfContents]]

imaptest was originally written for stress testing [http://www.dovecot.org/ Dovecot] IMAP server. Its primary testing method is still stress testing, but now it also parses the incoming IMAP replies and complains if anything looks broken. imaptest works with any IMAP server.
Line 7: Line 9:
 1. Download and compile Dovecot v1.1 sources. imaptest uses its library functions.  1. Download and compile Dovecot v1.1 sources. imaptest uses its library functions. Dovecot doesn't have any external dependencies.
Line 10: Line 12:
 1. Download http://www.dovecot.org/tools/imaptest.c to Dovecot sources' root directory (i.e. where the {{{configure}}} script exists)
 1. imaptest's header comment shows the compile command for gcc, use it or something similar.

You may want to modify the default configuration from #defines at the top of the file. This isn't required, but if you run imaptest often this way you don't have to give the same parameters every time.
 1. Download and compile http://dovecot.org/nightly/imaptest/imaptest-latest.tar.gz
  * {{{./configure --with-dovecot=../dovecot-1.1 && make}}}
  * {{{--with-dovecot=<path>}}} parameter is used to specify path to Dovecot v1.1 sources' root directory.
 1. Either {{{make install}}} or run {{{src/imaptest}}} directly.

You may want to modify the default configuration from #defines in {{{src/settings.h}}}. This isn't required, but if you run imaptest often, this way you don't have to give the same parameters every time.

== Mercurial repository ==

You can also get imaptest sources from [http://www.selenic.com/mercurial/ Mercurial] repository:

{{{
hg clone http://hg.dovecot.org/imaptest/
}}}

You can update it later with:

{{{
hg pull
hg update
}}}

You can see the latest changes using Mercurial's [http://hg.dovecot.org/imaptest/ web interface].
Line 20: Line 41:
 * user=tss: Username template. You can use multiple random users and domains by giving %d in the template. So for example {{{user%d}}} returns user1..user100 and {{{user%d@domain%d.org}}} returns user1..100@domain1..100.org. The 1..100 range is hardcoded to the binary. You can change them from {{{USER_RAND}}} and {{{DOMAIN_RAND}}} defines.  * user=$USER: Username template. You can use multiple random users and domains by giving %d in the template. So for example {{{user%d}}} returns user1..user100 and {{{user%d@domain%d.org}}} returns user1..100@domain1..100.org. The 1..100 range is hardcoded to the binary. You can change them from {{{USER_RAND}}} and {{{DOMAIN_RAND}}} defines.
Line 22: Line 43:
 * mbox=/home/tss/mail/dovecot-crlf: Path to mbox file where to append messages from. See below for how this is used.  * mbox=~/mail/dovecot-crlf: Path to mbox file where to append messages from. See below for how this is used.
Line 27: Line 48:
 * msgs=30: Try to keep the mailbox size around this many messages.
Line 32: Line 54:
 * rawlog: Write rawlog.* files for all connections containing their input and output.
Line 36: Line 59:
 * own_msgs: Assigns an owner client for each message. Complain if flags for a message is changed by a non-owner session.
 * own_flags: Assigns an owner client for each flag and keyword. Complain if they're changed by another session.
Line 38: Line 63:

== Append mbox ==

When saving messages, imaptest needs to get the messages from somewhere. mbox=path parameter specifies path to a file in mbox format that's used. Messages are sequentially appended from there. Once imaptest reaches the last message, it wraps back to appending the first message.

Currently imaptest's state tracking expects that Message-IDs are unique within the mbox, otherwise it gives bogus errors. If you really want to avoid changing the Message-IDs, use no_tracking setting to disable state tracking.

You can get a test mbox file from for example http://dovecot.org/tmp/dovecot-crlf. It's a 10MB file containing messages from Dovecot mailing list with unique Message-ID headers.

== States ==

States can be specified using full or short names. You specify the probabilities in percents, so for example {{{list=30}}} means that LIST command is executed with 30% probability.

The second probability is mainly useful for APPEND, where it controls how often to append multiple messages with MULTIAPPEND extension, so for example {{{append=80,30}}} means that APPEND command is executed with 80% probability, and after each appended message there's a 30% chance of the APPEND command continuing.

You can disable all except LOGIN, LOGOUT and SELECT states by giving "-" parameter. For example: {{{./imaptest - select=0 append=100,0 logout=0}}} will do nothing but a LOGIN followed by APPENDs.

|| Name || Short name || Default% || Description ||
|| AUTHENTICATE || Auth || 0 || Authentication with AUTHENTICATE PLAIN command ||
|| LOGIN || Logi || 100 || Authentication with LOGIN command ||
|| LIST || List || 50 || LIST "" * ||
|| MCREATE || LCre || 0 || CREATE test/x/y mailboxes randomly. '/' separator is hardcoded currently. ||
|| MDELETE || LDel || 0 || DELETE test/x/y mailboxes randomly. ||
|| STATUS || Stat || 50 || STATUS (MESSAGES UNSEEN RECENT) ||
|| SELECT || Sele || 100 || SELECT mailbox (required for most states below) ||
|| FETCH || Fetc || 100 || FETCH n:m (random fields) where n:m is a range with random start for 100 messages (or all messages if mailbox has less than 100 messages). Randomly fetched fields are: UID, FLAGS, ENVELOPE, INTERNALDATE, BODY, BODYSTRUCTURE and BODY.PEEK[HEADER.FIELDS (random headers)]. Random headers are: From, To, Cc, Subject, Message-ID, In-Reply-To, References, Delivered-To. ||
|| FETCH2 || Fet2 || 100,30 || FETCH n (BODY.PEEK[]) where n is a random message ||
|| SEARCH || Sear || 0 || SEARCH BODY hello ||
|| SORT || Sort || 0 || SORT (SUBJECT) US-ASCII for ALL or FLAGGED randomly ||
|| THREAD || Thre || 0 || THREAD REFERENCES US-ASCII ALL ||
|| COPY || Copy || 33,5 || COPY random number of messages ||
|| STORE || Stor || 50 || STORE <random-range> [+-]FLAGS[.SILENT] <random flags and keywords>. Only $Label1..5 are used as keywords. SILENT is used if checkpointing is disabled. \Deleted flags aren't set. ||
|| DELETE || Dele || 100 || STORE <random-range> +FLAGS[.SILENT] \Deleted ||
|| EXPUNGE || Expu || 100 || EXPUNGE ||
|| APPEND || Appe || 100,5 || APPEND messages to mailbox. MULTIAPPEND extension is used if possible. The counter shows number of APPEND commands, so with MULTIAPPEND it doesn't match the actual number of appended messages. ||
|| NOOP || Noop || 0 || NOOP command. ||
|| CHECK || Chec || 0 || CHECK command. ||
|| LOGOUT || Logo || 100 || LOGOUT command. ||
|| DISCONNECT || Disc || 0 || Disconnect without LOGOUT. ||
|| DELAY || Dela || 0 || 1 second delay ||
|| CHECKPOINT! || ChkP || 0 || Use checkpoint parameter to change this. The counter shows number of client connections successfully checkpointed ||

== State tracking ==

imaptest should catch the following errors:

 * Referring to sequences that haven't been announced with EXISTS.
 * Sequence <-> UID map changes unexpectedly
 * Message's static metadata changes unexpectedly. Messages are uniquely identified by their Message-ID: header.
  * BODY, BODYSTRUCTURE, ENVELOPE and RFC822.SIZE are currently tracked.
  * BODY[HEADER.FIELDS (..)] are tracked for some headers.
  * IMAP servers handle FETCHes for expunged messages differently. imaptest tries to catch these.
 * Message's INTERNALDATE changes unexpectedly.
 * FETCH FLAGS listing keywords that haven't been announced with untagged FLAGS.
 * Untagged FLAGS reply dropping a keyword that's still in use.
 * Flags or keywords changed unexpectedly (when own_msgs enabled)
 * Non-atomic flag or keyword updates. For example if session 1 does "STORE +FLAGS \Seen" and session 2 does "STORE +FLAGS \Draft" at the same time, with some servers the result may be either \Seen or \Draft instead of both. Testing this requires enabling own_flags.

Checkpointing works by letting all the pending commands finish. Then CHECK command is sent to all sessions. Once they're done, imaptest verifies that all clients' mailbox state looks exactly the same:

 * Number of messages is the same
 * Sequence <-> UID map is the same
 * Flags and keywords are the same
 * \Recent flag for a message exists in only one session (all mailboxes are SELECTed currently)
 * \Recent flag for a message has never existed. This works only with logout=0 and only if we know flags for all messages (so fetch is required also). Expunging less often (e.g. expunge=10) makes this check work better.
 * Summing up RECENT count from all sessions matches the message count. This is done only if at some point one session's RECENT count has matched message count. imaptest tries to make this happen by expunging all existing messages from mailbox.

== Examples ==

Basics:

{{{
./imaptest host=127.0.0.1 port=143 user=testuser password=testpass mbox=dovecot.mbox
}}}

Test IMAP server compliancy:

{{{
./imaptest checkpoint=1
./imaptest checkpoint=1 logout=0 expunge=10
./imaptest checkpoint=1 rawlog no_pipelining
./imaptest own_msgs expunge=5 logout=1
./imaptest own_flags expunge=5 logout=1
}}}

If you want to do benchmarking, you can set it to run for specified amount of time, and you should also give random number generator the same seed every time:

{{{
./imaptest seed=123 secs=300
}}}

If you want to test/benchmark only the login+logout speed, use:

{{{
./imaptest - select=0
}}}

To create a lot of long-running clients doing STATUS and NOOPs:

{{{
./imaptest clients=100 - logout=0 status=50 noop=50 delay=100
}}}

Benchmarking how fast messages can be saved:

{{{
./imaptest - append=100,0 logout=0 msgs=10000000
}}}

To test copying messages:

{{{
./imaptest copybox=Trash
}}}

By default LOGIN command is used. If you want to try AUTHENTICATE PLAIN:

{{{
./imaptest auth=100
}}}

== Server Status ==

 * Checkpoint: checkpoint parameter works. When issuing a CHECK command in all sessions, their state looks identical.
 * \Recent: Exactly one session sees a new message as \Recent - no more and no less.
 * Atomic flags: Flags and keywords can be added/removed in multiple sessions without one session dropping changes made by other (own_flags test)
 * Expunges: How messages expunged by another session are handled.
  * A: Expunging a message from one session keeps it available for other sessions until they can be notified that it's expunged. Flag updates to expunged messages may not be visible to other sessions though.
  * B: Some cached fields may be returned for expunged messages, but message bodies can't be retrieved anymore. Flags can still be updated within the session (but updates may not be visible to other sessions).
  * C: Even message flags can't be updated for expunged messages.

|| '''Server''' || '''Checkpoint''' || '''\Recent''' || '''Atomic flags''' || '''Expunges''' ||
|| Dovecot 1.0 || Yes || Unreliable || Bugs || B ||
|| Dovecot 1.1.beta14+ || Yes || Yes || Yes (mbox broken) || B ||
|| UW-IMAP 2007, mix format || Yes || Yes || Bugs || A ||
|| UW-IMAP 2007a.DEV, mix format || Yes || Yes || Yes || A ||
|| Courier 4.3.0 || Yes || Unreliable || Yes || C ||
|| Cyrus 2.3.9 || No || Unreliable || Bugs || B? Buggy ||
|| Isode M-Box 14.2a0 || No || Unreliable || Yes || C ||
|| Zimbra 5.0.1 || Yes || Yes || Yes || C ||
|| Communigate Pro 5.2.0 |||||||||| Fails IMAP state checks in multiple ways ||

IMAP Server Tester

TableOfContents

imaptest was originally written for stress testing [http://www.dovecot.org/ Dovecot] IMAP server. Its primary testing method is still stress testing, but now it also parses the incoming IMAP replies and complains if anything looks broken. imaptest works with any IMAP server.

Compiling

  1. Download and compile Dovecot v1.1 sources. imaptest uses its library functions. Dovecot doesn't have any external dependencies.
    1. Get the latest beta from http://dovecot.org/releases/1.1/beta/

    2. Compiling goes the usual way: ./configure && make (there's no need for make install)

  2. Download and compile http://dovecot.org/nightly/imaptest/imaptest-latest.tar.gz

    • ./configure --with-dovecot=../dovecot-1.1 && make

    • --with-dovecot=<path> parameter is used to specify path to Dovecot v1.1 sources' root directory.

  3. Either make install or run src/imaptest directly.

You may want to modify the default configuration from #defines in src/settings.h. This isn't required, but if you run imaptest often, this way you don't have to give the same parameters every time.

Mercurial repository

You can also get imaptest sources from [http://www.selenic.com/mercurial/ Mercurial] repository:

hg clone http://hg.dovecot.org/imaptest/

You can update it later with:

hg pull
hg update

You can see the latest changes using Mercurial's [http://hg.dovecot.org/imaptest/ web interface].

Running

The most important parameters (and their defaults) are:

  • host=127.0.0.1: Host name/IP address where to connect.
  • port=143: Port what to use.
  • user=$USER: Username template. You can use multiple random users and domains by giving %d in the template. So for example user%d returns user1..user100 and user%d@domain%d.org returns user1..100@domain1..100.org. The 1..100 range is hardcoded to the binary. You can change them from USER_RAND and DOMAIN_RAND defines.

  • password=pass: Password to use for all users. There's currently no way to use different passwords for different users.
  • mbox=~/mail/dovecot-crlf: Path to mbox file where to append messages from. See below for how this is used.

Other useful parameters include:

  • clients=10: Number of simultaneous client connections to use.
  • secs=n: Run imaptest n seconds and then exit.
  • msgs=30: Try to keep the mailbox size around this many messages.
  • box=INBOX: Mailbox to use for testing. INBOX is the default.
  • copybox=n: When testing COPY command, this specifies the destination mailbox.
  • seed=n: Seed to use for random generator. Setting this to some specific value makes repeated benchmarks a bit more reliable, because the used commands should be the same then.
  • disconnect_quit: If a client gets disconnected, quit. Use logout=0 parameter with this one. This is useful when debugging some problem in the server.
  • no_pipelining: Don't send multiple commands at once to server.
  • rawlog: Write rawlog.* files for all connections containing their input and output.

Selecting what to test:

  • <state>=<probability>[,<probability2>]: State probabilities to use. See below.

  • checkpoint=n: Run a checkpoint every n seconds. See below.
  • own_msgs: Assigns an owner client for each message. Complain if flags for a message is changed by a non-owner session.
  • own_flags: Assigns an owner client for each flag and keyword. Complain if they're changed by another session.
  • random: Switch randomly between states.
  • no_tracking: Don't track and complain about IMAP state. Makes it use a bit less CPU.

Append mbox

When saving messages, imaptest needs to get the messages from somewhere. mbox=path parameter specifies path to a file in mbox format that's used. Messages are sequentially appended from there. Once imaptest reaches the last message, it wraps back to appending the first message.

Currently imaptest's state tracking expects that Message-IDs are unique within the mbox, otherwise it gives bogus errors. If you really want to avoid changing the Message-IDs, use no_tracking setting to disable state tracking.

You can get a test mbox file from for example http://dovecot.org/tmp/dovecot-crlf. It's a 10MB file containing messages from Dovecot mailing list with unique Message-ID headers.

States

States can be specified using full or short names. You specify the probabilities in percents, so for example list=30 means that LIST command is executed with 30% probability.

The second probability is mainly useful for APPEND, where it controls how often to append multiple messages with MULTIAPPEND extension, so for example append=80,30 means that APPEND command is executed with 80% probability, and after each appended message there's a 30% chance of the APPEND command continuing.

You can disable all except LOGIN, LOGOUT and SELECT states by giving "-" parameter. For example: ./imaptest - select=0 append=100,0 logout=0 will do nothing but a LOGIN followed by APPENDs.

Name

Short name

Default%

Description

AUTHENTICATE

Auth

0

Authentication with AUTHENTICATE PLAIN command

LOGIN

Logi

100

Authentication with LOGIN command

LIST

List

50

LIST "" *

MCREATE

LCre

0

CREATE test/x/y mailboxes randomly. '/' separator is hardcoded currently.

MDELETE

LDel

0

DELETE test/x/y mailboxes randomly.

STATUS

Stat

50

STATUS (MESSAGES UNSEEN RECENT)

SELECT

Sele

100

SELECT mailbox (required for most states below)

FETCH

Fetc

100

FETCH n:m (random fields) where n:m is a range with random start for 100 messages (or all messages if mailbox has less than 100 messages). Randomly fetched fields are: UID, FLAGS, ENVELOPE, INTERNALDATE, BODY, BODYSTRUCTURE and BODY.PEEK[HEADER.FIELDS (random headers)]. Random headers are: From, To, Cc, Subject, Message-ID, In-Reply-To, References, Delivered-To.

FETCH2

Fet2

100,30

FETCH n (BODY.PEEK[]) where n is a random message

SEARCH

Sear

0

SEARCH BODY hello

SORT

Sort

0

SORT (SUBJECT) US-ASCII for ALL or FLAGGED randomly

THREAD

Thre

0

THREAD REFERENCES US-ASCII ALL

COPY

Copy

33,5

COPY random number of messages

STORE

Stor

50

STORE <random-range> [+-]FLAGS[.SILENT] <random flags and keywords>. Only $Label1..5 are used as keywords. SILENT is used if checkpointing is disabled. \Deleted flags aren't set.

DELETE

Dele

100

STORE <random-range> +FLAGS[.SILENT] \Deleted

EXPUNGE

Expu

100

EXPUNGE

APPEND

Appe

100,5

APPEND messages to mailbox. MULTIAPPEND extension is used if possible. The counter shows number of APPEND commands, so with MULTIAPPEND it doesn't match the actual number of appended messages.

NOOP

Noop

0

NOOP command.

CHECK

Chec

0

CHECK command.

LOGOUT

Logo

100

LOGOUT command.

DISCONNECT

Disc

0

Disconnect without LOGOUT.

DELAY

Dela

0

1 second delay

CHECKPOINT!

ChkP

0

Use checkpoint parameter to change this. The counter shows number of client connections successfully checkpointed

State tracking

imaptest should catch the following errors:

  • Referring to sequences that haven't been announced with EXISTS.
  • Sequence <-> UID map changes unexpectedly

  • Message's static metadata changes unexpectedly. Messages are uniquely identified by their Message-ID: header.
    • BODY, BODYSTRUCTURE, ENVELOPE and RFC822.SIZE are currently tracked.
    • BODY[HEADER.FIELDS (..)] are tracked for some headers.
    • IMAP servers handle FETCHes for expunged messages differently. imaptest tries to catch these.
  • Message's INTERNALDATE changes unexpectedly.
  • FETCH FLAGS listing keywords that haven't been announced with untagged FLAGS.
  • Untagged FLAGS reply dropping a keyword that's still in use.
  • Flags or keywords changed unexpectedly (when own_msgs enabled)
  • Non-atomic flag or keyword updates. For example if session 1 does "STORE +FLAGS \Seen" and session 2 does "STORE +FLAGS \Draft" at the same time, with some servers the result may be either \Seen or \Draft instead of both. Testing this requires enabling own_flags.

Checkpointing works by letting all the pending commands finish. Then CHECK command is sent to all sessions. Once they're done, imaptest verifies that all clients' mailbox state looks exactly the same:

  • Number of messages is the same
  • Sequence <-> UID map is the same

  • Flags and keywords are the same
  • \Recent flag for a message exists in only one session (all mailboxes are SELECTed currently)
  • \Recent flag for a message has never existed. This works only with logout=0 and only if we know flags for all messages (so fetch is required also). Expunging less often (e.g. expunge=10) makes this check work better.
  • Summing up RECENT count from all sessions matches the message count. This is done only if at some point one session's RECENT count has matched message count. imaptest tries to make this happen by expunging all existing messages from mailbox.

Examples

Basics:

./imaptest host=127.0.0.1 port=143 user=testuser password=testpass mbox=dovecot.mbox

Test IMAP server compliancy:

./imaptest checkpoint=1
./imaptest checkpoint=1 logout=0 expunge=10
./imaptest checkpoint=1 rawlog no_pipelining
./imaptest own_msgs expunge=5 logout=1
./imaptest own_flags expunge=5 logout=1

If you want to do benchmarking, you can set it to run for specified amount of time, and you should also give random number generator the same seed every time:

./imaptest seed=123 secs=300

If you want to test/benchmark only the login+logout speed, use:

./imaptest - select=0

To create a lot of long-running clients doing STATUS and NOOPs:

./imaptest clients=100 - logout=0 status=50 noop=50 delay=100

Benchmarking how fast messages can be saved:

./imaptest - append=100,0 logout=0 msgs=10000000

To test copying messages:

./imaptest copybox=Trash

By default LOGIN command is used. If you want to try AUTHENTICATE PLAIN:

./imaptest auth=100

Server Status

  • Checkpoint: checkpoint parameter works. When issuing a CHECK command in all sessions, their state looks identical.
  • \Recent: Exactly one session sees a new message as \Recent - no more and no less.
  • Atomic flags: Flags and keywords can be added/removed in multiple sessions without one session dropping changes made by other (own_flags test)
  • Expunges: How messages expunged by another session are handled.
    • A: Expunging a message from one session keeps it available for other sessions until they can be notified that it's expunged. Flag updates to expunged messages may not be visible to other sessions though.
    • B: Some cached fields may be returned for expunged messages, but message bodies can't be retrieved anymore. Flags can still be updated within the session (but updates may not be visible to other sessions).
    • C: Even message flags can't be updated for expunged messages.

Server

Checkpoint

\Recent

Atomic flags

Expunges

Dovecot 1.0

Yes

Unreliable

Bugs

B

Dovecot 1.1.beta14+

Yes

Yes

Yes (mbox broken)

B

UW-IMAP 2007, mix format

Yes

Yes

Bugs

A

UW-IMAP 2007a.DEV, mix format

Yes

Yes

Yes

A

Courier 4.3.0

Yes

Unreliable

Yes

C

Cyrus 2.3.9

No

Unreliable

Bugs

B? Buggy

Isode M-Box 14.2a0

No

Unreliable

Yes

C

Zimbra 5.0.1

Yes

Yes

Yes

C

Communigate Pro 5.2.0

Fails IMAP state checks in multiple ways

None: ImapTest (last edited 2023-11-07 21:14:43 by TimoSirainen)