Simple comparison benchmark for repositories (rel: JAMES-393)

[Stefano Bagnara <ap...@bago.org>]

Hi all,

I set up a benchmark consisting in:
1) create 1200 users in the db
2) start 20 sender threads: each one make continuos connections and each
time send 1 to 5 messages sized 0 to 1000 bytes to a random user (from the
1200). [I used postal for this task]
3) start 10 reader threads: each one make continuos connections and each
time read and delete a maximum of 10 messages for a random user (from the
1200). [I used rabid for this task]
10 minutes testbed.

I ran the tester application from a linux machine connected via 100mbit
ethernet to my own laptop (winXP) running james.

I tested the following configurations:
file repositories
derby repositories
mysql repositories (mysql installed in the same machine)
hsqldb repositories

I checked the following running data:
Messages sent
Messages read
Size of the spool at the end of the test
Maximum Memory used by (james)
Errors
I/O operations
I/O bytes.

       hsqldb mysql derby  file
sent    16488 16528 16218 12636
read    10584 10685 10309  6228
spool       0     0     0  4145
mem      120M   40M   90M   45M
input    100K  100K   90K 5000K
output   100K  100K  150K 1000K
ibytes   70MB   70M   75M   70M
obytes  180MB   30M  300M   70M

Problems:

*file repositories*
I got 5 times this error (probably a BUG in avalon file repository locking):
pop3server: Unable to STAT mail box 
javax.mail.MessagingException: Exception while retrieving mail: Cannot lock
Mail1123576906293-494 to remove it
	at
org.apache.james.mailrepository.AvalonMailRepository.retrieve(AvalonMailRepo
sitory.java:357)
	at
org.apache.james.pop3server.POP3Handler.stat(POP3Handler.java:387)
	at
org.apache.james.pop3server.POP3Handler.doPASS(POP3Handler.java:535)
	at
org.apache.james.pop3server.POP3Handler.parseCommand(POP3Handler.java:467)
	at
org.apache.james.pop3server.POP3Handler.handleConnection(POP3Handler.java:26
3)
	at
org.apache.james.util.connection.ServerConnection$ClientConnectionRunner.run
(ServerConnection.java:415)
	at
org.apache.excalibur.thread.impl.ExecutableRunnable.execute(ExecutableRunnab
le.java:90)
	at
org.apache.excalibur.thread.impl.WorkerThread.run(WorkerThread.java:136)

*hsqldb*
I've got a 1 time crash using hypersonic, not sure what happened.

*derby*
I've got 1 "Server timed out on read" during the derby test.


We surely cannot use such a simple test as a "rule" to the fastest solution
but here are some considerations:
1) file repositories are the less expensive in memory usage but the
performance are poor and I hit a bug 5 time in 10 minutes! Comparing to
mysql I had similar memory usage (ok you need mysql running) and really
better performance/reliability.
2) mysql is the fastest one but considering the I/O of mysqld is the one
that makes more use of the filesystem. Memory usage for james+mysqld was
less than james with derby or james with hsqldb.
3) mysql was not working with the default org.gjt.mm.mysql bundled driver
(report an authentication issues but in fact the protocol has changed)
4) looking at the few differences switching database I thought that most of
the cpu is used by james itself and not by the db in this test.
5) hsqldb jars are really smaller than derby (500K vs 2300K) and has similar
performance: at a first glance I found derby more reliable (hsqldb crashed
one time)

PS: I changed most of my sqlResource definition for createTable changing the
primary key from "message_name, repository_name" to "repository_name,
message_name". I'm not sure why some times we used one and sometime the
other. IMHO it's always better "repository_name, message_name". (should we
change this for every db?)

Stefano

PS: Does JIRA intercept mailing list messages with bug tag?
http://issues.apache.org/jira/browse/JAMES-393


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org