You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Bernd Fondermann <bf...@brainlounge.de> on 2006/03/10 23:18:36 UTC
Re: Long Running Test
Hi,
Finally, I managed to run the same scenario as described below with
2.3.0a1. 80603 mails sent, 0 lost. Performance seems to be basically
identical which is good news.
I will continue with more mails per minute, larger attachments and
longer runs...
And I'm looking forward to try and reproduce Stefanos improvements in
big mail handling (20MB) soon after they have been commited.
Also, I'd like to encourage people to look at JAMES-442 and comment on
the test tool itself and the test which should be run. Personally, I'd
like to have memory footprint data recorded and
multithreading/scalability/errorhandling improved. But what's more?
Ralf Hauser suggested to base it on JMeter which sounds like a good idea
to me. Any thoughts on this from someone already familiar with the
JMeter codebase?
Bernd
Bernd Fondermann wrote:
> Hi,
>
> I conducted the first longer test run using the code from Jira JAMES-442.
>
> The test ran without any problems under vanilla 2.2.0 for 24hrs, 80612
> mails were sent, 0 lost.
>
> It simulates the following scenario:
>
> 50 internal users.
> 100 external addresses.
>
> | mail |
> | proc.|
> | per | | MIME part size|
> | min |from| to | text | binary |
> +------+----+----+------+--------+
> | 20 |int |ext | 1000 | 1000 |
> | 10 |ext |int | 500 | - |
> | 1 |ext |int | 200 | 30000 |
> | 20 |int |int | 500 | - |
> | 1 |int |int | 100 | 30000 |
> ======
> 52
>
> What numbers would make more realistic average usage scenarios? More
> mails? More users?
>
> There is a problem running the tool under 2.3.0a1, hopefully I can get
> it working over the weekend for a first comparison of 2.2 and 2.3 branches.
>
> Bernd
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Long Running Test
Posted by Bernd Fondermann <bf...@brainlounge.de>.
Stefano Bagnara wrote:
> Bernd Fondermann wrote:
>
>> Hi,
>>
>> Finally, I managed to run the same scenario as described below with
>> 2.3.0a1. 80603 mails sent, 0 lost. Performance seems to be basically
>> identical which is good news.
>
>
>
> How did you find the performance are the same?
> I'm not sure I understood the test...
>
> You send always 52 mails per minute so you should spool 52*1440 (minutes
> per hour) = 74880 message.
>
> Why your results are 80600 for both servers?
Oops, a typo in my write-ups. 56 mails per minute have been sent.
Probably due to timer glitches, not the full number of 56*1440 = 80640
was reached.
> To find the performance you should as many mails as james is able to
> handle and slow down only when the spool start increasing it size.
> A good way would be to set a number of messages to send in "queue" and
> send more new messages only after receiving a few from the remote
> delivery (Mail Gateway'ing) or reading them from the pop3 server (not
> sure I explained what I mean).
>
> Scenario:
> 1. queue size = 100
> 2. postage start sending 100 messages to the server, checking pop3
> folders and checking incoming smtp messages.
> 3. after a message is received by the remote delivery or read from the
> pop3 server, postage send a new message to the server.
>
> This way the total messages sent depends on the spooling speed and not
> on the messages per minute you set.
>
> To be more compliant with your "scenario" configuration we can think
> that your numbers are not meant "per minute" but "per send unit".
>
> You know you have 52 messages "per send unit" in your scenario. You
> start sending a number of unit (1 to 10 units seems reasonable), then
> you send a new one only after every 52 messages you receive from the
> remote delivery or pop3 server.
>
> Does it make sense? Is this possible with "postage architecture"?
Makes perfect sense to me. This would be a stress test application.
What I intended to achieve in the first place is to
* put a steady, reproducable load on James
* gather as much data about successful delivery and errors as possible
* try to mimic real-life load
thus making two runs of the same configuration for different James
builds comparable in terms of: "Is the build I have here still capable
of delivering all messages as the last major version was - or is
something broken?"
The result of the whole test run would be boolean. This could be
executed daily by continuous integration tools.
But of course, the "adaptive" behavior you are describing would still be
an option to see when we "max out" the server. Seems to be more fun, too :-)
Bernd
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Long Running Test
Posted by Stefano Bagnara <ap...@bago.org>.
Bernd Fondermann wrote:
> Hi,
>
> Finally, I managed to run the same scenario as described below with
> 2.3.0a1. 80603 mails sent, 0 lost. Performance seems to be basically
> identical which is good news.
How did you find the performance are the same?
I'm not sure I understood the test...
You send always 52 mails per minute so you should spool 52*1440 (minutes
per hour) = 74880 message.
Why your results are 80600 for both servers?
To find the performance you should as many mails as james is able to
handle and slow down only when the spool start increasing it size.
A good way would be to set a number of messages to send in "queue" and
send more new messages only after receiving a few from the remote
delivery (Mail Gateway'ing) or reading them from the pop3 server (not
sure I explained what I mean).
Scenario:
1. queue size = 100
2. postage start sending 100 messages to the server, checking pop3
folders and checking incoming smtp messages.
3. after a message is received by the remote delivery or read from the
pop3 server, postage send a new message to the server.
This way the total messages sent depends on the spooling speed and not
on the messages per minute you set.
To be more compliant with your "scenario" configuration we can think
that your numbers are not meant "per minute" but "per send unit".
You know you have 52 messages "per send unit" in your scenario. You
start sending a number of unit (1 to 10 units seems reasonable), then
you send a new one only after every 52 messages you receive from the
remote delivery or pop3 server.
Does it make sense? Is this possible with "postage architecture"?
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: Long Running Test
Posted by "Noel J. Bergman" <no...@devtech.com>.
Bernd,
> Noel J. Bergman wrote:
>>>I'm +1 on java 5 dependencies for tests/support tools if this simplify
>>>the development. Don't know what other committers think about this.
>> *ONLY* for such test/support tools.
> it should not limit the tool's usage to j5. it could be an optional
> run-time and not a compile-time dependency.
Fine. I'm simply saying that I would not vote for a J5 requirement for
JAMES, itself, at this time.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Long Running Test
Posted by Bernd Fondermann <bf...@brainlounge.de>.
Noel J. Bergman wrote:
>>I'm +1 on java 5 dependencies for tests/support tools if this simplify
>>the development. Don't know what other committers think about this.
>
>
> *ONLY* for such test/support tools.
it should not limit the tool's usage to j5. it could be an optional
run-time and not a compile-time dependency.
Bernd
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: Long Running Test
Posted by "Noel J. Bergman" <no...@devtech.com>.
> I'm +1 on java 5 dependencies for tests/support tools if this simplify
> the development. Don't know what other committers think about this.
*ONLY* for such test/support tools.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Long Running Test
Posted by Stefano Bagnara <ap...@bago.org>.
Bernd Fondermann wrote:
>> (It would be cool to have a switch to say to the VM "try to use less
>> than X mb of heap" so that it automatically GC at that level but don't
>> send OutOfMemoryErrors when more memory is needed to complete the
>> operation)
>
> Most elegant solution I see is to use JSE 5's build-in JMX mechanism
> (http://java.sun.com/j2se/1.5.0/docs/api/java/lang/management/MemoryMXBean.html),
> but this would of course not be possible under Java 1.4 out-of-the-box.
> But maybe a backport exists somewhere.
I already use them in a few local tests but they are limited, anyway.
I'm +1 on java 5 dependencies for tests/support tools if this simplify
the development. Don't know what other committers think about this.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Long Running Test
Posted by Bernd Fondermann <bf...@brainlounge.de>.
Stefano Bagnara wrote:
> Bernd Fondermann wrote:
>
>> Hi,
>>
>> Finally, I managed to run the same scenario as described below with
>> 2.3.0a1. 80603 mails sent, 0 lost. Performance seems to be basically
>> identical which is good news.
>
>
> Good to know!
> Which environment did you test? (mysql/derby db? db/file/dbfile
> repositories?)
> Have you done any comparison of the different dbs?
James and test client were both run locally on the same machine.
I used file repositories for the first comparison of the major verions.
Different repositories come next.
Currently I get OutOfMemory exceptions -- within _postage_ after
recording data for more than 110000 mails. So, lots of improvement still
to be done.
> It is difficult to check memory footprint in java. The only way is to
> limit the JVM resources and see how it works.
>
> (It would be cool to have a switch to say to the VM "try to use less
> than X mb of heap" so that it automatically GC at that level but don't
> send OutOfMemoryErrors when more memory is needed to complete the
> operation)
Most elegant solution I see is to use JSE 5's build-in JMX mechanism
(http://java.sun.com/j2se/1.5.0/docs/api/java/lang/management/MemoryMXBean.html),
but this would of course not be possible under Java 1.4 out-of-the-box.
But maybe a backport exists somewhere.
Bernd
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: Long Running Test
Posted by "Noel J. Bergman" <no...@devtech.com>.
> > Personally, I'd like to have memory footprint data recorded and
> > multithreading/scalability/errorhandling improved.
> It is difficult to check memory footprint in java. The only way is
> to limit the JVM resources and see how it works.
But we can still use the heapdump tools to check for memory leaks. I do
that prior to releases, and whenever something seems to be leaking. We
cannot do anything about what the underlying JVM does, but we can see the
objects being created, destroyed, preserved and their allocation sites.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Long Running Test
Posted by Stefano Bagnara <ap...@bago.org>.
Bernd Fondermann wrote:
> Hi,
>
> Finally, I managed to run the same scenario as described below with
> 2.3.0a1. 80603 mails sent, 0 lost. Performance seems to be basically
> identical which is good news.
Good to know!
Which environment did you test? (mysql/derby db? db/file/dbfile
repositories?)
Have you done any comparison of the different dbs?
> And I'm looking forward to try and reproduce Stefanos improvements in
> big mail handling (20MB) soon after they have been commited.
The patch is more difficult than I though, but I'm working on it, now.
Hope to share something soon. I'm writing new tests because of the
"critical" changes to the core MimeMessageWrapper.
> Also, I'd like to encourage people to look at JAMES-442 and comment on
> the test tool itself and the test which should be run. Personally, I'd
> like to have memory footprint data recorded and
> multithreading/scalability/errorhandling improved. But what's more?
It is difficult to check memory footprint in java. The only way is to
limit the JVM resources and see how it works.
(It would be cool to have a switch to say to the VM "try to use less
than X mb of heap" so that it automatically GC at that level but don't
send OutOfMemoryErrors when more memory is needed to complete the operation)
> Ralf Hauser suggested to base it on JMeter which sounds like a good idea
> to me. Any thoughts on this from someone already familiar with the
> JMeter codebase?
Sorry, I just used it once for a web scenario but I don't know anything
about its codebase/architecture.
Stefano
PS: I didn't have the time to check out your postage work, but I think
that is a good add-on to james test suite. Maybe we should create its
own subproject instead of integrating it on the main codebase.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Long Running Test
Posted by Stefano Bagnara <ap...@bago.org>.
Bernd Fondermann wrote:
>> I'm writing it to test an "advanced" remote delivery that try to reuse
>> smtp connections to send multiple messages to the same domain, cache
>> smtp errors, and more.
>
> Cool. Sounds like this could improve throughput.
I've started working on this about one year ago, but the spare time is
so short and James issue list is so long, that I never tested it for
real yet. Just trying to complete a bunch of unit tests before promoting
the mailets to my personal mail server.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Long Running Test
Posted by Bernd Fondermann <bf...@brainlounge.de>.
Stefano Bagnara wrote:
> Norman Maurer wrote:
>
>> A good tool for benchmark mailservers is postal:
>> http://freshmeat.net/projects/postal/
>>
>> Maybe this will help all these peolpe which are intressted in thorughput
>> etc.
>
>
> We (at least I and Noel) already use postal/rabid to test James, but
> Bernd is creating a different tool that should be easier and more
> powerful for James testing.
I used that, too. But it is very limited: You can put load on James, but
you don't really know if messages were really spooled to the appropriate
receiver or just were dumped somewhere and never processed (thus giving
you high performance!) without checking by hand. This is not appropriate
for automated end-to-end tests.
>
> Later I will provide a "mocked javamail transport" I'm writing to test
> remotedelivery. This should help creating more realistic benchmarks
> (unreachable servers, errors in smtp server, retries, bounces).
>
> I'm writing it to test an "advanced" remote delivery that try to reuse
> smtp connections to send multiple messages to the same domain, cache
> smtp errors, and more.
>
Cool. Sounds like this could improve throughput.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Long Running Test
Posted by Stefano Bagnara <ap...@bago.org>.
Norman Maurer wrote:
> A good tool for benchmark mailservers is postal:
> http://freshmeat.net/projects/postal/
>
> Maybe this will help all these peolpe which are intressted in thorughput
> etc.
We (at least I and Noel) already use postal/rabid to test James, but
Bernd is creating a different tool that should be easier and more
powerful for James testing.
Later I will provide a "mocked javamail transport" I'm writing to test
remotedelivery. This should help creating more realistic benchmarks
(unreachable servers, errors in smtp server, retries, bounces).
I'm writing it to test an "advanced" remote delivery that try to reuse
smtp connections to send multiple messages to the same domain, cache
smtp errors, and more.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Long Running Test
Posted by Norman Maurer <nm...@byteaction.de>.
A good tool for benchmark mailservers is postal:
http://freshmeat.net/projects/postal/
Maybe this will help all these peolpe which are intressted in thorughput
etc.
Am Sonntag, den 12.03.2006, 17:13 +0100 schrieb Stefano Bagnara:
> Bernd Fondermann wrote:
> > Hi,
> >
> > Finally, I managed to run the same scenario as described below with
> > 2.3.0a1. 80603 mails sent, 0 lost. Performance seems to be basically
> > identical which is good news.
>
>
> How did you find the performance are the same?
> I'm not sure I understood the test...
>
> You send always 52 mails per minute so you should spool 52*1440 (minutes
> per hour) = 74880 message.
>
> Why your results are 80600 for both servers?
>
> To find the performance you should as many mails as james is able to
> handle and slow down only when the spool start increasing it size.
> A good way would be to set a number of messages to send in "queue" and
> send more new messages only after receiving a few from the remote
> delivery (Mail Gateway'ing) or reading them from the pop3 server (not
> sure I explained what I mean).
>
> Scenario:
> 1. queue size = 100
> 2. postage start sending 100 messages to the server, checking pop3
> folders and checking incoming smtp messages.
> 3. after a message is received by the remote delivery or read from the
> pop3 server, postage send a new message to the server.
>
> This way the total messages sent depends on the spooling speed and not
> on the messages per minute you set.
>
> To be more compliant with your "scenario" configuration we can think
> that your numbers are not meant "per minute" but "per send unit".
>
> You know you have 52 messages "per send unit" in your scenario. You
> start sending a number of unit (1 to 10 units seems reasonable), then
> you send a new one only after every 52 messages you receive from the
> remote delivery or pop3 server.
>
> Does it make sense? Is this possible with "postage architecture"?
>
> Stefano
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>