unit test progress

[Bernd Fondermann <bf...@brainlounge.de>]

hi,

some more unit tests have been added to JAMES-427.

as suggested, i looked into the pre-v2and3-merger-trunk tag. the tests 
contained in there were run in a stand-alone app against a full-fledged 
James instance. what we have now is a much more lightweight approach 
with very short turn-around cycles. it turns out to be real fun and i 
can only recommend to others adding more tests :-)

even though some tested units depend on a little bit too much other 
objects they want to have injected beforehand.

nevertheless, i got some hints from the pre-v2and3 stuff on possible 
future tests.

a specific class from this set was named "POP3Hammering" to reproduce a 
certain "stuck file" issue under windows. is this still relevant?

from the way how i read it, the pre-v2and3 suite was also intended to be 
a stress/load testing utility. this is different from unit testing. i 
will start working in this area soon.

a rather nasty thing i discovered was that RemoteManagerHandler makes 
the optimistic assumption that the UsersRepository he accesses holds 
JamesUser instances, while the repository's interface contract only 
assures to return User instances. it's not really a bug, since it 
obviously works, but it hinders somewhat exchanging the underlying 
implementation ;-)

   bernd




---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

health tests [Re: unit test progress]

[Bernd Fondermann <bf...@brainlounge.de>]

Stefano Bagnara wrote:

> It would be cool to have a performance/stress test that allow to test
> common setups. A benchmark could measure memory used and time needed to
> perform a given task.

yes, that would be excellent!
such tests could also be 'health tests' to 'certify' an intermediate or 
release build. it is my impression that maybe releases are held up 
because it's too hard to tell if 'everything is ok'.

when running tools like postal, rabid and smtpsink for some days i could 
never easily tell if things went good or bad - oh, at least when the 
server broke down, i could ;-)

what i would love to have would be:

* tester specifies the number of internal users and external users.
* test environment generates all internal users in the tested James server
* tester specifies, how much mails should be floating around every minute:
   a. coming in from external users for internal users
   b. from internal users to internal users
   c. going out from internal users to external
   (some users may have significant differences in inbound/outbound 
traffic, or even mostly internal mail traffic)
* for each set a,b,c the tester defines avg + max message size
* for each set a,b,c the tester defines how much percent have 
attachments, and what the max attachment size is
* tester specifies for incoming mail how much spam should be simulated, 
every 'spam' mail is marked with a special header.
* tester specifies the start/endtime of the test
* test generates mail traffic according to the given profiles
* test monitors the server and samples all data and makes it accessible 
to tester

does that sound reasonable?


   bernd



---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: unit test progress

[Siegfried Goeschl <si...@it20one.at>]

Not bad either ... :-)

Siegfried Goeschl

Stefano Bagnara wrote:
> Siegfried Goeschl wrote:
> 
>>Hi Stefano,
>>
>>maybe I can provide a few tips and tricks to improve testing
> 
> 
> Thank you for the tips!
> 
> 
>>+) Jakarta CommonsEmail uses Dumbster to test SMTP sending and testing
>>the result - see http://sourceforge.net/projects/dumbster and
>>http://quintanasoft.com/dumbster/
> 
> 
> I think I like much more the mock transport solution described here:
> http://bill.dudney.net/roller/comments/bill/Weblog/mocking_javamail
> 
> James is already a mail server and we probably don't need dumbster: we
> could simply send the message to our own smtpserver. RemoteDelivery and
> SMTPServer are totally different services so using one to test the other
> makes sense. Using a mocked transport we probably can simulate various
> errors/problems/conditions for the remote servers.
> 
> Stefano
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: unit test progress

[Stefano Bagnara <ap...@bago.org>]

Siegfried Goeschl wrote:
> Hi Stefano,
> 
> maybe I can provide a few tips and tricks to improve testing

Thank you for the tips!

> +) Jakarta CommonsEmail uses Dumbster to test SMTP sending and testing
> the result - see http://sourceforge.net/projects/dumbster and
> http://quintanasoft.com/dumbster/

I think I like much more the mock transport solution described here:
http://bill.dudney.net/roller/comments/bill/Weblog/mocking_javamail

James is already a mail server and we probably don't need dumbster: we
could simply send the message to our own smtpserver. RemoteDelivery and
SMTPServer are totally different services so using one to test the other
makes sense. Using a mocked transport we probably can simulate various
errors/problems/conditions for the remote servers.

Stefano

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: unit test progress

[Siegfried Goeschl <si...@it20one.at>]

Hi Stefano,

maybe I can provide a few tips and tricks to improve testing

+) Jakarta CommonsEmail uses Dumbster to test SMTP sending and testing 
the result - see http://sourceforge.net/projects/dumbster and 
http://quintanasoft.com/dumbster/

+) for the YAAFI Avalon Container I use various interceptors (see 
http://jakarta.apache.org/turbine/fulcrum/fulcrum-yaafi/interceptors/index.html) 
to measure performance and the JAMON library I use might be of interest 
for you - see http://jamonapi.sourceforge.net/ and 
http://jamonapi.sourceforge.net/JAMonAdmin.html. It is a simple library 
where you call start() and stop() to generate an HTML report in the 
backgorund - check out the implementation of 
http://jakarta.apache.org/turbine/fulcrum/fulcrum-yaafi/xref/index.html
And using JAMON has nothing to do with Avalon - it is just a library.

+) I wrote a simple AdviceService for YAAFI to attach the various 
interceptors to any Java object implementing interfaces which is 
currently used to add performance monitoring to another service 
framework. It is just one way of many others to add extra behaviour to 
existing services (spell it AOP)

Cheers,

Siegfried Goeschl




Stefano Bagnara wrote:
>>a specific class from this set was named "POP3Hammering" to reproduce a
>>certain "stuck file" issue under windows. is this still relevant?
> 
> 
> I think I've found this issue again while hammering my james trunk using
> file repositories. file repositories locking has something wrong.
> 
> 
>>from the way how i read it, the pre-v2and3 suite was also intended to be
>>a stress/load testing utility. this is different from unit testing. i
>>will start working in this area soon.
> 
> 
> It would be cool to have a performance/stress test that allow to test
> common setups. A benchmark could measure memory used and time needed to
> perform a given task.
> 
> I would also like to have a remote delivery test (not the RemoteDelivery
> mailet itself, but the remote delivery behaviour): I wrote a whole
> replacement for RemoteDelivery that handle multiple message per
> connection and try to optimize the outgoing throughput few months ago
> but I neved had time to test it! I don't know how this could be
> tested... maybe by providing a mocked smtp transport.
> 
> Stefano
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: unit test progress

[Stefano Bagnara <ap...@bago.org>]

> a specific class from this set was named "POP3Hammering" to reproduce a
> certain "stuck file" issue under windows. is this still relevant?

I think I've found this issue again while hammering my james trunk using
file repositories. file repositories locking has something wrong.

> from the way how i read it, the pre-v2and3 suite was also intended to be
> a stress/load testing utility. this is different from unit testing. i
> will start working in this area soon.

It would be cool to have a performance/stress test that allow to test
common setups. A benchmark could measure memory used and time needed to
perform a given task.

I would also like to have a remote delivery test (not the RemoteDelivery
mailet itself, but the remote delivery behaviour): I wrote a whole
replacement for RemoteDelivery that handle multiple message per
connection and try to optimize the outgoing throughput few months ago
but I neved had time to test it! I don't know how this could be
tested... maybe by providing a mocked smtp transport.

Stefano

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org