Testing as a separate project - Re: cvs commit: jakarta-james/proposals/TLP proposal.txt

[ha...@kodemuse.com]

----- Original Message -----
From: danny@apache.org
>           Sub-Projects:
>                   Mailet API:
>                   Mailet Applications:

How about 'Protocol Tests' as a separate sub project

Objective would be to have Protocol tests for 

- RFC Compliance
- Load Testing
- Performance Tesing.

These tests would be implementation neutral and could be used to benchmark James implementation against Sendmail, Exim, Exchange etc.

This would not replace the need for James Unit testing for individual components.

Vote/Comments...

Harmeet

Re: Testing as a separate project - Re: cvs commit: jakarta-james/proposals/TLP proposal.txt

[Harmeet Bedi <ha...@kodemuse.com>]

----- Original Message -----
From: "Noel J. Bergman" <no...@devtech.com>
> Harmeet,
>
> I have to say that I am unsure.  Yes, I think that there should be a
testing
> sub-project.  But is this something that needs to be dealt with now, or
> afterwards as a PMC?

Either way seems good. It maybe a new proposal idea, but testing has been
talked about for a looong time. A separate project may be the best way to
pull together, focus testing effort and invite wider participation for
testing.

>
> > Objective would be to have Protocol tests for
> > - RFC Compliance
> > - Load Testing
> > - Performance Tesing.
>
> Yes in concept.  And tools for doing those things.  One thing I don't know
> is whether or not the testing sub-project ought to work with the JMeter
> folks to get our protocols into that code, or whatever project might
replace
> it.

There are lot of Mail Servers, open source and otherwise. It would be useful
and help harness energy and provide a common testing project for Mail
Protocol Testing.

There is good code and ideas already.

I think this combination can yield a good structure to build tests on.
- Ant tasks (already there.)
- Protocol Simulation ability (in 2 places in James.)
- A scripting/template facility to tie things together (Written locally but
not committed. Requires library addition, long emails etc)
- Possible Junit tie in - (already there)


The main problem would be writing the large number of tests. For this we
could interest developers from other opensource projects. There seems to be
a lot of common value that can be realized.

Having a general testing project would have these benefits.
- Interest developers from other open source or commercial projects. There
seems to be a lot of protocol servers and testing could provide common value
to all at the same time help James.
- Testing is a big and important task. It would be good to have a separate
validation project.

Regd: JMeter and other testing projects. I think we should investigate this.
load/performance analysis would be esp. useful to reuse.
I think it is good to leverage where possible, but we should have rfc
testing for mail protocol as the primary goal. I don't think JMeter would
care much about testing n variants of SMTP protocol.


>
> Testing could become a pretty consuming project
>

Exactly. If we can have a James neutral sub-project defination we may be
able to solicit help from other mail server projects and reduce our testing
burden. At the same time build a community that benefits from this.

> requiring someone to focus
> on it every bit as much as a release manager has to focus on a release.

I have no problem with doing this. Darrell has done a lot of work. You have
a lot of ideas on this. There may be others passionate about this too.

Harmeet


--
To unsubscribe, e-mail:   <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>

RE: Testing as a separate project - Re: cvs commit: jakarta-james/proposals/TLP proposal.txt

["Noel J. Bergman" <no...@devtech.com>]

> > Testing could become a pretty consuming project requiring someone to
focus
> > on it every bit as much as a release manager has to focus on a release.

> This is another good point, its all very well to propose sub-projects, but
> they do need to demonstrate some kind of community prepared to actually
put
> in the effort.

Right.  And as much as I might have interest in the test bed, the most I
think I could offer would be kibbitzing because I expect that we will all be
consumed by the v3 development.  So whomever would decide to work on testing
is going to have to decide that they want to focus on testing as their
primary contribution for a while.  Do you agree?

	--- Noel


--
To unsubscribe, e-mail:   <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>

RE: Testing as a separate project - Re: cvs commit: jakarta-james/proposals/TLP proposal.txt

[Danny Angus <da...@apache.org>]

> I have to say that I am unsure.  Yes, I think that there should 
> be a testing
> sub-project.  But is this something that needs to be dealt with now, or
> afterwards as a PMC?

IMO its not a primary goal, and worth as it is, is probably best dealt with internally if/when we get that far. I can't see it swaying the board's decision.


> Testing could become a pretty consuming project requiring someone to focus
> on it every bit as much as a release manager has to focus on a release.

This is another good point, its all very well to propose sub-projects, but they do need to demonstrate some kind of community prepared to actually put in the effort.


--
To unsubscribe, e-mail:   <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>

RE: Testing as a separate project - Re: cvs commit: jakarta-james/proposals/TLP proposal.txt

["Noel J. Bergman" <no...@devtech.com>]

Harmeet,

I have to say that I am unsure.  Yes, I think that there should be a testing
sub-project.  But is this something that needs to be dealt with now, or
afterwards as a PMC?

> Objective would be to have Protocol tests for
> - RFC Compliance
> - Load Testing
> - Performance Tesing.

Yes in concept.  And tools for doing those things.  One thing I don't know
is whether or not the testing sub-project ought to work with the JMeter
folks to get our protocols into that code, or whatever project might replace
it.

Testing could become a pretty consuming project requiring someone to focus
on it every bit as much as a release manager has to focus on a release.

	--- Noel


--
To unsubscribe, e-mail:   <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>