New Mailet API - Was Re: what competition is doing

[Mike Heath <mi...@cycletime.com>]

I really like the idea of collaborating on a new mailet API.  Getting
more parties involved will help create a better mail processing
platform.

As I've been porting the SMTP protocol handler over to MINA and after
having written a few mailets for various tasks, I would like throw in a
couple of things I would like to see in the new mailet specification.

- Mailet staging - It would be really nice to be able to have different
stages for mailet execution.  The first stage would start when the SMTP
DATA command is received.  This mailets would only be able to process
based on data such as: remote IP, helo/ehlo host name, recipients,
sender, user, whether TLS is enabled or not etc.  This would also make
it possible to reject the incoming message before it has been received
in its entirety.  The second stage would start after all the headers
have been received and the third stage would start when the entire
message has been received.
- Mailets as deployable units - I don't know if this should go into the
new mailet specification itself but it would be nice if mailets could be
made into discrete deployable units.  I would like to be able to take a
jar file with my mailets, matchers and some type of deployment
descriptor and just give it to the mail server.  I think this would make
it much simpler for administrators to use third-party mailets.  I can
see this happening with JBoss as this seams to follow closely with their
easy deploy philosophy.
- Multiple mailets per matcher.  I have a couple of different matchers
that are expensive to run (Naive Bayes classifiers) that I would like to
have multiple mailets process.  To do this I've had to create a separate
spool and my James config file has gotten very messy.
- Process to allow mailets to run for a non-match.  Besides specifying
which mailets should run on a positive match, it would be nice to be
able to specify mailets that should run on an negative match.

Just my $0.02.

-Mike



---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: New Mailet API - Was Re: what competition is doing

[Laurent Rouvet <la...@roovay.com>]

Serge Knystautas wrote:
 >
> One thing that always bug me about webapps is how they restart.  Why 
> should there be any downtime?  I'm going to have a new classloader and 
> fully initialize it... don't tear down the first instance until the 
> second is fully initialized, then cutover and hand new connections to 
> the new classloader instance, and eventually the old classloader 
> instance will finish handling those requests and can be GC'd.  Does this 
> bug anyone else?
> 
> The only reason I can't see doing this is if you were changing socket 
> bindings, which neither servlets nor mailets should.
> 

In the mailets context i think that you are right, but in the webapp 
context, remember that you may have to manager the session!

However, in the cas of a compatible backend (DB or EBJ,...), the 2 
webapp may carry on run and the container send the request with a old 
session id to the old webapp.
So it's not soo easy....  fortunately in the mailet context we don't 
have a session. ;-)


Laurent Rouvet.

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

RE: New Mailet API - Was Re: what competition is doing

[Mike Heath <mi...@cycletime.com>]

On Tue, 2005-05-03 at 16:32 -0400, Noel J. Bergman wrote:
<snip>
> > I'm pretty excited about ActiveMQ (http://activemq.codehaus.org/)
> > which we could bundle, and then provide any JMS integration.
> 
> We can look, but I'm concerned about performance.
> 
> HOWEVER, one of the OTHER major things to look at is breaking out SMTP, the
> mailet container (main pipeline), POP3 and IMAP into separate processes.
> That way, an admin can run just protocol handlers as root, and give more
> limited access to the main pipeline.  And we can cluster the pipeline.

I too have been looking closely at integrating ActiveMQ.  It seams like
an excellent approach to solving cluster problems.  I really like Noel's
idea for separating out the protocol handlers too.

If performance is an issue, the ActiveMQ project already has some JMeter
performance tests, http://activemq.codehaus.org/JMeter+Performance
+Tests.  It should be a fairly straight forward process to see how
ActiveMQ compares to James' existing solution.

-Mike


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

RE: New Mailet API - Was Re: what competition is doing

["Noel J. Bergman" <no...@devtech.com>]

Serge Knystautas wrote:

> One thing that always bug me about webapps is how they restart.

No reason at all why we can't handle it "right" for our needs.  Messages in
the processor would continue until it was done.  When the processor is
finished, it would exit.  New messages arriving for that processor, by name,
would be handled by the revised configuration.

Sort of like a graceful restart with httpd.

> The only reason I can't see doing this is if you were changing socket
> bindings, which neither servlets nor mailets should.

Non-issue for mailets, since they should only process mail that is already
in the spool.  And should never see sockets or similar.

> I'm pretty excited about ActiveMQ (http://activemq.codehaus.org/)
> which we could bundle, and then provide any JMS integration.

We can look, but I'm concerned about performance.

HOWEVER, one of the OTHER major things to look at is breaking out SMTP, the
mailet container (main pipeline), POP3 and IMAP into separate processes.
That way, an admin can run just protocol handlers as root, and give more
limited access to the main pipeline.  And we can cluster the pipeline.

	--- Noel


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: New Mailet API - Was Re: what competition is doing

[Serge Knystautas <se...@lokitech.com>]

Noel J. Bergman wrote:
>>- Mailets as deployable units
> 
> Most things should be deployable.  Matchers and Mailets are container
> managed components.  We need to decide on exactly what/where the container
> is.  I have suggested that the Mailet container is the processor, and we can
> have different processors with different capabilities and even different
> Mailet Api versions.  Keeping this analogy, if we might look at a
> classloader with the container, so tearing down the processor like a web
> app.  The issue there would be handling currently unique things, such as how
> the new mailing list manager code works.

One thing that always bug me about webapps is how they restart.  Why 
should there be any downtime?  I'm going to have a new classloader and 
fully initialize it... don't tear down the first instance until the 
second is fully initialized, then cutover and hand new connections to 
the new classloader instance, and eventually the old classloader 
instance will finish handling those requests and can be GC'd.  Does this 
bug anyone else?

The only reason I can't see doing this is if you were changing socket 
bindings, which neither servlets nor mailets should.

> So something to give serious thought to, and then implement.
> 
> Also, I would like to see spooling and scheduling pulled out of the
> matcher/mailet and into the container code.  That way we can have
> RemoteDelivery type scheduling, but for other things, such as DNS retries,
> etc.  The spool interface should handle it, but we need to do some code and
> config movement.

Did we get anywhere with thinking JMS could do this?  I'm pretty excited 
about ActiveMQ (http://activemq.codehaus.org/) which we could bundle, 
and then provide any JMS integration.  Probably performance issues, but 
not sure.

> Again, I really like the ideas you are coming up with, and your enthusiasm.
> I'm just saying that there is a lot of prior discussion on much of these
> sorts of topics.  More discussions than volunteers available to do the work.

+1 as I am exemplifying. :)

-- 
Serge Knystautas
Lokitech >> software . strategy . design >> http://www.lokitech.com
p. 301.656.5501
e. sergek@lokitech.com

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

RE: New Mailet API - Was Re: what competition is doing

["Noel J. Bergman" <no...@devtech.com>]

Mike,

Good ideas.  However, you might want to review the Wiki and mailing list for
a lot of similar ideas that have been worked over already.

> As I've been porting the SMTP protocol handler over to MINA

We should talk about that, soon.  I am contemplating if that is really the
right place to do it, or if we should do it in the connection support code.
Benefits to each way.  Please see my discussion with Trustin regarding how
the servlet API was mapped to NIO, and the subsequent code changes in MINA.

> - Mailet staging - It would be really nice to be able to have different
> stages for mailet execution.  The first stage would start when the SMTP
> DATA command is received.

Mailets are designed to be used after a message is received and ready to be
processed.  We can talk about some other pluggable component for use inside
of protocol handlers, as appropriate.

> This would also make it possible to reject the incoming message

Please do not confuse matchers and mailets.  There is also the idea of using
a the Matcher interface within the protocol handlers, to better implement
flexible fast-fail.  We have some code already available for that, which we
can look to deploy now that we have the merged codebase coming together.

> - Mailets as deployable units

Most things should be deployable.  Matchers and Mailets are container
managed components.  We need to decide on exactly what/where the container
is.  I have suggested that the Mailet container is the processor, and we can
have different processors with different capabilities and even different
Mailet Api versions.  Keeping this analogy, if we might look at a
classloader with the container, so tearing down the processor like a web
app.  The issue there would be handling currently unique things, such as how
the new mailing list manager code works.

So something to give serious thought to, and then implement.

Also, I would like to see spooling and scheduling pulled out of the
matcher/mailet and into the container code.  That way we can have
RemoteDelivery type scheduling, but for other things, such as DNS retries,
etc.  The spool interface should handle it, but we need to do some code and
config movement.

> - Multiple mailets per matcher.

Been on the list for ages.  See the Wiki for some proposals.  But this is
really quite trivial to do today.  Just use a processor.

> - Process to allow mailets to run for a non-match.

I think that's also on the list for the more advanced config.

Again, I really like the ideas you are coming up with, and your enthusiasm.
I'm just saying that there is a lot of prior discussion on much of these
sorts of topics.  More discussions than volunteers available to do the work.

	--- Noel


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: New Mailet API - Was Re: what competition is doing

[ap...@bago.org]

> - Mailet staging - It would be really nice to be able to have 
> different stages for mailet execution.  The first stage would 
> start when the SMTP DATA command is received.  This mailets 
> would only be able to process based on data such as: remote 
> IP, helo/ehlo host name, recipients, sender, user, whether 
> TLS is enabled or not etc.  This would also make it possible 
> to reject the incoming message before it has been received in 
> its entirety.  The second stage would start after all the 
> headers have been received and the third stage would start 
> when the entire message has been received.

I don't fully agree.
I think that is not correct to close a connection during a DATA session, so
I think that there should be NO stage when we received the headers.
I also don't like the "remote IP, helo/ehlo host name, recipients, sender,
user, whether TLS is enabled or not" stage: this should be modularized but
not the matcher/mailet way IMHO.

> - Mailets as deployable units - I don't know if this should 
> go into the new mailet specification itself but it would be 
> nice if mailets could be made into discrete deployable units. 
>  I would like to be able to take a jar file with my mailets, 
> matchers and some type of deployment descriptor and just give 
> it to the mail server.  I think this would make it much 
> simpler for administrators to use third-party mailets.  I can 
> see this happening with JBoss as this seams to follow closely 
> with their easy deploy philosophy.

Can you provide an example?
A mailet need to be configured in a processor and probably associated to a
matcher. You can't just plug them and go. 

Do you mean you would like to have an
AdministrablePluggableHotdeployableSpoolManager that allow configuration a
pluggability of matchers/mailet on the fly via a remote interface?

> - Multiple mailets per matcher.  I have a couple of different 
> matchers that are expensive to run (Naive Bayes classifiers) 
> that I would like to have multiple mailets process.  To do 
> this I've had to create a separate spool and my James config 
> file has gotten very messy.
> - Process to allow mailets to run for a non-match.  Besides 
> specifying which mailets should run on a positive match, it 
> would be nice to be able to specify mailets that should run 
> on an negative match.

I think that there is some confusion between matcher and mailets,
specifications and james implementations.
Mailet API does not specify that there should be a matcher per mailet and
multiple matcher/mailet couples in a processor. The current Mailet API
simply provide interfaces to match none/partial/full recipients for a given
mail (matcher) or to service a given mail (mailet).

A more advanced container could provide a full featured workflow designer
with multiple matchers and multiple mailets or everything else.
Currently you can simply execute more mailets for a single matcher by using
the ToProcessor mailet in you matcher and then using as many mailet as you
want in the destination processor (all with match="All").

IMHO current Matcher/Mailet api are not enough to have REAL, RE-USABLE
mailets between different mailet containers: most James mailets currently
use and need concepts specific to james, repositories, states, attributes
that are not defined by the mailet api.

Stefano


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org