You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mailet-api@james.apache.org by Alan Williamson <al...@blog-city.com> on 2006/11/11 10:55:21 UTC
Introductions ...
Good morning one and all, and it's good to see the Mailet API getting
some attention specifically outside of JAMES. I joined this list after
spotting a blog entry from Danny requesting a discussion.
One of the main reasons I am interested in the MailetAPI is because I
love the abstraction. Being someone who was involved in the early
ServletAPI evolution, this particular interface was a little gem lost in
a sea of rocks.
I won't go into my dislike for JAMES, except to say, I was surprised to
see no one else had any real alternative engines for the Mailet API. So
at SpikeSource, I headed up a project to do just that. We built and
are using, our own Mailet engine. We have open sourced the project and
it's available here:
http://developer.spikesource.com/projects/mailcatcher/
Some docs:
http://alan.blog-city.com/mailcatcher_0_9.htm
It has proved to be very successful internally and we are using it for a
wide range of projects. We here there are plans for it to be bundled in
the next release of an application server.
However the Mailet API itself, while beautiful is missing a number of
significant hooks that we have started to address and will update the
main MailCatcher project soon.
Namely there is no call-back when the MAIL FROM or RCPT TO is invoked by
a remote SMTP server.
The problem with the current implementation of the API is that one must
accept the whole email before you can do any processing as to whether or
not you want it. For spam control, this is completely unacceptable.
You want to nuke the connection ASAP and only spend the resources to
accept an email you really want. To that end we've added a new
interface that deals with this.
So I am looking forward to being part of this discussion and promoting
the use of the Mailet API to hopefully the point where other vendors and
developers want to start supporting it also.
After all - isn't email still the killer app?
--
Alan Williamson, CTO Blog-City Ltd
w: http://www.blog-city.com/
b: http://alan.blog-city.com/
Re: Introductions ...
Posted by Danny Angus <da...@gmail.com>.
On 11/11/06, Alan Williamson <al...@blog-city.com> wrote:
> After all - isn't email still the killer app?
I still think so,
Re: Introductions ... re stefano
Posted by Danny Angus <da...@gmail.com>.
On 11/11/06, Alan Williamson <al...@blog-city.com> wrote:
> If the JAMES implementation is already doing this functionality; then it
> sounds as if it should be relatively trivial to move this handler
> functional out as an interface and include it as part of the official
> Mailet API.
Like Noel I'm not 100% sure if it should be in the API, however it
might be an optional extension to it.
d.
Re: Introductions ... re stefano
Posted by Alan Williamson <al...@blog-city.com>.
Stefano Bagnara wrote:
>> Namely there is no call-back when the MAIL FROM or RCPT TO is invoked
>> by a remote SMTP server.
>
> I agree this is a big missing piece.
>
> I think we should provide a more fine API for this. Sometimes mailets
> only care of connection data, sometimes of SMTP envelope data (sender,
> recipients...), sometimes only of mime message header, sometimes of the
> full body, sometimes of the mime structure of the body, sometimes
> combinations of the previous.
>
> I think we should provide a mean to make the container aware of this.
> Now that even james started supporting fastfail operations (in protocol
> handlers) it would be cool if we all could agree on something like
> "mailets" for the in-protocol operations.
*nods* it would be wonderful if we could agree on this. At the moment
we've implemented it as a separate Interface. You register the listener
and when the emails come in, then you have a hook as to what you want to
do with it. As you said to "fastfail operations" (i love that phrase).
If the JAMES implementation is already doing this functionality; then it
sounds as if it should be relatively trivial to move this handler
functional out as an interface and include it as part of the official
Mailet API.
btw - i wasn't expecting a reply at the weekend, so thank you.
Re: Introductions ...
Posted by Stefano Bagnara <ap...@bago.org>.
Alan Williamson wrote:
> Good morning one and all, and it's good to see the Mailet API getting
> some attention specifically outside of JAMES. I joined this list after
> spotting a blog entry from Danny requesting a discussion.
Unfortunately we never gave this list too much publicity. We should fix
this problem and the blong entry from Danny is a start :-)
> [...]
> However the Mailet API itself, while beautiful is missing a number of
> significant hooks that we have started to address and will update the
> main MailCatcher project soon.
>
> Namely there is no call-back when the MAIL FROM or RCPT TO is invoked by
> a remote SMTP server.
I agree this is a big missing piece.
I think we should provide a more fine API for this. Sometimes mailets
only care of connection data, sometimes of SMTP envelope data (sender,
recipients...), sometimes only of mime message header, sometimes of the
full body, sometimes of the mime structure of the body, sometimes
combinations of the previous.
I think we should provide a mean to make the container aware of this.
Now that even james started supporting fastfail operations (in protocol
handlers) it would be cool if we all could agree on something like
"mailets" for the in-protocol operations.
> The problem with the current implementation of the API is that one must
> accept the whole email before you can do any processing as to whether or
> not you want it. For spam control, this is completely unacceptable.
I agree
> You want to nuke the connection ASAP and only spend the resources to
> accept an email you really want. To that end we've added a new
> interface that deals with this.
>
> So I am looking forward to being part of this discussion and promoting
> the use of the Mailet API to hopefully the point where other vendors and
> developers want to start supporting it also.
Do you know anything about what we are doing with the smtp handlerapi in
Apache James Server?
I ask this to understand what is the common knowledge and if I need to
provide some detail about previous discussions on this topic.
Stefano
Re: Introductions ...
Posted by Stefano Bagnara <ap...@bago.org>.
Danny Angus wrote:
> On 11/11/06, Alan Williamson <al...@blog-city.com> wrote:
>
>> I am a little confused as to why Danny/Noel believe it isn't part of the
>> Mailet API? What is the remit of the Mailet API? What is its purpose
>> in life if not a hook for developers to process mails?
>
> Yeah, but the API doesn't assume that the system doing the processing
> is also implementing any of the mail protocols. It could be that the
> mail is all sitting in an archive and needs to be mined for some
> reason.
The current API does, we never agreed that this is the final Mailet API
goal.
Imho it is a big limit of the current mailet apis and we should fix it.
> Therefore protocol related API's are out of scope, but that doesn't
> make it a bad idea.
I don't care if we are in protocol or outside it. I think we only need
to add a finer api that let us to write "portable plugins" (implementing
interfaces defined by our api) that are able to handle a wide range of
usages:
- partial matching (only parts of the current Mail object)
- change tracking api: maybe it could be useful to add listeners/event
to track changes to a Mail, so to know if it is ghosted, duplicated,
rerouted or anything similar.
Imho there are a lot of ways to improve the current mailet apis without
the need to tie to SMTP or other transfer protocols.
Stefano
Re: Introductions ...
Posted by Stefano Bagnara <ap...@bago.org>.
Noel J. Bergman wrote:
> Alan Williamson wrote:
>>> Danny Angus wrote:
>>> Alan Williamson wrote:
>>>> I am a little confused as to why Danny/Noel believe it isn't part of the
>>>> Mailet API? What is the remit of the Mailet API? What is its purpose
>>>> in life if not a hook for developers to process mails?
>>> Yeah, but the API doesn't assume that the system doing the processing
>>> is also implementing any of the mail protocols. It could be that the
>>> mail is all sitting in an archive and needs to be mined for some
>>> reason.
>> Gotcha, that makes sense from that aspect.
>>> Therefore protocol related API's are out of scope, but that doesn't
>>> make it a bad idea.
>> I guess this is always the balance, purist vs. reality. In reality the
>> majority of mailets are probably being called upon the SMTP trigger,
>> therefore it makes sense to involve that protocols behaviour in the API.
>
> The Mailet API is driving mailets in a container that receives messages.
> Usually those arrive in some sort of spool. The spool could be distributed,
> and processed on systems that don't have anything to do with SMTP (or other
> protocol) transports.
>
> Look at it this way: the Mailet API has about as much in common with the
> in-protocol handler API as a Message-Driven Bean has in common with the ORB.
> They have totally different semantics, event structures, transport and
> transaction issues, etc. This is why I object to conflating the two areas.
I think that we have misunderstanding due to what handlerapi is in james
server and the concept of handling the data as soon as possible in a
stream based protocol.
>> you could make a strong argument, that the Matcher classes are
>> invoked at the wrong part of the chain. Maybe they should be
>> invoked as and when the email address is available, and not
>> when the whole body has arrived.
>
> Well, that is because you're conflating two different issues: Matchers,
> which deal with selecting messags in a message pipeline, and in-protocol
> handlers that act on events as they happen during protocol transfer.
Currently Matchers can do almost everything a Mailet can do: they can
change the mail. Mailet also can do almost everything.
Imho we should put more constraint on Matchers: imho it would be better
to use them as message selectors and they should not be able to change
the message, but we already use some of them to add attributes and
similar things.
> Where we will have far more agreement, and where we are refactoring JAMES
> code today, is that there are commonalities and places where you want to
> factor out underlying behavior that can be shared by in-protocol handlers
> and matcher/mailet code. So SpamAssassin can be invoked by an in-protocol
> handler (more and more mail *filtering* code is done in the handlers), or it
> could be invoked in the message pipeline.
Exactly, it would be good if we can find interfaces to access the
"underlying behaviours" so to make that common part, usable by both the
protocol handlers or by the "current" Mailets a common standard.
> As I said, more and more we're moving to handle filtering in the protocol
> handlers, and have just "application" type code in the pipeline, e.g.,
> mailing list managers, delivery support, etc.
>
> --- Noel
I agree, and that's why we should try to "evolve" the current API to be
more friendly with the new scenario where we need to have answers to our
processing early.
Myabe we can do this adding new unrelated apis, but maybe many things
can be done by improving the current interfaces.
Stefano
Re: Fast-Fail Operations in the Mailet API?
Posted by Danny Angus <da...@gmail.com>.
or perhaps more accurately
org.apache.mailapi.mailet
org.apache.mailapi.handler
org.apache.mailapi.service
RE: Fast-Fail Operations in the Mailet API?
Posted by "Noel J. Bergman" <no...@devtech.com>.
Danny Angus wrote:
> Noel J. Bergman wrote:
> > - Mailet API
> > - Handler API
> > - Common Services
> I believe that there is a place for all three of these things, and
> that the first two both depend upon the third one, but that neither
> of the first two need to be a mandatory requirement of an API which
> specifies the other one.
Exactly. :-)
> I think we're really close to agreeing that the scope of an API
> project can and possibly should include all three (I think we're
> just getting hung up on names)
Hence my repeated reference to nomenclature, and illustration of the
differences in the component lifecycles.
> I do still maintain that there are use-cases which do not include
> a requirement for implementing protocols.
> [... use cases ...]
Total agreement.
> I would be 100% in favour of developing the three API's under the
> mailet api banner, as long as it was done in a manner which clearly
> signposted the options which implementors have
Exactly. We just need to be careful to not conflate the requirements for
the different container types.
Of course, if a developer wants to intentionally embed a synchronous mailet
container as an onMessage handler, that's fine. At least two of us have
illustrated that usage in the past, and there are some use cases.
--- Noel
Re: Fast-Fail Operations in the Mailet API?
Posted by Danny Angus <da...@gmail.com>.
On 11/12/06, Noel J. Bergman <no...@devtech.com> wrote:
> - Mailet API
> - Handler API
> - Common Services
I believe that there is a place for all three of these things, and
that the first two both depend upon the third one, but that neither of
the first two need to be a mandatory requirement of an API which
specifies the other one.
Lets consider the J2ME jsr standards, there is a core and extensions.
Or Portals and CMS, there are a number of standards which can be
combined or used in isolation.
I think we're really close to agreeing that the scope of an API
project can and possibly should include all three (I think we're just
getting hung up on names), so that implementors can optionally
implement any sensible combination to build any sensible container,
but I do still maintain that there are use-cases which do not include
a requirement for implementing protocols.
One of these is a mail processor which has access to a spool which is
filled by something else. This may or may not be a thing which
implements the handler api.
Another is a piece of client software which uses JavaMail and POP3 or
IMAP to aggregate mail from a number of internet accounts.
I would be 100% in favour of developing the three API's under the
mailet api banner, as long as it was done in a manner which clearly
signposted the options which implementors have, and to my mind that
has to include seperate names and distinct packages.
org.apache.mail.mailet
org.apache.mail.handlers
org.apache.mail.services
d.
Re: Fast-Fail Operations in the Mailet API?
Posted by Stefano Bagnara <ap...@bago.org>.
Noel J. Bergman wrote:
> The lifecycle of Servlets, HTTP Servlets, JSPs, even Portlets and SIP
> Servlets are similar. But let's take a look at these others ... consider:
>
> Matcher: asynch stateless message router based on recipient lists
> match(Mail) method
In the current APIs the matcher is not a router, only a matcher/selector.
And I don't understand the "asynch" part of this definition. You call
the service(Mail) method and it process the Mail synchronously. What is
asynchronous?
> Mailet: asynch stateless message transformer/broker -- "the app"
> service(Mail) method
Again I don't see how it is "asynch": the only asynch part is how we
decided to implement it in james using the JamesSpoolManager.
> In-protocol Handler API:
> Stateful and synchronous
I don't think we want to put the current handler api as part of the
mailet apis, but I think it worth anyway discussing this
stateless/stateful synch/asynch topic.
The current handlers are not stateful, they are stateless like the
mailets. We should also make sure they are threadsafe like mailets
because they could be used for multiple connections at the same time.
And they are synchronous as the Mailets are: you call a method and wait
for it to finish: no callback here, no callback in the mailets.
> Arranged in a chain similar to a Filter Chain
> onConnect -- connection from remote client
> on<Command> -- receipt of a specific command
> onMessage -- receipt of a completed message
onMessage is really similar to what we do in the Mailet.service method.
on<Command> is what we should avoid in the Mailet APIs, but I think that
an "onEnvelope" receiving only the Mail part (without message) would
already be something that would give us the opportunity to refactor many
mailets to use this method instead of the whole "service" and would let
us to use it early in the processing or in processing that do not care
of the included MimeMessage.
Please note that this is only an example and something to be analyzed
and discussed much more.. But I think we should not exclude this
possibility now.
> They have in common the thing that you don't like: the
> "onMessage/match/service" method, which takes a completed message. The SMTP
> protocol handler is one source for such messages to the Mailet container.
> So is FetchMail. So is the NNTP protocol handler. So could be IMAP or
> Jabber. Etc.
>
> The Mailet pipeline deals with messages and routing, and is independent of
> protocol. It could not care less how messages are inserted into the spool.
> As you noted, a Mailet Container is protocol independent. There is a
> specific contract that the container has with its components. The contract
> that exists with protocol handling containers and *their* components
> (in-protocol handlers) is different. It would be entirely unreasonable (and
> quite poor design) to insist that all containers implement both contracts.
Can you summarize the 2 contracts?
I don't agree on the stateful/stateless and the asynch/synch differences
you reported, but maybe there are other critical differences. We should
really create a list of the 2 scenario peculiarities.
>>> As I said, more and more we're moving to handle filtering in the
> protocol
>>> handlers, and have just "application" type code in the pipeline, e.g.,
>>> mailing list managers, delivery support, etc.
>
>> But that is JAMES specific. I am not interested in JAMES.
>
> We have already said that if there is interest in the in-protocol handler
> API, that we'd be interested in exploring it, too. But we would be talking
> about three things:
>
> - Mailet API
> - Handler API
> - Common Services
>
> I'm curious to see if Common Services turns out to be the most contentious
> part.
Well, maybe we can split the whole thing in the 3 "package" you define.
For me it means renaming "Mailet API" to "Common Services" in my mind
and keep talking about how to provide common services that could be
defined once and be used both in the protocol, in the spoolmanager or
for the email client inbox rules.
Stefano
Re: Fast-Fail Operations in the Mailet API?
Posted by Stefano Bagnara <ap...@bago.org>.
Alan Williamson wrote:
> Danny Angus wrote:
>>> But to say it is outside of the current Mailet API, is wrong IMO. The
>>> Mailet API is for the routing and handling of emails -- all we are
>>> talking about is allowing the developer to make this decision much
>>> faster/earlier.
>>
>> Actually the mailet API implies a particular process which starts with
>> the full content of the message having already entered the system
>
> okay i can buy that. As with the ServletAPI, i think we should consider
> the possible of both. For example, the ServletAPI allows a developer to
> process the incoming stream themselves if they want to kill the
> connection at any point they can.
This is the very example I had in my mind :-)
> BTW - I don't suggest for a minute we have a "getMailetInputStream()"
> method, as that is way too ugly for even consideration! :)
I agree!
>> FastFail implies that an interest in the conversation carried out
>> during the protocol can result in decisions being taken *before* the
>> message has entered the system.
>
> Well before the complete message has arrived. But surely we are already
> protocol dependent. Think about it.
>
> We have data that arrives on "MAIL FROM" and "RCPT TO" and that data is
> presented in the Mailet API (aka the Matcher).
>
> HOWEVER ... this information is not necessarily in the actual "DATA"
> part. Therefore, if one was to dump the mail packet to the file system
> (using whatever) then a lot of this data is lost already.
>
> So to say the Mailet API has nothing to do with protocol is
> contradicting the current API and specification. Is it not?
>
> So my thought process is not to actually change anything, but to make
> the Matcher stuff be called as soon as any email data arrives; not just
> when they are all in.
You could create a partial Mail object as soon as the DATA command is
issued. This would give you everything but the body. If the
matcher/mailet try to read the body it will force to wait for the data
to be streamed, otherwise it could do its own matching/processing
anyway. The main problem is that we can't do this because until the data
is completed we don't know if that message is really arrived or not. We
can't start routing a message until we are not sure we "received" it
(until we received the CRLF.CRLF command and replied we accepted it).
What I think is that we should add the missing pieces to make this
"behaviour" effective and working.
Stefano
Re: Fast-Fail Operations in the Mailet API?
Posted by Danny Angus <da...@gmail.com>.
On 11/13/06, Alan Williamson <al...@blog-city.com> wrote:
> We have data that arrives on "MAIL FROM" and "RCPT TO" and that data is
> presented in the Mailet API (aka the Matcher).
No we have data which *may* have arrived at the boundary of our
system in that way but equally may not. It may have arrived thanks to
POP3's RETR, conceivably as a "Message/rfc822" attachment, or even
directly injected into the spool generated by another system.
It is not relevant to either thing that there is a shared concept of
"invoke decision making" what is relevant is that both invocation
frameworks can share the same investment in decision making logic.
d.
Re: Fast-Fail Operations in the Mailet API?
Posted by Alan Williamson <al...@blog-city.com>.
Danny Angus wrote:
>> But to say it is outside of the current Mailet API, is wrong IMO. The
>> Mailet API is for the routing and handling of emails -- all we are
>> talking about is allowing the developer to make this decision much
>> faster/earlier.
>
> Actually the mailet API implies a particular process which starts with
> the full content of the message having already entered the system
okay i can buy that. As with the ServletAPI, i think we should consider
the possible of both. For example, the ServletAPI allows a developer to
process the incoming stream themselves if they want to kill the
connection at any point they can.
BTW - I don't suggest for a minute we have a "getMailetInputStream()"
method, as that is way too ugly for even consideration! :)
> FastFail implies that an interest in the conversation carried out
> during the protocol can result in decisions being taken *before* the
> message has entered the system.
Well before the complete message has arrived. But surely we are already
protocol dependent. Think about it.
We have data that arrives on "MAIL FROM" and "RCPT TO" and that data is
presented in the Mailet API (aka the Matcher).
HOWEVER ... this information is not necessarily in the actual "DATA"
part. Therefore, if one was to dump the mail packet to the file system
(using whatever) then a lot of this data is lost already.
So to say the Mailet API has nothing to do with protocol is
contradicting the current API and specification. Is it not?
So my thought process is not to actually change anything, but to make
the Matcher stuff be called as soon as any email data arrives; not just
when they are all in.
Re: Fast-Fail Operations in the Mailet API?
Posted by Danny Angus <da...@gmail.com>.
> Add a mean for Matcher to specify that they don't care of the message
> but only of the envelope.
+1
>This would allow us to use it in systems wher
> the envelope does not exists. At the same time add a mean to specify
> that the matcher/mailet does not care of the enveloper but only of the
> mimemessage (or only of the mime headers): This would be useful to write
> a "incoming mail rule engine based on mailet apis" for a java based
> email client.
+1
>
> Another improvement to the api would be to give access to the message as
> a stream or as a a mime-tree of streams:
+10,000
> Adding "Fastfail" to the Mailet APIs means (to me) adding some more
> specific object/method/interface to allow processing of partial (or
> better "specific") subset of the current Mail object and not publishing
> JAMES Server (incomplete) handlerapis that are instead a command pattern
> we used to keep our smtpserver modular and more easy to be mantained.
I think the discussion is about how to invoke the processing, that
requires us to think about how you would invoke bespoke processing
during the protocol handling. Thats where the handler api comes in.
> WDYT? Is this something in the middle we could agree upon?
I think we already agree what the components are, we're just
disagreeing with Alan about whether or not the fast fail stuff should
be in the mailet API or a separate fast-fail/handler API.
d.
Re: Fast-Fail Operations in the Mailet API?
Posted by Stefano Bagnara <ap...@bago.org>.
Someone (with James Server knowledge) probably misunderstood my proposal
to support fastfail operations in Mailet API.
I think that publishing what we are currently defining "handlerapi" in
james server as a standard API is a bad idea.
At the same time I think that Mailet APIs must provide a mean to let
"mailets" (or something else) to better interact with scenarios where we
don't have the full message.
And here are some examples of what I think we should/could add to the api.
Add a mean for Matcher to specify that they don't care of the message
but only of the envelope. This would allow us to use it in systems wher
the envelope does not exists. At the same time add a mean to specify
that the matcher/mailet does not care of the enveloper but only of the
mimemessage (or only of the mime headers): This would be useful to write
a "incoming mail rule engine based on mailet apis" for a java based
email client.
Another improvement to the api would be to give access to the message as
a stream or as a a mime-tree of streams: this would allow to write a
more effective spam tool based on the mailet api. (I'm referring to
something like the SpamAssassin rules and the way SA manage targets of
the checks).
This would give us a lot of potentiality: we could parse the message
once, in the container and let multiple mailets to use the parsed data,
we could avoid to even care of the message if all of our mailets do not
read it, and so on.
Summary:
Adding "Fastfail" to the Mailet APIs means (to me) adding some more
specific object/method/interface to allow processing of partial (or
better "specific") subset of the current Mail object and not publishing
JAMES Server (incomplete) handlerapis that are instead a command pattern
we used to keep our smtpserver modular and more easy to be mantained.
WDYT? Is this something in the middle we could agree upon?
Stefano
Re: Fast-Fail Operations in the Mailet API?
Posted by Danny Angus <da...@gmail.com>.
On 11/13/06, Alan Williamson <al...@blog-city.com> wrote:
> But to say it is outside of the current Mailet API, is wrong IMO. The
> Mailet API is for the routing and handling of emails -- all we are
> talking about is allowing the developer to make this decision much
> faster/earlier.
Actually the mailet API implies a particular process which starts with
the full content of the message having already entered the system
FastFail implies that an interest in the conversation carried out
during the protocol can result in decisions being taken *before* the
message has entered the system.
The decision making process may or may not be the same, and the
services used in the decision making are quite likely to be the same.
But crucially the initial assumptions are different, the scenarios are
different, the process to which the decision making is attached is
different and the possible outcomes of the decision making have a
significantly smaller scope.
d.
Re: Fast-Fail Operations in the Mailet API?
Posted by Stefano Bagnara <ap...@bago.org>.
Alan Williamson wrote:
> Noel i am going to ignore the majority of your email, because in essence
> it boils down to your statement here:
>
>> The Mailet pipeline deals with messages and routing, and is
>> independent of
>> protocol. It could not care less how messages are inserted into the
>> spool.
>
> I fully agree with this statement. The key phrase here is "routing".
I agree also, but I want to make sure everyone is aware of the fact that
current API does not route anything: it matches a partial list or
recipients (from 0 to all recipients).
We can evolve it to return a destination for each recipient, but this is
not there: so it does not route anything. It simply match now.
Stefano
Re: Fast-Fail Operations in the Mailet API?
Posted by Alan Williamson <al...@blog-city.com>.
Noel i am going to ignore the majority of your email, because in essence
it boils down to your statement here:
> The Mailet pipeline deals with messages and routing, and is independent of
> protocol. It could not care less how messages are inserted into the spool.
I fully agree with this statement. The key phrase here is "routing".
This isn't about conflating anything, this is about giving the necessary
hooks within the API to intelligently decide on whether or not we wish
to "route" this email any further through the system.
So forget about procotols, forget about where the email is coming from,
it boils down to the simple question:
"Do i want to accept this email?"
Now, ironically enough the Mailet API already answers this question in
the Matcher classes. So I am not sure where your arguments are coming from.
We need to move this processing/decision making further up the chain.
We need to provide developers with the ability to get a peek at the
message as soon as possible.
Yes, the examples you give FetchMail, IMAP (does JAMES even support this
yet?), NNTP (which to be honest with you, anyone using Mailets/JavaMail
to work with an NNTP server is just darn right crazy; there is one
protocol that should never have been in the JavaMail API. We can duke
that one out later!) will all still work. If they do not wish to
override/implement the interface then they don't.
But to say it is outside of the current Mailet API, is wrong IMO. The
Mailet API is for the routing and handling of emails -- all we are
talking about is allowing the developer to make this decision much
faster/earlier.
As for how the container implementation; how the container wants to
handle the non-SMTP cases is up to them. For my own part, I am only
interested in the 99.9999% of cases when emails originate via SMTP.
RE: Fast-Fail Operations in the Mailet API?
Posted by "Noel J. Bergman" <no...@devtech.com>.
Alan Williamson wrote:
> Looks like we are going to disagree on this issue
That depends on whether you want to agree on standardizing on an API for
in-protocol handlers or insist on conflating the requirements of two
distinct and separate containers.
> This sort of dogma bogged down the early days of the Servlet API; many
> argued that servlets were far more than HTTP. They were right; however
> 99.999999% of all servlets are attached to the HTTP protocol. Therefore
> it quickly made sense to push HTTP functionality into the specification.
The lifecycle of Servlets, HTTP Servlets, JSPs, even Portlets and SIP
Servlets are similar. But let's take a look at these others ... consider:
Matcher: asynch stateless message router based on recipient lists
match(Mail) method
Mailet: asynch stateless message transformer/broker -- "the app"
service(Mail) method
In-protocol Handler API:
Stateful and synchronous
Arranged in a chain similar to a Filter Chain
onConnect -- connection from remote client
on<Command> -- receipt of a specific command
onMessage -- receipt of a completed message
They have in common the thing that you don't like: the
"onMessage/match/service" method, which takes a completed message. The SMTP
protocol handler is one source for such messages to the Mailet container.
So is FetchMail. So is the NNTP protocol handler. So could be IMAP or
Jabber. Etc.
The Mailet pipeline deals with messages and routing, and is independent of
protocol. It could not care less how messages are inserted into the spool.
As you noted, a Mailet Container is protocol independent. There is a
specific contract that the container has with its components. The contract
that exists with protocol handling containers and *their* components
(in-protocol handlers) is different. It would be entirely unreasonable (and
quite poor design) to insist that all containers implement both contracts.
> > As I said, more and more we're moving to handle filtering in the
protocol
> > handlers, and have just "application" type code in the pipeline, e.g.,
> > mailing list managers, delivery support, etc.
> But that is JAMES specific. I am not interested in JAMES.
We have already said that if there is interest in the in-protocol handler
API, that we'd be interested in exploring it, too. But we would be talking
about three things:
- Mailet API
- Handler API
- Common Services
I'm curious to see if Common Services turns out to be the most contentious
part.
--- Noel
Fast-Fail Operations in the Mailet API?
Posted by Alan Williamson <al...@blog-city.com>.
Looks like we are going to disagree on this issue Noel
Noel J. Bergman wrote:
> Well, that is because you're conflating two different issues: Matchers,
> which deal with selecting messags in a message pipeline, and in-protocol
> handlers that act on events as they happen during protocol transfer.
The message pipeline as you call it, in this instance is SMTP. The
pipeline *is* by definition a transfer; moving data from one place to
another. Doesn't really matter HOW that happens, thats protocol
specific. By your own logic, the fast-fail operations belong in the
Mailet API.
But lets wind back here, because I think you're being maybe a little too
pragmatic here.
"The Mailet API is an API designed to facilitate the development
and deployment of a configurable email processing applications"
That is lifted straight from the Mailet WIKI.
There is nothing in that statement that suggests putting in the
hooks/interfaces I have suggested would violate that statement; infact,
one argues it strengthens it.
> As I said, more and more we're moving to handle filtering in the protocol
> handlers, and have just "application" type code in the pipeline, e.g.,
> mailing list managers, delivery support, etc.
But that is JAMES specific. I am not interested in JAMES. If JAMES is
the only container implementing the Mailet API then this whole
discussion is somewhat moot and academic. How you guys handle it
doesn't really matter, since no one else but JAMES is using it.
If however, the group is keen for the Mailet API to grow beyond JAMES
then one has to look at where the vast majority of Mailet development is
going to be done by Java developers; listening to emails coming in from
SMTP. Can we at least agree on that? Everything else, JAMES will
probably do for them anyway.
This sort of dogma bogged down the early days of the Servlet API; many
argued that servlets were far more than HTTP. They were right; however
99.999999% of all servlets are attached to the HTTP protocol. Therefore
it quickly made sense to push HTTP functionality into the specification.
all fun fun on the frontline of API design!
:)
RE: Introductions ...
Posted by "Noel J. Bergman" <no...@devtech.com>.
Alan Williamson wrote:
>> Danny Angus wrote:
>> Alan Williamson wrote:
>>> I am a little confused as to why Danny/Noel believe it isn't part of the
>>> Mailet API? What is the remit of the Mailet API? What is its purpose
>>> in life if not a hook for developers to process mails?
>> Yeah, but the API doesn't assume that the system doing the processing
>> is also implementing any of the mail protocols. It could be that the
>> mail is all sitting in an archive and needs to be mined for some
>> reason.
> Gotcha, that makes sense from that aspect.
>> Therefore protocol related API's are out of scope, but that doesn't
>> make it a bad idea.
> I guess this is always the balance, purist vs. reality. In reality the
> majority of mailets are probably being called upon the SMTP trigger,
> therefore it makes sense to involve that protocols behaviour in the API.
The Mailet API is driving mailets in a container that receives messages.
Usually those arrive in some sort of spool. The spool could be distributed,
and processed on systems that don't have anything to do with SMTP (or other
protocol) transports.
Look at it this way: the Mailet API has about as much in common with the
in-protocol handler API as a Message-Driven Bean has in common with the ORB.
They have totally different semantics, event structures, transport and
transaction issues, etc. This is why I object to conflating the two areas.
> you could make a strong argument, that the Matcher classes are
> invoked at the wrong part of the chain. Maybe they should be
> invoked as and when the email address is available, and not
> when the whole body has arrived.
Well, that is because you're conflating two different issues: Matchers,
which deal with selecting messags in a message pipeline, and in-protocol
handlers that act on events as they happen during protocol transfer.
Where we will have far more agreement, and where we are refactoring JAMES
code today, is that there are commonalities and places where you want to
factor out underlying behavior that can be shared by in-protocol handlers
and matcher/mailet code. So SpamAssassin can be invoked by an in-protocol
handler (more and more mail *filtering* code is done in the handlers), or it
could be invoked in the message pipeline.
As I said, more and more we're moving to handle filtering in the protocol
handlers, and have just "application" type code in the pipeline, e.g.,
mailing list managers, delivery support, etc.
--- Noel
Re: Introductions ...
Posted by Alan Williamson <al...@blog-city.com>.
Danny Angus wrote:
>> I am a little confused as to why Danny/Noel believe it isn't part of the
>> Mailet API? What is the remit of the Mailet API? What is its purpose
>> in life if not a hook for developers to process mails?
>
> Yeah, but the API doesn't assume that the system doing the processing
> is also implementing any of the mail protocols. It could be that the
> mail is all sitting in an archive and needs to be mined for some
> reason.
Gotcha, that makes sense from that aspect.
> Therefore protocol related API's are out of scope, but that doesn't
> make it a bad idea.
I guess this is always the balance, purist vs. reality. In reality the
majority of mailets are probably being called upon the SMTP trigger,
therefore it makes sense to involve that protocols behaviour in the API.
However, you could make it completely protocol agnostic and treat it a
little like how Servlets manage HEAD/GET/POST/DELETE etc.
For example, we are discussing whether or not we want to process the
mail. Therefore, you could make a strong argument, that the Matcher
classes are invoked at the wrong part of the chain. Maybe they should
be invoked as and when the email address is available, and not when the
whole body has arrived.
So for SMTP that would be in response to a MAIL FROM / RCPT TO; for IMAP
that would be when the email was downloaded (does JAMES support IMAP
btw?); etc.
Right ... i really have to go for tonight, i have 3 boys waiting for
their father to watch a movie!!! Have a good evening gentlemen.
Re: Introductions ...
Posted by "Andrew C. Oliver" <an...@superlinksoftware.com>.
Serge Knystautas wrote:
> On 11/11/06, Alan Williamson <al...@blog-city.com> wrote:
>
> I think originally it was just addressing mail processing in general,
> which some people liked because it opened the door for mailets with
> other protocols (NNTP, jabber, etc...). I came around to wanting it
> exclusively for SMTP. I think that while mime and some other notions
> are shared between SMTP, NNTP, and others, the differences are too
> wide to have one API effectively solve all those problems.
>
> I'm +1 to adding the handler API stuff into the mailet API.
>
I don't feel exceptionally strongly about it because we're architected
roughly the same way if I understand
"the handler stuff" correctly (each command is separate with a
reflectional lookup that finds that command)
minus that last step of configuration. However, this design decision is
probably not a great thing to force
with the mailet API for implementors. Assuming other mailservers (say a
postfix) overcame the cumbersome
requirement of Java to plug this in, somehow I doubt they'll be willing
to rearchitect in this manner. Though
the system of queues, message "objects" and listeners is fairly universal.
-Andy
Re: Introductions ...
Posted by Serge Knystautas <sk...@gmail.com>.
On 11/11/06, Alan Williamson <al...@blog-city.com> wrote:
> > Therefore protocol related API's are out of scope, but that doesn't
> > make it a bad idea.
>
> I guess this is always the balance, purist vs. reality. In reality the
> majority of mailets are probably being called upon the SMTP trigger,
> therefore it makes sense to involve that protocols behaviour in the API.
I think originally it was just addressing mail processing in general,
which some people liked because it opened the door for mailets with
other protocols (NNTP, jabber, etc...). I came around to wanting it
exclusively for SMTP. I think that while mime and some other notions
are shared between SMTP, NNTP, and others, the differences are too
wide to have one API effectively solve all those problems.
I'm +1 to adding the handler API stuff into the mailet API.
--
Serge Knystautas
Lokitech >> software . strategy . design >> http://www.lokitech.com
p. 301.656.5501
e. sergek@lokitech.com
Re: Introductions ...
Posted by Danny Angus <da...@gmail.com>.
On 11/11/06, Alan Williamson <al...@blog-city.com> wrote:
> I am a little confused as to why Danny/Noel believe it isn't part of the
> Mailet API? What is the remit of the Mailet API? What is its purpose
> in life if not a hook for developers to process mails?
Yeah, but the API doesn't assume that the system doing the processing
is also implementing any of the mail protocols. It could be that the
mail is all sitting in an archive and needs to be mined for some
reason.
Therefore protocol related API's are out of scope, but that doesn't
make it a bad idea.
d.
Re: Introductions ...
Posted by Alan Williamson <al...@blog-city.com>.
Danny Angus wrote:
> On 11/11/06, Noel J. Bergman <no...@devtech.com> wrote:
>
>> That isn't part of the Mailet API at all, and doesn't belong there
>> IMO. In
>> JAMES, we have pluggable in-protocol handlers that are connected to such
>> events. I do not know that anyone wants to standardize on our
>> in-protocol
>> plug-in architecture, which we're still evolving, but that seems separate
>> from Mailets.
>
> I agree that it is separate from mailets, but theres no reason why it
> couldn;t be another independant API which others could implement, It
> does solve the problem by providing an abstraction and it would be (as
> we already know) a different way to invoke some of the same underlying
> services, so there are some similarities.
yeah, i agree with Noel that it shouldn't be part of the actual class:
org.apache.mailet.Mailet
But it should be another interface, that sits within the
[org.apache.mailet] package.
Isn't the Mailet API everything under this package?
I am a little confused as to why Danny/Noel believe it isn't part of the
Mailet API? What is the remit of the Mailet API? What is its purpose
in life if not a hook for developers to process mails?
please forgive me, i have just joined the list, so i am a little unclear
as to the scope of this list and the scope of Mailets.
RE: Introductions ...
Posted by "Noel J. Bergman" <no...@devtech.com>.
Danny Angus wrote:
> > That isn't part of the Mailet API at all, and doesn't belong there IMO.
In
> > JAMES, we have pluggable in-protocol handlers that are connected to such
> > events. I do not know that anyone wants to standardize on our
in-protocol
> > plug-in architecture, which we're still evolving, but that seems
separate
> > from Mailets.
> I agree that it is separate from mailets, but theres no reason why it
> couldn't be another independant API which others could implement
Absolutely. My point is not to conflate the two separate APIs.
--- Noel
Re: Introductions ...
Posted by Danny Angus <da...@gmail.com>.
On 11/11/06, Noel J. Bergman <no...@devtech.com> wrote:
> That isn't part of the Mailet API at all, and doesn't belong there IMO. In
> JAMES, we have pluggable in-protocol handlers that are connected to such
> events. I do not know that anyone wants to standardize on our in-protocol
> plug-in architecture, which we're still evolving, but that seems separate
> from Mailets.
I agree that it is separate from mailets, but theres no reason why it
couldn;t be another independant API which others could implement, It
does solve the problem by providing an abstraction and it would be (as
we already know) a different way to invoke some of the same underlying
services, so there are some similarities.
d.
Re: Introductions ...
Posted by Norman Maurer <nm...@byteaction.de>.
Danny Angus schrieb:
> On 11/11/06, Noel J. Bergman <no...@devtech.com> wrote:
>
>> That isn't part of the Mailet API at all, and doesn't belong there
>> IMO. In
>> JAMES, we have pluggable in-protocol handlers that are connected to such
>> events. I do not know that anyone wants to standardize on our
>> in-protocol
>> plug-in architecture, which we're still evolving, but that seems
>> separate
>> from Mailets.
>
> I agree that it is separate from mailets, but theres no reason why it
> couldn;t be another independant API which others could implement, It
> does solve the problem by providing an abstraction and it would be (as
> we already know) a different way to invoke some of the same underlying
> services, so there are some similarities.
>
> d.
I fully agree with you danny... It whould be really cool to provide such
an API!
bye
Norman
Re: Introductions ... re noel
Posted by Stefano Bagnara <ap...@bago.org>.
Alan Williamson wrote:
>> I agree with the latter, but the former is addressed by the
>> aforementioned
>> in-protocol handlers. Perhaps you're just behind the times with JAMES.
>
> I respectfully disagree Noel.
>
> I don't wish to talk about JAMES. Not because I don't value it as a
> project but because IMHO it has no purpose on a discussion list about
> the Mailet API.
I partially agree with this statement: I don't want us to make the big
mistake to discuss an abstract api without concrete goals.
I think future "implementors" of the API are the one that should
influence the API itself, and that we all (implementors) should work on
something concrete and only after having tested it as working in a real
environment try to standardize the API.
We should start creating a list of limitations of the current APIs and a
list of things we believe should be feasible from a Mailet API user (the
developer) perspective.
Once (if) we'll have agreed what features we want to support and what
features we want to keep out from the apis we can discuss the best api
to provide the required extensibility.
> We no more want to talk about Tomcat or JBoss when it comes to
> discussing the Servlet API. JAMES is merely *one* container
> implementation of the Mailet API - we now have an alternative
> implementation in MailCatcher.
>
> JAMES is probably suffering from being only the rooster in the farm; but
> that is something we would like to address and offer people a choice
> when it comes to Mailet containers.
>
> If one wants to use the Servlet API again as an example, i see JAMES as
> the full blown "J2EE" reference implementation, where as the Mailet
> stuff, is like the Servlet Containers (ServletExec, Tomcat etc).
>
> So the question boils down to this; does one want to see the Mailet API
> grow as a separate project, or is this really a JAMES discussion?
I think we should grow it as a separate project, but I think we should
keep discussing about how currently James Server and MailCacher (and any
other implementation around) is trying to solve commons problem with the
current mailet apis or without them.
To understand better each other we should probably study each other
products. I already download mailcatcher.zip ...
Stefano
RE: Introductions ... re noel
Posted by "Noel J. Bergman" <no...@devtech.com>.
> does one want to see the Mailet API grow as a separate project,
> or is this really a JAMES discussion?
That still isn't the point. The point is that you are conflating two
separate and not really related issues: the Mailet API, which deals with
selecting and processing messages; and a protocol handler API for dealing
with in-protocol events. I am happy to discuss either or both, but not to
conflate the two. Again, as I said in my initial reply:
we have pluggable in-protocol handlers that are connected to such
events. I do not know that anyone wants to standardize on our in-protocol
plug-in architecture, which we're still evolving, but that seems separate
from Mailets.
The "fast-fail" (the original name for it, but "in-protocol handler" is
better since failure is just one application of it) API is separate from
matchers and mailets.
I also don't agree with your thought that:
> i agree with Noel that it shouldn't be part of the actual class:
> org.apache.mailet.Mailet
> But it should be another interface, that sits within the
> [org.apache.mailet] package.
I don't see it as being part of the Mailet package, but that's a
nomenclature issue. If there is interest in standardizing on that API, too,
that's still a fine idea.
--- Noel
Re: Introductions ... re noel
Posted by Alan Williamson <al...@blog-city.com>.
> I agree with the latter, but the former is addressed by the aforementioned
> in-protocol handlers. Perhaps you're just behind the times with JAMES.
I respectfully disagree Noel.
I don't wish to talk about JAMES. Not because I don't value it as a
project but because IMHO it has no purpose on a discussion list about
the Mailet API.
We no more want to talk about Tomcat or JBoss when it comes to
discussing the Servlet API. JAMES is merely *one* container
implementation of the Mailet API - we now have an alternative
implementation in MailCatcher.
JAMES is probably suffering from being only the rooster in the farm; but
that is something we would like to address and offer people a choice
when it comes to Mailet containers.
If one wants to use the Servlet API again as an example, i see JAMES as
the full blown "J2EE" reference implementation, where as the Mailet
stuff, is like the Servlet Containers (ServletExec, Tomcat etc).
So the question boils down to this; does one want to see the Mailet API
grow as a separate project, or is this really a JAMES discussion?
RE: Introductions ...
Posted by "Noel J. Bergman" <no...@devtech.com>.
> However the Mailet API itself, while beautiful is missing a number of
> significant hooks that we have started to address and will update the
> main MailCatcher project soon.
> Namely there is no call-back when the MAIL FROM or RCPT TO is invoked by
> a remote SMTP server.
That isn't part of the Mailet API at all, and doesn't belong there IMO. In
JAMES, we have pluggable in-protocol handlers that are connected to such
events. I do not know that anyone wants to standardize on our in-protocol
plug-in architecture, which we're still evolving, but that seems separate
from Mailets.
> The problem with the current implementation of the API is that one must
> accept the whole email before you can do any processing as to whether or
> not you want it. For spam control, this is completely unacceptable.
I agree with the latter, but the former is addressed by the aforementioned
in-protocol handlers. Perhaps you're just behind the times with JAMES.
--- Noel