You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by "Noel J. Bergman" <no...@devtech.com> on 2005/05/08 05:43:58 UTC
Wanted: Volunteers for Fast-Fail Proposal/Implementation
There are a variety of changes to make to JAMES now, and here is one that is
high on my list. Since I am starting a 3 week stint on the road, I am
hoping to get a few others to collaborate on this with me.
Two items I want to run by the list.
In-Protocol processor for SMTP handler:
Once we have a message ready for posting, instead of just
putting it in the spool, we post it as a "pre-spool" and
call a processor on it.
The processor can use matchers/mailets, but in the end,
the message either gets spooled to some processor (e.g.,
root) or we get an SMTP response to return in-protocol
to the sender.
That should unify and clear up remaining issues. It builds on some of the
work started by Alex, incorporates other ideas that have been put forth,
keeps us as consistent as possible with the normal pipeline (we should be
able to re-use all matchers, a number of mailets, and processor code; and
gives us more flexibility. One could even post to different starting points
in the pipeline if desired.
Each SMTP server would have a unique in-protocol, bootstrap, processor:
<smtpserver ...>
...
<processor ...>
</processor>
...
</smtpserver>
This would allow an SMTP server that uses SMTPS and requires SMTP AUTH to
handle e-mail differently from a normal SMTP server that handles arbitrary
messages.
As Vincenzo said when I mentioned this to him, "It's a spool like the
others, but [the] the smtp connection/protocol is kept open, waiting to send
back [an SMTP response]." It should not be used for processing that does
not effect in-protocol responses, since we need to complete all processing
before we can generate the response to the message.
The processor handles messages, which means late in the SMTP process. The
other of the two things I want to discuss is a change to how the SMTP
handler is structured. I want to suggest a chained approach, although I'm
not entirely sure of which of a couple directions would be best.
The code currently in the handler that pulls and parses messages would be
factored out. The code that it dispatches to would be formalized in an
interface for the protocol handler. Each event: connection, HELO, MAIL
FROM, RCPT TO, etc. would have a corresponding method in the protocol
specific interface. Our code would implement that interface, but so would
fast-fail filters. The outside loop, either MINA based or socket-based,
would take the each complete command and dispatch it to the chain, at which
point either the container would iterate over the handler chain, or we would
use a doNext type approach. This would give us a nice pluggable
architecture for fast-fail on a per-event basis.
Combined, these would allow Virtual User Tsble mapping during protocol
handling, allowing better response to invalid addresses; DNS checks; spam
checks, etc.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Wanted: Volunteers for Fast-Fail Proposal/Implementation
Posted by Danny Angus <da...@gmail.com>.
On 10/05/05, Noel J. Bergman <no...@devtech.com> wrote:
> Danny Angus wrote:
> > I would propose an alternative architecture for James (not the mailet
> > API yet) which looks like this..
>
> I am feeling a bit snarky at the moment, but did you and the others who
> replied, happen to read the REST of my e-mail, or just the first part?
Guilty as charged. I did but it didn't register as I was obsessing
about the first bit.
Sorry it was both disrespectful and unprofessional of me.
> Everyone developed tunnel vision on the in-process processor, and not one of
> you commented on the protocol handler chain, which had already addressed
> every one of your concerns. To repeat, with emphasis:
Ok, I'll address it this time..
> "The *OTHER* of the *TWO* things I want to discuss is a change to how the
> SMTP handler is structured. I want to suggest a chained approach, although
> I'm not entirely sure of which of a couple directions would be best.
Still with you ;-)
>
> The code currently in the handler that pulls and parses messages would be
> factored out. The code that it dispatches to would be formalized in an
> interface for the protocol handler.
Yes we're still at the same party...
> Each event: connection, HELO, MAIL
> FROM, RCPT TO, etc. would have a corresponding method in the protocol
> specific interface.
Extrapolate this to its logical conclusion, have the protocol contain
a Map of Command object delegates, and select the delegate based upon
the command itself as the key.
> Our code would implement that interface, but so would
> fast-fail filters.
I'm proposing that the filters, i'd called them rules, would be called
by the command handlers, using the event listener pattern.
> The outside loop, either MINA based or socket-based,
> would take the each complete command and dispatch it to the chain, at which
> point either the container would iterate over the handler chain, or we would
> use a doNext type approach. This would give us a nice pluggable
> architecture for fast-fail on a per-event basis."
>
> I even laid out how this fits in with Mike Heath's work on MINA, and how we
> can accommodate the use of *either* MINA or the current socket-based code.
>
> The in-protocol processor and protocol handler chain are complementary
> solutions.
Not arguing anymore!
In summary I still think the in-process-processor is a bit hard to
swallow probably but this is probaly my fault because I don't
understand the requirement.I also think that the fast fail requirement
should be satisfied by a wholly assembled approach to protocols,
towards which your proposal goes some way, but not as far as I would
go.
d.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
MINA SMTP hanlder
Posted by ap...@bago.org.
> The MINA based SMTP handler I've developed already supports
> extensions as Danny has described. For examples, The
> STARTTLS command handler was just plugged into the framework
> and didn't require modifying anything else. My current
> design still needs some additional extensions points and
> improved manageability (like the ability to remove an extension
> dynamically) but I think it's a step in the right direction.
Hi Mike,
I'm interested in your SMTP implementation's extensibility because I would
like to support ESMTP-DSN extension and I need to handle additional strings
passed after the MAIL FROM: and the RCPT TO: is this going to be allowed
from your extension mechanism?
PS: ESMTP-DSN requests are in the form of:
MAIL FROM: <te...@domain.com> ENVID=test
RCPT TO: <te...@domain.com> NOTIFY=DELAY,SUCCESS
ORCPT=rfc822;original-recipient@domain.com
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Wanted: Volunteers for Fast-Fail Proposal/Implementation
Posted by Mike Heath <mi...@cycletime.com>.
On Mon, 2005-05-09 at 10:09 +0100, Danny Angus wrote:
<snip>
> 1/ it tries to force the mailet architecture into a role for which it
> isn;t suited
> 2/ it doesn't satisfy the requirement that the server be capable of
> terminating the conversation after any command which resolves to a
> predicted temprorary _or_ permamnent failure to deliver.
>
>
> I would propose an alternative architecture for James (not the mailet
> API yet) which looks like this..
>
> ----
> ProtocolHandler - parses commands per protocol spec.
> CommandHandler - matches command(s) and responds to the protocol
> handler with a response code & message.
> CommandRule - called by command handler to validate command, responds
> with a response code & message
> ----
> Configuration process assigns command handlers to protocol handlers,
> and assigns CommandRules to command handlers.
> ----
> In this way we build up functionality out of simple-to-develop &
> simple-to-deploy classes.
>
> This (IMHO);
> - matches the simplicity which has been one of mailet's strengths
> - is flexible and doesn't pre-judge individual requirements
> - meets the general requirement for fast-fail, minimising resource
> consumption by unwanted messages, and adherence to protocol RFC's
> - architecture can be applied to all protocols (not just smtp)
> - future proofs existing protocols by encapsulating commands such that
> commands can be modified, added to, reused etc. etc.
>
In spite of the fact that I brought up a similar idea to Noel's on the
list a while ago, I've changed my stance and have to agree with Danny.
I like Danny's above comments because they resonate really well with the
work I've been doing with MINA. As I've thought about this problem, it
seams like a much better design to modularize the SMTP handler to
facilitate fast fails. Trying to pull the mailet system in seams to be
opening a can of worms.
<snip>
> I'd like to have a crack at this too, along the lines mentioned,
> because I believe that this will also give us an opportunity for ESMTP
> support to accept externally developed Extensions, and for Extensions
> to be deployed by configuration (declaratively).
>
> It would also provide a framework for releasing minimal compliant
> protocol handlers, such as IMAP and next-generation SMTP and adding to
> or replacing features of these with no impact on existing functions.
The MINA based SMTP handler I've developed already supports extensions
as Danny has described. For examples, The STARTTLS command handler was
just plugged into the framework and didn't require modifying anything
else. My current design still needs some additional extensions points
and improved manageability (like the ability to remove an extension
dynamically) but I think it's a step in the right direction.
Danny, I'm interested in your ideas and would like to take a look at
your architecture and see where I can make improvements to what I'm
doing.
-Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Wanted: Volunteers for Fast-Fail Proposal/Implementation
Posted by Danny Angus <da...@gmail.com>.
On 09/05/05, apache@bago.org <ap...@bago.org> wrote:
> In a REAL SMTP environment I think we need to allow custom behaviour in
> reply to the "RCPT TO" (recipient not local, relay not allowed) and in reply
> to the "DATA" commands (this is spam, mail too big, no virus please, other
> content filters).
My proposal addresses this.
> The latter could be addressed with the pre-processor (but please find a
> solution to my question: what happen if the preprocessor splits the mail?)
> but the rcpt reply not.
My proposal would have any rule in any command handler of any protocol
capable of returning a hard or soft fail response with a custom
message generated in response to the exact mode of failure, by the
rule that judged the command to fail.
> For the "RCPT TO:" stuff I think we should add a specific handler, but I
> would hardcode the management of this handler and leave for later the
> automatic extensibility for each command!
Why? If we simply refactor the doXYZ() methods into
ProtocolCommand.execute(String command, Object state)
we get the architecture. I did HELO, EHLO & RCPT TO last night.
> Can you point to other situations in the SMTP session that needs
> extensibility at deploy time?
To add extended commands per ESMTP without having to modify the core
SMTP commands.
d.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Wanted: Volunteers for Fast-Fail Proposal/Implementation
Posted by ap...@bago.org.
In a REAL SMTP environment I think we need to allow custom behaviour in
reply to the "RCPT TO" (recipient not local, relay not allowed) and in reply
to the "DATA" commands (this is spam, mail too big, no virus please, other
content filters).
The latter could be addressed with the pre-processor (but please find a
solution to my question: what happen if the preprocessor splits the mail?)
but the rcpt reply not.
For the "RCPT TO:" stuff I think we should add a specific handler, but I
would hardcode the management of this handler and leave for later the
automatic extensibility for each command!
Can you point to other situations in the SMTP session that needs
extensibility at deploy time?
Perhaps one could add a orb lookup when accepting a connection and reply a
"451 you are a spammer, we are not here for you" instead of the "220 server
ready" but is the only thing I can imagine: the big deal now is in the "RCPT
TO", the virtusertable and non-existant/full mailboxes.
Stefano
> I would propose an alternative architecture for James (not
> the mailet API yet) which looks like this..
>
> ----
> ProtocolHandler - parses commands per protocol spec.
> CommandHandler - matches command(s) and responds to the
> protocol handler with a response code & message.
> CommandRule - called by command handler to validate command,
> responds with a response code & message
> ----
> Configuration process assigns command handlers to protocol
> handlers, and assigns CommandRules to command handlers.
> ----
> In this way we build up functionality out of
> simple-to-develop & simple-to-deploy classes.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Wanted: Volunteers for Fast-Fail Proposal/Implementation
Posted by ap...@bago.org.
> > > In-Protocol processor for SMTP handler:
>
> > I don't like this, sorry mate, because (pls read all of this, it is
> > *constructive*):
>
> I am feeling a bit snarky at the moment, but did you and the
> others who replied, happen to read the REST of my e-mail, or
> just the first part?
Hi Noel,
I read both parts, but I've doubt on the first one: the In-protocol
processor.
I think that we should first improve (add) delivery status tracking in every
james part, then we can simply add the "in-protocol" handler or make the
root processor synchronized with the smtphandler (so that it returns SMTP
errorcodes instead of bounces).
We need informations from mailet about what they did with our messages: we
need this for your in-protocol processor but I think we more need this for a
good DSN behaviour. I already tried to discuss some solutions for this but
haven't received much feedback.
Look at the comment here: http://issues.apache.org/jira/browse/JAMES-322
In fact we need a delivery status tracking object that mailets and the
linearprocessor (perhaps the mailetcontext too) will update with their
informations: SMTP in-protocol processor or DSNBounce mailet will then use
this object to decide what to do.
Is there an extensible way to provide this feature? Any idea?
I think that once resolved the DSN tracking issue for ESMTP-DSN support then
we will have a simpler solution than the in-protocol processor.
I agree about your second proposal: I would like further details and better
I would like to know if Mike's work on the MINA based SMTP handler already
provide this behaviour (when there is ready code is better to start from
there instead of running utopic discussions ;-) ).
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Wanted: Volunteers for Fast-Fail Proposal/Implementation
Posted by ap...@bago.org.
> "The *OTHER* of the *TWO* things I want to discuss is a
> change to how the SMTP handler is structured. I want to
> suggest a chained approach, although I'm not entirely sure of
> which of a couple directions would be best.
Let me try to do an example use-case and understand your solution:
Let's say we have a james configured with many processors and many
matcher/mailets that provide specific
aliasing/forwarding/virtusertable/localdelivery. If we want to add fast-fail
per recipient then we need to create a tree of SMTPHandler plugins that
provides functionalities similar to the mailet we used and add a
configuration to the SMTPHandler to specify our processing. Am i correct?
I really don't like the configuration replication but I think there is no
chance to reuse current matcher/mailets and their configuration to produce a
per recipient fast-fail.
I still hope we can find a solution where we move configurations without
replicate them.
If we need to create an object chain (SMTP handler plugins) that given the
currenct SMTP transaction status (sender IP, sender MAIL FROM, the last
recipient RCPT TO) is able to run through virtusertables, aliasing,
forwarding, local user table and provide a good reply (mailbox full, mailbox
disabled, relay not allowed/mailbox not local, user unknown) (We can even
supports the VRFY command this way), then why should't we use the same
objects in a custom mailet running as the first mailet in the root
processor? I mean: why should I configure the SMTP server to check my
virtusertable and localusertable and then do it again in my
spooler/processors? I don't like james to run twice the same code for each
mail!
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: Wanted: Volunteers for Fast-Fail Proposal/Implementation
Posted by "Noel J. Bergman" <no...@devtech.com>.
Danny Angus wrote:
> > Two items I want to run by the list.
>
> > In-Protocol processor for SMTP handler:
> I don't like this, sorry mate, because (pls read all of this, it is
> *constructive*):
> I would propose an alternative architecture for James (not the mailet
> API yet) which looks like this..
I am feeling a bit snarky at the moment, but did you and the others who
replied, happen to read the REST of my e-mail, or just the first part?
Everyone developed tunnel vision on the in-process processor, and not one of
you commented on the protocol handler chain, which had already addressed
every one of your concerns. To repeat, with emphasis:
"The *OTHER* of the *TWO* things I want to discuss is a change to how the
SMTP handler is structured. I want to suggest a chained approach, although
I'm not entirely sure of which of a couple directions would be best.
The code currently in the handler that pulls and parses messages would be
factored out. The code that it dispatches to would be formalized in an
interface for the protocol handler. Each event: connection, HELO, MAIL
FROM, RCPT TO, etc. would have a corresponding method in the protocol
specific interface. Our code would implement that interface, but so would
fast-fail filters. The outside loop, either MINA based or socket-based,
would take the each complete command and dispatch it to the chain, at which
point either the container would iterate over the handler chain, or we would
use a doNext type approach. This would give us a nice pluggable
architecture for fast-fail on a per-event basis."
I even laid out how this fits in with Mike Heath's work on MINA, and how we
can accommodate the use of *either* MINA or the current socket-based code.
The in-protocol processor and protocol handler chain are complementary
solutions.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Wanted: Volunteers for Fast-Fail Proposal/Implementation
Posted by Danny Angus <da...@gmail.com>.
On 08/05/05, Noel J. Bergman <no...@devtech.com> wrote:
> There are a variety of changes to make to JAMES now, and here is one that is
> high on my list. Since I am starting a 3 week stint on the road, I am
> hoping to get a few others to collaborate on this with me.
>
> Two items I want to run by the list.
>
> In-Protocol processor for SMTP handler:
>
> Once we have a message ready for posting, instead of just
> putting it in the spool, we post it as a "pre-spool" and
> call a processor on it.
>
> The processor can use matchers/mailets, but in the end,
> the message either gets spooled to some processor (e.g.,
> root) or we get an SMTP response to return in-protocol
> to the sender.
Noel,
I don't like this, sorry mate, because (pls read all of this, it is
*constructive*):
1/ it tries to force the mailet architecture into a role for which it
isn;t suited
2/ it doesn't satisfy the requirement that the server be capable of
terminating the conversation after any command which resolves to a
predicted temprorary _or_ permamnent failure to deliver.
I would propose an alternative architecture for James (not the mailet
API yet) which looks like this..
----
ProtocolHandler - parses commands per protocol spec.
CommandHandler - matches command(s) and responds to the protocol
handler with a response code & message.
CommandRule - called by command handler to validate command, responds
with a response code & message
----
Configuration process assigns command handlers to protocol handlers,
and assigns CommandRules to command handlers.
----
In this way we build up functionality out of simple-to-develop &
simple-to-deploy classes.
This (IMHO);
- matches the simplicity which has been one of mailet's strengths
- is flexible and doesn't pre-judge individual requirements
- meets the general requirement for fast-fail, minimising resource
consumption by unwanted messages, and adherence to protocol RFC's
- architecture can be applied to all protocols (not just smtp)
- future proofs existing protocols by encapsulating commands such that
commands can be modified, added to, reused etc. etc.
I have been working on an architecture for mailet, which is attempting
to make better use of polymorphism to increase the number of roles
that can be played by a single class, and reduce the number of
distinct responsibilities in the system.
I'd like to have a crack at this too, along the lines mentioned,
because I believe that this will also give us an opportunity for ESMTP
support to accept externally developed Extensions, and for Extensions
to be deployed by configuration (declaratively).
It would also provide a framework for releasing minimal compliant
protocol handlers, such as IMAP and next-generation SMTP and adding to
or replacing features of these with no impact on existing functions.
d.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Wanted: Volunteers for Fast-Fail Proposal/Implementation
Posted by ap...@bago.org.
> In-Protocol processor for SMTP handler:
> [...]
> The processor can use matchers/mailets, but in the end,
> the message either gets spooled to some processor (e.g.,
> root) or we get an SMTP response to return in-protocol
> to the sender.
Let me understand:
1. at the end of this first processor you write the response code to the
"DATA" smtp code?
2. What does it happens to a message with 2 recipients one of them being
unavailable? Remember that each matcher of this pre-processor could split
the mail in multiple messages and you would have multiple output for a
single input. How do you think to handle this?
3. What informations will be added to the mail in order to have correct SMTP
responses?
4. a mailetContext.sendMail will then put the messages inside this processor
or in the spooler's root processor?
> [...]
> The code currently in the handler that pulls and parses
> messages would be factored out. The code that it dispatches
> to would be formalized in an interface for the protocol
> handler. Each event: connection, HELO, MAIL FROM, RCPT TO,
> etc. would have a corresponding method in the protocol
> specific interface. Our code would implement that interface,
> but so would fast-fail filters. The outside loop, either
> MINA based or socket-based, would take the each complete
> command and dispatch it to the chain, at which point either
> the container would iterate over the handler chain, or we
> would use a doNext type approach. This would give us a nice
> pluggable architecture for fast-fail on a per-event basis.
>
> Combined, these would allow Virtual User Tsble mapping during
> protocol handling, allowing better response to invalid
> addresses; DNS checks; spam checks, etc.
Again, if you move virtusertable in a smtp handler or something that is not
the spooler then how do you handle messages sent by mailetContext.sendMail?
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org