You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-user@james.apache.org by Maxime Levesque <ma...@webhelp.com> on 2002/12/05 15:59:15 UTC
fault tolerance of smtp server and mailets
I have the following questions regarding James :
i) Suppose Jame's smtp server is midway in swallowing an email and
I pull the plug on the machine, what will happen ?
ii) When a mailet's service method gets called, is the mail read
from the spool file ?
iii) If a mailet's service method is midway in execution and I pull the
plug, what happens ?
In general, how can one achieve guaranteed "once and only once" email
receival with James ?
Suppose I have the following email pipeline, and I want the guarantee
that : "if an email makes it's way to the smtp middleman, it will
eventually result in one execution of myMailet's service method." :
EMailCLient---> smtpd(sendmail?)--->james(SMTP->james-spool)->myMailet
Should I configure sendmail to retry upon failure (lets say, retry up to
5 times with a 10 minute sleep in between) ?
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance of smtp server and mailets
Posted by Danny Angus <da...@apache.org>.
Maxime,
> From what I understand, the spool is like a producer/consumer queue,
> from which the JamesSmtp server
> enqueues mails, and mailet threads asynchronously dequeue mails for
> processing. The enqueue and dequeue operations
> both have a small window where duplication can occur.
Almost.. in fact the dequeue operation occurs in two stages, first a mail is locked by james, preventing its access by other de-spooling threads.
Processing by the mailets occurs.
The mail is then removed, or moved to another spool (for another processor).
The opportunity for duplication of mail occurs in the processing step, where a mailet may have completed processing and mark the mail to be removed from the spool, or moved to another spool.
Termination at this point is possible before the actual removal/move is actioned.
The spool action is only carried out by the container after the service method returns.
The likelyhood of this being encoutered during an abnormal termination obvoiusly increases as the time after marking the message but before the service method returns increases.
In normal operation the de-spooling of messages into the processor containing your mailet does not result in duplication, what happens in practice is the reverse, that another thread may obtain a lock on a message which no-longer exists.
d.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance of smtp server and mailets
Posted by Danny Angus <da...@apache.org>.
I agree, even if it were possible, and I don't think it can be, mailets cannot be atomic actions.
All we can ever do is try to make failure behaviour predictable.
> -----Original Message-----
> From: Aaron Knauf [mailto:aknauf@xtra.co.nz]
> Sent: 08 December 2002 05:50
> To: James Users List
> Subject: Re: fault tolerance of smtp server and mailets
>
>
> Noel,
>
> I understand the issues here. The point that I was trying to make was
> that it is actually not possible to absolutely guarantee "once and only
> once". There will always be a window, however small, for either a loss
> or a duplicate (in the case of JAMES, a duplicate).
>
> JAMES currently takes all reasonable measures toward "once and only
> once". Hunting for an absolute guarantee (even if it were possible)
> would not really make JAMES any more useful to 99.9% of users.
>
> Cheers
>
> ADK
>
>
>
> Noel J. Bergman wrote:
> > Aaron,
> >
> > There are distributed transaction managers. Moot point in this context.
> >
> > Let's consider a simple example. Let's say that James accepts
> a message and
> > the sender's connection is pulled at just the wrong point. James has
> > spooled the message, but the sender can't get the notification.
> >
> > RFC 1123 states that 'When the receiver-SMTP accepts a piece of mail (by
> > sending a "250 OK" message in response to DATA), it is accepting
> > responsibility for delivering or relaying the message. It must take this
> > responsibility seriously, i.e., it MUST NOT lose the message
> for frivolous
> > reasons, e.g., because the host later crashes or because of a
> predictable
> > resource shortage.' Therefore, James may see that the
> sender's connection
> > has gone bye-bye, but that doesn't mean James should not send
> the message.
> >
> > Meanwhile, the sender doesn't know if the message was
> sucessfully sent. RFC
> > 1123 (amongst others) states that the sender "MUST be able to
> [retry] until
> > a delivery attempt succeeds."
> >
> > Sending e-mail is not a database transaction. James tries to
> be as reliable
> > as possible. James should not send the "250" until after the
> message is in
> > the spool. To do otherwise would be a defect requiring a fix. Once the
> > message is in the spool, it stays there until after a sucessful transfer
> > elsewhere.
> >
> > If you need guaranteed delivery over SMTP, you could have the
> destination
> > send an ACK after it has successfully processed the message. The sender
> > continues to periodically resend the message until receiving
> the ACK. The
> > receiver must discard duplicates.
> >
> >
> >>It may be possible to track the progress of a mail through the processor
> >
> >
> > As you noted, this would require making a database update for each
> > transition between mailets. There is a different solution,
> available today.
> > Configure a separate processor element for each non-idempotent
> operation.
> > That way there will be a spooler transition recording the
> change where it is
> > necessary.
> >
> > --- Noel
> >
> >
> > --
> > To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: fault tolerance of smtp server and mailets
Posted by Aaron Knauf <ak...@xtra.co.nz>.
Noel,
I understand the issues here. The point that I was trying to make was
that it is actually not possible to absolutely guarantee "once and only
once". There will always be a window, however small, for either a loss
or a duplicate (in the case of JAMES, a duplicate).
JAMES currently takes all reasonable measures toward "once and only
once". Hunting for an absolute guarantee (even if it were possible)
would not really make JAMES any more useful to 99.9% of users.
Cheers
ADK
Noel J. Bergman wrote:
> Aaron,
>
> There are distributed transaction managers. Moot point in this context.
>
> Let's consider a simple example. Let's say that James accepts a message and
> the sender's connection is pulled at just the wrong point. James has
> spooled the message, but the sender can't get the notification.
>
> RFC 1123 states that 'When the receiver-SMTP accepts a piece of mail (by
> sending a "250 OK" message in response to DATA), it is accepting
> responsibility for delivering or relaying the message. It must take this
> responsibility seriously, i.e., it MUST NOT lose the message for frivolous
> reasons, e.g., because the host later crashes or because of a predictable
> resource shortage.' Therefore, James may see that the sender's connection
> has gone bye-bye, but that doesn't mean James should not send the message.
>
> Meanwhile, the sender doesn't know if the message was sucessfully sent. RFC
> 1123 (amongst others) states that the sender "MUST be able to [retry] until
> a delivery attempt succeeds."
>
> Sending e-mail is not a database transaction. James tries to be as reliable
> as possible. James should not send the "250" until after the message is in
> the spool. To do otherwise would be a defect requiring a fix. Once the
> message is in the spool, it stays there until after a sucessful transfer
> elsewhere.
>
> If you need guaranteed delivery over SMTP, you could have the destination
> send an ACK after it has successfully processed the message. The sender
> continues to periodically resend the message until receiving the ACK. The
> receiver must discard duplicates.
>
>
>>It may be possible to track the progress of a mail through the processor
>
>
> As you noted, this would require making a database update for each
> transition between mailets. There is a different solution, available today.
> Configure a separate processor element for each non-idempotent operation.
> That way there will be a spooler transition recording the change where it is
> necessary.
>
> --- Noel
>
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance of smtp server and mailets
Posted by "Noel J. Bergman" <no...@devtech.com>.
> The receiver must discard duplicates.
If anyone actually cares, the other thing that happens is that the ACK is
re-sent, on the grounds that it may have been lost on the way to the sender.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance of smtp server and mailets
Posted by "Noel J. Bergman" <no...@devtech.com>.
Aaron,
There are distributed transaction managers. Moot point in this context.
Let's consider a simple example. Let's say that James accepts a message and
the sender's connection is pulled at just the wrong point. James has
spooled the message, but the sender can't get the notification.
RFC 1123 states that 'When the receiver-SMTP accepts a piece of mail (by
sending a "250 OK" message in response to DATA), it is accepting
responsibility for delivering or relaying the message. It must take this
responsibility seriously, i.e., it MUST NOT lose the message for frivolous
reasons, e.g., because the host later crashes or because of a predictable
resource shortage.' Therefore, James may see that the sender's connection
has gone bye-bye, but that doesn't mean James should not send the message.
Meanwhile, the sender doesn't know if the message was sucessfully sent. RFC
1123 (amongst others) states that the sender "MUST be able to [retry] until
a delivery attempt succeeds."
Sending e-mail is not a database transaction. James tries to be as reliable
as possible. James should not send the "250" until after the message is in
the spool. To do otherwise would be a defect requiring a fix. Once the
message is in the spool, it stays there until after a sucessful transfer
elsewhere.
If you need guaranteed delivery over SMTP, you could have the destination
send an ACK after it has successfully processed the message. The sender
continues to periodically resend the message until receiving the ACK. The
receiver must discard duplicates.
> It may be possible to track the progress of a mail through the processor
As you noted, this would require making a database update for each
transition between mailets. There is a different solution, available today.
Configure a separate processor element for each non-idempotent operation.
That way there will be a spooler transition recording the change where it is
necessary.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: fault tolerance of smtp server and mailets
Posted by Aaron Knauf <ak...@xtra.co.nz>.
As I understand the two-phase commit protocol (and I will admit, I am no
expert on this) the same small window exists even there.
You basically say:
db1: prepare to perform some operation;
db2: prepare to perform some operation;
On success of both prepare operations:
db1: commit;
db2: commit;
On failure of either prepare operation:
db1: rollback;
db2: rollback;
If there is a server/link failure between the two commits, you still get
out of sync.
In the JAMES case, the db's are the repository and the remote
server/local mailbox/etc.
As far as I can see, it is actually not possible to eliminate this
window of error. The only thing that can be done is to make it as
miniscule as possible. I would say that has been achieved.
There is possibly one improvement that could be investigated. It may be
possible to track the progress of a mail through the processor, so that
when it resumes after a crash, it resumes at the same point -
essentially, this would mean saving state after every mailet. I am not
sure that this is really something that belong in JAMES' feature set,
however. Applications requiring this feature are probably a small
minority and nothing prevents them from implementing it themselves. (It
would be quite simple.) For the rest of the JAMES user base, the
performance hit incurred by saving state between every mailet would
probably be unwelcome.
ADK
Noel J. Bergman wrote:
> This is a transaction issue. But there is no mechanism within SMTP for a
> two-phase commit. There is always some chance that between storing the
> message and advising the client that the system could crash. For that
> matter, after sending an acknowledgment to the client, there is some chance
> that the packet could be lost. This effects receipt and relaying by James.
>
> Internally, mail is queued between the handler and the spool manager, and
> then within the spool manager for each transition between processors. Not
> for each mailet.
>
> What you want is a transaction, so that the process of your storing the
> mailet in the database, and the message being removed from the spool appears
> to be atomic. But there is no two-phase commit within the spooling
> mechanism. If your mailet puts the message in a database, as LocalDelivery
> can, then your mailet should set the state to GHOST, and the message will be
> removed from the spool, so that the only copy will be the one you saved. As
> I said, there is a small window between storing the message in the database,
> and when it is cancelled from the spool. Very unlikely, but possible.
>
> --- Noel
>
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Automatic reply?
Posted by "Noel J. Bergman" <no...@devtech.com>.
Siraj,
Yes, there is a mailet (NotifySender). But you have to put the <mailet>
into the config. There is nothing in the user repository to do this
automatically in James v2.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Automatic reply?
Posted by Siraj Shakil <sh...@yahoo.com>.
Hello all,
Is there any method to automatically send a reply to a
sender using Maillet? Something like "Out of office"
autoresponder?
Thanks in advance.
Regards, Shakil
__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance of smtp server and mailets
Posted by Danny Angus <da...@apache.org>.
> Is the assumption about James correct (that it cannot lose a mail
> provided that the sender retries until success) ?
Not behaving this way would be a bug, and a bad one at that, but bad things happen.
I guess you have to make a value judgement based upon the fact that james is designed to work as you have described, and by and large is seen to do so in practice.
Any concrete guarantee of performance is not really possible.
d.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance of smtp server and mailets
Posted by "Noel J. Bergman" <no...@devtech.com>.
Maxime,
If your definition of uniqueness is based upon the message content, and it
cannot ever be the case that the same content is legitimately reproduced,
you could construct a mailet that added an MD5 hash header, e.g., X-MD5:
[....] to the mail message. You would have to decide if the message body
was sufficient, or if there were headers unrelated to the specific delivery
that needed to be included.
If you use db or dbfile protocol, your spool should be entirely reliable.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: fault tolerance of smtp server and mailets
Posted by Maxime Levesque <ma...@webhelp.com>.
> 2) On the receiver side (James) :
> if(!alreadyExistsInPersistentStore(mail))
> persist(mail);
>
>
>
>There is no GUID for e-mail, so how would you implement that generically?
>:-) But in a controlled environment, this is basically what I mentioned
>last week.
>
>
Here's how I would implement the
"alreadyExistsInPersistentStore(mail)" without GUID :
- An exact match on some canonical representation of a Mail (perhaps
javax.mail.Message ?)
Now, one would ask : How can this be implemented cheaply ?
- With hashing :
i) store the mail in a DB table along with it's hash code
ii) alreadyExistsInPersistentStore(mail) would first retrieve
from the DB all emails with the
same hash code, (the better the hash function, the fewer there
will be), and then do an exact
comparison on all of them, returning true if it finds an exact
match.
Note that this will eliminate mails that are an exact copy of a
previously received mail, but
this is allowable for my app, its even desired behavior.
So if James can guaranty "at least once receival" (when sender is a
"retry until acknowledge of success" like
most SMTP servers are), adding the simple algo above, in a mailet will
give the "at most once",
the end result being "exactly once" delivery.
Is the assumption about James correct (that it cannot lose a mail
provided that the sender retries until success) ?
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance of smtp server and mailets
Posted by "Noel J. Bergman" <no...@devtech.com>.
> 1) On the sender side (the SMTP source) :
What you describe is what most SMTP servers do already. They send until
they believe that it was received successfully. You receive the 250
response code from James or you continue to send the message.
> 2) On the receiver side (James) :
> if(!alreadyExistsInPersistentStore(mail))
> persist(mail);
There is no GUID for e-mail, so how would you implement that generically?
:-) But in a controlled environment, this is basically what I mentioned
last week.
> The only prerequisite for this is that SMTP provides for some kind of
> acknowledgement of receipt, and that the SMTP source implementation
> uses this ack. to delete the mail from its "to send" list...
There is already an acknowledgement of receipt. The real prerequisite is
that the SMTP sender provide a GUID that can be used by the receiver to deal
with duplicates. Look again at your logic.
If you are doing this in a controlled environment, your custom sender and
your custom mailet can use this approach.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance of smtp server and mailets
Posted by Danny Angus <da...@apache.org>.
Maxime,
I'm not quite sure where you're going with this, but its as likely to be my misunderstanding as anything, so please bear with me..
> The only prerequisite for this is that SMTP provides for some kind of
> acknowledgement of receipt,
> and that the SMTP source implementation uses this ack. to delete the
> mail from its "to send" list...
>
> Does SMTP have this ?
As far as SMTP is concerned it should in practice offer once and once only receipt for all reasonable purposes.
It doesn't *guarantee* it, for the obvious reason that it is often impossible to deliver absolutes, but relatively easy to deliver accuracy to the nth degree.
What you're suggesting is in fact the way in which James behaves during its SMTP receive-and-spool, failure to spool causes a "4.x.x" "transient error" code to be returned prompting the sender to re-try at their convenience.
The issue I think we started discussing was that James de-spooling and subsequent processing is a lazier system, offering no-losses but at the price of messages possibly passing at least part-way through more than once.
d.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: fault tolerance of smtp server and mailets
Posted by Maxime Levesque <ma...@webhelp.com>.
A two phase commit would do the job indeed, but is not the only way
(at one condition se (*1)), a much
simpler approach, one that is better suited for James would be :
1) On the sender side (the SMTP source) :
boolean success = false;
while(!success) {
success = send(mail);// It's a prerequisite that send returns
"true" if and only if an acknowledgement is received,
// and that the later is sent if and only it the message was
persisted. (*1)
if(success)
deleteFromLocalStore(mail); // or "persist the fact that
'mail' was sent.
sleep(x);
}
2) On the receiver side (James) :
while(true) {
mail = receiveMail(); // a blocking call that unblocks when an
email gets in...
if(!alreadyExistsInPersistentStore(mail))
persist(mail);
}
The (1) part insures "at least once" guaranty of delivery, while (2)
by detecting duplicates takes care of the
"at most once" part guaranty.
The only prerequisite for this is that SMTP provides for some kind of
acknowledgement of receipt,
and that the SMTP source implementation uses this ack. to delete the
mail from its "to send" list...
Does SMTP have this ?
Note that this also takes care of DB failures, since the ACK. is
sent only when successfully stored.
Noel J. Bergman wrote:
>This is a transaction issue. But there is no mechanism within SMTP for a
>two-phase commit. There is always some chance that between storing the
>message and advising the client that the system could crash. For that
>matter, after sending an acknowledgment to the client, there is some chance
>that the packet could be lost. This effects receipt and relaying by James.
>
>Internally, mail is queued between the handler and the spool manager, and
>then within the spool manager for each transition between processors. Not
>for each mailet.
>
>What you want is a transaction, so that the process of your storing the
>mailet in the database, and the message being removed from the spool appears
>to be atomic. But there is no two-phase commit within the spooling
>mechanism. If your mailet puts the message in a database, as LocalDelivery
>can, then your mailet should set the state to GHOST, and the message will be
>removed from the spool, so that the only copy will be the one you saved. As
>I said, there is a small window between storing the message in the database,
>and when it is cancelled from the spool. Very unlikely, but possible.
>
> --- Noel
>
>
>--
>To unsubscribe, e-mail: <ma...@jakarta.apache.org>
>For additional commands, e-mail: <ma...@jakarta.apache.org>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance of smtp server and mailets
Posted by "Noel J. Bergman" <no...@devtech.com>.
This is a transaction issue. But there is no mechanism within SMTP for a
two-phase commit. There is always some chance that between storing the
message and advising the client that the system could crash. For that
matter, after sending an acknowledgment to the client, there is some chance
that the packet could be lost. This effects receipt and relaying by James.
Internally, mail is queued between the handler and the spool manager, and
then within the spool manager for each transition between processors. Not
for each mailet.
What you want is a transaction, so that the process of your storing the
mailet in the database, and the message being removed from the spool appears
to be atomic. But there is no two-phase commit within the spooling
mechanism. If your mailet puts the message in a database, as LocalDelivery
can, then your mailet should set the state to GHOST, and the message will be
removed from the spool, so that the only copy will be the one you saved. As
I said, there is a small window between storing the message in the database,
and when it is cancelled from the spool. Very unlikely, but possible.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance and mail loss
Posted by Alan Gerhard <al...@GerCom.Com>.
~I would add to the top of that list:
~
~Don't be patronizing:
it was not my intent ...
my apologies to the list
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: fault tolerance and mail loss
Posted by Aaron Knauf <ak...@xtra.co.nz>.
> ?????
> http://careers.usatoday.com/service/usa/national/content/news/onthejob/2001-03-2
> 9-people-skills
Alan,
I would add to the top of that list:
Don't be patronising:
People /know/ you're arrogant when you patronisingly tell them to work
on their people skills.
The people that post to this list are not the most sweetly spoken lot,
it's true. They do, however, make a genuine and completely voluntary
attempt to help out with the JAMES-related problems of all and sundry.
The statements made may be blunt, but they are generally on-topic. You
wont get very far by making offensive personal remarks.
ADK
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance and mail loss
Posted by "Noel J. Bergman" <no...@devtech.com>.
> I am under some restrictions as to which version of James I am
> sanctioned to run. I am aware of the tremendous amount of work
> has been done in getting to 2.1...
> Until then I am running 20a3 and have found issues that I believed
> have not been addressed
> therefore my arrogance and chest pounding about the apparently faulty
> operation of a severed db connection.
Well, maybe table pounding, but that's about it. As far as Danny and I can
tell, the problem doesn't exist in the current code. But you can't run the
fixed code, so what would you have us do? Other than get v2.1 officially
released. :-)
As I said, in the future I hope that we can fairly rapidly get out new
release versions in response to defects.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance and mail loss
Posted by Alan Gerhard <al...@GerCom.Com>.
Noel -
~If you can demonstrate that James v2.1 provides a 250 return
~code to the
~sender when using a JDBC spool and a failed database (and where the JDBC
~driver is providing failure notification), then we have a
~serious defect to fix.
I am sorry - I cannot do that.
I am under some restrictions as to which version of James I am sanctioned to
run. I am aware of the tremendous amount of work has been done in getting to 2.1
...
Unfortunately I cannot 'officially' install and run James 2.1 until it has been
'officially released' by the James Group. Until then I am running 20a3 and have
found issues that I believed have not been addressed - therefore my arrogance
and chest pounding about the apparently faulty operation of a severed db
connection.
it's been raised and the issues - if any - that need addressing are being
attended to.
alan
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance and mail loss
Posted by "Noel J. Bergman" <no...@devtech.com>.
Alan,
I have been listening, and I certainly know your name.
Personally, I haven't used 2.0a3 since June. By my automated count, there
have been ~600 patches made to James since 2.0a3. I am bothered that there
have been critical fixes that have not been part of a Release Build. It is
my goal that when a defect is located in the future, that we can quickly put
out a fixed Release, as we just did to fix a resource leak impacting Oracle
users.
If you can demonstrate that James v2.1 provides a 250 return code to the
sender when using a JDBC spool and a failed database (and where the JDBC
driver is providing failure notification), then we have a serious defect to
fix. I cannot produce such an error.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance and mail loss
Posted by Alan Gerhard <al...@GerCom.Com>.
~into the spool is ever removed from it. That happens only if a mailet sets
~the status to GHOST, which happens after the message is successfully
~transferred to another store, or a determination is made by a mailet to discard
it.
yes, agreed, understood, no loss; duplication; two coins.
~I see nothing in your messages to james-dev that adds any additional
~information. Are you saying that the JDBC driver you are using reports
~success even if there is a failure?
?????
http://careers.usatoday.com/service/usa/national/content/news/onthejob/2001-03-2
9-people-skills
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance and mail loss
Posted by "Noel J. Bergman" <no...@devtech.com>.
> The latest stable version of James, as stated (20022106),
> does not notify the sender at all when the db is down and
> James cannot process the mail.
Does 20022106 translate as June 21, 2002? What stable release build are you
talking about? James 2.0a3? If so, I can believe it. I fixed quite a few
errors in the handling of exceptions.
To be sure, I just re-ran a test where I stopped the database during a
postal test. Postal immediately started receiving 451 errors:
Server error:451 Error processing message:
Exception spooling message:
Exception caught while storing mail
Container: java.sql.SQLException:
Communication link failure:
java.net.SocketException;
The 25NOV2002 code in the Milestone directory should be what gets released.
The only remaining work items for the official release of James v2.1 are
related to cleaning up the documentation, and a formal release vote.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance and mail loss
Posted by Danny Angus <da...@apache.org>.
> The latest stable version of James, as stated (20022106), does
> not notify the
> sender at all when the db is down and James cannot process the mail.
>
> ALL MAIL SENT TO JAMES WHEN IN THIS STATE IS LOST AND WITHOUT NOTIFICATION
According to my tests this isn't the case could you confirm this and send a trascript of an SMTP session and/or logfiles, so we can investigate further.
d.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance and mail loss
Posted by Alan Gerhard <al...@GerCom.Com>.
~> In point of fact, if the database is unavailable when the handler
~> receives e-mail from the sender, it will return an error to
~the sender.
~> no, this is incorrect as i have pointed out in the dev list.
~As has been demonstrated to you, if the database is down, and the JDBC
~driver reports the error, a 451 error is returned to the
~sender. That's working in James already. What you were told is that there are
planned
~improvements so that James will reacquire connections within
~requiring a restart after a db server crash. That does not amount to a
~window for data loss.
Noel -
The latest stable version of James, as stated (20022106), does not notify the
sender at all when the db is down and James cannot process the mail.
ALL MAIL SENT TO JAMES WHEN IN THIS STATE IS LOST AND WITHOUT NOTIFICATION
alan
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance and mail loss
Posted by "Noel J. Bergman" <no...@devtech.com>.
> we are not running a guaranteed message deliver service within James ...
> there is a window - however small - of either loss or duplicates;
> depending on how the transfer validation was implemented.
I disagree that loss and duplication are two heads of the same coin,
semantically speaking.
> i may be anal about this but SMTP is not a guaranteed delivery mechanism
and
> neither is the internal spool/queue that we are using.
Read the code. Line 321 of JamesSpoolManager.java is the only place in all
of James where a message that has been successfully inserted into the spool
is ever removed from it. That happens only if a mailet sets the status to
GHOST, which happens after the message is successfully transfered to another
store, or a determination is made by a mailet to discard it.
As I have said, once the message goes into the spool it remains in the spool
until it has been successfully stored elsewhere or requested to be
destroyed.
> the window of loss/duplication is very small - but there IS a window.
> we can not pretend that it doesn't exist
A window for duplication, yes. Show me the window for loss. You will have
to posit a defect in the code, not a window, because removal from the spool
does not occur until afterwards.
> In point of fact, if the database is unavailable when the handler
> receives e-mail from the sender, it will return an error to the sender.
> no, this is incorrect as i have pointed out in the dev list.
As has been demonstrated to you, if the database is down, and the JDBC
driver reports the error, a 451 error is returned to the sender. That's
working in James already. What you were told is that there are planned
improvements so that James will reacquire connections within requiring a
restart after a db server crash. That does not amount to a window for data
loss.
I see nothing in your messages to james-dev that adds any additional
information. Are you saying that the JDBC driver you are using reports
success even if there is a failure?
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance and mail loss
Posted by Alan Gerhard <al...@GerCom.Com>.
~From: Noel J. Bergman
~ There is a window within which to PREVENT loss, there is the remote
~possibility of a duplicate transaction.
we are not running a guaranteed message deliver service within James ... there
is a window - however small - of either loss or duplicates; depending on how the
transfer validation was implemented.
i may be anal about this but SMTP is not a guaranteed delivery mechanism and
neither is the internal spool/queue that we are using. for all intents and
purposes, we can rest assured that the window of loss/duplication is very
small - but there IS a window. we can not pretend that it doesn't exist or that
SMTP is a guaranteed message delivery system; that's just plain foolish.
~> i have scenario that the sender receives a positive from James
~> when it in fact cannot respond - positive or negative as the
~> db is unavailable.
~In point of fact, if the database is unavailable when the
~handler receives e-mail from the sender, it will return an error to the sender.
no, this is incorrect as i have pointed out in the dev list. i will need to
upgrade to the next official release or run several unsanctioned patches to
correct this - but for a commercial ISP, it cannot be expected to run a
development version. as far as i am concerned, this is a problem that is being
addressed - but it does exist.
alan
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance and mail loss
Posted by "Noel J. Bergman" <no...@devtech.com>.
Alan,
Read the thread again. There is no window of LOSS for Maxime. There is a
window within which to PREVENT loss, there is the remote possibility of a
duplicate transaction.
> i have scenario that the sender receives a positive from James
> when it in fact cannot respond - positive or negative as the
> db is unavailable.
In point of fact, if the database is unavailable when the handler receives
e-mail from the sender, it will return an error to the sender. I have had
this occur, as has Danny in his testing.
Once a message is in the spool it is not removed until it has been
successfully transfered again. If the mail could not be delivered to a
mailbox because of a database server failure, it should remain in the spool
or be moved to the error respository. I'd have to check that part of the
pipeline's error handling. At the moment, I'd recommend that the error
repository use the file:// protocol.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
fault tolerance and mail loss
Posted by Alan Gerhard <al...@GerCom.Com>.
maxime -
> From what I understand, the spool is like a producer/consumer queue,
> ....
>both have a small window where duplication can occur.
i have been carrying on a similar discussion over in developer's list which
probably belongs here - thus this response and slight change of topic.
the robustness of the *Processor Pipeline* is of great importance, especially to
commercial IPSs and from what I have followed your discussions to date, my gut
reaction is that 'Yes, there is a window in which the messaging services of
James can fail.' but we first must review the term and concept of 'guaranteed
messaging' and how it relates to the SMTP protocol - which unfortunately can
not, does not and will not guarantee a delivery - though it does come close.
for your scenario, the window of loss is rather small and well within the
industry expected bounds of reliability - which is all any system can hope for.
your main concern to how the spool processes the mail and how it may or may not
be lost or doubled and what the chances are of that happening are. This becomes
more of an issue between the queue and the targeted repository and any
guarantees that may exist between the two - which i don't believe there are any
though i may be mistaken. Implementing a messaging g service (JMS) between the
two or introducing robust code to ensure a low percent of mail-loss possibility
can be done though i suspect discussions of this will need to take place prior ,
etc. etc.
anyway, how my discussion falls within your is that i am concerned about
properly processing the SMTP protocol within the *Processor Pipeline* - where i
have scenario that the sender receives a positive from James when it in fact
cannot respond - positive or negative as the db is unavailable.
different situations but similar concerns.
thanks,
alan
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: fault tolerance of smtp server and mailets
Posted by Maxime Levesque <ma...@webhelp.com>.
Noel J. Bergman wrote:
>Maxime,
>
>James will place the e-mail ONCE and ONLY once into the spool from SMTP
>handler. At any given point after that, it will be somewhere in the spool
>until it goes into a delivery mailet. So there is a tiny window between
>James spooling the message, and telling the sender that all is well.
>
>There is no means to guarantee that a mid-processor mailet will receive the
>message only once. If james is stopped during processing, when it starts
>again, messages will start at the top of the same processor they were last
>recorded as having been in. However, none of that means that they are
>duplicated.
>
From what I understand, the spool is like a producer/consumer queue,
from which the JamesSmtp server
enqueues mails, and mailet threads asynchronously dequeue mails for
processing. The enqueue and dequeue operations
both have a small window where duplication can occur.
Is this accurate ?
Because if mymailet does a 'put in database', the end result will be a
duplicate...
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance of smtp server and mailets
Posted by "Noel J. Bergman" <no...@devtech.com>.
Maxime,
James will place the e-mail ONCE and ONLY once into the spool from SMTP
handler. At any given point after that, it will be somewhere in the spool
until it goes into a delivery mailet. So there is a tiny window between
James spooling the message, and telling the sender that all is well.
There is no means to guarantee that a mid-processor mailet will receive the
message only once. If james is stopped during processing, when it starts
again, messages will start at the top of the same processor they were last
recorded as having been in. However, none of that means that they are
duplicated.
There is an extremely small window (consider it a transaction) between when
LocalDelivery or RemoteDelivery deliver the message to its final
destination, and when they mark it as having been delivered.
So those two transactional windows, both very small, represent the only
opportunity for a message to be delivered more than once, unless I am
overlooking something.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance of smtp server and mailets
Posted by "Noel J. Bergman" <no...@devtech.com>.
Danny,
[switching mailing lists]
I agree that it isn't a high priority. And an alternative would be for the
Mail object passed into the mailet to delegate the state change call to the
context, so that we close the window without changing the mailet code. The
object would just be a further specialized MailImpl.
--- Noel
-----Original Message-----
From: Danny Angus [mailto:danny@apache.org]
Sent: Tuesday, December 10, 2002 6:56
To: James Users List
Subject: RE: fault tolerance of smtp server and mailets
Apart from this instance I've never heard a demand for such strict control,
so I don't generally think it would be worthwhile.
Making the call to change the status actually carry out the writes might be
worth considering, perhaps it should be a method called on the context like
container.setState(this, mail.GHOST) as I think we both agree that the
de-coupling opens a "window of opportunity", to quote a popular military
phrase.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance of smtp server and mailets
Posted by Danny Angus <da...@apache.org>.
Apart from this instance I've never heard a demand for such strict control, so I don't generally think it would be worthwhile.
Making the call to change the status actually carry out the writes might be worth considering, perhaps it should be a method called on the context like .. container.setState(this, mail.GHOST) as I think we both agree that the de-coupling opens a "window of opportunity", to quote a popular military phrase.
> I don't know if we want to look at anything related to this for v3. We
> would have to agree that mail *status* is always kept in a database, even
> when mail content is stored in the file-system. I am amenable to
> that as a
> discussion.
>
> --- Noel
>
>
> --
> To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance of smtp server and mailets
Posted by "Noel J. Bergman" <no...@devtech.com>.
> But you did say you'd realized how.. "I realized earlier today how to" ;-)
Yes. I didn't say that the solution was general. :-) If you want an
atomic transaction, you have to put yourself in a position to have one.
Right now, that means we need to control a database transaction within a
mailet, and use that to create atomicity.
> I've pondered it for these past few days and been unable to see how one
> could guarantee that a mail was only processed, in the general situation
You can't do it without an enclosing transaction manager.
> Alternatives based on making internal lists won't help you unless the
> action of adding/removing or comparing mail with lists is atomic
> itself and atomic with the processing you are trying to control.
Exactly. :-) Even in a case where you use some GUID, you need to be able to
persist the effect of the mailet and the change to the GUID related status
in an atomic transaction.
As you said, in the case of the current "storm in a tea-cup", a database
solution can be constructed.
I don't know if we want to look at anything related to this for v3. We
would have to agree that mail *status* is always kept in a database, even
when mail content is stored in the file-system. I am amenable to that as a
discussion.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance of smtp server and mailets
Posted by Danny Angus <da...@apache.org>.
Noel,
> I said that "the mailet has to be specially written to do it." What I
> didn't say was how.
But you did say you'd realized how.. "I realized earlier today how to" ;-)
I've pondered it for these past few days and been unable to see how one could guarantee that a mail was only processed, in the general situation, once if the processing takes a finite period.
You would be compelled to assume either the start, or the end of processing was the gate which couldn't be passed twice, and these are the two options you would have to choose:
1/ gate is at the start, pass gate but processing is not completed, mail isn't re-processed and is "lost"
2/ gate is at the end, mail is processed successfully but terminated before it passes the gate, it is re-processed and "duplicated"
Alternatives based on making internal lists won't help you unless the action of adding/removing or comparing mail with lists is atomic itself and atomic with the processing you are trying to control.
Of course in the specific case which started this discussion (aka. storm in a tea-cup) where database inserts are concerned this might be achieved through the use of 100% available database clusters and by the use of the mail-uid as a unique key in the database.
In this situation duplication will fail because of duplicate values for a unique key, and db availability can be arranged to avoid any losses in all but the most extreme circumstances. (else banks wouldn't use them ;)
d.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance of smtp server and mailets
Posted by "Noel J. Bergman" <no...@devtech.com>.
> > Actually, I realized earlier today how to completely eliminate the
> > window within a custom mailet under carefully controlled
> > circumstances,
> I believe it could be done, but how accurate would it be? could you
> guarentee *no* message is *ever* processed twice?
I said that "the mailet has to be specially written to do it." What I
didn't say was how. You need to take over control within the mailet, and
wrap your database updates within a JDBC transaction: both your operational
post to the database, and the state change. Very dependent upon knowing how
James works and in setting up the configuration -- NOT a portable solution.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance of smtp server and mailets
Posted by Danny Angus <da...@apache.org>.
> It occured to me that this window can be minimized by the simple
> process of
> putting a non-idempotent mailet in its own processor, or having it set the
> state.
Yes it potentially moves the write and the flagging operations close together
>Actually, I realized earlier today how to completely eliminate the
> window within a custom mailet under carefully controlled
> circumstances,
I believe it could be done, but how accurate would it be? could you guarentee *no* message is *ever* processed twice?
> the mailet has to be specially written to do it. Not something that would
> generally be done.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance of smtp server and mailets
Posted by "Noel J. Bergman" <no...@devtech.com>.
> The processing is not atomic, therefore an oportunity for processing to
> terminate abruptly with mail "between" states will *always* be present.
> James deals with this by re-starting processing from the last recorded
> state, allowing the possibility for some mail to pass through some
> processing twice.
It occured to me that this window can be minimized by the simple process of
putting a non-idempotent mailet in its own processor, or having it set the
state. Actually, I realized earlier today how to completely eliminate the
window within a custom mailet under carefully controlled circumstances, but
the mailet has to be specially written to do it. Not something that would
generally be done.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance of smtp server and mailets
Posted by Danny Angus <da...@apache.org>.
Maxime,
> i) I need my app to "listen" for emails arriving via smtp, and
> store them in a database table
James SMTP handler will do this correctly unless you require a different db schema than James spool/mail repository tables.
> v) I need the guarantee that no mail loss can occur, and neither can
> duplicates
> (every email arriving from sendmail will get processed once and
> only once,
> no matter how often my James process gets restarted).
>
> It seems from your answer that I have to add this fault tolerance, for
> James does not
> have it build in.
Actually James is tolerant of failures, and *should* not experience losses.
However the price is that it can't avoid duplication of some tasks.
The processing is not atomic, therefore an oportunity for processing to terminate abruptly with mail "between" states will *always* be present. James deals with this by re-starting processing from the last recorded state, allowing the possibility for some mail to pass through some processing twice.
> 1) Configure my sendmail 'source' to "retry upon failure". Lets say,
> upon failure,
> retry every 5 minutes up to 10 times.
Sendmail should do this as shipped, though not as rapidly as you may want.
> 2) Detect and ignore duplicates :
>
> myRobustMailet.service(Mail m) {
> if(isDuplicate(mail))
> return;
>
> put mail in DB;
> }
Yes.
> In my case, my mail source is a sendmail instance, and the typical
> journey for emails today is typically
> a serie of "store and forward" along a chain of relaying processes
> through SMTP, not all of them James instances (yet ;) ) !
I understand this, but was simply questing why you think you need single sendmail instance as the sole source for messages into James.
I accept that there *are* legitimate reasons for it, but wanted to make sure you're not going to unecessary additional effort to achieve your goal.
d.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: fault tolerance of smtp server and mailets
Posted by Maxime Levesque <ma...@webhelp.com>.
>> In general, how can one achieve guaranteed "once and only once" email
>>receival with James ?
>>
>>
>
>I don't understand what "guaranteed once and only once email receival" is, James will receive a mail or not receive it.
>If you mean how to ensure that a mailet will only every process a mail once it is not 100% possible.
>There will always be a chance that a mail being processed when james is killed will be re-processed when it is re-started.
>
>
Let my clarify what I meant by 'guarantee of once and only once
processing' :
i) I need my app to "listen" for emails arriving via smtp, and
store them in a database table
ii) I assume that mails are arriving from an SMTP relaying process
(sendmail)
iii) I cannot assume that my James VM will not get restarted once in
a while
iv) For the simplicity of this discussion, we will assume that the
SMTP source
is absolutely reliable.
v) I need the guarantee that no mail loss can occur, and neither can
duplicates
(every email arriving from sendmail will get processed once and
only once,
no matter how often my James process gets restarted).
It seems from your answer that I have to add this fault tolerance, for
James does not
have it build in.
Here's a sketch of the approach I'm considering :
1) Configure my sendmail 'source' to "retry upon failure". Lets say,
upon failure,
retry every 5 minutes up to 10 times.
This solves the "at least once" part of the guarantee, but is likely
to cause duplicates.
Now for the "at most once" guarantee :
2) Detect and ignore duplicates :
myRobustMailet.service(Mail m) {
if(isDuplicate(mail))
return;
put mail in DB;
}
>Sendmail is irrelevant here, your mail client should connect straight to James, James is an SMTP Mail server too.
>If you have to use sendmail for other reasons, fine, but it changes nothing, sendmail and James both know how to send and recieve mail properly. Any valid SMTP configuration of either will result in the correct transmission of your messages.
>
>
In my case, my mail source is a sendmail instance, and the typical
journey for emails today is typically
a serie of "store and forward" along a chain of relaying processes
through SMTP, not all of them James instances (yet ;) ) !
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: fault tolerance of smtp server and mailets
Posted by Danny Angus <da...@apache.org>.
Maxime,
> i) Suppose Jame's smtp server is midway in swallowing an email and
> I pull the plug on the machine, what will happen ?
The connection will be terminated and the sending MTA will time-out and (presumably) re-try.
> ii) When a mailet's service method gets called, is the mail read
> from the spool file ?
No, it is read (from file or database) only when it is fetched from the spool and inserted into the top of the relevant processor.
> iii) If a mailet's service method is midway in execution and I pull the
> plug, what happens ?
Its easier if I answer a different question first ;-)
"What happens to a mail as it is processed"
1/ the spool manager reads a mail from the spool
2/ based upon its "state" it is inserted into a linear processor
3/ the processor is responsible for passing the mail through its mailets/matchers
4/ only when the "state" changes is the mail written to the spool
So.. if a mail reaches a mailet and is processed, but the plug is pulled, then that mail *will* be passed through the processor, including any mailets it had previously passed through, again.
If a mailet has to ensure that it only processes mail once that mailet would currently have to take its own steps to ensure that this was the case.
> In general, how can one achieve guaranteed "once and only once" email
> receival with James ?
I don't understand what "guaranteed once and only once email receival" is, James will receive a mail or not receive it.
If you mean how to ensure that a mailet will only every process a mail once it is not 100% possible.
There will always be a chance that a mail being processed when james is killed will be re-processed when it is re-started.
> Suppose I have the following email pipeline, and I want the guarantee
> that : "if an email makes it's way to the smtp middleman, it will
> eventually result in one execution of myMailet's service method." :
>
> EMailCLient---> smtpd(sendmail?)--->james(SMTP->james-spool)->myMailet
>
> Should I configure sendmail to retry upon failure (lets say, retry up to
> 5 times with a 10 minute sleep in between) ?
Sendmail is irrelevant here, your mail client should connect straight to James, James is an SMTP Mail server too.
If you have to use sendmail for other reasons, fine, but it changes nothing, sendmail and James both know how to send and recieve mail properly. Any valid SMTP configuration of either will result in the correct transmission of your messages.
d.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>