You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by "Noel J. Bergman" <no...@devtech.com> on 2006/06/03 20:18:56 UTC
Proposal: new approach to spooling
Concepts:
- Each processor is a named queue entry. This is
not a change from today, except that these may
be real queues in the JMS sense of the word if
the underlying queue manager uses JMS. But the
approach should NOT be JSM/MQ specific. It
should work just fine with JDBC-based queues.
- A queue entry would normally contain a JAMES Mail
object. JavaMail Message objects would be
contained, or more likely referenced, by a JAMES
Mail object. I say "normally" just in case we
want to permit other message types, e.g., JAMES
control messages, to be posted, since JAMES
processors effectively become services.
- Each processor is a transaction. What this means
is that we can try to wrap a global transaction
around behavior within the processor. This will
have some impact on Mailets, too.
- Each processor is associated with a queue manager
and, optionally, a retry schedule.
- I believe that a queue implementation independent
scheduler that provides the next time at which a
message should be processed may be sufficient.
Each queue entry would carry a timestamp before
which it should not be processed. "Restarting"
the queue would be as simple as changing that
timestamp entry.
- A new RETRY Mail state can be set to rollback the
transaction and put the Mail back into the queue.
We should decide on commit and rollback semantics.
- The processor acquires a new attribute that explicitly
sets the fall-through state. The default shall be the
new RETRY state, except for messages that exhause the
retry schedule.
There are all sorts of ways to express this in XML, one of which might be:
<processor name="<queue-name>"
[onException="..."]
[fallthrough="<message-state>" def: RETRY]
[class="<class>" def: LinearProcessor] >
<queue [queueManager="<class>"> def: DefaultQueueManager] >
<!-- queue configuration -->
</queue>
<schedule [class="<class>" def: DefaultScheduler]>
<!-- see RemoteDelivery -->
</schedule>
<mailet ...>
</processor>
Despite that this defines the queue manager as a child of the processor, we
would invert that in code, with the queue manager calling the processor. I
believe that thread management is defined squarely on the queue manager.
The concept still needs fleshing out, but I wanted to air the thoughts.
This gives a lot of flexibility in implementation. For example, one might
implement a processor as an MDB. The processor is responsible for workflow
within the transaction, and for bridging between the queue manager and the
mailet API.
For this reason, I would not define interfaces for the queue manager and
processor, but rather define semantics and responsibilities. The Mailet API
is the inviolate API that all processors would be expected to support. A
processor would accept the queue entry from the queue manager, invoke the
mailet pipeline, handle any exceptions, etc. A queue manager would handle
acquisition of appropriate entries from the queue queue, threading, calling
the processor, etc. I envision enhancements to the ToProcessor mailet, and
related calls, and am still debating how queue aliases might be handled,
which can be quite powerful.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Proposal: new approach to spooling
Posted by Danny Angus <da...@gmail.com>.
Oh, great, just when all the systems at work are conspiring against me
the James listserver decides to kick me when I'm down .
:-D :-D
d.
On 14/06/06, Noel J. Bergman <no...@devtech.com> wrote:
> > Oh drawing 2 didn't work! RETRY..
>
> The list server is stripping almost all attachments.
>
> --- Noel
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: Proposal: new approach to spooling
Posted by "Noel J. Bergman" <no...@devtech.com>.
> Oh drawing 2 didn't work! RETRY..
The list server is stripping almost all attachments.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Proposal: new approach to spooling
Posted by Danny Angus <da...@gmail.com>.
Oh drawing 2 didn't work! RETRY..
Re: Proposal: new approach to spooling
Posted by Danny Angus <da...@gmail.com>.
On 03/06/06, Noel J. Bergman <no...@devtech.com> wrote:
> Concepts:
>
> - Each processor is a named queue entry. This is
> not a change from today, except that these may
> be real queues in the JMS sense of the word
JMS +1
>
> - A queue entry would normally contain a JAMES Mail
> object. JavaMail Message objects would be
> contained, or more likely referenced, by a JAMES
> Mail object. I say "normally" just in case we
> want to permit other message types, e.g., JAMES
> control messages, to be posted, since JAMES
> processors effectively become services.
Not refrenced, where is the referenced object going to live?
The JMS queue may have been populated by another system at another location.
In the JMS vision we can imagine a partial James running SMTP-inbound
and publishing to a queue,
that queue is then subscribed by another queue on another system the
partial James running on the other system will be dequeueing and
processing.
Outbound messages can be routed to other queues on yet more systems,
even just for regular old mail routing I can envisage an outbound
queue and a local delivery queue.
(see drawings for examples)
> - Each processor is a transaction. What this means
> is that we can try to wrap a global transaction
> around behavior within the processor. This will
> have some impact on Mailets, too.
I think we have to think a bit harder about transaction boundaries.
> - Each processor is associated with a queue manager
> and, optionally, a retry schedule.
I think there should be one queue with multiple topics.
> - I believe that a queue implementation independent
> scheduler that provides the next time at which a
> message should be processed may be sufficient.
> Each queue entry would carry a timestamp before
> which it should not be processed. "Restarting"
> the queue would be as simple as changing that
> timestamp entry.
Queue implementations should include this kind of detail.
Remember that we want to use queues, not invent them.
> - A new RETRY Mail state can be set to rollback the
> transaction and put the Mail back into the queue.
> We should decide on commit and rollback semantics.
Use exceptions and jdbc transactions, if a mailet throws a recoverable
exception the dequeue is rolledback, if it throws an unrecoverable
exception the message is dequeued into the error queue.
> - The processor acquires a new attribute that explicitly
> sets the fall-through state. The default shall be the
> new RETRY state, except for messages that exhause the
> retry schedule.
Me no like, no retry, and certainly not default. If a message can't be
processed an exception should rollback the dequeue (see above)
> There are all sorts of ways to express this in XML
Yes aren't there, remember that you will also need to allow the vendor
specific configuration of the JMS implementation, and message
selection,
multiple or single subscribers, and in-order or not-in-order delivery.
> Despite that this defines the queue manager as a child of the processor, we
> would invert that in code, with the queue manager calling the processor. I
> believe that thread management is defined squarely on the queue manager.
It has to be that way, examine message driven beans for an example of
how to use JMS.
Imagine james deployed as MDB's
> The concept still needs fleshing out, but I wanted to air the thoughts.
> This gives a lot of flexibility in implementation. For example, one might
> implement a processor as an MDB.
Exactly. and all the config you'd need would be the queue and or topic.
> The processor is responsible for workflow
> within the transaction, and for bridging between the queue manager and the
> mailet API.
d.
RE: Proposal: new approach to spooling
Posted by "Noel J. Bergman" <no...@devtech.com>.
Stefano Bagnara wrote:
> 1) What are the benefits of inverting the queue <=> processor relation?
See my replies to Steve. I'm after several capabilities, and did consider
several alternative syntactic approaches, but only care about relative ease
of expression and functional capability.
> 2) Are you willing to actually implement this or is this just a
discussion?
I'd expect to work on it. I'd also expect that more than one person would
work on different aspects of it.
> I'm +1 to experiment and refactor the code to add some of the features
> you describe here (e.g: fallthrough, transactions) as a progressive
> evolution.
We already have fallthrough, but adding that as an attribute would be a
simple and nice enhancement. Transactions are rather more work.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Proposal: new approach to spooling
Posted by Stefano Bagnara <ap...@bago.org>.
1) What are the benefits of inverting the queue <=> processor relation?
It seems less clear than the current way and I don't see specific
advantages.
2) Are you willing to actually implement this or is this just a discussion?
I'm +1 to experiment and refactor the code to add some of the features
you describe here (e.g: fallthrough, transactions) as a progressive
evolution.
About this proposal itself I'm +0 because imho it is too much things and
I'm not able to evaluate the impact on our codebase.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: PROPOSAL: planned change to allow generic (re)-scheduling
Posted by Bernd Fondermann <be...@googlemail.com>.
On 8/25/06, Stefano Bagnara <ap...@bago.org> wrote:
> Bernd Fondermann wrote:
> >> I also have a lot of things in my todo list, so I at least try to review
> >> the proposal and try to understand where it is better or worst of what I
> >> already use.
> >
> > well, I think we should eventually merge (the best of) both approaches
> > if they are addressing the same thing.
> > its not about either one being better or worse.
>
> Exactly what I thought. My english is not so good but by "WHERE it is
> better or worse" I meant "what parts are better and what are worse, so
> we can take the best from both".
oops, sorry, my bad. I read "whether" instead of "where".
Bernd (leaving to buy glasses)
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: PROPOSAL: planned change to allow generic (re)-scheduling
Posted by Stefano Bagnara <ap...@bago.org>.
Bernd Fondermann wrote:
>> I also have a lot of things in my todo list, so I at least try to review
>> the proposal and try to understand where it is better or worst of what I
>> already use.
>
> well, I think we should eventually merge (the best of) both approaches
> if they are addressing the same thing.
> its not about either one being better or worse.
Exactly what I thought. My english is not so good but by "WHERE it is
better or worse" I meant "what parts are better and what are worse, so
we can take the best from both".
I currently don't see a real connection between the main proposal from
Noel and this sub-proposal, but I guess I don't really have understood
right this last proposal so I will wait for Noel to have more time to
explain.
I currenlty still think that the service approach is better and more
clean/clear but I'm looking forward for further infos on this proposal.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: PROPOSAL: planned change to allow generic (re)-scheduling
Posted by Bernd Fondermann <be...@googlemail.com>.
On 8/24/06, Stefano Bagnara <ap...@bago.org> wrote:
> Bernd Fondermann wrote:
> > You say "New approach to spooling". How does the proposal affect
> > spooling 'in general'?
> >
> > How is this related to past propositions (IIRC) to evolve RD from a
> > mailet to something better, e.g. an own processor?
>
> Maybe you're referring to my proposal to create a RemoteDeliveryManager
> that, like the JamesSpoolManager, is a top level component and has its
> own mailet pipeline.
Exactly.
> I already use this approach in one of my production machine and I
> planned to adapt it to the current trunk but Noel started the other
> thread to plan a different behaviour as soon as I started talking about
> this one, so I waited to understand what he was planning to do.
> I also have a lot of things in my todo list, so I at least try to review
> the proposal and try to understand where it is better or worst of what I
> already use.
>
well, I think we should eventually merge (the best of) both approaches
if they are addressing the same thing.
its not about either one being better or worse.
Bernd
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: PROPOSAL: planned change to allow generic (re)-scheduling
Posted by Stefano Bagnara <ap...@bago.org>.
Bernd Fondermann wrote:
> You say "New approach to spooling". How does the proposal affect
> spooling 'in general'?
>
> How is this related to past propositions (IIRC) to evolve RD from a
> mailet to something better, e.g. an own processor?
Maybe you're referring to my proposal to create a RemoteDeliveryManager
that, like the JamesSpoolManager, is a top level component and has its
own mailet pipeline.
I already use this approach in one of my production machine and I
planned to adapt it to the current trunk but Noel started the other
thread to plan a different behaviour as soon as I started talking about
this one, so I waited to understand what he was planning to do.
I also have a lot of things in my todo list, so I at least try to review
the proposal and try to understand where it is better or worst of what I
already use.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: PROPOSAL: planned change to allow generic (re)-scheduling
Posted by Bernd Fondermann <be...@googlemail.com>.
Hi Noel,
Nice proposal. I have questions about the 'big picture':
Although I understand that you say the Scheduler is working with the
RemoteDelivery, is this a proposal to either
o replace RemoteDelivery or greater parts from it in short term
o take one aspect of RD and make it available in another context
o establish a parallel approach which is intended to replace RD when
time is due?
You say "New approach to spooling". How does the proposal affect
spooling 'in general'?
How is this related to past propositions (IIRC) to evolve RD from a
mailet to something better, e.g. an own processor?
I support any effort to forster 'seperations of concerns' so I should
be fine with the proposal in any case.
Thanks,
Bernd
On 8/20/06, Noel J. Bergman <no...@devtech.com> wrote:
> I have a really busy couple of weeks in front of me, but once that is done,
> I should have time to work on this for v3.0.
>
> This is an approach to implement the scheduling aspect of my "New Approach
> to Spooling" proposla. It is intentionally compatible with the existing
> Mailet API.
>
> The change will consist of three parts: a processor, a mailet and a utility
> class. The processor would be defined, to use a simplified example without
> configuration options, as:
>
> <processor name="SCHEDULER">
> <mailet class="Scheduler"/>
> </processor>
>
> which does nothing for you unless you know what the Scheduler mailet is, how
> messages get there, and where they go afterwards.
>
> The Scheduler mailet will be derived from the part of RemoteDelivery that
> receives a message on a spool thread, and has a separate thread processing
> an internal queue. We won't need more than one scheduler thread. When a
> message's schedule time occurs, the mailet will post it to the designated
> destination.
>
> The utility class will contain the actual scheduler, whose core pseudo-code
> can be described as:
>
> schedule.retry(mail)
> boolean scheduled = false;
> retries = getRetryCount(mail);
> if (retries < getMaxRetries()) {
> setRetryTime(mail) { mail.setAttribute("org.apache.james.RETRY_TIME",
> nextTime(mail)); }
> setRetryCount(mail) {
> mail.setAttribute("org.apache.james.RETRY_COUNT", ++retries); }
> // WARNING: THESE TWO OPERATIONS MUST BE PERFORMED IN THIS SEQUENCE
> SINCE BOTH DEPEND UPON THE MAIL'S STATE
> setRetryDest(mail) { mail.setAttribute("org.apache.james.RETRY_DEST",
> mail.getState()); }
> setScheduler(mail) { mail.setState(schedule_processor); }
> scheduled = true;
> }
> return scheduled
>
> The code looks at the message, decides if there are more retries available,
> and if so posts the message so that the scheduler knows when to post it
> back, and to which processor. Yes, the rescheduling would be on a processor
> boundary, which will be the transaction boundary once we properly support
> transactions.
>
> The schedule object would be constructed from a retry schedule such as the
> one we have for RemoteDelivery. Actually, I'll do an abstract class with
> the code for scheduling and an abstract method "nextTime" method, and then a
> specific implementation making use of the code from RemoteDelivery, to
> separate mechanics from policy. RemoteDelivery would receive messages and
> perform something like:
>
> if (!deliver(mail) && !schedule.retry(mail)) {
> process a bounce
> }
>
> Other mailets, such as ClamAV, could contain similar code to schedule
> retries if a resource they need is not available (and optionally notify the
> administrator of the problem). Additional work will be done to allow
> matchers to also rescheduling.
>
> Comments solicited. This is just a first out.
>
> --- Noel
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: PROPOSAL: planned change to allow generic (re)-scheduling
Posted by Stefano Bagnara <ap...@bago.org>.
Noel J. Bergman wrote:
> This is an approach to implement the scheduling aspect of my "New Approach
> to Spooling" proposla. It is intentionally compatible with the existing
> Mailet API.
As a reminder, I already made comments to the "New Approach to Spooling"
and I have a (stalled, maybe working) work in progress on the same area.
> The change will consist of three parts: a processor, a mailet and a utility
> class. The processor would be defined, to use a simplified example without
> configuration options, as:
>
> <processor name="SCHEDULER">
> <mailet class="Scheduler"/>
> </processor>
have the processor name "SCHEDULER" some special meaning? I don't
understand it from this mail.
> which does nothing for you unless you know what the Scheduler mailet is, how
> messages get there, and where they go afterwards.
>
> The Scheduler mailet will be derived from the part of RemoteDelivery that
> receives a message on a spool thread, and has a separate thread processing
> an internal queue. We won't need more than one scheduler thread. When a
> message's schedule time occurs, the mailet will post it to the designated
> destination.
> [...]
> The schedule object would be constructed from a retry schedule such as the
> one we have for RemoteDelivery. Actually, I'll do an abstract class with
> the code for scheduling and an abstract method "nextTime" method, and then a
> specific implementation making use of the code from RemoteDelivery, to
> separate mechanics from policy. RemoteDelivery would receive messages and
> perform something like:
>
> if (!deliver(mail) && !schedule.retry(mail)) {
> process a bounce
> }
How do you handle the accept? Currently the spoolrepository interface
does not give you access to mail attributes.
In practice, how do you ignore mails until the next attempt time? What
is the message state and the repository of delayed messages?
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
PROPOSAL: planned change to allow generic (re)-scheduling
Posted by "Noel J. Bergman" <no...@devtech.com>.
I have a really busy couple of weeks in front of me, but once that is done,
I should have time to work on this for v3.0.
This is an approach to implement the scheduling aspect of my "New Approach
to Spooling" proposla. It is intentionally compatible with the existing
Mailet API.
The change will consist of three parts: a processor, a mailet and a utility
class. The processor would be defined, to use a simplified example without
configuration options, as:
<processor name="SCHEDULER">
<mailet class="Scheduler"/>
</processor>
which does nothing for you unless you know what the Scheduler mailet is, how
messages get there, and where they go afterwards.
The Scheduler mailet will be derived from the part of RemoteDelivery that
receives a message on a spool thread, and has a separate thread processing
an internal queue. We won't need more than one scheduler thread. When a
message's schedule time occurs, the mailet will post it to the designated
destination.
The utility class will contain the actual scheduler, whose core pseudo-code
can be described as:
schedule.retry(mail)
boolean scheduled = false;
retries = getRetryCount(mail);
if (retries < getMaxRetries()) {
setRetryTime(mail) { mail.setAttribute("org.apache.james.RETRY_TIME",
nextTime(mail)); }
setRetryCount(mail) {
mail.setAttribute("org.apache.james.RETRY_COUNT", ++retries); }
// WARNING: THESE TWO OPERATIONS MUST BE PERFORMED IN THIS SEQUENCE
SINCE BOTH DEPEND UPON THE MAIL'S STATE
setRetryDest(mail) { mail.setAttribute("org.apache.james.RETRY_DEST",
mail.getState()); }
setScheduler(mail) { mail.setState(schedule_processor); }
scheduled = true;
}
return scheduled
The code looks at the message, decides if there are more retries available,
and if so posts the message so that the scheduler knows when to post it
back, and to which processor. Yes, the rescheduling would be on a processor
boundary, which will be the transaction boundary once we properly support
transactions.
The schedule object would be constructed from a retry schedule such as the
one we have for RemoteDelivery. Actually, I'll do an abstract class with
the code for scheduling and an abstract method "nextTime" method, and then a
specific implementation making use of the code from RemoteDelivery, to
separate mechanics from policy. RemoteDelivery would receive messages and
perform something like:
if (!deliver(mail) && !schedule.retry(mail)) {
process a bounce
}
Other mailets, such as ClamAV, could contain similar code to schedule
retries if a resource they need is not available (and optionally notify the
administrator of the problem). Additional work will be done to allow
matchers to also rescheduling.
Comments solicited. This is just a first out.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org