You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by Aidan Skinner <ai...@apache.org> on 2009/11/07 19:00:54 UTC
qpid.periodic exchange type
So, somebody asked me on IRC about delaying messages - sending a
message but not having it delivered for a while. We don't do that, so
I asked about their use case and they want to be able to put a bunch
of "do this" messages onto a queue and having them drip off slowly.
That would be quite messy to implement, so another (maybe more AMQP-y)
way of doing this might be to have a periodic exchange which you send
messages too and then it routes it onto another exchange (you must
specify another exchange on the X-Additional-Exchange) with the same
routing key. The exchange keeps a copy of the message, and sends a new
one to the same routing key and exchange at intervals specified in an
X-Period (float, hz) header.
There are apparently a few things that implement similar features,
like celery, that sit above AMQP but are a bit heavy if you just want
this. I'd like to get this into the Java broker for 0.7, I expect
it'll need a bit of iteration to get the interface right.
- Aidan
--
Apache Qpid - AMQP, JMS, other messaging love http://qpid.apache.org
"A witty saying proves nothing" - Voltaire
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: qpid.periodic exchange type
Posted by Gordon Sim <gs...@redhat.com>.
On 11/10/2009 03:33 AM, Robert Godfrey wrote:
> 2009/11/10 Aidan Skinner<ai...@gmail.com>
>> Re-reading my original mail I wasn't entirely clear about the
>> > use-case: this was someone who basically wanted to publish the same
>> > message every N minutes. Think cron with a really basic client that
>> > sends a message in a static file. the deliver-after stuff was their
>> > top level stab at doing it in AMQP.
>> >
> So - I'm tempted to say this is not really a Message Queuing use case...
That's my initial reaction also. What are the drawbacks with a cron job
and a really simple client?
> I can see that the deliver-after type thing could be generally useful
What is/are the use-case(s) you have in mind for that?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: qpid.periodic exchange type
Posted by Robert Godfrey <ro...@gmail.com>.
2009/11/10 Aidan Skinner <ai...@gmail.com>
> On Sun, Nov 8, 2009 at 5:15 PM, Robert Godfrey <ro...@gmail.com>
> wrote:
> > 2009/11/8 Aidan Skinner <ai...@apache.org>
> >
> >> The original message is delivered and persisted as normal. The
> >> exchange would hold onto a copy of that message, and generate an
> >> entirely new, identical one at the specified interval.
> >>
> >> Putting this logic at the exchange level seems better to me because
> >> it avoids further complicating the subscription implementation. It'll
> >> also map a bit cleaner into 1.0 semantics (as a non-destructive link).
> >>
> >>
> > I don't think we would need to complicate subscription implementation
> unless
> > we wanted to implement this by having selectors which compared message
> > fields to the present time (or some other condition that would need to be
> > periodically reevaluated). I was thinking that we would implement like
> TTL
> > - i.e. the message state on the queue is changed when the time-driven
> > condition changes (in this case when it becomes live)... As happens
> already
> > in the queue, the state change will cause the subscribers to go
> re-examine
> > the message to see if they can now deliver it. Other than sorting out
> how
> > we do timed events a bit, this seems like a fairly minor change at the
> queue
> > level.
>
> I hadn't realised we could signal state change to the subscriber to
> have it reevaluate all the messages, that does make it much simpler.
> AIUI messages which had an expired TTL were cleaned up by the
> housekeeping thread after the pointer had already moved on.
>
>
Yes - this is basically how release works in general in the Java Broker
since other subscribers may have already advanced to a point beyond the
place where the released message exists.
> > Having exchanges that retain state would be a big departure, and as
> Robert
> > points out would complicate persistence. From a 1-0 point of view, there
> > will be no such thing as exchanges, so I'm not sure I really see the
> > advantage there. Basically the difference is between hard-coding the
> entry
> > point/address for the message (by fixing the exchange type) or adding a
> new
> Re-reading my mail I wasn't entirely clear about the use-case: this
> was someone who basically wanted to publish the same message every N
> minutes. Think cron with a really basic client that sends a message in
> a static file. the deliver-after stuff was a top level stab at doing
> it in AMQP.
>
OK, that is somewhat different, although from a 1-0 perspective you don't
really want to make them the "same" message as we say that queues should
reject duplicates.
>
> > capability to queues. I much prefer the latter, and I also think it will
> be
> > cheaper to implement
>
> So, the main functional difference I see with an "X-deliver-after"
> header is that the publisher has to occasionally push a fresh batch of
> messages. Eventually all the messages would be consumed and they'd
> have to send a new batch. I'm not sure that I see much benefit in
> having a cron job that fires every 3 hours instead of every 15
> minutes.
>
> With the exchange implementation, they'd only need to publish 1. Well,
> 2. Somewhere in the back of my mind, with the exchange idea, I thought
> they'd publish a cancel message to stop the flow. That has a couple of
> problems from a resiliency perspective, not least client-death
> (although we could detect that, it would start to get a touch messy).
>
> We could move all this to the client and have, say, a
> PeriodicPublisher which spawns a separate thread. That way, if the
> publishing client dies, the flow stops. The user ends up tied to our
> client library, but since the alternatives are language/framework
> dependent anyway that's not a huge cost. The main downside I see to
> this approach is that we'd need to reimplement this in each client,
> whereas the header or exchange routes avoid that.
>
> The other cost is that at least one extra thread gets spawned. I
> guess, in the Java client at least, we might be able to tie it into
> the dispatcher somehow but that could get hairy..
>
> Re-reading my original mail I wasn't entirely clear about the
> use-case: this was someone who basically wanted to publish the same
> message every N minutes. Think cron with a really basic client that
> sends a message in a static file. the deliver-after stuff was their
> top level stab at doing it in AMQP.
>
So - I'm tempted to say this is not really a Message Queuing use case... I
can see that the deliver-after type thing could be generally useful
(although you'd want to make it a per-queue option)... Having messages come
back from the dead periodically seems... odd... and having something that
periodically generates new messages is essentially wishing to write an
application inside the broker... 1-0 would define this as some sort of
service node... Even in 0-8/0-9/0-10 I would be more tempted to write this
as a custom queue than a custom exchange.
In terms of consumption, are you expecting multiple consumers to share the
processing of these messages or is there just one consumer? In the one
consumer case you could use my LVQ implementation (when I check it in
properly) to read a message which details the periodicity on startup and
then just spawns a thread in the client to perform the task at the
periodicity required... Listening for updates to the periodicity in case it
is changed.
If no-one is listening do you care about having a persistent copy of the
message...? if no-one is listening for 2xperiod do you expect them to
process 2 separate messages on startup or just the one? In the latter case
I could buy the case for this being some sort of exchange in 0-8/0-9/0-10
speak
Anyways I'm tempted to implement the deliver-after thing as a general option
on queues...
Cheers,
Rob
> - Aidan
> --
> Apache Qpid - AMQP, JMS, other messaging love http://qpid.apache.org
> "A witty saying proves nothing" - Voltaire
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>
Re: qpid.periodic exchange type
Posted by Aidan Skinner <ai...@gmail.com>.
On Sun, Nov 8, 2009 at 5:15 PM, Robert Godfrey <ro...@gmail.com> wrote:
> 2009/11/8 Aidan Skinner <ai...@apache.org>
>
>> The original message is delivered and persisted as normal. The
>> exchange would hold onto a copy of that message, and generate an
>> entirely new, identical one at the specified interval.
>>
>> Putting this logic at the exchange level seems better to me because
>> it avoids further complicating the subscription implementation. It'll
>> also map a bit cleaner into 1.0 semantics (as a non-destructive link).
>>
>>
> I don't think we would need to complicate subscription implementation unless
> we wanted to implement this by having selectors which compared message
> fields to the present time (or some other condition that would need to be
> periodically reevaluated). I was thinking that we would implement like TTL
> - i.e. the message state on the queue is changed when the time-driven
> condition changes (in this case when it becomes live)... As happens already
> in the queue, the state change will cause the subscribers to go re-examine
> the message to see if they can now deliver it. Other than sorting out how
> we do timed events a bit, this seems like a fairly minor change at the queue
> level.
I hadn't realised we could signal state change to the subscriber to
have it reevaluate all the messages, that does make it much simpler.
AIUI messages which had an expired TTL were cleaned up by the
housekeeping thread after the pointer had already moved on.
> Having exchanges that retain state would be a big departure, and as Robert
> points out would complicate persistence. From a 1-0 point of view, there
> will be no such thing as exchanges, so I'm not sure I really see the
> advantage there. Basically the difference is between hard-coding the entry
> point/address for the message (by fixing the exchange type) or adding a new
Re-reading my mail I wasn't entirely clear about the use-case: this
was someone who basically wanted to publish the same message every N
minutes. Think cron with a really basic client that sends a message in
a static file. the deliver-after stuff was a top level stab at doing
it in AMQP.
> capability to queues. I much prefer the latter, and I also think it will be
> cheaper to implement
So, the main functional difference I see with an "X-deliver-after"
header is that the publisher has to occasionally push a fresh batch of
messages. Eventually all the messages would be consumed and they'd
have to send a new batch. I'm not sure that I see much benefit in
having a cron job that fires every 3 hours instead of every 15
minutes.
With the exchange implementation, they'd only need to publish 1. Well,
2. Somewhere in the back of my mind, with the exchange idea, I thought
they'd publish a cancel message to stop the flow. That has a couple of
problems from a resiliency perspective, not least client-death
(although we could detect that, it would start to get a touch messy).
We could move all this to the client and have, say, a
PeriodicPublisher which spawns a separate thread. That way, if the
publishing client dies, the flow stops. The user ends up tied to our
client library, but since the alternatives are language/framework
dependent anyway that's not a huge cost. The main downside I see to
this approach is that we'd need to reimplement this in each client,
whereas the header or exchange routes avoid that.
The other cost is that at least one extra thread gets spawned. I
guess, in the Java client at least, we might be able to tie it into
the dispatcher somehow but that could get hairy..
Re-reading my original mail I wasn't entirely clear about the
use-case: this was someone who basically wanted to publish the same
message every N minutes. Think cron with a really basic client that
sends a message in a static file. the deliver-after stuff was their
top level stab at doing it in AMQP.
- Aidan
--
Apache Qpid - AMQP, JMS, other messaging love http://qpid.apache.org
"A witty saying proves nothing" - Voltaire
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: qpid.periodic exchange type
Posted by Robert Godfrey <ro...@gmail.com>.
2009/11/8 Aidan Skinner <ai...@apache.org>
> The original message is delivered and persisted as normal. The
> exchange would hold onto a copy of that message, and generate an
> entirely new, identical one at the specified interval.
>
> Putting this logic at the exchange level seems better to me because
> it avoids further complicating the subscription implementation. It'll
> also map a bit cleaner into 1.0 semantics (as a non-destructive link).
>
>
I don't think we would need to complicate subscription implementation unless
we wanted to implement this by having selectors which compared message
fields to the present time (or some other condition that would need to be
periodically reevaluated). I was thinking that we would implement like TTL
- i.e. the message state on the queue is changed when the time-driven
condition changes (in this case when it becomes live)... As happens already
in the queue, the state change will cause the subscribers to go re-examine
the message to see if they can now deliver it. Other than sorting out how
we do timed events a bit, this seems like a fairly minor change at the queue
level.
Having exchanges that retain state would be a big departure, and as Robert
points out would complicate persistence. From a 1-0 point of view, there
will be no such thing as exchanges, so I'm not sure I really see the
advantage there. Basically the difference is between hard-coding the entry
point/address for the message (by fixing the exchange type) or adding a new
capability to queues. I much prefer the latter, and I also think it will be
cheaper to implement
Cheers,
Rob
Re: qpid.periodic exchange type
Posted by Aidan Skinner <ai...@apache.org>.
The original message is delivered and persisted as normal. The
exchange would hold onto a copy of that message, and generate an
entirely new, identical one at the specified interval.
Putting this logic at the exchange level seems better to me because
it avoids further complicating the subscription implementation. It'll
also map a bit cleaner into 1.0 semantics (as a non-destructive link).
On 11/7/09, Robert Greig <ro...@gmail.com> wrote:
> 2009/11/7 Robert Godfrey <ro...@gmail.com>:
>> It would seem to me that it would be better for this use-case to instead
>> implement something more analogous to time-to-live... only rather than
>> that
>> signaling the time at which the message would be archived it would signal
>> the time at which the message becomes available for consumption. I'm not
>> too keen on Exchanges that hold on to messages...
>
> Yes, I agree. How would you handle persistent messages if you hold
> them at the exchange?
>
> RG
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>
--
Sent from Google Mail for mobile | mobile.google.com
Apache Qpid - AMQP, JMS, other messaging love http://qpid.apache.org
"A witty saying proves nothing" - Voltaire
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: qpid.periodic exchange type
Posted by Robert Greig <ro...@gmail.com>.
2009/11/7 Robert Godfrey <ro...@gmail.com>:
> It would seem to me that it would be better for this use-case to instead
> implement something more analogous to time-to-live... only rather than that
> signaling the time at which the message would be archived it would signal
> the time at which the message becomes available for consumption. I'm not
> too keen on Exchanges that hold on to messages...
Yes, I agree. How would you handle persistent messages if you hold
them at the exchange?
RG
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: qpid.periodic exchange type
Posted by Robert Godfrey <ro...@gmail.com>.
It would seem to me that it would be better for this use-case to instead
implement something more analogous to time-to-live... only rather than that
signaling the time at which the message would be archived it would signal
the time at which the message becomes available for consumption. I'm not
too keen on Exchanges that hold on to messages...
-- Rob
2009/11/7 Aidan Skinner <ai...@apache.org>
> So, somebody asked me on IRC about delaying messages - sending a
> message but not having it delivered for a while. We don't do that, so
> I asked about their use case and they want to be able to put a bunch
> of "do this" messages onto a queue and having them drip off slowly.
>
> That would be quite messy to implement, so another (maybe more AMQP-y)
> way of doing this might be to have a periodic exchange which you send
> messages too and then it routes it onto another exchange (you must
> specify another exchange on the X-Additional-Exchange) with the same
> routing key. The exchange keeps a copy of the message, and sends a new
> one to the same routing key and exchange at intervals specified in an
> X-Period (float, hz) header.
>
> There are apparently a few things that implement similar features,
> like celery, that sit above AMQP but are a bit heavy if you just want
> this. I'd like to get this into the Java broker for 0.7, I expect
> it'll need a bit of iteration to get the interface right.
>
> - Aidan
>
> --
> Apache Qpid - AMQP, JMS, other messaging love http://qpid.apache.org
> "A witty saying proves nothing" - Voltaire
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>