Message history and replay

[Colin Crist <co...@hermesjms.com>]

Hi,

A quick question...

A common recovery scenario is requesting replay on a queue from a given,
known message, either by its ID or by something in the header. Its saves all
that mucking about with XA and makes recovery the normal way of system
startup rather then some exceptional case.

This means when a message has reached all its recipients it should never be
marked for removal from the durable message store. 

As a user, I would also like to control when messages get removed as part of
my end of day or even weekly processs, probably based on header property
selectors. 

I don't believe this is supported in AMQP in the wire protocol (nor perhaps
should it be) - is it maybe something that could be done via some
standardised command messages to a "replay" exchange?

Any thoughts on implementation difficulty? 

Regards,

Colin.

Re: Message history and replay

[Bhupendra Bhardwaj <bh...@gmail.com>]

Hi,

Another usecase from developers point of view, which makes replaying a
desired feature, is that when the consumer applicaitons themselves are
performing some tests (volume testing) or testing results of some fixes,
they would like to replay some set of messages again and again. Which is
easier than producing and sending those all again or requesting producing
applications to send those messages again.


Regards,
Bhupendra


On 1/23/07, John O'Hara <jo...@gmail.com> wrote:
>
> I know this use case too.
> (Its what STS does for those who know that system).
>
> The key point is: Messaging as the recovery point.
>
> The most visible, unwavering use case is where you lose an applications
> RDBMS and you recover it from a backup.
> You've missed some events in the recovered copy, so you need to replay all
> the upstream systems.
> Often this is not programmatically controlled by the application, but an
> administrator asking for the replay.
>
> The other variant is more like yours (and FIX) where replay from the
> middleware is used as a recovery journal.  In FIX this can involve
> recovering all the way from the start of day even if you fell over at 4pm
> (this style of journalling recovery is infamous for its slow startup, but
> people keep using it because its simple and it works).
>
> In STS, messages were held for 4 days (a bank holiday weekend worth!), the
> message DB was 100MB.
>
> Just because the likes of MQ can't do this, and archives to a separate
> queue, doesnt mean AMQP should support this.
> I think its just 'rewinding a pointer' on the queue being read by the
> client.
>
> This does not involve exchanges at all in my view.  I'm reading my inbox
> again, not redelivering the mail.
> There is also the question of message identity on replay.  I would suggest
> the identity should not change, esp. considering the recovery use case.
>
> Other requirements - for queues there is also the requirement to:
> 1) Park messages that have arrived in error, but you don't want to throw
> them away; you'll process them later
> 2) Resubmit messages off a dead letter queue
>
> Another one to write up.
> John
>
> On 22/01/07, Colin Crist <co...@hermesjms.com> wrote:
> >
> >
> > Hi Alan,
> >
> > I've seen other replies to this but I had this in draft so will take it
> > mail
> > by mail..
> >
> > The scenario you describe is different to my requirement - at least if I
> > read you correctly.
> >
> > My requirement is a consumer initiated replay of messages from a queue
> > where
> > the messages are originating in any exchange. Tyically the consumers
> > recovery is to "fill in a gap" of messages it may have missed as its
> > avoiding XA and is optimistically writing to is store (RDBMS or
> whatever)
> > before doing a 1 phase commit in the right order. Additionally, as its
> > messaging and not a database, just if I've commited my messages is no
> > reason
> > not to want them again, at least not in my ideal solution :) Messaging
> is
> > the recovery point.
> >
> > Replay would be onto a different queue to maintain ordering (but with
> > dupes
> > of course, idempotent consumers is my contract) whilst new messages
> would
> > be
> > getting delivered to the main queue (with a subset on the head being
> > duplicated in the replay queue). The replayed messages would be drained
> by
> > the client from the replay queue and then normal message consumption
> would
> > resume on the main queue.
> >
> > From my point of view, the contract is with the queue the consumer is
> > reading from and not the originating exchange(s). I am most interested
> in
> > this with a pubsub exchange so I think core broker support is needed to
> > access the store and exchange chaining is not enough? There may be other
> > messages on the queue from other exchanges and they also need replaying.
> > The
> > consumer is not interested in the upstream exchange.
> >
> > The consumer would always recover like this as its starting from some
> > known
> > stable point in history. There needs to be some "rendezvous" (old skool
> > term
> > I know) with a "replay complete" message on the replay queue in order to
> > flip back to the main queue. A valid recovery point could also be a
> > configuration defined "start of day" timestamp for the most erratic
> > consumers.
> >
> > I'd like to use a hard messaging backbone as the ulimate point of
> > recovery.
> > I have bad consumers and want to rely on my messaging to fix my world.
> >
> > Please correct me if I am missing any AMQP behavour as I'm only causally
> > aware of the spec from reading the docs and not working with it at this
> > point and I wont "get it" until I work with it closer.
> >
> > More later,
> >
> > Many thanks all,
> >
> > Colin.
> >
> > >
> > > On Thu, 2007-01-18 at 15:52 +0000, Colin Crist wrote:
> > > > Hi,
> > > >
> > > > A quick question...
> > > >
> > > > A common recovery scenario is requesting replay on a queue from a
> > > > given, known message, either by its ID or by something in
> > > the header.
> > > > Its saves all that mucking about with XA and makes recovery
> > > the normal
> > > > way of system startup rather then some exceptional case.
> > > >
> > > > This means when a message has reached all its recipients it should
> > > > never be marked for removal from the durable message store.
> > > >
> > > > As a user, I would also like to control when messages get
> > > removed as
> > > > part of my end of day or even weekly processs, probably based on
> > > > header property selectors.
> > > >
> > > > I don't believe this is supported in AMQP in the wire protocol (nor
> > > > perhaps should it be) - is it maybe something that could be
> > > done via
> > > > some standardised command messages to a "replay" exchange?
> > > >
> > > > Any thoughts on implementation difficulty?
> > >
> > > You could represent this in AMQP protocol using a custom
> > > exchange type.
> > >
> > > A ReplayExchange stores all messages sent to it (until
> > > cleaned up by some administrative function.)
> > >
> > > When binding a queue to a replay exchange you MAY specify an
> > > argument "replay-from=N". A queue so bound will be
> > > "pre-filled" with the range of message starting from the Nth
> > > message up to the most recent message sent to the exchange.
> > > Thereafter the exchange behaves like a fan-out exchange, each
> > > new message being added to all the bound queues.
> > >
> > > That gives you replay entirely within the standard protocol,
> > > using a standard extension point to add this as proprietary
> > > broker functionality.
> > >
> > > Note that to combine replay with other exchange types
> > > requires exchange chaining which has been discussed several
> > > times but is not yet an agreed part of the protocol - this
> > > may be another use case for that discussion
> > >
> > > Cheers,
> > > Alan.
> > >
> > >
> >
> >
> >
>
>

Re: Message history and replay

[John O'Hara <jo...@gmail.com>]

I know this use case too.
(Its what STS does for those who know that system).

The key point is: Messaging as the recovery point.

The most visible, unwavering use case is where you lose an applications
RDBMS and you recover it from a backup.
You've missed some events in the recovered copy, so you need to replay all
the upstream systems.
Often this is not programmatically controlled by the application, but an
administrator asking for the replay.

The other variant is more like yours (and FIX) where replay from the
middleware is used as a recovery journal.  In FIX this can involve
recovering all the way from the start of day even if you fell over at 4pm
(this style of journalling recovery is infamous for its slow startup, but
people keep using it because its simple and it works).

In STS, messages were held for 4 days (a bank holiday weekend worth!), the
message DB was 100MB.

Just because the likes of MQ can't do this, and archives to a separate
queue, doesnt mean AMQP should support this.
I think its just 'rewinding a pointer' on the queue being read by the
client.

This does not involve exchanges at all in my view.  I'm reading my inbox
again, not redelivering the mail.
There is also the question of message identity on replay.  I would suggest
the identity should not change, esp. considering the recovery use case.

Other requirements - for queues there is also the requirement to:
1) Park messages that have arrived in error, but you don't want to throw
them away; you'll process them later
2) Resubmit messages off a dead letter queue

Another one to write up.
John

On 22/01/07, Colin Crist <co...@hermesjms.com> wrote:
>
>
> Hi Alan,
>
> I've seen other replies to this but I had this in draft so will take it
> mail
> by mail..
>
> The scenario you describe is different to my requirement - at least if I
> read you correctly.
>
> My requirement is a consumer initiated replay of messages from a queue
> where
> the messages are originating in any exchange. Tyically the consumers
> recovery is to "fill in a gap" of messages it may have missed as its
> avoiding XA and is optimistically writing to is store (RDBMS or whatever)
> before doing a 1 phase commit in the right order. Additionally, as its
> messaging and not a database, just if I've commited my messages is no
> reason
> not to want them again, at least not in my ideal solution :) Messaging is
> the recovery point.
>
> Replay would be onto a different queue to maintain ordering (but with
> dupes
> of course, idempotent consumers is my contract) whilst new messages would
> be
> getting delivered to the main queue (with a subset on the head being
> duplicated in the replay queue). The replayed messages would be drained by
> the client from the replay queue and then normal message consumption would
> resume on the main queue.
>
> From my point of view, the contract is with the queue the consumer is
> reading from and not the originating exchange(s). I am most interested in
> this with a pubsub exchange so I think core broker support is needed to
> access the store and exchange chaining is not enough? There may be other
> messages on the queue from other exchanges and they also need replaying.
> The
> consumer is not interested in the upstream exchange.
>
> The consumer would always recover like this as its starting from some
> known
> stable point in history. There needs to be some "rendezvous" (old skool
> term
> I know) with a "replay complete" message on the replay queue in order to
> flip back to the main queue. A valid recovery point could also be a
> configuration defined "start of day" timestamp for the most erratic
> consumers.
>
> I'd like to use a hard messaging backbone as the ulimate point of
> recovery.
> I have bad consumers and want to rely on my messaging to fix my world.
>
> Please correct me if I am missing any AMQP behavour as I'm only causally
> aware of the spec from reading the docs and not working with it at this
> point and I wont "get it" until I work with it closer.
>
> More later,
>
> Many thanks all,
>
> Colin.
>
> >
> > On Thu, 2007-01-18 at 15:52 +0000, Colin Crist wrote:
> > > Hi,
> > >
> > > A quick question...
> > >
> > > A common recovery scenario is requesting replay on a queue from a
> > > given, known message, either by its ID or by something in
> > the header.
> > > Its saves all that mucking about with XA and makes recovery
> > the normal
> > > way of system startup rather then some exceptional case.
> > >
> > > This means when a message has reached all its recipients it should
> > > never be marked for removal from the durable message store.
> > >
> > > As a user, I would also like to control when messages get
> > removed as
> > > part of my end of day or even weekly processs, probably based on
> > > header property selectors.
> > >
> > > I don't believe this is supported in AMQP in the wire protocol (nor
> > > perhaps should it be) - is it maybe something that could be
> > done via
> > > some standardised command messages to a "replay" exchange?
> > >
> > > Any thoughts on implementation difficulty?
> >
> > You could represent this in AMQP protocol using a custom
> > exchange type.
> >
> > A ReplayExchange stores all messages sent to it (until
> > cleaned up by some administrative function.)
> >
> > When binding a queue to a replay exchange you MAY specify an
> > argument "replay-from=N". A queue so bound will be
> > "pre-filled" with the range of message starting from the Nth
> > message up to the most recent message sent to the exchange.
> > Thereafter the exchange behaves like a fan-out exchange, each
> > new message being added to all the bound queues.
> >
> > That gives you replay entirely within the standard protocol,
> > using a standard extension point to add this as proprietary
> > broker functionality.
> >
> > Note that to combine replay with other exchange types
> > requires exchange chaining which has been discussed several
> > times but is not yet an agreed part of the protocol - this
> > may be another use case for that discussion
> >
> > Cheers,
> > Alan.
> >
> >
>
>
>

Re: Message history and replay

[Marnie McCormack <ma...@googlemail.com>]

I think you can add info to the JIRA as comments if that helps ?

Bfn,
Marnie


On 1/30/07, Carl Trieloff <cc...@redhat.com> wrote:
>
>
> Colin,
>
> Can you submit the test as a patch. We are busy working out how to add a
> someone to JIRA before they are a commiter and what that bar should be.
> We have agreed that the bar for JIRA should be lower but I am not clear
> whether we need a vote on that from an apache process point of view.
> When I asked for your user id, I should have first sent mail to clear
> this up. Do you know if other projects call a vote to add wiki access
> for existing apache committers?
>
> Sorry about the confusion, hope to have sorted shortly.
> Carl.
>
>
>
>
>
> Colin Crist wrote:
> > Hi Marnie,
> >
> > Thanks, I started to create a page earlier (prompted by the JIRA entry)
> and
> > it didn't let me save and I lost my edits. Brrrr. Luckily it was only a
> > paragraph. Odd that I could add a page but not save it. Maybe there is a
> > draft somewhere from the confluence auto-save but I can't find it.
> >
> > Can someone look at getting me the wherewithall to add pages? If I can't
> as
> > I'm not a commiter I can put the content on my confluence and email
> someone
> > the markup and images later - not a problem.
> >
> > Colin.
> >
> >
> >> -----Original Message-----
> >> From: Marnie McCormack [mailto:marnie.mccormack@googlemail.com]
> >> Sent: 30 January 2007 13:56
> >> To: qpid-dev@incubator.apache.org
> >> Subject: Re: Message history and replay
> >>
> >> Hi All,
> >>
> >> I think we're in danger of talking lots but not moving the
> >> requirements or implementation details forward.
> >>
> >> To help us keep track of the various use cases and suggested
> >> solutions I've created a Qpid JIRA with Colin's original
> >> opening post as details:
> >>
> >> http://issues.apache.org/jira/browse/QPID-329
> >>
> >> I appreciate that we'll probably need a corresponding JIRA on
> >> the AMQP system too, but think it's best to let Colin spend
> >> the time he kindly volunteered on the use cases etc before
> >> anything can usefully be raised there.
> >>
> >> Colin - I can't assign a JIRA to you at present. Can't
> >> remember what policy is for adding non-committers to the JIRA
> >> qpid developers group though. Carl can you clarify please ? Thx !
> >>
> >> Hth,
> >> Marnie
> >>
> >>
> >
> >
> >
>
>
>

Re: Message history and replay

[Carl Trieloff <cc...@redhat.com>]

Colin,

Can you submit the test as a patch. We are busy working out how to add a 
someone to JIRA before they are a commiter and what that bar should be. 
We have agreed that the bar for JIRA should be lower but I am not clear 
whether we need a vote on that from an apache process point of view. 
When I asked for your user id, I should have first sent mail to clear 
this up. Do you know if other projects call a vote to add wiki access 
for existing apache committers?

Sorry about the confusion, hope to have sorted shortly.
Carl.





Colin Crist wrote:
> Hi Marnie,
>
> Thanks, I started to create a page earlier (prompted by the JIRA entry) and
> it didn't let me save and I lost my edits. Brrrr. Luckily it was only a
> paragraph. Odd that I could add a page but not save it. Maybe there is a
> draft somewhere from the confluence auto-save but I can't find it.
>
> Can someone look at getting me the wherewithall to add pages? If I can't as
> I'm not a commiter I can put the content on my confluence and email someone
> the markup and images later - not a problem.
>
> Colin. 
>
>   
>> -----Original Message-----
>> From: Marnie McCormack [mailto:marnie.mccormack@googlemail.com] 
>> Sent: 30 January 2007 13:56
>> To: qpid-dev@incubator.apache.org
>> Subject: Re: Message history and replay
>>
>> Hi All,
>>
>> I think we're in danger of talking lots but not moving the 
>> requirements or implementation details forward.
>>
>> To help us keep track of the various use cases and suggested 
>> solutions I've created a Qpid JIRA with Colin's original 
>> opening post as details:
>>
>> http://issues.apache.org/jira/browse/QPID-329
>>
>> I appreciate that we'll probably need a corresponding JIRA on 
>> the AMQP system too, but think it's best to let Colin spend 
>> the time he kindly volunteered on the use cases etc before 
>> anything can usefully be raised there.
>>
>> Colin - I can't assign a JIRA to you at present. Can't 
>> remember what policy is for adding non-committers to the JIRA 
>> qpid developers group though. Carl can you clarify please ? Thx !
>>
>> Hth,
>> Marnie
>>
>>     
>
>
>   

RE: Message history and replay

[Colin Crist <co...@hermesjms.com>]

Hi Marnie,

Thanks, I started to create a page earlier (prompted by the JIRA entry) and
it didn't let me save and I lost my edits. Brrrr. Luckily it was only a
paragraph. Odd that I could add a page but not save it. Maybe there is a
draft somewhere from the confluence auto-save but I can't find it.

Can someone look at getting me the wherewithall to add pages? If I can't as
I'm not a commiter I can put the content on my confluence and email someone
the markup and images later - not a problem.

Colin. 

> -----Original Message-----
> From: Marnie McCormack [mailto:marnie.mccormack@googlemail.com] 
> Sent: 30 January 2007 13:56
> To: qpid-dev@incubator.apache.org
> Subject: Re: Message history and replay
> 
> Hi All,
> 
> I think we're in danger of talking lots but not moving the 
> requirements or implementation details forward.
> 
> To help us keep track of the various use cases and suggested 
> solutions I've created a Qpid JIRA with Colin's original 
> opening post as details:
> 
> http://issues.apache.org/jira/browse/QPID-329
> 
> I appreciate that we'll probably need a corresponding JIRA on 
> the AMQP system too, but think it's best to let Colin spend 
> the time he kindly volunteered on the use cases etc before 
> anything can usefully be raised there.
> 
> Colin - I can't assign a JIRA to you at present. Can't 
> remember what policy is for adding non-committers to the JIRA 
> qpid developers group though. Carl can you clarify please ? Thx !
> 
> Hth,
> Marnie
> 

Re: Message history and replay

[Marnie McCormack <ma...@googlemail.com>]

Hi All,

I think we're in danger of talking lots but not moving the requirements or
implementation details forward.

To help us keep track of the various use cases and suggested solutions I've
created a Qpid JIRA with Colin's original opening post as details:

http://issues.apache.org/jira/browse/QPID-329

I appreciate that we'll probably need a corresponding JIRA on the AMQP
system too, but think it's best to let Colin spend the time he kindly
volunteered on the use cases etc before anything can usefully be raised
there.

Colin - I can't assign a JIRA to you at present. Can't remember what policy
is for adding non-committers to the JIRA qpid developers group though. Carl
can you clarify please ? Thx !

Hth,
Marnie

Re: Message history and replay

[Martin Ritchie <ri...@apache.org>]

On 29/01/07, Alan Conway <ac...@redhat.com> wrote:
> On Mon, 2007-01-29 at 17:53 +0000, Gordon Sim wrote:
> > Alan Conway wrote:
> > > That raises an interesting question: should AMQP support (custom) Queue
> > > types?
> >
> > There is an 'arguments' field of type table that could be used to
> > indicate queue type (or desired characteristics) albeit in a
> > non-standard way.
>
> That is true. However the question remains - is it better to provide a
> single Queue class that does everything (including replay) or a
> selection of Queue classes with different features? I'm not sure of the
> right answer. I am leaning towards the swiss-army-queue because AMQP
> does not provide any way to compose features, so separate Queues would
> suffer from the "but I want it to do X AND Y" problem and because I'm
> guessing the list of things that anyone would want to a queue to do is
> bounded and fairly small - but I'm not sure of that last point either
> yet.
>
> Cheers,
> Alan.

You could of course dynamically change the type of queue in use
depending on the usage. e.g. We could write a queue that is slightly
more efficient when you don't have queue browsers or JMS selectors but
when a client connects that requests this then the DeliveryManager
(The guts of our AMQ Queue) could be changed as "normally" there is no
list of messages. I say "normally" as you only have a list stored on
the broker when there are no consumers or not sufficiently fast
consumption.


-- 
Martin Ritchie

Re: Message history and replay

[John O'Hara <jo...@gmail.com>]

Please please please please start with a single "smart queue".

It's almost worthwhile treating the queue as an application in itself, it
could be very sophisticated.
It's also where AMQP based products will have most differentiation.

In fact, you could imagine having the queue as a library that could be used
entirely standalone away from Qpid -- a bit like embedding the Derby DB
inside an application.  You could embed a Qpid Queue inside, say, an email
or RSS reader; or use it in some replication software, or whatever needs
persistent or 'smart' queueing.  Like a Unix pipe on steroids.

See what I mean?

John


On 29/01/07, Alan Conway <ac...@redhat.com> wrote:
>
> On Mon, 2007-01-29 at 17:53 +0000, Gordon Sim wrote:
> > Alan Conway wrote:
> > > That raises an interesting question: should AMQP support (custom)
> Queue
> > > types?
> >
> > There is an 'arguments' field of type table that could be used to
> > indicate queue type (or desired characteristics) albeit in a
> > non-standard way.
>
> That is true. However the question remains - is it better to provide a
> single Queue class that does everything (including replay) or a
> selection of Queue classes with different features? I'm not sure of the
> right answer. I am leaning towards the swiss-army-queue because AMQP
> does not provide any way to compose features, so separate Queues would
> suffer from the "but I want it to do X AND Y" problem and because I'm
> guessing the list of things that anyone would want to a queue to do is
> bounded and fairly small - but I'm not sure of that last point either
> yet.
>
> Cheers,
> Alan.
>
>

Re: Message history and replay

[Alan Conway <ac...@redhat.com>]

On Mon, 2007-01-29 at 17:53 +0000, Gordon Sim wrote:
> Alan Conway wrote:
> > That raises an interesting question: should AMQP support (custom) Queue
> > types? 
> 
> There is an 'arguments' field of type table that could be used to 
> indicate queue type (or desired characteristics) albeit in a 
> non-standard way.

That is true. However the question remains - is it better to provide a
single Queue class that does everything (including replay) or a
selection of Queue classes with different features? I'm not sure of the
right answer. I am leaning towards the swiss-army-queue because AMQP
does not provide any way to compose features, so separate Queues would
suffer from the "but I want it to do X AND Y" problem and because I'm
guessing the list of things that anyone would want to a queue to do is
bounded and fairly small - but I'm not sure of that last point either
yet.

Cheers,
Alan.

Re: Message history and replay

[Gordon Sim <gs...@redhat.com>]

Alan Conway wrote:
> That raises an interesting question: should AMQP support (custom) Queue
> types? 

There is an 'arguments' field of type table that could be used to 
indicate queue type (or desired characteristics) albeit in a 
non-standard way.

RE: Message history and replay

[Alan Conway <ac...@redhat.com>]

On Mon, 2007-01-22 at 21:27 +0000, Colin Crist wrote:
> Hi Alan,
> 
> I've seen other replies to this but I had this in draft so will take it mail
> by mail..
> 
> The scenario you describe is different to my requirement - at least if I
> read you correctly.
[snip: argues that replay is a property of a Queue, not an Exchange.
> Please correct me if I am missing any AMQP behavour as I'm only causally
> aware of the spec from reading the docs and not working with it at this
> point and I wont "get it" until I work with it closer.

The main reason I suggested a ReplayExchange is because custom exchanges
are a recognized extension point in AMQP already, but I agree replay
seems like a job for a queue. Queues are the storage agents in AMQP, the
role of an Exchange is selection or routing. Clearly replay is all about
storage. However we can't extend the behavior of a Queue without
changing the protocol.

That raises an interesting question: should AMQP support (custom) Queue
types? Is it better to add replay (and probably other functionality)
into a single swiss-army-Queue or to create a selection of Queue types
(e.g. ReplayableQueue) that provide different functionality. Both
approaches have pros & cons: the swiss-army-queue risks becoming over
complex and penalizing simple users with undesired overhead from complex
use cases. On the other hand multiple queue types suffer from inability
to combine features provided by different queue types in a single queue?

Cheers,
Alan.

RE: Message history and replay

[Colin Crist <co...@hermesjms.com>]

Hi Alan,

I've seen other replies to this but I had this in draft so will take it mail
by mail..

The scenario you describe is different to my requirement - at least if I
read you correctly.

My requirement is a consumer initiated replay of messages from a queue where
the messages are originating in any exchange. Tyically the consumers
recovery is to "fill in a gap" of messages it may have missed as its
avoiding XA and is optimistically writing to is store (RDBMS or whatever)
before doing a 1 phase commit in the right order. Additionally, as its
messaging and not a database, just if I've commited my messages is no reason
not to want them again, at least not in my ideal solution :) Messaging is
the recovery point.

Replay would be onto a different queue to maintain ordering (but with dupes
of course, idempotent consumers is my contract) whilst new messages would be
getting delivered to the main queue (with a subset on the head being
duplicated in the replay queue). The replayed messages would be drained by
the client from the replay queue and then normal message consumption would
resume on the main queue.

>From my point of view, the contract is with the queue the consumer is
reading from and not the originating exchange(s). I am most interested in
this with a pubsub exchange so I think core broker support is needed to
access the store and exchange chaining is not enough? There may be other
messages on the queue from other exchanges and they also need replaying. The
consumer is not interested in the upstream exchange.

The consumer would always recover like this as its starting from some known
stable point in history. There needs to be some "rendezvous" (old skool term
I know) with a "replay complete" message on the replay queue in order to
flip back to the main queue. A valid recovery point could also be a
configuration defined "start of day" timestamp for the most erratic
consumers.

I'd like to use a hard messaging backbone as the ulimate point of recovery.
I have bad consumers and want to rely on my messaging to fix my world. 

Please correct me if I am missing any AMQP behavour as I'm only causally
aware of the spec from reading the docs and not working with it at this
point and I wont "get it" until I work with it closer.

More later,

Many thanks all,

Colin.

> 
> On Thu, 2007-01-18 at 15:52 +0000, Colin Crist wrote:
> > Hi,
> > 
> > A quick question...
> > 
> > A common recovery scenario is requesting replay on a queue from a 
> > given, known message, either by its ID or by something in 
> the header. 
> > Its saves all that mucking about with XA and makes recovery 
> the normal 
> > way of system startup rather then some exceptional case.
> > 
> > This means when a message has reached all its recipients it should 
> > never be marked for removal from the durable message store.
> > 
> > As a user, I would also like to control when messages get 
> removed as 
> > part of my end of day or even weekly processs, probably based on 
> > header property selectors.
> > 
> > I don't believe this is supported in AMQP in the wire protocol (nor 
> > perhaps should it be) - is it maybe something that could be 
> done via 
> > some standardised command messages to a "replay" exchange?
> > 
> > Any thoughts on implementation difficulty? 
> 
> You could represent this in AMQP protocol using a custom 
> exchange type.
> 
> A ReplayExchange stores all messages sent to it (until 
> cleaned up by some administrative function.)
> 
> When binding a queue to a replay exchange you MAY specify an 
> argument "replay-from=N". A queue so bound will be 
> "pre-filled" with the range of message starting from the Nth 
> message up to the most recent message sent to the exchange. 
> Thereafter the exchange behaves like a fan-out exchange, each 
> new message being added to all the bound queues.
> 
> That gives you replay entirely within the standard protocol, 
> using a standard extension point to add this as proprietary 
> broker functionality.
> 
> Note that to combine replay with other exchange types 
> requires exchange chaining which has been discussed several 
> times but is not yet an agreed part of the protocol - this 
> may be another use case for that discussion
> 
> Cheers,
> Alan.
> 
> 

RE: Message history and replay

[Colin Crist <co...@hermesjms.com>]

 

Hi Carl,

Carl Trieloff wote:

> This has been asked for before. I would interested to see if 
> an exchange could be used. I also think this might bring up 
> the topic of chained exchanges again.
> 
> Colin, do you have any interest in creating a prototype 
> ReplyExchange to see how well it works?
> Carl.

Very much so. 

I'd like to spend the next few weeks, until I have time to look at any
implemenation, thrashing out what such a ReplayExchange really means. What
were the other requirements you've been given? Perhaps mine are quite
specific and there is a more general pattern that can be applied?

Regards,

Colin.

Re: Message history and replay

[Carl Trieloff <cc...@redhat.com>]

Another very common case is consumers joining late and wanting a full 
reply of the queue.

more comments in-line.

Marnie McCormack wrote:
> I've been trying to imagine the scenarios in which a user will make 
> use of a ReplayExchange.
>  
> I thought that the most common usage would be the following:
>  
> - the receiving application has lost some messages sent to it and 
> needs them to be resent from a point in time or a message ID
>  
> (The only other recovery scenario I thought likely was broker failure, 
> which would be handled by restarted (with persistence) and the 
> undelivered messages would already remain in the BDB store.)
>  
> So, the bit I'm not sure about is this - you have no way of knowing in 
> advance the point at which replay might be needed from (unless using 
> client acks - which work with ordinary queues ?).
>  
> Thus your recipient app starts up again (or recognises it has lost 
> messages) and then wants to replay. So, the queue it is reading from 
> is rebound to a replay exchange (with a marker ID) ?

Reply is usually only use for new connection or full logs. A client 
should not loose messages and if it does it should error.

> Does that mean that it is actually reading from a different queue than 
> normal operation,
yes this is how it is typically done.

> or that the queue is bound to a different exchange type ?
>  
> The logic behind this could be quite complex - in that the balance 
> between normal operation and replay operation in subtle and would 
> involve the receiving client understanding the AMQP behaviour at quite 
> a low level ?
>  
> Apologies if I've missed a simple flow for replay :-)
>  
> Regards,
> Marnie
>  
> On 1/22/07, *Carl Trieloff* <cctrieloff@redhat.com 
> <ma...@redhat.com>> wrote:
>
>
>     This has been asked for before. I would interested to see if an
>     exchange
>     could be used. I also think this
>     might bring up the topic of chained exchanges again.
>
>     Colin, do you have any interest in creating a prototype
>     ReplyExchange to
>     see how well it works?
>     Carl.
>
>     Alan Conway wrote:
>     > On Thu, 2007-01-18 at 15:52 +0000, Colin Crist wrote:
>     >
>     >> Hi,
>     >>
>     >> A quick question...
>     >>
>     >> A common recovery scenario is requesting replay on a queue from
>     a given,
>     >> known message, either by its ID or by something in the header.
>     Its saves all
>     >> that mucking about with XA and makes recovery the normal way of
>     system
>     >> startup rather then some exceptional case.
>     >>
>     >> This means when a message has reached all its recipients it
>     should never be
>     >> marked for removal from the durable message store.
>     >>
>     >> As a user, I would also like to control when messages get
>     removed as part of
>     >> my end of day or even weekly processs, probably based on header
>     property
>     >> selectors.
>     >>
>     >> I don't believe this is supported in AMQP in the wire protocol
>     (nor perhaps
>     >> should it be) - is it maybe something that could be done via some
>     >> standardised command messages to a "replay" exchange?
>     >>
>     >> Any thoughts on implementation difficulty?
>     >>
>     >
>     > You could represent this in AMQP protocol using a custom
>     exchange type.
>     >
>     > A ReplayExchange stores all messages sent to it (until cleaned
>     up by
>     > some administrative function.)
>     >
>     > When binding a queue to a replay exchange you MAY specify an
>     argument
>     > "replay-from=N". A queue so bound will be "pre-filled" with the
>     range of
>     > message starting from the Nth message up to the most recent
>     message sent
>     > to the exchange. Thereafter the exchange behaves like a fan-out
>     > exchange, each new message being added to all the bound queues.
>     >
>     > That gives you replay entirely within the standard protocol, using a
>     > standard extension point to add this as proprietary broker
>     > functionality.
>     >
>     > Note that to combine replay with other exchange types requires
>     exchange
>     > chaining which has been discussed several times but is not yet
>     an agreed
>     > part of the protocol - this may be another use case for that
>     discussion
>     >
>     > Cheers,
>     > Alan.
>     >
>     >
>
>
>

Re: Message history and replay

[John O'Hara <jo...@gmail.com>]

It is definitely true that message throughput may increase if we let the
B-tree grow a bit.
It removes contention between the delete process and the enqueue process.

I've actually seen this happen on the STS application (for those who've
heard of it).
Having a reaper/garbage collector/vacuum/re-org process is a good way to go.

John

On 25/01/07, Robert Greig <ro...@gmail.com> wrote:
>
> On 25/01/07, Marnie McCormack <ma...@googlemail.com> wrote:
> > I don't know about how BDB manages indexing (or it's equivalent) - I'm
> > supposing this would be key to how fast updates to growing tables would
> be ?
>
> BDB is essentially just a btree. We index the delivery table on
> message id, so the update would be to the value not the key. The
> update should in fact be cheaper than deletion of the row.
>
> We may also choose to create a secondary index on delete date.
>
> > Certainly backup/restore processes would be expected to lengthen -
> though no
> > way of knowing by what factor.
>
> I think (hope!) the backup and restore times will be proportional to
> the number of messages in the store, irrespective of whether they are
> delivered or pending delivery.
>
> RG
>

Re: Message history and replay

[Robert Greig <ro...@gmail.com>]

On 25/01/07, Marnie McCormack <ma...@googlemail.com> wrote:

> Re below - surely one key point in this discussion around replay thus far
> has included the idea that delivered messages stay in the store. Thus, the
> store (and queues) could potentially get far larger than otherwise .....

Yes but this is still a known value and if the impact on
backup/recovery is known then it's not really a problem?

i.e. we are not saying that by supporting replay backup will become O(n^2).

RG

Re: Message history and replay

[John O'Hara <jo...@gmail.com>]

True.

On 25/01/07, Marnie McCormack <ma...@googlemail.com> wrote:
>
> In summary, I don't dispute anything we've so far discussed.
>
> I would however posit that we should not lose sight of the impact(s) of
> any
> solution on the various affected elements (store space, recovery/backup
> times, index rebuilding requirements ? etc).
>
> This is the key point I'm making - along with testing our various
> assumptions about performance :-) We need to able to explain/forecast the
> likely impact on an application of introducing reply - so we can explain
> it
> to our users.
>
> The only other point I'd make is my opening one - we need to be careful
> where we want to draw the line between message replay and running reports
> etc against retained messages. Sure we might want to support message
> selectors etc, but there has to be a line and SQL style features is surely
> it.
>
> As many users will be currently using an RDBMS for replay style
> functionality we should be careful not to go down the path of adding
> features to take their RDBMS reqs out of the picture. This would almost
> certainly be a mistake !
>
> Marnie
>
>
> On 1/25/07, John O'Hara <jo...@gmail.com> wrote:
> >
> > That's fine.
> > A design assumption should be to hold 5 days messages in the store. This
> > is
> > more of a space than a performance issue.  People who need this can pay
> > for
> > the disk.
> >
> > That means that a queue size could be 500% of a day's activity.
> > If I were to use Qpid in serious business application, that could be 1
> > million 16KB messages.  About 100GB in total.
> >
> > Looking forward, I'd imagine that the way disk and compute are going the
> > architecture will need to cater for large storage sizes.  Hitachi will
> > release a 1TB PC hard disk this quarter, and it won't cost a fortune.
> >
> > John
> >
> >
> >
> > On 25/01/07, Marnie McCormack <ma...@googlemail.com> wrote:
> > >
> > > Hi,
> > >
> > > Re below - surely one key point in this discussion around replay thus
> > far
> > > has included the idea that delivered messages stay in the store. Thus,
> > the
> > > store (and queues) could potentially get far larger than otherwise
> .....
> > >
> > > Marnie
> > >
> > >
> > > On 1/25/07, Robert Greig <ro...@gmail.com> wrote:
> > > >
> > > > I think (hope!) the backup and restore times will be proportional to
> > > > the number of messages in the store, irrespective of whether they
> are
> > > > delivered or pending delivery.
> > > >
> > > > RG
> > > >
> > >
> > >
> >
> >
>
>

Re: Message history and replay

[Marnie McCormack <ma...@googlemail.com>]

In summary, I don't dispute anything we've so far discussed.

I would however posit that we should not lose sight of the impact(s) of any
solution on the various affected elements (store space, recovery/backup
times, index rebuilding requirements ? etc).

This is the key point I'm making - along with testing our various
assumptions about performance :-) We need to able to explain/forecast the
likely impact on an application of introducing reply - so we can explain it
to our users.

The only other point I'd make is my opening one - we need to be careful
where we want to draw the line between message replay and running reports
etc against retained messages. Sure we might want to support message
selectors etc, but there has to be a line and SQL style features is surely
it.

As many users will be currently using an RDBMS for replay style
functionality we should be careful not to go down the path of adding
features to take their RDBMS reqs out of the picture. This would almost
certainly be a mistake !

Marnie


On 1/25/07, John O'Hara <jo...@gmail.com> wrote:
>
> That's fine.
> A design assumption should be to hold 5 days messages in the store. This
> is
> more of a space than a performance issue.  People who need this can pay
> for
> the disk.
>
> That means that a queue size could be 500% of a day's activity.
> If I were to use Qpid in serious business application, that could be 1
> million 16KB messages.  About 100GB in total.
>
> Looking forward, I'd imagine that the way disk and compute are going the
> architecture will need to cater for large storage sizes.  Hitachi will
> release a 1TB PC hard disk this quarter, and it won't cost a fortune.
>
> John
>
>
>
> On 25/01/07, Marnie McCormack <ma...@googlemail.com> wrote:
> >
> > Hi,
> >
> > Re below - surely one key point in this discussion around replay thus
> far
> > has included the idea that delivered messages stay in the store. Thus,
> the
> > store (and queues) could potentially get far larger than otherwise .....
> >
> > Marnie
> >
> >
> > On 1/25/07, Robert Greig <ro...@gmail.com> wrote:
> > >
> > > I think (hope!) the backup and restore times will be proportional to
> > > the number of messages in the store, irrespective of whether they are
> > > delivered or pending delivery.
> > >
> > > RG
> > >
> >
> >
>
>

Re: Message history and replay

[John O'Hara <jo...@gmail.com>]

That's fine.
A design assumption should be to hold 5 days messages in the store. This is
more of a space than a performance issue.  People who need this can pay for
the disk.

That means that a queue size could be 500% of a day's activity.
If I were to use Qpid in serious business application, that could be 1
million 16KB messages.  About 100GB in total.

Looking forward, I'd imagine that the way disk and compute are going the
architecture will need to cater for large storage sizes.  Hitachi will
release a 1TB PC hard disk this quarter, and it won't cost a fortune.

John



On 25/01/07, Marnie McCormack <ma...@googlemail.com> wrote:
>
> Hi,
>
> Re below - surely one key point in this discussion around replay thus far
> has included the idea that delivered messages stay in the store. Thus, the
> store (and queues) could potentially get far larger than otherwise .....
>
> Marnie
>
>
> On 1/25/07, Robert Greig <ro...@gmail.com> wrote:
> >
> > I think (hope!) the backup and restore times will be proportional to
> > the number of messages in the store, irrespective of whether they are
> > delivered or pending delivery.
> >
> > RG
> >
>
>

Re: Message history and replay

[Marnie McCormack <ma...@googlemail.com>]

Hi,

Re below - surely one key point in this discussion around replay thus far
has included the idea that delivered messages stay in the store. Thus, the
store (and queues) could potentially get far larger than otherwise .....

Marnie


On 1/25/07, Robert Greig <ro...@gmail.com> wrote:
>
> I think (hope!) the backup and restore times will be proportional to
> the number of messages in the store, irrespective of whether they are
> delivered or pending delivery.
>
> RG
>

Re: Message history and replay

[Robert Greig <ro...@gmail.com>]

On 25/01/07, Marnie McCormack <ma...@googlemail.com> wrote:
> I don't know about how BDB manages indexing (or it's equivalent) - I'm
> supposing this would be key to how fast updates to growing tables would be ?

BDB is essentially just a btree. We index the delivery table on
message id, so the update would be to the value not the key. The
update should in fact be cheaper than deletion of the row.

We may also choose to create a secondary index on delete date.

> Certainly backup/restore processes would be expected to lengthen - though no
> way of knowing by what factor.

I think (hope!) the backup and restore times will be proportional to
the number of messages in the store, irrespective of whether they are
delivered or pending delivery.

RG

Re: Message history and replay

[Marnie McCormack <ma...@googlemail.com>]

I don't know about how BDB manages indexing (or it's equivalent) - I'm
supposing this would be key to how fast updates to growing tables would be ?

Certainly backup/restore processes would be expected to lengthen - though no
way of knowing by what factor.

M


On 1/25/07, Robert Greig <ro...@gmail.com> wrote:
>
> On 25/01/07, Marnie McCormack <ma...@googlemail.com> wrote:
>
> > IMHO I think the salient distinction (more to Robert's message) is that
> the
> > user perceives the *messaging* component of the application architecture
> to
> > have slowed.
> >
> > The overall end-to-end might be cheaper, but we need to mitigate the
> > impression that Qpid performance has degraded. So, whatever the cost and
> the
> > potential mitigation (recommended batch sizes etc) we need to be clear
> about
> > it.
>
> That is assuming that making messages replayable has a negative impact
> on performance. I don't think this is necessarily true. Certainly the
> way I envisage it working with the current BDB store would not impact
> performace  - you would replace deletion of a row in the delivery
> table with an update to a row.
>
> RG
>

Re: Message history and replay

[Robert Greig <ro...@gmail.com>]

On 25/01/07, Marnie McCormack <ma...@googlemail.com> wrote:

> IMHO I think the salient distinction (more to Robert's message) is that the
> user perceives the *messaging* component of the application architecture to
> have slowed.
>
> The overall end-to-end might be cheaper, but we need to mitigate the
> impression that Qpid performance has degraded. So, whatever the cost and the
> potential mitigation (recommended batch sizes etc) we need to be clear about
> it.

That is assuming that making messages replayable has a negative impact
on performance. I don't think this is necessarily true. Certainly the
way I envisage it working with the current BDB store would not impact
performace  - you would replace deletion of a row in the delivery
table with an update to a row.

RG

Re: Message history and replay

[Marnie McCormack <ma...@googlemail.com>]

I don't dispute that this is a useful addition to Qpid functionality, but am
merely concerned that we think carefully about performance trade-offs that
might arise :-)

IMHO I think the salient distinction (more to Robert's message) is that the
user perceives the *messaging* component of the application architecture to
have slowed.

The overall end-to-end might be cheaper, but we need to mitigate the
impression that Qpid performance has degraded. So, whatever the cost and the
potential mitigation (recommended batch sizes etc) we need to be clear about
it.

So, yes - let's do it, but let's think about the various architectures we'd
recommend for use with it while we do. Btw 1000/msgs sec for persistent
messages of any size is not an easy win !

Regards,
Marnie


On 1/24/07, John O'Hara <jo...@gmail.com> wrote:
>
> We will be supporting replayable messages.
> It's one of the first 'services' that we as users of other middleware have
> built when it doesn't exist.
>
> As Colin points out (and all the FIX users in the world) this is not a
> strange requirement, and our applications have had call to use the
> facility
> very routinely.
>
> A queue is a store, it is a resouce.  That makes Qpid a resource manager.
> Obviously you shouldnt store messages in queues for ever, but it is a
> valid
> first-class persistent-store for data between applications.
>
> Performance in this type of application (to say 1000 messages per second
> de-queuing) is achieve by running larger transactions against the queue
> and
> committing in batches.  This enables the slow part of the operation, the
> write to the log/journal (dept on architecture choice) to happen less
> frequently.
>
> John
>
>
>
> On 24/01/07, Robert Greig <ro...@gmail.com> wrote:
> >
> > On 24/01/07, Marnie McCormack <ma...@googlemail.com> wrote:
> >
> > > I can see how this would help, but I do think we need to consider the
> > likely
> > > performance trade-off for persistent messaging very carefully in this
> > > solution. It's hard to make persistent messaging really quick anyway
> (or
> > at
> > > least other messaging vendors find it hard :-) so we'll need to test
> > > comparatively so users would understand the 'cost' of replay-ability.
> >
> > I think that users make the messages replayable in some other way if
> > the messaging system does not support this (e.g. slap them into a
> > database before putting them onto the queue is an obvious example...).
> > That is almost certainly more expensive in terms of performance than
> > the messaging system making them replayable via its store
> > implementation.
> >
> > RG
> >
>
>

Re: Message history and replay

[John O'Hara <jo...@gmail.com>]

We will be supporting replayable messages.
It's one of the first 'services' that we as users of other middleware have
built when it doesn't exist.

As Colin points out (and all the FIX users in the world) this is not a
strange requirement, and our applications have had call to use the facility
very routinely.

A queue is a store, it is a resouce.  That makes Qpid a resource manager.
Obviously you shouldnt store messages in queues for ever, but it is a valid
first-class persistent-store for data between applications.

Performance in this type of application (to say 1000 messages per second
de-queuing) is achieve by running larger transactions against the queue and
committing in batches.  This enables the slow part of the operation, the
write to the log/journal (dept on architecture choice) to happen less
frequently.

John



On 24/01/07, Robert Greig <ro...@gmail.com> wrote:
>
> On 24/01/07, Marnie McCormack <ma...@googlemail.com> wrote:
>
> > I can see how this would help, but I do think we need to consider the
> likely
> > performance trade-off for persistent messaging very carefully in this
> > solution. It's hard to make persistent messaging really quick anyway (or
> at
> > least other messaging vendors find it hard :-) so we'll need to test
> > comparatively so users would understand the 'cost' of replay-ability.
>
> I think that users make the messages replayable in some other way if
> the messaging system does not support this (e.g. slap them into a
> database before putting them onto the queue is an obvious example...).
> That is almost certainly more expensive in terms of performance than
> the messaging system making them replayable via its store
> implementation.
>
> RG
>

Re: Message history and replay

[Robert Greig <ro...@gmail.com>]

On 24/01/07, Marnie McCormack <ma...@googlemail.com> wrote:

> I can see how this would help, but I do think we need to consider the likely
> performance trade-off for persistent messaging very carefully in this
> solution. It's hard to make persistent messaging really quick anyway (or at
> least other messaging vendors find it hard :-) so we'll need to test
> comparatively so users would understand the 'cost' of replay-ability.

I think that users make the messages replayable in some other way if
the messaging system does not support this (e.g. slap them into a
database before putting them onto the queue is an obvious example...).
That is almost certainly more expensive in terms of performance than
the messaging system making them replayable via its store
implementation.

RG

Re: Message history and replay

[Marnie McCormack <ma...@googlemail.com>]

Hi All,

I'll add some comments/thoughts to the wiki page when possible.

In the meantime, I'm just wondering about the notion that replaying messages
from a persistent store is likely to be the easy option when an RDBMS is
knackered. Can't help wondering if we're making some assumptions about the
manageability/reliability of BDB vs other vendor dbs .....

I can see how this would help, but I do think we need to consider the likely
performance trade-off for persistent messaging very carefully in this
solution. It's hard to make persistent messaging really quick anyway (or at
least other messaging vendors find it hard :-) so we'll need to test
comparatively so users would understand the 'cost' of replay-ability.

Bfn,
Marnie

On 1/24/07, Carl Trieloff <cc...@redhat.com> wrote:
>
>
> what is your apache cwiki user-id?
>
> Carl.
>
>
> Colin Crist wrote:
> > Sure, I'll put some notes together.
> >
> > I don't seem to have permissions to add any pages - can you give me
> access
> > or alternatively I can put the pages on my own confluence and get
> someone to
> > move them later.
> >
> > Colin.
> >
> >
> >> -----Original Message-----
> >> From: Carl Trieloff [mailto:cctrieloff@redhat.com]
> >> Sent: 24 January 2007 18:07
> >> To: qpid-dev@incubator.apache.org
> >> Subject: Re: Message history and replay
> >>
> >>
> >> Colin,
> >>
> >> Would you be willing to create a page on the wiki for this
> >> cwiki.apache.org/qpid and lay out all the use cases there so
> >> we don't loose them. then we can add designs etc there also
> >> as we explore this topic.
> >>
> >> Carl.
> >>
> >>
> >> Colin Crist wrote:
> >>
> >>> I believe the customary apache response here is:
> >>>
> >>> +1
> >>>
> >>> Colin.
> >>>
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: John O'Hara [mailto:john.r.ohara@gmail.com]
> >>>> Sent: 23 January 2007 16:56
> >>>> To: qpid-dev@incubator.apache.org
> >>>> Subject: Re: Message history and replay
> >>>>
> >>>> Two options:
> >>>> 1) Make the 'rewind' accessible to the client on connect
> >>>> 2) Make it an administrative function (reset next message to X)
> >>>>
> >>>> Both are valid.
> >>>> Also retentions of several days should be valid subject to
> >>>>
> >> resource
> >>
> >>>> constraints.
> >>>>
> >>>>
> >>>> Also Queues should have attributes related to this:
> >>>> a) Archive Retention Period  *0*, x seconds, y messages
> >>>> b) Rewindable Y / *N*
> >>>> c) Client Can Rewind Y / *N*
> >>>> d) Archive treatment - *Delete* or Copy-out
> >>>> e) Archive interval - *30s* (queue garbage collect period)
> >>>> f) Archive TTL-deleted messages - Y / *N*
> >>>>
> >>>> Defaults in *.
> >>>>
> >>>> Queues are smart resources.  We need to spec out what they can do
> >>>> (and what
> >>>> *subset* of that we propose to standardise over on the AMQP WG).
> >>>> Here's a great place to put value added features.
> >>>>
> >>>> John
> >>>>
> >>>> On 23/01/07, Colin Crist <co...@hermesjms.com> wrote:
> >>>>
> >>>>
> >>>>> I had thought a replay queue is where the messages would be
> >>>>>
> >>>>>
> >>>> put I'm no
> >>>>
> >>>>
> >>>>> longer convinced.
> >>>>>
> >>>>> In another mail John O'Hara uses the the reading email analogy.
> >>>>> Continuing this to its logical conclusion could mean that
> >>>>>
> >> replay is
> >>
> >>>>> actually an option when you open the queue for reading. It
> >>>>>
> >>>>>
> >>>> avoids the
> >>>>
> >>>>
> >>>>> client having to muck about with another queue.
> >>>>>
> >>>>> This is not the traditional view of a queue but is more
> >>>>>
> >>>>>
> >>>> general and I
> >>>>
> >>>>
> >>>>> like it.
> >>>>>
> >>>>> So, how about a queue contains all messages put on it
> >>>>>
> >> since it was
> >>
> >>>>> last "purged". When a queue is opened for reading, it can
> >>>>>
> >> be opened
> >>
> >>>>> with a selector and one part of that selector is whether
> >>>>>
> >> or not the
> >>
> >>>>> message has been commited by a consumer or not.
> >>>>>
> >>>>> A consumer could then always open the queue and request all
> >>>>>
> >>>>>
> >>>> messages,
> >>>>
> >>>>
> >>>>> committed or not, from a known point, e.g. a trade ID or
> >>>>>
> >>>>>
> >>>> message ID.
> >>>>
> >>>>
> >>>>> The key thing here I suppose is that this type of consumer
> >>>>>
> >>>>>
> >>>> is writing
> >>>>
> >>>>
> >>>>> the message in some other form into a database and so
> >>>>>
> >>>>>
> >>>> always knows the
> >>>>
> >>>>
> >>>>> last thing written.
> >>>>> A nice benefit is that if the database is really hosed you
> >>>>>
> >>>>>
> >>>> could get
> >>>>
> >>>>
> >>>>> replay from start of day. This would definitely help
> >>>>>
> >>>>>
> >>>> implement DR for
> >>>>
> >>>>
> >>>>> purely message based systems.
> >>>>>
> >>>>> So, can I specialise queues and layer this functionality on
> >>>>>
> >>>>>
> >>>> top of the
> >>>>
> >>>>
> >>>>> spec in the APID implementation with some kind of
> >>>>>
> >> provider specific
> >>
> >>>>> header property?
> >>>>>
> >>>>> Colin.
> >>>>>
> >>>>>
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Marnie McCormack [mailto:marnie.mccormack@googlemail.com]
> >>>>>> Sent: 22 January 2007 19:54
> >>>>>> To: qpid-dev@incubator.apache.org; cctrieloff@redhat.com
> >>>>>> Subject: Re: Message history and replay
> >>>>>>
> >>>>>> I've been trying to imagine the scenarios in which a user
> >>>>>>
> >>>>>>
> >>>> will make
> >>>>
> >>>>
> >>>>>> use of a ReplayExchange.
> >>>>>>
> >>>>>> I thought that the most common usage would be the following:
> >>>>>>
> >>>>>> - the receiving application has lost some messages sent
> >>>>>>
> >> to it and
> >>
> >>>>>> needs them to be resent from a point in time or a message ID
> >>>>>>
> >>>>>> (The only other recovery scenario I thought likely was broker
> >>>>>> failure, which would be handled by restarted (with
> >>>>>> persistence) and the undelivered messages would already
> >>>>>>
> >> remain in
> >>
> >>>>>> the BDB store.)
> >>>>>>
> >>>>>> So, the bit I'm not sure about is this - you have no way
> >>>>>>
> >>>>>>
> >>>> of knowing
> >>>>
> >>>>
> >>>>>> in advance the point at which replay might be needed
> >>>>>>
> >> from (unless
> >>
> >>>>>> using client acks - which work with ordinary queues ?).
> >>>>>>
> >>>>>> Thus your recipient app starts up again (or recognises it
> >>>>>>
> >>>>>>
> >>>> has lost
> >>>>
> >>>>
> >>>>>> messages) and then wants to replay. So, the queue it is
> >>>>>>
> >>>>>>
> >>>> reading from
> >>>>
> >>>>
> >>>>>> is rebound to a replay exchange (with a marker
> >>>>>> ID) ? Does that mean that it is actually reading from a
> >>>>>>
> >> different
> >>
> >>>>>> queue than normal operation, or that the queue is bound to a
> >>>>>> different exchange type ?
> >>>>>>
> >>>>>> The logic behind this could be quite complex - in that
> >>>>>>
> >>>>>>
> >>>> the balance
> >>>>
> >>>>
> >>>>>> between normal operation and replay operation in subtle
> >>>>>>
> >> and would
> >>
> >>>>>> involve the receiving client understanding the AMQP behaviour at
> >>>>>> quite a low level ?
> >>>>>>
> >>>>>> Apologies if I've missed a simple flow for replay :-)
> >>>>>>
> >>>>>> Regards,
> >>>>>> Marnie
> >>>>>>
> >>>>>> On 1/22/07, Carl Trieloff <cc...@redhat.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>> This has been asked for before. I would interested to see if an
> >>>>>>> exchange could be used. I also think this might bring up
> >>>>>>>
> >>>>>>>
> >>>>>> the topic of
> >>>>>>
> >>>>>>
> >>>>>>> chained exchanges again.
> >>>>>>>
> >>>>>>> Colin, do you have any interest in creating a prototype
> >>>>>>>
> >>>>>>>
> >>>>>> ReplyExchange
> >>>>>>
> >>>>>>
> >>>>>>> to see how well it works?
> >>>>>>> Carl.
> >>>>>>>
> >>>>>>> Alan Conway wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Thu, 2007-01-18 at 15:52 +0000, Colin Crist wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>> A quick question...
> >>>>>>>>>
> >>>>>>>>> A common recovery scenario is requesting replay on a
> >>>>>>>>>
> >>>>>>>>>
> >>>> queue from
> >>>>
> >>>>
> >>>>>>>>> a
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>> given,
> >>>>>>>
> >>>>>>>
> >>>>>>>>> known message, either by its ID or by something in the
> >>>>>>>>>
> >>>>>>>>>
> >>>>>> header. Its
> >>>>>>
> >>>>>>
> >>>>>>> saves all
> >>>>>>>
> >>>>>>>
> >>>>>>>>> that mucking about with XA and makes recovery the
> >>>>>>>>>
> >>>>>>>>>
> >>>> normal way of
> >>>>
> >>>>
> >>>>>>>>> system startup rather then some exceptional case.
> >>>>>>>>>
> >>>>>>>>> This means when a message has reached all its recipients
> >>>>>>>>>
> >>>>>>>>>
> >>>>>> it should
> >>>>>>
> >>>>>>
> >>>>>>> never be
> >>>>>>>
> >>>>>>>
> >>>>>>>>> marked for removal from the durable message store.
> >>>>>>>>>
> >>>>>>>>> As a user, I would also like to control when messages
> >>>>>>>>>
> >>>>>>>>>
> >>>>>> get removed
> >>>>>>
> >>>>>>
> >>>>>>>>> as
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>> part of
> >>>>>>>
> >>>>>>>
> >>>>>>>>> my end of day or even weekly processs, probably
> >>>>>>>>>
> >>>>>>>>>
> >>>> based on header
> >>>>
> >>>>
> >>>>>>> property
> >>>>>>>
> >>>>>>>
> >>>>>>>>> selectors.
> >>>>>>>>>
> >>>>>>>>> I don't believe this is supported in AMQP in the wire
> >>>>>>>>>
> >>>>>>>>>
> >>>>>> protocol (nor
> >>>>>>
> >>>>>>
> >>>>>>> perhaps
> >>>>>>>
> >>>>>>>
> >>>>>>>>> should it be) - is it maybe something that could be done
> >>>>>>>>>
> >>>>>>>>>
> >>>>>> via some
> >>>>>>
> >>>>>>
> >>>>>>>>> standardised command messages to a "replay" exchange?
> >>>>>>>>>
> >>>>>>>>> Any thoughts on implementation difficulty?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> You could represent this in AMQP protocol using a custom
> >>>>>>>>
> >>>>>>>>
> >>>>>> exchange type.
> >>>>>>
> >>>>>>
> >>>>>>>> A ReplayExchange stores all messages sent to it (until
> >>>>>>>>
> >>>>>>>>
> >>>>>> cleaned up by
> >>>>>>
> >>>>>>
> >>>>>>>> some administrative function.)
> >>>>>>>>
> >>>>>>>> When binding a queue to a replay exchange you MAY specify an
> >>>>>>>> argument "replay-from=N". A queue so bound will be
> >>>>>>>>
> >>>>>>>>
> >>>>>> "pre-filled" with
> >>>>>>
> >>>>>>
> >>>>>>>> the range of message starting from the Nth message up
> >>>>>>>>
> >> to the most
> >>
> >>>>>>>> recent message sent to the exchange. Thereafter the
> >>>>>>>>
> >>>>>>>>
> >>>>>> exchange behaves
> >>>>>>
> >>>>>>
> >>>>>>>> like a fan-out exchange, each new message being added to
> >>>>>>>>
> >>>>>>>>
> >>>>>> all the bound queues.
> >>>>>>
> >>>>>>
> >>>>>>>> That gives you replay entirely within the standard
> >>>>>>>>
> >>>>>>>>
> >>>>>> protocol, using a
> >>>>>>
> >>>>>>
> >>>>>>>> standard extension point to add this as proprietary broker
> >>>>>>>> functionality.
> >>>>>>>>
> >>>>>>>> Note that to combine replay with other exchange types
> >>>>>>>>
> >>>>>>>>
> >>>> requires
> >>>>
> >>>>
> >>>>>>>> exchange chaining which has been discussed several times
> >>>>>>>>
> >>>>>>>>
> >>>>>> but is not
> >>>>>>
> >>>>>>
> >>>>>>>> yet an agreed part of the protocol - this may be
> >>>>>>>>
> >> another use case
> >>
> >>>>>>>> for that discussion
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>> Alan.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
> >
> >
> >
>
>
>

Re: Message history and replay

[Carl Trieloff <cc...@redhat.com>]

what is your apache cwiki user-id?

Carl.


Colin Crist wrote:
> Sure, I'll put some notes together.
>
> I don't seem to have permissions to add any pages - can you give me access
> or alternatively I can put the pages on my own confluence and get someone to
> move them later.
>
> Colin. 
>
>   
>> -----Original Message-----
>> From: Carl Trieloff [mailto:cctrieloff@redhat.com] 
>> Sent: 24 January 2007 18:07
>> To: qpid-dev@incubator.apache.org
>> Subject: Re: Message history and replay
>>
>>
>> Colin,
>>
>> Would you be willing to create a page on the wiki for this 
>> cwiki.apache.org/qpid and lay out all the use cases there so 
>> we don't loose them. then we can add designs etc there also 
>> as we explore this topic.
>>
>> Carl.
>>
>>
>> Colin Crist wrote:
>>     
>>> I believe the customary apache response here is:
>>>
>>> +1
>>>
>>> Colin. 
>>>
>>>   
>>>       
>>>> -----Original Message-----
>>>> From: John O'Hara [mailto:john.r.ohara@gmail.com]
>>>> Sent: 23 January 2007 16:56
>>>> To: qpid-dev@incubator.apache.org
>>>> Subject: Re: Message history and replay
>>>>
>>>> Two options:
>>>> 1) Make the 'rewind' accessible to the client on connect
>>>> 2) Make it an administrative function (reset next message to X)
>>>>
>>>> Both are valid.
>>>> Also retentions of several days should be valid subject to 
>>>>         
>> resource 
>>     
>>>> constraints.
>>>>
>>>>
>>>> Also Queues should have attributes related to this:
>>>> a) Archive Retention Period  *0*, x seconds, y messages
>>>> b) Rewindable Y / *N*
>>>> c) Client Can Rewind Y / *N*
>>>> d) Archive treatment - *Delete* or Copy-out
>>>> e) Archive interval - *30s* (queue garbage collect period)
>>>> f) Archive TTL-deleted messages - Y / *N*
>>>>
>>>> Defaults in *.
>>>>
>>>> Queues are smart resources.  We need to spec out what they can do 
>>>> (and what
>>>> *subset* of that we propose to standardise over on the AMQP WG).  
>>>> Here's a great place to put value added features.
>>>>
>>>> John
>>>>
>>>> On 23/01/07, Colin Crist <co...@hermesjms.com> wrote:
>>>>     
>>>>         
>>>>> I had thought a replay queue is where the messages would be
>>>>>       
>>>>>           
>>>> put I'm no
>>>>     
>>>>         
>>>>> longer convinced.
>>>>>
>>>>> In another mail John O'Hara uses the the reading email analogy. 
>>>>> Continuing this to its logical conclusion could mean that 
>>>>>           
>> replay is 
>>     
>>>>> actually an option when you open the queue for reading. It
>>>>>       
>>>>>           
>>>> avoids the
>>>>     
>>>>         
>>>>> client having to muck about with another queue.
>>>>>
>>>>> This is not the traditional view of a queue but is more
>>>>>       
>>>>>           
>>>> general and I
>>>>     
>>>>         
>>>>> like it.
>>>>>
>>>>> So, how about a queue contains all messages put on it 
>>>>>           
>> since it was 
>>     
>>>>> last "purged". When a queue is opened for reading, it can 
>>>>>           
>> be opened 
>>     
>>>>> with a selector and one part of that selector is whether 
>>>>>           
>> or not the 
>>     
>>>>> message has been commited by a consumer or not.
>>>>>
>>>>> A consumer could then always open the queue and request all
>>>>>       
>>>>>           
>>>> messages,
>>>>     
>>>>         
>>>>> committed or not, from a known point, e.g. a trade ID or
>>>>>       
>>>>>           
>>>> message ID. 
>>>>     
>>>>         
>>>>> The key thing here I suppose is that this type of consumer
>>>>>       
>>>>>           
>>>> is writing
>>>>     
>>>>         
>>>>> the message in some other form into a database and so
>>>>>       
>>>>>           
>>>> always knows the
>>>>     
>>>>         
>>>>> last thing written.
>>>>> A nice benefit is that if the database is really hosed you
>>>>>       
>>>>>           
>>>> could get
>>>>     
>>>>         
>>>>> replay from start of day. This would definitely help
>>>>>       
>>>>>           
>>>> implement DR for
>>>>     
>>>>         
>>>>> purely message based systems.
>>>>>
>>>>> So, can I specialise queues and layer this functionality on
>>>>>       
>>>>>           
>>>> top of the
>>>>     
>>>>         
>>>>> spec in the APID implementation with some kind of 
>>>>>           
>> provider specific 
>>     
>>>>> header property?
>>>>>
>>>>> Colin.
>>>>>
>>>>>       
>>>>>           
>>>>>> -----Original Message-----
>>>>>> From: Marnie McCormack [mailto:marnie.mccormack@googlemail.com]
>>>>>> Sent: 22 January 2007 19:54
>>>>>> To: qpid-dev@incubator.apache.org; cctrieloff@redhat.com
>>>>>> Subject: Re: Message history and replay
>>>>>>
>>>>>> I've been trying to imagine the scenarios in which a user
>>>>>>         
>>>>>>             
>>>> will make
>>>>     
>>>>         
>>>>>> use of a ReplayExchange.
>>>>>>
>>>>>> I thought that the most common usage would be the following:
>>>>>>
>>>>>> - the receiving application has lost some messages sent 
>>>>>>             
>> to it and 
>>     
>>>>>> needs them to be resent from a point in time or a message ID
>>>>>>
>>>>>> (The only other recovery scenario I thought likely was broker 
>>>>>> failure, which would be handled by restarted (with
>>>>>> persistence) and the undelivered messages would already 
>>>>>>             
>> remain in 
>>     
>>>>>> the BDB store.)
>>>>>>
>>>>>> So, the bit I'm not sure about is this - you have no way
>>>>>>         
>>>>>>             
>>>> of knowing
>>>>     
>>>>         
>>>>>> in advance the point at which replay might be needed 
>>>>>>             
>> from (unless 
>>     
>>>>>> using client acks - which work with ordinary queues ?).
>>>>>>
>>>>>> Thus your recipient app starts up again (or recognises it
>>>>>>         
>>>>>>             
>>>> has lost
>>>>     
>>>>         
>>>>>> messages) and then wants to replay. So, the queue it is
>>>>>>         
>>>>>>             
>>>> reading from
>>>>     
>>>>         
>>>>>> is rebound to a replay exchange (with a marker
>>>>>> ID) ? Does that mean that it is actually reading from a 
>>>>>>             
>> different 
>>     
>>>>>> queue than normal operation, or that the queue is bound to a 
>>>>>> different exchange type ?
>>>>>>
>>>>>> The logic behind this could be quite complex - in that
>>>>>>         
>>>>>>             
>>>> the balance
>>>>     
>>>>         
>>>>>> between normal operation and replay operation in subtle 
>>>>>>             
>> and would 
>>     
>>>>>> involve the receiving client understanding the AMQP behaviour at 
>>>>>> quite a low level ?
>>>>>>
>>>>>> Apologies if I've missed a simple flow for replay :-)
>>>>>>
>>>>>> Regards,
>>>>>> Marnie
>>>>>>
>>>>>> On 1/22/07, Carl Trieloff <cc...@redhat.com> wrote:
>>>>>>         
>>>>>>             
>>>>>>> This has been asked for before. I would interested to see if an 
>>>>>>> exchange could be used. I also think this might bring up
>>>>>>>           
>>>>>>>               
>>>>>> the topic of
>>>>>>         
>>>>>>             
>>>>>>> chained exchanges again.
>>>>>>>
>>>>>>> Colin, do you have any interest in creating a prototype
>>>>>>>           
>>>>>>>               
>>>>>> ReplyExchange
>>>>>>         
>>>>>>             
>>>>>>> to see how well it works?
>>>>>>> Carl.
>>>>>>>
>>>>>>> Alan Conway wrote:
>>>>>>>           
>>>>>>>               
>>>>>>>> On Thu, 2007-01-18 at 15:52 +0000, Colin Crist wrote:
>>>>>>>>
>>>>>>>>             
>>>>>>>>                 
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> A quick question...
>>>>>>>>>
>>>>>>>>> A common recovery scenario is requesting replay on a
>>>>>>>>>               
>>>>>>>>>                   
>>>> queue from
>>>>     
>>>>         
>>>>>>>>> a
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>> given,
>>>>>>>           
>>>>>>>               
>>>>>>>>> known message, either by its ID or by something in the
>>>>>>>>>               
>>>>>>>>>                   
>>>>>> header. Its
>>>>>>         
>>>>>>             
>>>>>>> saves all
>>>>>>>           
>>>>>>>               
>>>>>>>>> that mucking about with XA and makes recovery the
>>>>>>>>>               
>>>>>>>>>                   
>>>> normal way of
>>>>     
>>>>         
>>>>>>>>> system startup rather then some exceptional case.
>>>>>>>>>
>>>>>>>>> This means when a message has reached all its recipients
>>>>>>>>>               
>>>>>>>>>                   
>>>>>> it should
>>>>>>         
>>>>>>             
>>>>>>> never be
>>>>>>>           
>>>>>>>               
>>>>>>>>> marked for removal from the durable message store.
>>>>>>>>>
>>>>>>>>> As a user, I would also like to control when messages
>>>>>>>>>               
>>>>>>>>>                   
>>>>>> get removed
>>>>>>         
>>>>>>             
>>>>>>>>> as
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>> part of
>>>>>>>           
>>>>>>>               
>>>>>>>>> my end of day or even weekly processs, probably
>>>>>>>>>               
>>>>>>>>>                   
>>>> based on header
>>>>     
>>>>         
>>>>>>> property
>>>>>>>           
>>>>>>>               
>>>>>>>>> selectors.
>>>>>>>>>
>>>>>>>>> I don't believe this is supported in AMQP in the wire
>>>>>>>>>               
>>>>>>>>>                   
>>>>>> protocol (nor
>>>>>>         
>>>>>>             
>>>>>>> perhaps
>>>>>>>           
>>>>>>>               
>>>>>>>>> should it be) - is it maybe something that could be done
>>>>>>>>>               
>>>>>>>>>                   
>>>>>> via some
>>>>>>         
>>>>>>             
>>>>>>>>> standardised command messages to a "replay" exchange?
>>>>>>>>>
>>>>>>>>> Any thoughts on implementation difficulty?
>>>>>>>>>
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>> You could represent this in AMQP protocol using a custom
>>>>>>>>             
>>>>>>>>                 
>>>>>> exchange type.
>>>>>>         
>>>>>>             
>>>>>>>> A ReplayExchange stores all messages sent to it (until
>>>>>>>>             
>>>>>>>>                 
>>>>>> cleaned up by
>>>>>>         
>>>>>>             
>>>>>>>> some administrative function.)
>>>>>>>>
>>>>>>>> When binding a queue to a replay exchange you MAY specify an 
>>>>>>>> argument "replay-from=N". A queue so bound will be
>>>>>>>>             
>>>>>>>>                 
>>>>>> "pre-filled" with
>>>>>>         
>>>>>>             
>>>>>>>> the range of message starting from the Nth message up 
>>>>>>>>                 
>> to the most 
>>     
>>>>>>>> recent message sent to the exchange. Thereafter the
>>>>>>>>             
>>>>>>>>                 
>>>>>> exchange behaves
>>>>>>         
>>>>>>             
>>>>>>>> like a fan-out exchange, each new message being added to
>>>>>>>>             
>>>>>>>>                 
>>>>>> all the bound queues.
>>>>>>         
>>>>>>             
>>>>>>>> That gives you replay entirely within the standard
>>>>>>>>             
>>>>>>>>                 
>>>>>> protocol, using a
>>>>>>         
>>>>>>             
>>>>>>>> standard extension point to add this as proprietary broker 
>>>>>>>> functionality.
>>>>>>>>
>>>>>>>> Note that to combine replay with other exchange types
>>>>>>>>             
>>>>>>>>                 
>>>> requires
>>>>     
>>>>         
>>>>>>>> exchange chaining which has been discussed several times
>>>>>>>>             
>>>>>>>>                 
>>>>>> but is not
>>>>>>         
>>>>>>             
>>>>>>>> yet an agreed part of the protocol - this may be 
>>>>>>>>                 
>> another use case 
>>     
>>>>>>>> for that discussion
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Alan.
>>>>>>>>
>>>>>>>>
>>>>>>>>             
>>>>>>>>                 
>>>>>>>           
>>>>>>>               
>>>>>       
>>>>>           
>>>   
>>>       
>>     
>
>
>   

RE: Message history and replay

[Colin Crist <co...@hermesjms.com>]

Sure, I'll put some notes together.

I don't seem to have permissions to add any pages - can you give me access
or alternatively I can put the pages on my own confluence and get someone to
move them later.

Colin. 

> -----Original Message-----
> From: Carl Trieloff [mailto:cctrieloff@redhat.com] 
> Sent: 24 January 2007 18:07
> To: qpid-dev@incubator.apache.org
> Subject: Re: Message history and replay
> 
> 
> Colin,
> 
> Would you be willing to create a page on the wiki for this 
> cwiki.apache.org/qpid and lay out all the use cases there so 
> we don't loose them. then we can add designs etc there also 
> as we explore this topic.
> 
> Carl.
> 
> 
> Colin Crist wrote:
> > I believe the customary apache response here is:
> >
> > +1
> >
> > Colin. 
> >
> >   
> >> -----Original Message-----
> >> From: John O'Hara [mailto:john.r.ohara@gmail.com]
> >> Sent: 23 January 2007 16:56
> >> To: qpid-dev@incubator.apache.org
> >> Subject: Re: Message history and replay
> >>
> >> Two options:
> >> 1) Make the 'rewind' accessible to the client on connect
> >> 2) Make it an administrative function (reset next message to X)
> >>
> >> Both are valid.
> >> Also retentions of several days should be valid subject to 
> resource 
> >> constraints.
> >>
> >>
> >> Also Queues should have attributes related to this:
> >> a) Archive Retention Period  *0*, x seconds, y messages
> >> b) Rewindable Y / *N*
> >> c) Client Can Rewind Y / *N*
> >> d) Archive treatment - *Delete* or Copy-out
> >> e) Archive interval - *30s* (queue garbage collect period)
> >> f) Archive TTL-deleted messages - Y / *N*
> >>
> >> Defaults in *.
> >>
> >> Queues are smart resources.  We need to spec out what they can do 
> >> (and what
> >> *subset* of that we propose to standardise over on the AMQP WG).  
> >> Here's a great place to put value added features.
> >>
> >> John
> >>
> >> On 23/01/07, Colin Crist <co...@hermesjms.com> wrote:
> >>     
> >>> I had thought a replay queue is where the messages would be
> >>>       
> >> put I'm no
> >>     
> >>> longer convinced.
> >>>
> >>> In another mail John O'Hara uses the the reading email analogy. 
> >>> Continuing this to its logical conclusion could mean that 
> replay is 
> >>> actually an option when you open the queue for reading. It
> >>>       
> >> avoids the
> >>     
> >>> client having to muck about with another queue.
> >>>
> >>> This is not the traditional view of a queue but is more
> >>>       
> >> general and I
> >>     
> >>> like it.
> >>>
> >>> So, how about a queue contains all messages put on it 
> since it was 
> >>> last "purged". When a queue is opened for reading, it can 
> be opened 
> >>> with a selector and one part of that selector is whether 
> or not the 
> >>> message has been commited by a consumer or not.
> >>>
> >>> A consumer could then always open the queue and request all
> >>>       
> >> messages,
> >>     
> >>> committed or not, from a known point, e.g. a trade ID or
> >>>       
> >> message ID. 
> >>     
> >>> The key thing here I suppose is that this type of consumer
> >>>       
> >> is writing
> >>     
> >>> the message in some other form into a database and so
> >>>       
> >> always knows the
> >>     
> >>> last thing written.
> >>> A nice benefit is that if the database is really hosed you
> >>>       
> >> could get
> >>     
> >>> replay from start of day. This would definitely help
> >>>       
> >> implement DR for
> >>     
> >>> purely message based systems.
> >>>
> >>> So, can I specialise queues and layer this functionality on
> >>>       
> >> top of the
> >>     
> >>> spec in the APID implementation with some kind of 
> provider specific 
> >>> header property?
> >>>
> >>> Colin.
> >>>
> >>>       
> >>>> -----Original Message-----
> >>>> From: Marnie McCormack [mailto:marnie.mccormack@googlemail.com]
> >>>> Sent: 22 January 2007 19:54
> >>>> To: qpid-dev@incubator.apache.org; cctrieloff@redhat.com
> >>>> Subject: Re: Message history and replay
> >>>>
> >>>> I've been trying to imagine the scenarios in which a user
> >>>>         
> >> will make
> >>     
> >>>> use of a ReplayExchange.
> >>>>
> >>>> I thought that the most common usage would be the following:
> >>>>
> >>>> - the receiving application has lost some messages sent 
> to it and 
> >>>> needs them to be resent from a point in time or a message ID
> >>>>
> >>>> (The only other recovery scenario I thought likely was broker 
> >>>> failure, which would be handled by restarted (with
> >>>> persistence) and the undelivered messages would already 
> remain in 
> >>>> the BDB store.)
> >>>>
> >>>> So, the bit I'm not sure about is this - you have no way
> >>>>         
> >> of knowing
> >>     
> >>>> in advance the point at which replay might be needed 
> from (unless 
> >>>> using client acks - which work with ordinary queues ?).
> >>>>
> >>>> Thus your recipient app starts up again (or recognises it
> >>>>         
> >> has lost
> >>     
> >>>> messages) and then wants to replay. So, the queue it is
> >>>>         
> >> reading from
> >>     
> >>>> is rebound to a replay exchange (with a marker
> >>>> ID) ? Does that mean that it is actually reading from a 
> different 
> >>>> queue than normal operation, or that the queue is bound to a 
> >>>> different exchange type ?
> >>>>
> >>>> The logic behind this could be quite complex - in that
> >>>>         
> >> the balance
> >>     
> >>>> between normal operation and replay operation in subtle 
> and would 
> >>>> involve the receiving client understanding the AMQP behaviour at 
> >>>> quite a low level ?
> >>>>
> >>>> Apologies if I've missed a simple flow for replay :-)
> >>>>
> >>>> Regards,
> >>>> Marnie
> >>>>
> >>>> On 1/22/07, Carl Trieloff <cc...@redhat.com> wrote:
> >>>>         
> >>>>> This has been asked for before. I would interested to see if an 
> >>>>> exchange could be used. I also think this might bring up
> >>>>>           
> >>>> the topic of
> >>>>         
> >>>>> chained exchanges again.
> >>>>>
> >>>>> Colin, do you have any interest in creating a prototype
> >>>>>           
> >>>> ReplyExchange
> >>>>         
> >>>>> to see how well it works?
> >>>>> Carl.
> >>>>>
> >>>>> Alan Conway wrote:
> >>>>>           
> >>>>>> On Thu, 2007-01-18 at 15:52 +0000, Colin Crist wrote:
> >>>>>>
> >>>>>>             
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> A quick question...
> >>>>>>>
> >>>>>>> A common recovery scenario is requesting replay on a
> >>>>>>>               
> >> queue from
> >>     
> >>>>>>> a
> >>>>>>>               
> >>>>> given,
> >>>>>           
> >>>>>>> known message, either by its ID or by something in the
> >>>>>>>               
> >>>> header. Its
> >>>>         
> >>>>> saves all
> >>>>>           
> >>>>>>> that mucking about with XA and makes recovery the
> >>>>>>>               
> >> normal way of
> >>     
> >>>>>>> system startup rather then some exceptional case.
> >>>>>>>
> >>>>>>> This means when a message has reached all its recipients
> >>>>>>>               
> >>>> it should
> >>>>         
> >>>>> never be
> >>>>>           
> >>>>>>> marked for removal from the durable message store.
> >>>>>>>
> >>>>>>> As a user, I would also like to control when messages
> >>>>>>>               
> >>>> get removed
> >>>>         
> >>>>>>> as
> >>>>>>>               
> >>>>> part of
> >>>>>           
> >>>>>>> my end of day or even weekly processs, probably
> >>>>>>>               
> >> based on header
> >>     
> >>>>> property
> >>>>>           
> >>>>>>> selectors.
> >>>>>>>
> >>>>>>> I don't believe this is supported in AMQP in the wire
> >>>>>>>               
> >>>> protocol (nor
> >>>>         
> >>>>> perhaps
> >>>>>           
> >>>>>>> should it be) - is it maybe something that could be done
> >>>>>>>               
> >>>> via some
> >>>>         
> >>>>>>> standardised command messages to a "replay" exchange?
> >>>>>>>
> >>>>>>> Any thoughts on implementation difficulty?
> >>>>>>>
> >>>>>>>               
> >>>>>> You could represent this in AMQP protocol using a custom
> >>>>>>             
> >>>> exchange type.
> >>>>         
> >>>>>> A ReplayExchange stores all messages sent to it (until
> >>>>>>             
> >>>> cleaned up by
> >>>>         
> >>>>>> some administrative function.)
> >>>>>>
> >>>>>> When binding a queue to a replay exchange you MAY specify an 
> >>>>>> argument "replay-from=N". A queue so bound will be
> >>>>>>             
> >>>> "pre-filled" with
> >>>>         
> >>>>>> the range of message starting from the Nth message up 
> to the most 
> >>>>>> recent message sent to the exchange. Thereafter the
> >>>>>>             
> >>>> exchange behaves
> >>>>         
> >>>>>> like a fan-out exchange, each new message being added to
> >>>>>>             
> >>>> all the bound queues.
> >>>>         
> >>>>>> That gives you replay entirely within the standard
> >>>>>>             
> >>>> protocol, using a
> >>>>         
> >>>>>> standard extension point to add this as proprietary broker 
> >>>>>> functionality.
> >>>>>>
> >>>>>> Note that to combine replay with other exchange types
> >>>>>>             
> >> requires
> >>     
> >>>>>> exchange chaining which has been discussed several times
> >>>>>>             
> >>>> but is not
> >>>>         
> >>>>>> yet an agreed part of the protocol - this may be 
> another use case 
> >>>>>> for that discussion
> >>>>>>
> >>>>>> Cheers,
> >>>>>> Alan.
> >>>>>>
> >>>>>>
> >>>>>>             
> >>>>>
> >>>>>           
> >>>
> >>>       
> >
> >
> >   
> 
> 

Re: Message history and replay

[Carl Trieloff <cc...@redhat.com>]

Colin,

Would you be willing to create a page on the wiki for this 
cwiki.apache.org/qpid and lay out all the
use cases there so we don't loose them. then we can add designs etc 
there also as we explore this topic.

Carl.


Colin Crist wrote:
> I believe the customary apache response here is:
>
> +1
>
> Colin. 
>
>   
>> -----Original Message-----
>> From: John O'Hara [mailto:john.r.ohara@gmail.com] 
>> Sent: 23 January 2007 16:56
>> To: qpid-dev@incubator.apache.org
>> Subject: Re: Message history and replay
>>
>> Two options:
>> 1) Make the 'rewind' accessible to the client on connect
>> 2) Make it an administrative function (reset next message to X)
>>
>> Both are valid.
>> Also retentions of several days should be valid subject to 
>> resource constraints.
>>
>>
>> Also Queues should have attributes related to this:
>> a) Archive Retention Period  *0*, x seconds, y messages
>> b) Rewindable Y / *N*
>> c) Client Can Rewind Y / *N*
>> d) Archive treatment - *Delete* or Copy-out
>> e) Archive interval - *30s* (queue garbage collect period)
>> f) Archive TTL-deleted messages - Y / *N*
>>
>> Defaults in *.
>>
>> Queues are smart resources.  We need to spec out what they 
>> can do (and what
>> *subset* of that we propose to standardise over on the AMQP 
>> WG).  Here's a great place to put value added features.
>>
>> John
>>
>> On 23/01/07, Colin Crist <co...@hermesjms.com> wrote:
>>     
>>> I had thought a replay queue is where the messages would be 
>>>       
>> put I'm no 
>>     
>>> longer convinced.
>>>
>>> In another mail John O'Hara uses the the reading email analogy. 
>>> Continuing this to its logical conclusion could mean that replay is 
>>> actually an option when you open the queue for reading. It 
>>>       
>> avoids the 
>>     
>>> client having to muck about with another queue.
>>>
>>> This is not the traditional view of a queue but is more 
>>>       
>> general and I 
>>     
>>> like it.
>>>
>>> So, how about a queue contains all messages put on it since it was 
>>> last "purged". When a queue is opened for reading, it can be opened 
>>> with a selector and one part of that selector is whether or not the 
>>> message has been commited by a consumer or not.
>>>
>>> A consumer could then always open the queue and request all 
>>>       
>> messages, 
>>     
>>> committed or not, from a known point, e.g. a trade ID or 
>>>       
>> message ID. 
>>     
>>> The key thing here I suppose is that this type of consumer 
>>>       
>> is writing 
>>     
>>> the message in some other form into a database and so 
>>>       
>> always knows the 
>>     
>>> last thing written.
>>> A nice benefit is that if the database is really hosed you 
>>>       
>> could get 
>>     
>>> replay from start of day. This would definitely help 
>>>       
>> implement DR for 
>>     
>>> purely message based systems.
>>>
>>> So, can I specialise queues and layer this functionality on 
>>>       
>> top of the 
>>     
>>> spec in the APID implementation with some kind of provider specific 
>>> header property?
>>>
>>> Colin.
>>>
>>>       
>>>> -----Original Message-----
>>>> From: Marnie McCormack [mailto:marnie.mccormack@googlemail.com]
>>>> Sent: 22 January 2007 19:54
>>>> To: qpid-dev@incubator.apache.org; cctrieloff@redhat.com
>>>> Subject: Re: Message history and replay
>>>>
>>>> I've been trying to imagine the scenarios in which a user 
>>>>         
>> will make 
>>     
>>>> use of a ReplayExchange.
>>>>
>>>> I thought that the most common usage would be the following:
>>>>
>>>> - the receiving application has lost some messages sent to it and 
>>>> needs them to be resent from a point in time or a message ID
>>>>
>>>> (The only other recovery scenario I thought likely was broker 
>>>> failure, which would be handled by restarted (with
>>>> persistence) and the undelivered messages would already remain in 
>>>> the BDB store.)
>>>>
>>>> So, the bit I'm not sure about is this - you have no way 
>>>>         
>> of knowing 
>>     
>>>> in advance the point at which replay might be needed from (unless 
>>>> using client acks - which work with ordinary queues ?).
>>>>
>>>> Thus your recipient app starts up again (or recognises it 
>>>>         
>> has lost 
>>     
>>>> messages) and then wants to replay. So, the queue it is 
>>>>         
>> reading from 
>>     
>>>> is rebound to a replay exchange (with a marker
>>>> ID) ? Does that mean that it is actually reading from a different 
>>>> queue than normal operation, or that the queue is bound to a 
>>>> different exchange type ?
>>>>
>>>> The logic behind this could be quite complex - in that 
>>>>         
>> the balance 
>>     
>>>> between normal operation and replay operation in subtle and would 
>>>> involve the receiving client understanding the AMQP behaviour at 
>>>> quite a low level ?
>>>>
>>>> Apologies if I've missed a simple flow for replay :-)
>>>>
>>>> Regards,
>>>> Marnie
>>>>
>>>> On 1/22/07, Carl Trieloff <cc...@redhat.com> wrote:
>>>>         
>>>>> This has been asked for before. I would interested to see if an 
>>>>> exchange could be used. I also think this might bring up
>>>>>           
>>>> the topic of
>>>>         
>>>>> chained exchanges again.
>>>>>
>>>>> Colin, do you have any interest in creating a prototype
>>>>>           
>>>> ReplyExchange
>>>>         
>>>>> to see how well it works?
>>>>> Carl.
>>>>>
>>>>> Alan Conway wrote:
>>>>>           
>>>>>> On Thu, 2007-01-18 at 15:52 +0000, Colin Crist wrote:
>>>>>>
>>>>>>             
>>>>>>> Hi,
>>>>>>>
>>>>>>> A quick question...
>>>>>>>
>>>>>>> A common recovery scenario is requesting replay on a 
>>>>>>>               
>> queue from 
>>     
>>>>>>> a
>>>>>>>               
>>>>> given,
>>>>>           
>>>>>>> known message, either by its ID or by something in the
>>>>>>>               
>>>> header. Its
>>>>         
>>>>> saves all
>>>>>           
>>>>>>> that mucking about with XA and makes recovery the 
>>>>>>>               
>> normal way of 
>>     
>>>>>>> system startup rather then some exceptional case.
>>>>>>>
>>>>>>> This means when a message has reached all its recipients
>>>>>>>               
>>>> it should
>>>>         
>>>>> never be
>>>>>           
>>>>>>> marked for removal from the durable message store.
>>>>>>>
>>>>>>> As a user, I would also like to control when messages
>>>>>>>               
>>>> get removed
>>>>         
>>>>>>> as
>>>>>>>               
>>>>> part of
>>>>>           
>>>>>>> my end of day or even weekly processs, probably 
>>>>>>>               
>> based on header
>>     
>>>>> property
>>>>>           
>>>>>>> selectors.
>>>>>>>
>>>>>>> I don't believe this is supported in AMQP in the wire
>>>>>>>               
>>>> protocol (nor
>>>>         
>>>>> perhaps
>>>>>           
>>>>>>> should it be) - is it maybe something that could be done
>>>>>>>               
>>>> via some
>>>>         
>>>>>>> standardised command messages to a "replay" exchange?
>>>>>>>
>>>>>>> Any thoughts on implementation difficulty?
>>>>>>>
>>>>>>>               
>>>>>> You could represent this in AMQP protocol using a custom
>>>>>>             
>>>> exchange type.
>>>>         
>>>>>> A ReplayExchange stores all messages sent to it (until
>>>>>>             
>>>> cleaned up by
>>>>         
>>>>>> some administrative function.)
>>>>>>
>>>>>> When binding a queue to a replay exchange you MAY specify an 
>>>>>> argument "replay-from=N". A queue so bound will be
>>>>>>             
>>>> "pre-filled" with
>>>>         
>>>>>> the range of message starting from the Nth message up to the 
>>>>>> most recent message sent to the exchange. Thereafter the
>>>>>>             
>>>> exchange behaves
>>>>         
>>>>>> like a fan-out exchange, each new message being added to
>>>>>>             
>>>> all the bound queues.
>>>>         
>>>>>> That gives you replay entirely within the standard
>>>>>>             
>>>> protocol, using a
>>>>         
>>>>>> standard extension point to add this as proprietary broker 
>>>>>> functionality.
>>>>>>
>>>>>> Note that to combine replay with other exchange types 
>>>>>>             
>> requires 
>>     
>>>>>> exchange chaining which has been discussed several times
>>>>>>             
>>>> but is not
>>>>         
>>>>>> yet an agreed part of the protocol - this may be another use 
>>>>>> case for that discussion
>>>>>>
>>>>>> Cheers,
>>>>>> Alan.
>>>>>>
>>>>>>
>>>>>>             
>>>>>
>>>>>           
>>>
>>>       
>
>
>   

RE: Message history and replay

[Colin Crist <co...@hermesjms.com>]

I believe the customary apache response here is:

+1

Colin. 

> -----Original Message-----
> From: John O'Hara [mailto:john.r.ohara@gmail.com] 
> Sent: 23 January 2007 16:56
> To: qpid-dev@incubator.apache.org
> Subject: Re: Message history and replay
> 
> Two options:
> 1) Make the 'rewind' accessible to the client on connect
> 2) Make it an administrative function (reset next message to X)
> 
> Both are valid.
> Also retentions of several days should be valid subject to 
> resource constraints.
> 
> 
> Also Queues should have attributes related to this:
> a) Archive Retention Period  *0*, x seconds, y messages
> b) Rewindable Y / *N*
> c) Client Can Rewind Y / *N*
> d) Archive treatment - *Delete* or Copy-out
> e) Archive interval - *30s* (queue garbage collect period)
> f) Archive TTL-deleted messages - Y / *N*
> 
> Defaults in *.
> 
> Queues are smart resources.  We need to spec out what they 
> can do (and what
> *subset* of that we propose to standardise over on the AMQP 
> WG).  Here's a great place to put value added features.
> 
> John
> 
> On 23/01/07, Colin Crist <co...@hermesjms.com> wrote:
> >
> >
> > I had thought a replay queue is where the messages would be 
> put I'm no 
> > longer convinced.
> >
> > In another mail John O'Hara uses the the reading email analogy. 
> > Continuing this to its logical conclusion could mean that replay is 
> > actually an option when you open the queue for reading. It 
> avoids the 
> > client having to muck about with another queue.
> >
> > This is not the traditional view of a queue but is more 
> general and I 
> > like it.
> >
> > So, how about a queue contains all messages put on it since it was 
> > last "purged". When a queue is opened for reading, it can be opened 
> > with a selector and one part of that selector is whether or not the 
> > message has been commited by a consumer or not.
> >
> > A consumer could then always open the queue and request all 
> messages, 
> > committed or not, from a known point, e.g. a trade ID or 
> message ID. 
> > The key thing here I suppose is that this type of consumer 
> is writing 
> > the message in some other form into a database and so 
> always knows the 
> > last thing written.
> > A nice benefit is that if the database is really hosed you 
> could get 
> > replay from start of day. This would definitely help 
> implement DR for 
> > purely message based systems.
> >
> > So, can I specialise queues and layer this functionality on 
> top of the 
> > spec in the APID implementation with some kind of provider specific 
> > header property?
> >
> > Colin.
> >
> > > -----Original Message-----
> > > From: Marnie McCormack [mailto:marnie.mccormack@googlemail.com]
> > > Sent: 22 January 2007 19:54
> > > To: qpid-dev@incubator.apache.org; cctrieloff@redhat.com
> > > Subject: Re: Message history and replay
> > >
> > > I've been trying to imagine the scenarios in which a user 
> will make 
> > > use of a ReplayExchange.
> > >
> > > I thought that the most common usage would be the following:
> > >
> > > - the receiving application has lost some messages sent to it and 
> > > needs them to be resent from a point in time or a message ID
> > >
> > > (The only other recovery scenario I thought likely was broker 
> > > failure, which would be handled by restarted (with
> > > persistence) and the undelivered messages would already remain in 
> > > the BDB store.)
> > >
> > > So, the bit I'm not sure about is this - you have no way 
> of knowing 
> > > in advance the point at which replay might be needed from (unless 
> > > using client acks - which work with ordinary queues ?).
> > >
> > > Thus your recipient app starts up again (or recognises it 
> has lost 
> > > messages) and then wants to replay. So, the queue it is 
> reading from 
> > > is rebound to a replay exchange (with a marker
> > > ID) ? Does that mean that it is actually reading from a different 
> > > queue than normal operation, or that the queue is bound to a 
> > > different exchange type ?
> > >
> > > The logic behind this could be quite complex - in that 
> the balance 
> > > between normal operation and replay operation in subtle and would 
> > > involve the receiving client understanding the AMQP behaviour at 
> > > quite a low level ?
> > >
> > > Apologies if I've missed a simple flow for replay :-)
> > >
> > > Regards,
> > > Marnie
> > >
> > > On 1/22/07, Carl Trieloff <cc...@redhat.com> wrote:
> > > >
> > > >
> > > > This has been asked for before. I would interested to see if an 
> > > > exchange could be used. I also think this might bring up
> > > the topic of
> > > > chained exchanges again.
> > > >
> > > > Colin, do you have any interest in creating a prototype
> > > ReplyExchange
> > > > to see how well it works?
> > > > Carl.
> > > >
> > > > Alan Conway wrote:
> > > > > On Thu, 2007-01-18 at 15:52 +0000, Colin Crist wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> A quick question...
> > > > >>
> > > > >> A common recovery scenario is requesting replay on a 
> queue from 
> > > > >> a
> > > > given,
> > > > >> known message, either by its ID or by something in the
> > > header. Its
> > > > saves all
> > > > >> that mucking about with XA and makes recovery the 
> normal way of 
> > > > >> system startup rather then some exceptional case.
> > > > >>
> > > > >> This means when a message has reached all its recipients
> > > it should
> > > > never be
> > > > >> marked for removal from the durable message store.
> > > > >>
> > > > >> As a user, I would also like to control when messages
> > > get removed
> > > > >> as
> > > > part of
> > > > >> my end of day or even weekly processs, probably 
> based on header
> > > > property
> > > > >> selectors.
> > > > >>
> > > > >> I don't believe this is supported in AMQP in the wire
> > > protocol (nor
> > > > perhaps
> > > > >> should it be) - is it maybe something that could be done
> > > via some
> > > > >> standardised command messages to a "replay" exchange?
> > > > >>
> > > > >> Any thoughts on implementation difficulty?
> > > > >>
> > > > >
> > > > > You could represent this in AMQP protocol using a custom
> > > exchange type.
> > > > >
> > > > > A ReplayExchange stores all messages sent to it (until
> > > cleaned up by
> > > > > some administrative function.)
> > > > >
> > > > > When binding a queue to a replay exchange you MAY specify an 
> > > > > argument "replay-from=N". A queue so bound will be
> > > "pre-filled" with
> > > > > the range of message starting from the Nth message up to the 
> > > > > most recent message sent to the exchange. Thereafter the
> > > exchange behaves
> > > > > like a fan-out exchange, each new message being added to
> > > all the bound queues.
> > > > >
> > > > > That gives you replay entirely within the standard
> > > protocol, using a
> > > > > standard extension point to add this as proprietary broker 
> > > > > functionality.
> > > > >
> > > > > Note that to combine replay with other exchange types 
> requires 
> > > > > exchange chaining which has been discussed several times
> > > but is not
> > > > > yet an agreed part of the protocol - this may be another use 
> > > > > case for that discussion
> > > > >
> > > > > Cheers,
> > > > > Alan.
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
> 

Re: Message history and replay

[John O'Hara <jo...@gmail.com>]

Two options:
1) Make the 'rewind' accessible to the client on connect
2) Make it an administrative function (reset next message to X)

Both are valid.
Also retentions of several days should be valid subject to resource
constraints.


Also Queues should have attributes related to this:
a) Archive Retention Period  *0*, x seconds, y messages
b) Rewindable Y / *N*
c) Client Can Rewind Y / *N*
d) Archive treatment - *Delete* or Copy-out
e) Archive interval - *30s* (queue garbage collect period)
f) Archive TTL-deleted messages - Y / *N*

Defaults in *.

Queues are smart resources.  We need to spec out what they can do (and what
*subset* of that we propose to standardise over on the AMQP WG).  Here's a
great place to put value added features.

John

On 23/01/07, Colin Crist <co...@hermesjms.com> wrote:
>
>
> I had thought a replay queue is where the messages would be put I'm no
> longer convinced.
>
> In another mail John O'Hara uses the the reading email analogy. Continuing
> this to its logical conclusion could mean that replay is actually an
> option
> when you open the queue for reading. It avoids the client having to muck
> about with another queue.
>
> This is not the traditional view of a queue but is more general and I like
> it.
>
> So, how about a queue contains all messages put on it since it was last
> "purged". When a queue is opened for reading, it can be opened with a
> selector and one part of that selector is whether or not the message has
> been commited by a consumer or not.
>
> A consumer could then always open the queue and request all messages,
> committed or not, from a known point, e.g. a trade ID or message ID. The
> key
> thing here I suppose is that this type of consumer is writing the message
> in
> some other form into a database and so always knows the last thing
> written.
> A nice benefit is that if the database is really hosed you could get
> replay
> from start of day. This would definitely help implement DR for purely
> message based systems.
>
> So, can I specialise queues and layer this functionality on top of the
> spec
> in the APID implementation with some kind of provider specific header
> property?
>
> Colin.
>
> > -----Original Message-----
> > From: Marnie McCormack [mailto:marnie.mccormack@googlemail.com]
> > Sent: 22 January 2007 19:54
> > To: qpid-dev@incubator.apache.org; cctrieloff@redhat.com
> > Subject: Re: Message history and replay
> >
> > I've been trying to imagine the scenarios in which a user
> > will make use of a ReplayExchange.
> >
> > I thought that the most common usage would be the following:
> >
> > - the receiving application has lost some messages sent to it
> > and needs them to be resent from a point in time or a message ID
> >
> > (The only other recovery scenario I thought likely was broker
> > failure, which would be handled by restarted (with
> > persistence) and the undelivered messages would already
> > remain in the BDB store.)
> >
> > So, the bit I'm not sure about is this - you have no way of
> > knowing in advance the point at which replay might be needed
> > from (unless using client acks - which work with ordinary queues ?).
> >
> > Thus your recipient app starts up again (or recognises it has
> > lost messages) and then wants to replay. So, the queue it is
> > reading from is rebound to a replay exchange (with a marker
> > ID) ? Does that mean that it is actually reading from a
> > different queue than normal operation, or that the queue is
> > bound to a different exchange type ?
> >
> > The logic behind this could be quite complex - in that the
> > balance between normal operation and replay operation in
> > subtle and would involve the receiving client understanding
> > the AMQP behaviour at quite a low level ?
> >
> > Apologies if I've missed a simple flow for replay :-)
> >
> > Regards,
> > Marnie
> >
> > On 1/22/07, Carl Trieloff <cc...@redhat.com> wrote:
> > >
> > >
> > > This has been asked for before. I would interested to see if an
> > > exchange could be used. I also think this might bring up
> > the topic of
> > > chained exchanges again.
> > >
> > > Colin, do you have any interest in creating a prototype
> > ReplyExchange
> > > to see how well it works?
> > > Carl.
> > >
> > > Alan Conway wrote:
> > > > On Thu, 2007-01-18 at 15:52 +0000, Colin Crist wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> A quick question...
> > > >>
> > > >> A common recovery scenario is requesting replay on a queue from a
> > > given,
> > > >> known message, either by its ID or by something in the
> > header. Its
> > > saves all
> > > >> that mucking about with XA and makes recovery the normal way of
> > > >> system startup rather then some exceptional case.
> > > >>
> > > >> This means when a message has reached all its recipients
> > it should
> > > never be
> > > >> marked for removal from the durable message store.
> > > >>
> > > >> As a user, I would also like to control when messages
> > get removed
> > > >> as
> > > part of
> > > >> my end of day or even weekly processs, probably based on header
> > > property
> > > >> selectors.
> > > >>
> > > >> I don't believe this is supported in AMQP in the wire
> > protocol (nor
> > > perhaps
> > > >> should it be) - is it maybe something that could be done
> > via some
> > > >> standardised command messages to a "replay" exchange?
> > > >>
> > > >> Any thoughts on implementation difficulty?
> > > >>
> > > >
> > > > You could represent this in AMQP protocol using a custom
> > exchange type.
> > > >
> > > > A ReplayExchange stores all messages sent to it (until
> > cleaned up by
> > > > some administrative function.)
> > > >
> > > > When binding a queue to a replay exchange you MAY specify an
> > > > argument "replay-from=N". A queue so bound will be
> > "pre-filled" with
> > > > the range of message starting from the Nth message up to the most
> > > > recent message sent to the exchange. Thereafter the
> > exchange behaves
> > > > like a fan-out exchange, each new message being added to
> > all the bound queues.
> > > >
> > > > That gives you replay entirely within the standard
> > protocol, using a
> > > > standard extension point to add this as proprietary broker
> > > > functionality.
> > > >
> > > > Note that to combine replay with other exchange types requires
> > > > exchange chaining which has been discussed several times
> > but is not
> > > > yet an agreed part of the protocol - this may be another use case
> > > > for that discussion
> > > >
> > > > Cheers,
> > > > Alan.
> > > >
> > > >
> > >
> > >
> > >
> >
>
>
>

RE: Message history and replay

[Colin Crist <co...@hermesjms.com>]

I had thought a replay queue is where the messages would be put I'm no
longer convinced.

In another mail John O'Hara uses the the reading email analogy. Continuing
this to its logical conclusion could mean that replay is actually an option
when you open the queue for reading. It avoids the client having to muck
about with another queue.

This is not the traditional view of a queue but is more general and I like
it.

So, how about a queue contains all messages put on it since it was last
"purged". When a queue is opened for reading, it can be opened with a
selector and one part of that selector is whether or not the message has
been commited by a consumer or not. 

A consumer could then always open the queue and request all messages,
committed or not, from a known point, e.g. a trade ID or message ID. The key
thing here I suppose is that this type of consumer is writing the message in
some other form into a database and so always knows the last thing written.
A nice benefit is that if the database is really hosed you could get replay
from start of day. This would definitely help implement DR for purely
message based systems.

So, can I specialise queues and layer this functionality on top of the spec
in the APID implementation with some kind of provider specific header
property? 

Colin.

> -----Original Message-----
> From: Marnie McCormack [mailto:marnie.mccormack@googlemail.com] 
> Sent: 22 January 2007 19:54
> To: qpid-dev@incubator.apache.org; cctrieloff@redhat.com
> Subject: Re: Message history and replay
> 
> I've been trying to imagine the scenarios in which a user 
> will make use of a ReplayExchange.
> 
> I thought that the most common usage would be the following:
> 
> - the receiving application has lost some messages sent to it 
> and needs them to be resent from a point in time or a message ID
> 
> (The only other recovery scenario I thought likely was broker 
> failure, which would be handled by restarted (with 
> persistence) and the undelivered messages would already 
> remain in the BDB store.)
> 
> So, the bit I'm not sure about is this - you have no way of 
> knowing in advance the point at which replay might be needed 
> from (unless using client acks - which work with ordinary queues ?).
> 
> Thus your recipient app starts up again (or recognises it has 
> lost messages) and then wants to replay. So, the queue it is 
> reading from is rebound to a replay exchange (with a marker 
> ID) ? Does that mean that it is actually reading from a 
> different queue than normal operation, or that the queue is 
> bound to a different exchange type ?
> 
> The logic behind this could be quite complex - in that the 
> balance between normal operation and replay operation in 
> subtle and would involve the receiving client understanding 
> the AMQP behaviour at quite a low level ?
> 
> Apologies if I've missed a simple flow for replay :-)
> 
> Regards,
> Marnie
> 
> On 1/22/07, Carl Trieloff <cc...@redhat.com> wrote:
> >
> >
> > This has been asked for before. I would interested to see if an 
> > exchange could be used. I also think this might bring up 
> the topic of 
> > chained exchanges again.
> >
> > Colin, do you have any interest in creating a prototype 
> ReplyExchange 
> > to see how well it works?
> > Carl.
> >
> > Alan Conway wrote:
> > > On Thu, 2007-01-18 at 15:52 +0000, Colin Crist wrote:
> > >
> > >> Hi,
> > >>
> > >> A quick question...
> > >>
> > >> A common recovery scenario is requesting replay on a queue from a
> > given,
> > >> known message, either by its ID or by something in the 
> header. Its
> > saves all
> > >> that mucking about with XA and makes recovery the normal way of 
> > >> system startup rather then some exceptional case.
> > >>
> > >> This means when a message has reached all its recipients 
> it should
> > never be
> > >> marked for removal from the durable message store.
> > >>
> > >> As a user, I would also like to control when messages 
> get removed 
> > >> as
> > part of
> > >> my end of day or even weekly processs, probably based on header
> > property
> > >> selectors.
> > >>
> > >> I don't believe this is supported in AMQP in the wire 
> protocol (nor
> > perhaps
> > >> should it be) - is it maybe something that could be done 
> via some 
> > >> standardised command messages to a "replay" exchange?
> > >>
> > >> Any thoughts on implementation difficulty?
> > >>
> > >
> > > You could represent this in AMQP protocol using a custom 
> exchange type.
> > >
> > > A ReplayExchange stores all messages sent to it (until 
> cleaned up by 
> > > some administrative function.)
> > >
> > > When binding a queue to a replay exchange you MAY specify an 
> > > argument "replay-from=N". A queue so bound will be 
> "pre-filled" with 
> > > the range of message starting from the Nth message up to the most 
> > > recent message sent to the exchange. Thereafter the 
> exchange behaves 
> > > like a fan-out exchange, each new message being added to 
> all the bound queues.
> > >
> > > That gives you replay entirely within the standard 
> protocol, using a 
> > > standard extension point to add this as proprietary broker 
> > > functionality.
> > >
> > > Note that to combine replay with other exchange types requires 
> > > exchange chaining which has been discussed several times 
> but is not 
> > > yet an agreed part of the protocol - this may be another use case 
> > > for that discussion
> > >
> > > Cheers,
> > > Alan.
> > >
> > >
> >
> >
> >
> 

Re: Message history and replay

[Marnie McCormack <ma...@googlemail.com>]

I've been trying to imagine the scenarios in which a user will make use of a
ReplayExchange.

I thought that the most common usage would be the following:

- the receiving application has lost some messages sent to it and needs them
to be resent from a point in time or a message ID

(The only other recovery scenario I thought likely was broker failure, which
would be handled by restarted (with persistence) and the undelivered
messages would already remain in the BDB store.)

So, the bit I'm not sure about is this - you have no way of knowing in
advance the point at which replay might be needed from (unless using client
acks - which work with ordinary queues ?).

Thus your recipient app starts up again (or recognises it has lost messages)
and then wants to replay. So, the queue it is reading from is rebound to a
replay exchange (with a marker ID) ? Does that mean that it is actually
reading from a different queue than normal operation, or that the queue is
bound to a different exchange type ?

The logic behind this could be quite complex - in that the balance between
normal operation and replay operation in subtle and would involve the
receiving client understanding the AMQP behaviour at quite a low level ?

Apologies if I've missed a simple flow for replay :-)

Regards,
Marnie

On 1/22/07, Carl Trieloff <cc...@redhat.com> wrote:
>
>
> This has been asked for before. I would interested to see if an exchange
> could be used. I also think this
> might bring up the topic of chained exchanges again.
>
> Colin, do you have any interest in creating a prototype ReplyExchange to
> see how well it works?
> Carl.
>
> Alan Conway wrote:
> > On Thu, 2007-01-18 at 15:52 +0000, Colin Crist wrote:
> >
> >> Hi,
> >>
> >> A quick question...
> >>
> >> A common recovery scenario is requesting replay on a queue from a
> given,
> >> known message, either by its ID or by something in the header. Its
> saves all
> >> that mucking about with XA and makes recovery the normal way of system
> >> startup rather then some exceptional case.
> >>
> >> This means when a message has reached all its recipients it should
> never be
> >> marked for removal from the durable message store.
> >>
> >> As a user, I would also like to control when messages get removed as
> part of
> >> my end of day or even weekly processs, probably based on header
> property
> >> selectors.
> >>
> >> I don't believe this is supported in AMQP in the wire protocol (nor
> perhaps
> >> should it be) - is it maybe something that could be done via some
> >> standardised command messages to a "replay" exchange?
> >>
> >> Any thoughts on implementation difficulty?
> >>
> >
> > You could represent this in AMQP protocol using a custom exchange type.
> >
> > A ReplayExchange stores all messages sent to it (until cleaned up by
> > some administrative function.)
> >
> > When binding a queue to a replay exchange you MAY specify an argument
> > "replay-from=N". A queue so bound will be "pre-filled" with the range of
> > message starting from the Nth message up to the most recent message sent
> > to the exchange. Thereafter the exchange behaves like a fan-out
> > exchange, each new message being added to all the bound queues.
> >
> > That gives you replay entirely within the standard protocol, using a
> > standard extension point to add this as proprietary broker
> > functionality.
> >
> > Note that to combine replay with other exchange types requires exchange
> > chaining which has been discussed several times but is not yet an agreed
> > part of the protocol - this may be another use case for that discussion
> >
> > Cheers,
> > Alan.
> >
> >
>
>
>

Re: Message history and replay

[Carl Trieloff <cc...@redhat.com>]

This has been asked for before. I would interested to see if an exchange 
could be used. I also think this
might bring up the topic of chained exchanges again.

Colin, do you have any interest in creating a prototype ReplyExchange to 
see how well it works?
Carl.

Alan Conway wrote:
> On Thu, 2007-01-18 at 15:52 +0000, Colin Crist wrote:
>   
>> Hi,
>>
>> A quick question...
>>
>> A common recovery scenario is requesting replay on a queue from a given,
>> known message, either by its ID or by something in the header. Its saves all
>> that mucking about with XA and makes recovery the normal way of system
>> startup rather then some exceptional case.
>>
>> This means when a message has reached all its recipients it should never be
>> marked for removal from the durable message store. 
>>
>> As a user, I would also like to control when messages get removed as part of
>> my end of day or even weekly processs, probably based on header property
>> selectors. 
>>
>> I don't believe this is supported in AMQP in the wire protocol (nor perhaps
>> should it be) - is it maybe something that could be done via some
>> standardised command messages to a "replay" exchange?
>>
>> Any thoughts on implementation difficulty? 
>>     
>
> You could represent this in AMQP protocol using a custom exchange type.
>
> A ReplayExchange stores all messages sent to it (until cleaned up by
> some administrative function.)
>
> When binding a queue to a replay exchange you MAY specify an argument
> "replay-from=N". A queue so bound will be "pre-filled" with the range of
> message starting from the Nth message up to the most recent message sent
> to the exchange. Thereafter the exchange behaves like a fan-out
> exchange, each new message being added to all the bound queues.
>
> That gives you replay entirely within the standard protocol, using a
> standard extension point to add this as proprietary broker
> functionality.
>
> Note that to combine replay with other exchange types requires exchange
> chaining which has been discussed several times but is not yet an agreed
> part of the protocol - this may be another use case for that discussion
>
> Cheers,
> Alan.
>
>   

Re: Message history and replay

[Alan Conway <ac...@redhat.com>]

On Thu, 2007-01-18 at 15:52 +0000, Colin Crist wrote:
> Hi,
> 
> A quick question...
> 
> A common recovery scenario is requesting replay on a queue from a given,
> known message, either by its ID or by something in the header. Its saves all
> that mucking about with XA and makes recovery the normal way of system
> startup rather then some exceptional case.
> 
> This means when a message has reached all its recipients it should never be
> marked for removal from the durable message store. 
> 
> As a user, I would also like to control when messages get removed as part of
> my end of day or even weekly processs, probably based on header property
> selectors. 
> 
> I don't believe this is supported in AMQP in the wire protocol (nor perhaps
> should it be) - is it maybe something that could be done via some
> standardised command messages to a "replay" exchange?
> 
> Any thoughts on implementation difficulty? 

You could represent this in AMQP protocol using a custom exchange type.

A ReplayExchange stores all messages sent to it (until cleaned up by
some administrative function.)

When binding a queue to a replay exchange you MAY specify an argument
"replay-from=N". A queue so bound will be "pre-filled" with the range of
message starting from the Nth message up to the most recent message sent
to the exchange. Thereafter the exchange behaves like a fan-out
exchange, each new message being added to all the bound queues.

That gives you replay entirely within the standard protocol, using a
standard extension point to add this as proprietary broker
functionality.

Note that to combine replay with other exchange types requires exchange
chaining which has been discussed several times but is not yet an agreed
part of the protocol - this may be another use case for that discussion

Cheers,
Alan.

RE: Message history and replay

[Colin Crist <co...@hermesjms.com>]

Hi Marnie,

There are certainly opertational issues to consider when deploying such
behaviour and its unlikely that it would be enabled out of the box :)

I think I'd go for a header property to mark the message as "recoverable" or
"replayable" so that I could run both types of behaviour concurrently and
even maybe keep the recoverable messages in a separate message store so I
can better monitor its size. There would be no need to try and cache any of
these messages anywhere, recovering clients would be happy to take the disk
read hit. Having two stores would complicate recovery as you'd have to
ensure that all messages in transit between the stores were flushed when a
client requests recovery so an exchange implementation would need support in
the core broker to do this - even if all it did was call something to say a
message has reached zero references and should it be deleted.

I've used this kind of recovery before and it worked very well but you need
the transport to support it natively and very few do. It would be real Qpid
differentiator and I'd be keen to bash out a prototype when I've got some
time next month.

Anyway, thanks for your brain time, talk l8r.

Colin.

> -----Original Message-----
> From: Marnie McCormack [mailto:marnie.mccormack@googlemail.com] 
> Sent: 18 January 2007 16:02
> To: qpid-dev@incubator.apache.org
> Subject: Re: Message history and replay
> 
> Hi Colin,
> 
> I guess my immediate reading of this is that it could be 
> mighty dangerous in the wrong hands i.e. a very large message 
> store might result if applied too liberally.
> 
> When we talked about archiving messages previously we had 
> imagined that timeToLive could come in to play here, though I 
> might well be abusing it's intended purpose.
> 
> I guess we might also need to be careful around anything 
> which might slow up the 'normal' startup model i.e. when we 
> need to point a new broker instance at an existing BDB store 
> and get the messages in queues for recovery.
> 
> I'll try to give it some more thought :-)
> 
> Bfn,
> Marnie
> 
> On 1/18/07, Colin Crist <co...@hermesjms.com> wrote:
> >
> >
> > Hi,
> >
> > A quick question...
> >
> > A common recovery scenario is requesting replay on a queue from a 
> > given, known message, either by its ID or by something in 
> the header. 
> > Its saves all that mucking about with XA and makes recovery 
> the normal 
> > way of system startup rather then some exceptional case.
> >
> > This means when a message has reached all its recipients it should 
> > never be marked for removal from the durable message store.
> >
> > As a user, I would also like to control when messages get 
> removed as 
> > part of my end of day or even weekly processs, probably based on 
> > header property selectors.
> >
> > I don't believe this is supported in AMQP in the wire protocol (nor 
> > perhaps should it be) - is it maybe something that could be 
> done via 
> > some standardised command messages to a "replay" exchange?
> >
> > Any thoughts on implementation difficulty?
> >
> > Regards,
> >
> > Colin.
> >
> >
> >
> >
> >
> 

Re: Message history and replay

[Marnie McCormack <ma...@googlemail.com>]

Hi Colin,

I guess my immediate reading of this is that it could be mighty dangerous in
the wrong hands i.e. a very large message store might result if applied too
liberally.

When we talked about archiving messages previously we had imagined that
timeToLive could come in to play here, though I might well be abusing it's
intended purpose.

I guess we might also need to be careful around anything which might slow up
the 'normal' startup model i.e. when we need to point a new broker instance
at an existing BDB store and get the messages in queues for recovery.

I'll try to give it some more thought :-)

Bfn,
Marnie

On 1/18/07, Colin Crist <co...@hermesjms.com> wrote:
>
>
> Hi,
>
> A quick question...
>
> A common recovery scenario is requesting replay on a queue from a given,
> known message, either by its ID or by something in the header. Its saves
> all
> that mucking about with XA and makes recovery the normal way of system
> startup rather then some exceptional case.
>
> This means when a message has reached all its recipients it should never
> be
> marked for removal from the durable message store.
>
> As a user, I would also like to control when messages get removed as part
> of
> my end of day or even weekly processs, probably based on header property
> selectors.
>
> I don't believe this is supported in AMQP in the wire protocol (nor
> perhaps
> should it be) - is it maybe something that could be done via some
> standardised command messages to a "replay" exchange?
>
> Any thoughts on implementation difficulty?
>
> Regards,
>
> Colin.
>
>
>
>
>

Re: Message history and replay

[Marnie McCormack <ma...@googlemail.com>]

Hi Colin,

Had a little more time to think about this and I believe that there are
potentially two distinct solutions, with pros & cons for each, which
possibly meet the needs of different applications/architectures.

Robert Greig & I discussed briefly and don't entirely agree on the pros/cons
:-)

*Replay with only BDBStore*
If your application has performance as its only really significant
requirement, then a solution which provided a fairly limited set of
operations for BDBStore replaying would be appropriate.

So, you might perhaps have a tool that can archive data (using a header
property) or replay it based on a property, not using SQL since that's not
the BDBStore way.
However, it'd be implementation expensive to try and provide anything more
feature rich.

*Replay with BDBStore and an RDBMS*
So, if you assume that for many users currently replaying data they're
probably using an RDBMS to do it (store/replay/archive/report etc), then
they might find the BDBStore approach fast but not as sophisticated as their
existing architecture.

In that case, it might be nice to be able to setup BDBStore to archive to an
RDBMS and be able to replay from thet RDBMS (mySQL or whatever). The users
can then migrate over any SQL scripts they use to select data from
replay/check the state of their archive/run reports etc - at the cost of a
less performant recovery model.

Could batch up the archive to RDBMS from BDBStore to make it more
efficient/more robust if RDBMS becomes unavailable.

Just some thoughts - hth.

Marnie


On 1/18/07, Colin Crist <co...@hermesjms.com> wrote:
>
>
> Typo.
>
> ^On the consumer side, WebMethods (!) for example in their^On the producer
> side, WebMethods (!) for example in their^
>
> Sry. Colin.
>
> > -----Original Message-----
> > From: Colin Crist [mailto:colincrist@hermesjms.com]
> > Sent: 18 January 2007 23:16
> > To: 'qpid-dev@incubator.apache.org'
> > Subject: RE: Message history and replay
> >
> >
> >
> > > > As a user, I would also like to control when messages get
> > > removed as
> > > > part of my end of day or even weekly processs, probably based on
> > > > header property selectors.
> > >
> > > Yes, very interesting and useful use case. It would improve
> > throughput
> > > for many apps since people would not have to commit the
> > whole message
> > > to their database before sending it to the message broker.
> >
> > On the consumer side, WebMethods (!) for example in their
> > core messaging platform can let you find out the last message
> > a client published when it reconnects. I didn't use it when I
> > was in WebMethods hell as our systems did not require such
> > behaviour but I found it interesting and could see scenarios
> > when it may help minimise downsteam duplicates and shorten
> > producer recovery time.
> >
> > I don't think this works for AMQP as it does not maintain the
> > exchange route history - or am I wrong?
> >
> > Personally, I can live without it and its best to keep
> > recovery downstream IMO. It's the consumer end that has to
> > deal with duplicates after all but I've found many systems
> > that on recovery will resend too much - all trades from start
> > of day being a common example that can give unexpected load
> > on the middleware and general network buzz.
> >
> > The lesson I found is that some producers are very simple and
> > when you outsource or buy in products you've just got to live
> > with them!
> >
> > > > I don't believe this is supported in AMQP in the wire
> > protocol (nor
> > > > perhaps should it be) - is it maybe something that could be
> > > done via
> > > > some standardised command messages to a "replay" exchange?
> > >
> > > > Any thoughts on implementation difficulty?
> > >
> > > In terms of our underlying architecture, this would be relatively
> > > easy. We have a delivery table that simply maps from message id to
> > > queue, and the message is stored separately. Messages are removed
> > > automatically when a ref count hits zero. The removal process is
> > > actually handled in the store module so it could do something else
> > > easily (e.g if you had a custom store).
> >
> > > The admin API could be enhanced to support re-enqueuing of
> > message ids
> > > or hard delete of messages.
> >
> > I'd see this kind of behaviour to be message based and
> > perhaps standardised in some addendum to the AMQP
> > specification that defines optional exchanges and payload
> > formats. Of course it makes sense to map it to an API but
> > outside of the core specification to keep it decoupled as
> > much as possible and reduce dependencies and so on. If I were
> > to have a client in a language not in the Qpid toolkit, it
> > would be good to have the messages and interaction well
> > defined so another implementation could leverage and help
> > drive it into a standard.
> >
> > Its hardly new and there are always things to learn from the
> > old dogs of middleware (with most of the current market share...)
> >
> > http://www-1.ibm.com/support/docview.wss?rs=203&uid=swg2400066
> > 8&loc=en_US&cs=utf-8&lang=en
> >
> > The same argument goes for configuration, management and
> > monitoring functionality too :) I'd like to see a market for
> > 3rd party tooling that works with all AMQP based products. A
> > whole other conversation when you and the spec are hitting 1.0...
> >
> > Hope this is not too far leftfield for this stage of the game
> > and you don't mind me throwing in my opinions...
> >
> > Regards,
> >
> > Colin.
> >
>
>
>

RE: Message history and replay

[Colin Crist <co...@hermesjms.com>]

Typo.

^On the consumer side, WebMethods (!) for example in their^On the producer
side, WebMethods (!) for example in their^

Sry. Colin.

> -----Original Message-----
> From: Colin Crist [mailto:colincrist@hermesjms.com] 
> Sent: 18 January 2007 23:16
> To: 'qpid-dev@incubator.apache.org'
> Subject: RE: Message history and replay
> 
>  
> 
> > > As a user, I would also like to control when messages get
> > removed as
> > > part of my end of day or even weekly processs, probably based on 
> > > header property selectors.
> > 
> > Yes, very interesting and useful use case. It would improve 
> throughput 
> > for many apps since people would not have to commit the 
> whole message 
> > to their database before sending it to the message broker.
> 
> On the consumer side, WebMethods (!) for example in their 
> core messaging platform can let you find out the last message 
> a client published when it reconnects. I didn't use it when I 
> was in WebMethods hell as our systems did not require such 
> behaviour but I found it interesting and could see scenarios 
> when it may help minimise downsteam duplicates and shorten 
> producer recovery time.
> 
> I don't think this works for AMQP as it does not maintain the 
> exchange route history - or am I wrong?
> 
> Personally, I can live without it and its best to keep 
> recovery downstream IMO. It's the consumer end that has to 
> deal with duplicates after all but I've found many systems 
> that on recovery will resend too much - all trades from start 
> of day being a common example that can give unexpected load 
> on the middleware and general network buzz. 
> 
> The lesson I found is that some producers are very simple and 
> when you outsource or buy in products you've just got to live 
> with them!
> 
> > > I don't believe this is supported in AMQP in the wire 
> protocol (nor 
> > > perhaps should it be) - is it maybe something that could be
> > done via
> > > some standardised command messages to a "replay" exchange?
> > 
> > > Any thoughts on implementation difficulty?
> > 
> > In terms of our underlying architecture, this would be relatively 
> > easy. We have a delivery table that simply maps from message id to 
> > queue, and the message is stored separately. Messages are removed 
> > automatically when a ref count hits zero. The removal process is 
> > actually handled in the store module so it could do something else 
> > easily (e.g if you had a custom store).
> 
> > The admin API could be enhanced to support re-enqueuing of 
> message ids 
> > or hard delete of messages.
> 
> I'd see this kind of behaviour to be message based and 
> perhaps standardised in some addendum to the AMQP 
> specification that defines optional exchanges and payload 
> formats. Of course it makes sense to map it to an API but 
> outside of the core specification to keep it decoupled as 
> much as possible and reduce dependencies and so on. If I were 
> to have a client in a language not in the Qpid toolkit, it 
> would be good to have the messages and interaction well 
> defined so another implementation could leverage and help 
> drive it into a standard.
> 
> Its hardly new and there are always things to learn from the 
> old dogs of middleware (with most of the current market share...)
> 
> http://www-1.ibm.com/support/docview.wss?rs=203&uid=swg2400066
> 8&loc=en_US&cs=utf-8&lang=en
> 
> The same argument goes for configuration, management and 
> monitoring functionality too :) I'd like to see a market for 
> 3rd party tooling that works with all AMQP based products. A 
> whole other conversation when you and the spec are hitting 1.0...
> 
> Hope this is not too far leftfield for this stage of the game 
> and you don't mind me throwing in my opinions...
> 
> Regards,
> 
> Colin.
> 

RE: Message history and replay

[Colin Crist <co...@hermesjms.com>]

 

> > As a user, I would also like to control when messages get 
> removed as 
> > part of my end of day or even weekly processs, probably based on 
> > header property selectors.
> 
> Yes, very interesting and useful use case. It would improve 
> throughput for many apps since people would not have to 
> commit the whole message to their database before sending it 
> to the message broker.

On the consumer side, WebMethods (!) for example in their core messaging
platform can let you find out the last message a client published when it
reconnects. I didn't use it when I was in WebMethods hell as our systems did
not require such behaviour but I found it interesting and could see
scenarios when it may help minimise downsteam duplicates and shorten
producer recovery time.

I don't think this works for AMQP as it does not maintain the exchange route
history - or am I wrong?

Personally, I can live without it and its best to keep recovery downstream
IMO. It's the consumer end that has to deal with duplicates after all but
I've found many systems that on recovery will resend too much - all trades
from start of day being a common example that can give unexpected load on
the middleware and general network buzz. 

The lesson I found is that some producers are very simple and when you
outsource or buy in products you've just got to live with them!

> > I don't believe this is supported in AMQP in the wire protocol (nor 
> > perhaps should it be) - is it maybe something that could be 
> done via 
> > some standardised command messages to a "replay" exchange?
> 
> > Any thoughts on implementation difficulty?
> 
> In terms of our underlying architecture, this would be 
> relatively easy. We have a delivery table that simply maps 
> from message id to queue, and the message is stored 
> separately. Messages are removed automatically when a ref 
> count hits zero. The removal process is actually handled in 
> the store module so it could do something else easily (e.g if 
> you had a custom store).

> The admin API could be enhanced to support re-enqueuing of 
> message ids or hard delete of messages.

I'd see this kind of behaviour to be message based and perhaps standardised
in some addendum to the AMQP specification that defines optional exchanges
and payload formats. Of course it makes sense to map it to an API but
outside of the core specification to keep it decoupled as much as possible
and reduce dependencies and so on. If I were to have a client in a language
not in the Qpid toolkit, it would be good to have the messages and
interaction well defined so another implementation could leverage and help
drive it into a standard.

Its hardly new and there are always things to learn from the old dogs of
middleware (with most of the current market share...)

http://www-1.ibm.com/support/docview.wss?rs=203&uid=swg24000668&loc=en_US&cs
=utf-8&lang=en

The same argument goes for configuration, management and monitoring
functionality too :) I'd like to see a market for 3rd party tooling that
works with all AMQP based products. A whole other conversation when you and
the spec are hitting 1.0...

Hope this is not too far leftfield for this stage of the game and you don't
mind me throwing in my opinions...

Regards,

Colin.

Re: Message history and replay

[Robert Greig <ro...@gmail.com>]

On 18/01/07, Colin Crist <co...@hermesjms.com> wrote:
>
> A common recovery scenario is requesting replay on a queue from a given,
> known message, either by its ID or by something in the header. Its saves all
> that mucking about with XA and makes recovery the normal way of system
> startup rather then some exceptional case.

Yes - let's just design our systems to handle duplicate and out of
order delivery and we can avoid XA in most cases.

> As a user, I would also like to control when messages get removed as part of
> my end of day or even weekly processs, probably based on header property
> selectors.

Yes, very interesting and useful use case. It would improve throughput
for many apps since people would not have to commit the whole message
to their database before sending it to the message broker.

> I don't believe this is supported in AMQP in the wire protocol (nor perhaps
> should it be) - is it maybe something that could be done via some
> standardised command messages to a "replay" exchange?

> Any thoughts on implementation difficulty?

In terms of our underlying architecture, this would be relatively
easy. We have a delivery table that simply maps from message id to
queue, and the message is stored separately. Messages are removed
automatically when a ref count hits zero. The removal process is
actually handled in the store module so it could do something else
easily (e.g if you had a custom store).

The admin API could be enhanced to support re-enqueuing of message ids
or hard delete of messages.

RG