You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by William Henry <wh...@redhat.com> on 2008/07/28 19:43:39 UTC
Admin: Moving messages from one queue to another.
Hi all,
There is a requirement to be able to administratively move messages from
one queue to another and I'm looking for input regarding the/a use case
for this requirement.
It has been suggested that this is really a query/criteria based move
and it's really at a broker level. i.e. broker X please move all
messages from user johnz from queue A to queue B. Or broker X please
move all messages with topic Y from queue A to queue B. Queue B might
just be some sort of suspend queue while an administrator finds out what
johnz really wants or why messages of topic Y are not getting consumed etc.
Has anyone any opinions or a concrete use case for this?
Regards,
William Henry
Re: Admin: Moving messages from one queue to another.
Posted by Martin Ritchie <ri...@apache.org>.
2008/7/30 Gordon Sim <gs...@redhat.com>:
> Ted Ross wrote:
>>
>> My two cents...
>>
>> I think this is an administrative function that is performed on-demand and
>> has no lasting affect on the normal operation of the broker (i.e. no filters
>> or redirects are used). I also think that this feature will be used
>> primarily as a diagnostic tool.
>>
>> Since messages are not modeled as management objects, this feature must be
>> coupled with a message-query feature. A management method (or set of
>> methods) can be implemented on queue objects to allow management users to
>> get information about the set of messages on a queue.
>
> [...]
>>
>> This query feature can be used to provide visibility into the content of a
>> blocked or slow-moving queue.
>
> In 0-10 you can already 'browse' the messages on a queue using the
> not-acquired mode on your subscription. That could be a good way to address
> this feature.
>
> Though not yet implemented in the c++ broker, the arguments to subscribe can
> be used to specify a selector (which is what we want for JMS support anyway)
> that restricts the messages actually delivered for that subscription.
>
> A non-acquired message can be removed from the queue by acquiring it (and
> acknowledging if required).
>
> We could even add an option to the arguments for subscribe for indicating
> that only the headers should be delivered for that subscription if needed to
> reduce the amount of data that needed to be transfered (would perhaps only
> be valid in not-acquired mode).
>
> [...]
>>
>> Note that message query and message move/copy will probably require that
>> the queue be locked during the duration of the operation so these may be
>> expensive features. It would be good to find a way to implement these on a
>> flowing queue without locking.
>
> Browsing and acquiring as described above will already implement any
> necessary locking.
>
I agree with Ted that these are one off actions on a queue rather than
predefined filters. e.g. A client is unable to process the message at
the front of the queue. Move this message to another queue so it can
be manually inspected and potentially returned to the former queue.
That said we are in the process of adding Binding level filters to the
Java broker so that we can correctly apply selectors on JMS Topics,
other filters should not be difficult to apply after that.
--
Martin Ritchie
Re: Admin: Moving messages from one queue to another.
Posted by Alan Conway <ac...@redhat.com>.
On Wed, 2008-07-30 at 10:49 +0100, Gordon Sim wrote:
> Ted Ross wrote:
> > My two cents...
> >
> > I think this is an administrative function that is performed on-demand
> > and has no lasting affect on the normal operation of the broker (i.e. no
> > filters or redirects are used). I also think that this feature will be
> > used primarily as a diagnostic tool.
> >
> > Since messages are not modeled as management objects, this feature must
> > be coupled with a message-query feature. A management method (or set of
> > methods) can be implemented on queue objects to allow management users
> > to get information about the set of messages on a queue.
> [...]
> > This query feature can be used to provide visibility into the content of
> > a blocked or slow-moving queue.
>
> In 0-10 you can already 'browse' the messages on a queue using the
> not-acquired mode on your subscription. That could be a good way to
> address this feature.
>
> Though not yet implemented in the c++ broker, the arguments to subscribe
> can be used to specify a selector (which is what we want for JMS support
> anyway) that restricts the messages actually delivered for that
> subscription.
>
> A non-acquired message can be removed from the queue by acquiring it
> (and acknowledging if required).
>
> We could even add an option to the arguments for subscribe for
> indicating that only the headers should be delivered for that
> subscription if needed to reduce the amount of data that needed to be
> transfered (would perhaps only be valid in not-acquired mode).
>
> [...]
> > Note that message query and message move/copy will probably require that
> > the queue be locked during the duration of the operation so these may be
> > expensive features. It would be good to find a way to implement these
> > on a flowing queue without locking.
>
> Browsing and acquiring as described above will already implement any
> necessary locking.
+1. I'd argue this should be done by an admin client rather than
complicating the broker unless there is a compelling use case that is
impossible to support without broker changes.
Re: Admin: Moving messages from one queue to another.
Posted by Gordon Sim <gs...@redhat.com>.
Ted Ross wrote:
> My two cents...
>
> I think this is an administrative function that is performed on-demand
> and has no lasting affect on the normal operation of the broker (i.e. no
> filters or redirects are used). I also think that this feature will be
> used primarily as a diagnostic tool.
>
> Since messages are not modeled as management objects, this feature must
> be coupled with a message-query feature. A management method (or set of
> methods) can be implemented on queue objects to allow management users
> to get information about the set of messages on a queue.
[...]
> This query feature can be used to provide visibility into the content of
> a blocked or slow-moving queue.
In 0-10 you can already 'browse' the messages on a queue using the
not-acquired mode on your subscription. That could be a good way to
address this feature.
Though not yet implemented in the c++ broker, the arguments to subscribe
can be used to specify a selector (which is what we want for JMS support
anyway) that restricts the messages actually delivered for that
subscription.
A non-acquired message can be removed from the queue by acquiring it
(and acknowledging if required).
We could even add an option to the arguments for subscribe for
indicating that only the headers should be delivered for that
subscription if needed to reduce the amount of data that needed to be
transfered (would perhaps only be valid in not-acquired mode).
[...]
> Note that message query and message move/copy will probably require that
> the queue be locked during the duration of the operation so these may be
> expensive features. It would be good to find a way to implement these
> on a flowing queue without locking.
Browsing and acquiring as described above will already implement any
necessary locking.
Re: Admin: Moving messages from one queue to another.
Posted by Ted Ross <tr...@redhat.com>.
William Henry wrote:
> Hi all,
>
> There is a requirement to be able to administratively move messages
> from one queue to another and I'm looking for input regarding the/a
> use case for this requirement.
>
> It has been suggested that this is really a query/criteria based move
> and it's really at a broker level. i.e. broker X please move all
> messages from user johnz from queue A to queue B. Or broker X please
> move all messages with topic Y from queue A to queue B. Queue B might
> just be some sort of suspend queue while an administrator finds out
> what johnz really wants or why messages of topic Y are not getting
> consumed etc.
>
> Has anyone any opinions or a concrete use case for this?
> Regards,
> William Henry
>
My two cents...
I think this is an administrative function that is performed on-demand
and has no lasting affect on the normal operation of the broker (i.e. no
filters or redirects are used). I also think that this feature will be
used primarily as a diagnostic tool.
Since messages are not modeled as management objects, this feature must
be coupled with a message-query feature. A management method (or set of
methods) can be implemented on queue objects to allow management users
to get information about the set of messages on a queue. For example:
- Get the top 10 most frequently seen reply-to fields and their
instance counts.
- Get the largest 10 message body sizes
- Get the top 10 most frequently seen [header-field] and instance counts.
- Get the number of messages with [header-field] matching a specified
value.
- Fill in a histogram of message sizes. The histogram buckets can be
supplied by the request or be fixed.
This query feature can be used to provide visibility into the content of
a blocked or slow-moving queue.
A companion feature can then be implemented that moves or copies all
messages meeting a certain criteria (i.e. size range or header field
match) to another queue. This allows the administrator to go to the
next level and actually view the full contents of messages (by
subscribing a consumer to the copied-to queue). It also allows the
administrator to remove messages from the queue to solve an application
problem.
Note that message query and message move/copy will probably require that
the queue be locked during the duration of the operation so these may be
expensive features. It would be good to find a way to implement these
on a flowing queue without locking.
-Ted
Re: Admin: Moving messages from one queue to another.
Posted by Ted Ross <tr...@redhat.com>.
michael goulish wrote:
> On Mon, 2008-07-28 at 11:43 -0600, William Henry wrote:
>
>> Hi all,
>>
>> There is a requirement to be able to administratively move messages from
>> one queue to another and I'm looking for input regarding the/a use case
>> for this requirement.
>>
>> It has been suggested that this is really a query/criteria based move
>> and it's really at a broker level. i.e. broker X please move all
>> messages from user johnz from queue A to queue B. Or broker X please
>> move all messages with topic Y from queue A to queue B. Queue B might
>> just be some sort of suspend queue while an administrator finds out what
>> johnz really wants or why messages of topic Y are not getting consumed etc.
>>
>> Has anyone any opinions or a concrete use case for this?
>>
>> Regards,
>> William Henry
>>
>>
>>
>
>
> This is a potentially rude and violent way to treat messages,and there
> had better be a good reason for it or the administrator could get sued
> or prosecuted, depending on the value of the message.
>
> But with that said, here are several real-life analogues:
>
>
> 1. The Deadbeat
> ------------
>
> User_X has quit paying his bills. We sent him a Final Notice thousands
> of milliseconds ago, but he has not responded. Admin starts diverting
> all his messages to a holding queue and sends him a message that says we
> are no longer delivering his messages. Then he pays up, and we move all
> his messages back to their respective queues. Note that in this
> scenario you have to remember where they came from (in the header?)
>
>
>
> 2. The Fast Lane
> --------------
>
> Due to extreme message traffic on some queues, some new queues are
> created. Certain (preferred) senders are re-routed to those queues, and
> receivers are told to subscribe. The new queues are given preferential
> treatment, so that the messages in them experience much lower latency
> for the duration of the emergency.
>
>
>
> 3. The Censor
> --------------
>
> A Very Important Customer informs us that all (or some) messages on
> certain queues must be diverted to other queues, where each one will be
> examined. Upon examination each message will either be diverted
> unchanged back to its original queue, or parts will be deleted before
> diversion back to its original queue, or a team of Navy Seals will be
> sent to the message's origin point.
>
Michael's use cases all assume that a filter is put into place on a
queue that looks for enqueued messages that match a certain criteria and
move them to another queue. I think this goes beyond the real
requirement. If these are valid use cases, I claim that this becomes an
exchange feature, not a queue feature. In other words, a redirect
capability would be needed in exchanges to place the messages on an
alternate queue.
-Ted
Re: Admin: Moving messages from one queue to another.
Posted by michael goulish <mg...@redhat.com>.
On Mon, 2008-07-28 at 11:43 -0600, William Henry wrote:
> Hi all,
>
> There is a requirement to be able to administratively move messages from
> one queue to another and I'm looking for input regarding the/a use case
> for this requirement.
>
> It has been suggested that this is really a query/criteria based move
> and it's really at a broker level. i.e. broker X please move all
> messages from user johnz from queue A to queue B. Or broker X please
> move all messages with topic Y from queue A to queue B. Queue B might
> just be some sort of suspend queue while an administrator finds out what
> johnz really wants or why messages of topic Y are not getting consumed etc.
>
> Has anyone any opinions or a concrete use case for this?
>
> Regards,
> William Henry
>
>
This is a potentially rude and violent way to treat messages,and there
had better be a good reason for it or the administrator could get sued
or prosecuted, depending on the value of the message.
But with that said, here are several real-life analogues:
1. The Deadbeat
------------
User_X has quit paying his bills. We sent him a Final Notice thousands
of milliseconds ago, but he has not responded. Admin starts diverting
all his messages to a holding queue and sends him a message that says we
are no longer delivering his messages. Then he pays up, and we move all
his messages back to their respective queues. Note that in this
scenario you have to remember where they came from (in the header?)
2. The Fast Lane
--------------
Due to extreme message traffic on some queues, some new queues are
created. Certain (preferred) senders are re-routed to those queues, and
receivers are told to subscribe. The new queues are given preferential
treatment, so that the messages in them experience much lower latency
for the duration of the emergency.
3. The Censor
--------------
A Very Important Customer informs us that all (or some) messages on
certain queues must be diverted to other queues, where each one will be
examined. Upon examination each message will either be diverted
unchanged back to its original queue, or parts will be deleted before
diversion back to its original queue, or a team of Navy Seals will be
sent to the message's origin point.