You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Xiong Zou <zo...@gmail.com> on 2014/01/21 08:28:07 UTC
bind a persistent queue to an exchange?
Hi All,
I am thinking to have a message source to publish messages to exchange
amq.topic with a routing key like USA.news, thus all client subscribers to
this exchange with routing key USA.news will receive a copy of each message
of USA.news.
Now I would like to have the client subscribers to be able to recover all
published USA.news messages once any of them re-started. How can I achieve
this purpose?
qpid-config command tool has following options:
qpid-config help
Usage: qpid-config [OPTIONS]
qpid-config [OPTIONS] exchanges [filter-string]
qpid-config [OPTIONS] queues [filter-string]
qpid-config [OPTIONS] add exchange <type> <name>
[AddExchangeOptions]
qpid-config [OPTIONS] del exchange <name>
qpid-config [OPTIONS] add queue <name> [AddQueueOptions]
qpid-config [OPTIONS] del queue <name> [DelQueueOptions]
qpid-config [OPTIONS] bind <exchange-name> <queue-name>
[binding-key]
<for type xml> [-f -|filename]
<for type header> [all|any] k1=v1 [, k2=v2...]
qpid-config [OPTIONS] unbind <exchange-name> <queue-name>
[binding-key]
qpid-config [OPTIONS] reload-acl
Can I use qpid-config to do following?
1. add a persistent queue "Persistent.USA.news",
qpid-config --durable add queue Persistent.USA.news
2. then bind this queue to exchange amq.topic
qpid-config bind amq.topic Persistent.USA.news USA.news
3. like ./drain 'Persistent.USA.news; {mode: browse}' to browse all existing
messages for recovery.
and how to ensure messages delivered to queue "Persistent.USA.news" to be
persistent even if qpidd restarted?
Or is there any other ways to achieve the same?
Looking forward to your advices!
Thanks a lot!
--
View this message in context: http://qpid.2158936.n2.nabble.com/bind-a-persistent-queue-to-an-exchange-tp7602992.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: bind a persistent queue to an exchange?
Posted by Gordon Sim <gs...@redhat.com>.
On 01/23/2014 03:37 AM, Xiong Zou wrote:
> I can see that QPID supports my design target. If all messages published
> will be persistent by default in C++ APIs as well, then that will be
> perfect.
In the c++ client messages are not durable by default. However you can
set them to be durable using the Message::setDurable() method.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: bind a persistent queue to an exchange?
Posted by Xiong Zou <zo...@gmail.com>.
Thank you Gordon, Rob and Frase for your kind help.
I can see that QPID supports my design target. If all messages published
will be persistent by default in C++ APIs as well, then that will be
perfect.
Cheers,
Xiong
--
View this message in context: http://qpid.2158936.n2.nabble.com/bind-a-persistent-queue-to-an-exchange-tp7602992p7603136.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: bind a persistent queue to an exchange?
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 22/01/14 02:02, Xiong Zou wrote:
> You mentioned that /*"You need to make sure the publisher(s) mark(s) the
> messages as persistent. Unfortunately at present the c++ broker doesn't
> provide a way to override that on the broker side."*/
>
I'm a little bit rusty on this bit, but isn't the *default* behaviour in
both the JMS and qpid::messaging client APIs that messages are marked as
persistent by default?
From the JMS JavaDocs:
*Field Summary*
|static int| |*DEFAULT_DELIVERY_MODE
<http://docs.oracle.com/javaee/5/api/javax/jms/Message.html#DEFAULT_DELIVERY_MODE>*|
The message producer's default delivery mode is |PERSISTENT|.
|static int| |*DEFAULT_PRIORITY
<http://docs.oracle.com/javaee/5/api/javax/jms/Message.html#DEFAULT_PRIORITY>*|
The message producer's default priority is 4.
|static long| |*DEFAULT_TIME_TO_LIVE
<http://docs.oracle.com/javaee/5/api/javax/jms/Message.html#DEFAULT_TIME_TO_LIVE>*|
The message producer's default time to live is unlimited; the
message never expires.
I'm pretty sure it's the same for qpid::messaging too.
So the comments from Gordon and Rob are true that you can't currently
override what has been specified by the client in the broker, but unless
your clients have *explicitly* set the delivery mode to
DeliverMode.NON_PERSISTENT then they will be marked as durable - and I'd
argue that if they've explicitly set it as NON_PERSISTENT then they are
not in any sort of strong position to argue with you if you happen to
lose their messages :-)
Cheers,
Frase
Re: bind a persistent queue to an exchange?
Posted by Rob Godfrey <ro...@gmail.com>.
On 22 January 2014 10:35, Gordon Sim <gs...@redhat.com> wrote:
> On 01/22/2014 02:02 AM, Xiong Zou wrote:
>
>> You mentioned that /*"You need to make sure the publisher(s) mark(s) the
>> messages as persistent. Unfortunately at present the c++ broker doesn't
>> provide a way to override that on the broker side."*/
>> Does it mean that we can still achieve messages' persistence via C++
>> publisher?
>>
>
> Certainly, you just need to call Message::setDurable(true) on all messages.
>
>
> How about a Java Broker? can Java Broker defines all message in a
>> particular queue to always be persistent, no matter whether a publisher
>> marks it persistent or not?
>>
>
> I'm not 100% sure, but I can't see any option for that in the docs or
> extensions age. Perhaps someone from the java broker development team can
> correct me if I'm wrong though.
>
>
No - you are correct - there is not currently an option for doing this in
the Java Broker, though it is something we've discussed previously.
I like your idea with amq.topic/USA.news; {link:{name:my-subscription,
>> durable:True}} plus the auto_delete_timeout. Can I set the timeout to be
>> a
>> whole day as auto_delete_timeout:86400 ?
>>
>
> Certainly. Make sure your queue has sufficient capacity (in memory and in
> the journal) to store the expected amount of traffic for a whole day, but
> otherwise there is no problem.
>
>
>
-- Rob
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: bind a persistent queue to an exchange?
Posted by Gordon Sim <gs...@redhat.com>.
On 01/22/2014 02:02 AM, Xiong Zou wrote:
> You mentioned that /*"You need to make sure the publisher(s) mark(s) the
> messages as persistent. Unfortunately at present the c++ broker doesn't
> provide a way to override that on the broker side."*/
> Does it mean that we can still achieve messages' persistence via C++
> publisher?
Certainly, you just need to call Message::setDurable(true) on all messages.
> How about a Java Broker? can Java Broker defines all message in a
> particular queue to always be persistent, no matter whether a publisher
> marks it persistent or not?
I'm not 100% sure, but I can't see any option for that in the docs or
extensions age. Perhaps someone from the java broker development team
can correct me if I'm wrong though.
> I like your idea with amq.topic/USA.news; {link:{name:my-subscription,
> durable:True}} plus the auto_delete_timeout. Can I set the timeout to be a
> whole day as auto_delete_timeout:86400 ?
Certainly. Make sure your queue has sufficient capacity (in memory and
in the journal) to store the expected amount of traffic for a whole day,
but otherwise there is no problem.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: bind a persistent queue to an exchange?
Posted by Xiong Zou <zo...@gmail.com>.
Hi Gordon,
Thanks a lot for your help again.
I am currently using C++ Broker and C++ Client APIs. I am designing a stock
trading solution, which shall be re-started daily. Thus we need only to keep
a day's messages Persistent with qpid for recovery purpose in case Client
APIs re-start. Sure the data can be huge for all the orders and reports
processed within a day for a retail customer.
You mentioned that /*"You need to make sure the publisher(s) mark(s) the
messages as persistent. Unfortunately at present the c++ broker doesn't
provide a way to override that on the broker side."*/
Does it mean that we can still achieve messages' persistence via C++
publisher? How about a Java Broker? can Java Broker defines all message in a
particular queue to always be persistent, no matter whether a publisher
marks it persistent or not?
I like your idea with amq.topic/USA.news; {link:{name:my-subscription,
durable:True}} plus the auto_delete_timeout. Can I set the timeout to be a
whole day as auto_delete_timeout:86400 ?
In this case, I can use either qpid-config or a JMS client to achive
house-keeping one whole copy of all messages exchanged within my system in a
day to be ready for recovery of any clients connected to qpidd broker.
Thanks & regards,
Xiong Zou
--
View this message in context: http://qpid.2158936.n2.nabble.com/bind-a-persistent-queue-to-an-exchange-tp7602992p7603053.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: bind a persistent queue to an exchange?
Posted by Gordon Sim <gs...@redhat.com>.
On 01/21/2014 07:28 AM, Xiong Zou wrote:
> Hi All,
>
> I am thinking to have a message source to publish messages to exchange
> amq.topic with a routing key like USA.news, thus all client subscribers to
> this exchange with routing key USA.news will receive a copy of each message
> of USA.news.
>
> Now I would like to have the client subscribers to be able to recover all
> published USA.news messages once any of them re-started. How can I achieve
> this purpose?
[...]
> Can I use qpid-config to do following?
> 1. add a persistent queue "Persistent.USA.news",
> qpid-config --durable add queue Persistent.USA.news
> 2. then bind this queue to exchange amq.topic
> qpid-config bind amq.topic Persistent.USA.news USA.news
> 3. like ./drain 'Persistent.USA.news; {mode: browse}' to browse all existing
> messages for recovery.
The problem with that is that if all the client subscribers browse the
queue, then the messages will never be deleted and the queue will grow
without limit.
Depending on use case, there are a couple of ways around that if this
approach is otherwise attractive. You can limit the queue to a certain
number of messages, with older ones discarded. That way the queue always
has the last N messages and subscribers can browse them all.
Another possible downside though is that when a subscriber restarts, it
will get all N messages, regardless of whether it already saw them.
Another possible solution to the growing queue is to use a 'last value
queue'. This works if the messages can be viewed as updates to some
previous message. You choose a logical key and when a new message with a
given key arrives, it causes the last message with the same key to be
removed.
> and how to ensure messages delivered to queue "Persistent.USA.news" to be
> persistent even if qpidd restarted?
You need to make sure the publisher(s) mark(s) the messages as
persistent. Unfortunately at present the c++ broker doesn't provide a
way to override that on the broker side.
> Or is there any other ways to achieve the same?
Which API are you using?
With JMS there is a specific durable subscriber API that you can use.
With the Qpid messaging API, you can just specify an appropriate address
when creating the receiver. E.g.
amq.topic/USA.news; {link:{name:my-subscription, durable:True}}
That will result in a persistent queue being bound to amq,topic with key
USA.news. You need to ensure that each subscriber chooses a unique name
for their subscription, as this will be used to identify the appropriate
queue when restarting.
Be aware that if a subscriber is inactive for a long period, or never
restarts for some reason, the queue will remain and will keep growing,
which may eventually cause problems.
One way to work around that problem is to add a timeout to the queue,
such that if it is inactive for a certain period it will be cleaned up.
You can do this using the qpid.auto_delete_timeout property, e.g. to
clean up after 5 minutes:
amq.topic/USA.news; {link:{name:my-subscription, durable:True,
x-declare:{auto-delete:True, arguments:{qpid,auto_delete_timeout:350}}}}
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org