You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by felixehm <fe...@cern.ch> on 2010/10/04 20:12:45 UTC
Qpid messaging design
Hi,
We're currently evaluating Qpid as a middleware solution and during this
process I stumbled upon some questions which I hoped to get clarified with
some of your help.
For the evaluation I have to implement a typical scenario on our site. 400
Metrics (numbers) are published every second and need to be retrieved <
300ms by max 10 subscribers. It should still be possible to subscribe to
individual values only.
In JMS words : 400 topics each subscribed by 10 Message listeners.
Now, reading the documentation and the examples the way to go is to send the
values via a topic exchange and set the routing key for each Message to the
according value identifier:
for (i=0;..){
message.getDeliveryProperties().setRoutingKey(routing_key+i);
async(session).messageTransfer(arg::content=message,
arg::destination="amq.topic");
}
(I'm aware that packing them into one message is very obvious and faster.
But for various reasons this is currently not possible in our environment.)
The subscriber on the other hand will subscribe using the same exchange, but
for each subscription a different routingKey.
for (i=0..){
session.queueDeclare(arg::queue=queue+i, arg::exclusive=false,
arg::autoDelete=true);
session.exchangeBind(arg::exchange="amq.topic", arg::queue=queue+i,
arg::bindingKey=metric1);
}
1. Is this the only way to realize the described scenario?
2. Why do I need an own queue per routing key per client (400 x 10)?
I've tried to move the queueDeclare outside the loop to re-use the
client-specific queue. I'd imagine that the server then dispatches messages
with the according routing key to this client's queue. But apparently this
is not allowed by the client library.
It might well be that I've misunderstood some concepts and would therefore
I'd appreciate any help on this.
Cheers,
Felix
--
View this message in context: http://apache-qpid-users.2158936.n2.nabble.com/Qpid-messaging-design-tp5600245p5600245.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Qpid messaging design
Posted by felixehm <fe...@cern.ch>.
Rajith Attapattu wrote:
>
> I would start using the new messaging API here.
> Read the following doc to get a good idea.
> http://qpid.apache.org/books/0.7/Programming-In-Apache-Qpid/html/index.html
>
Yes, already read this. But I don't see any section about asynchronous
reception of messages which is required in our case.
Rajith Attapattu wrote:
>
> Since you have 10 subscribers you could just have 10 queues, instead
> of 4000 queues.
> Each subscriber can create a queue and then bind it's queue to the 400
> topics.
>
> This is based on the assumption that you don't have to give equal
> priority for each topic, as topics that are more active will likely
> dominate the queues and the less active topics will have it's messages
> being processed few and far apart.
> We've had cases where customers had to treat each subscription equally
> and therefore 4000 queues were needed.
> In that case the subscribers would go in a round robin fashion
> consuming from each of it's 400 queues.
>
Yes. That's the case. The problem is that I can't find an example how I
should suppose to do this. I've posted our code attempt in the previous
message which didn't work.
Besides : would you recommend using the SubscriptionManager to tweak some
options or does the local queue speed up the reception?
Thanks,
Felix
--
View this message in context: http://apache-qpid-users.2158936.n2.nabble.com/Qpid-messaging-design-tp5600245p5603925.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Qpid messaging design
Posted by Rajith Attapattu <ra...@gmail.com>.
On Mon, Oct 4, 2010 at 2:12 PM, felixehm <fe...@cern.ch> wrote:
>
> Hi,
>
> We're currently evaluating Qpid as a middleware solution and during this
> process I stumbled upon some questions which I hoped to get clarified with
> some of your help.
>
> For the evaluation I have to implement a typical scenario on our site. 400
> Metrics (numbers) are published every second and need to be retrieved <
> 300ms by max 10 subscribers. It should still be possible to subscribe to
> individual values only.
>
> In JMS words : 400 topics each subscribed by 10 Message listeners.
>
I would start using the new messaging API here.
Read the following doc to get a good idea.
http://qpid.apache.org/books/0.7/Programming-In-Apache-Qpid/html/index.html
> Now, reading the documentation and the examples the way to go is to send the
> values via a topic exchange and set the routing key for each Message to the
> according value identifier:
> for (i=0;..){
> message.getDeliveryProperties().setRoutingKey(routing_key+i);
> async(session).messageTransfer(arg::content=message,
> arg::destination="amq.topic");
> }
> (I'm aware that packing them into one message is very obvious and faster.
> But for various reasons this is currently not possible in our environment.)
>
> The subscriber on the other hand will subscribe using the same exchange, but
> for each subscription a different routingKey.
> for (i=0..){
> session.queueDeclare(arg::queue=queue+i, arg::exclusive=false,
> arg::autoDelete=true);
> session.exchangeBind(arg::exchange="amq.topic", arg::queue=queue+i,
> arg::bindingKey=metric1);
> }
>
>
>
>
> 1. Is this the only way to realize the described scenario?
> 2. Why do I need an own queue per routing key per client (400 x 10)?
> I've tried to move the queueDeclare outside the loop to re-use the
> client-specific queue. I'd imagine that the server then dispatches messages
> with the according routing key to this client's queue. But apparently this
> is not allowed by the client library.
Since you have 10 subscribers you could just have 10 queues, instead
of 4000 queues.
Each subscriber can create a queue and then bind it's queue to the 400 topics.
This is based on the assumption that you don't have to give equal
priority for each topic, as topics that are more active will likely
dominate the queues and the less active topics will have it's messages
being processed few and far apart.
We've had cases where customers had to treat each subscription equally
and therefore 4000 queues were needed.
In that case the subscribers would go in a round robin fashion
consuming from each of it's 400 queues.
> It might well be that I've misunderstood some concepts and would therefore
> I'd appreciate any help on this.
>
> Cheers,
> Felix
>
>
> --
> View this message in context: http://apache-qpid-users.2158936.n2.nabble.com/Qpid-messaging-design-tp5600245p5600245.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
--
Regards,
Rajith Attapattu
Red Hat
http://rajith.2rlabs.com/
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: Qpid messaging design
Posted by felixehm <fe...@cern.ch>.
Hi Steve,
thanks for the quick reply.
Steve Huston wrote:
>
> Do you mean you want each client to declare one queue and then bind the
> amq.topic exchange to that queue using different routing keys? That
> should be possible.
>
Yes. This is a result from out tests we're run so far. The performance
numbers we got were not convincing at all (a 16 Core machine runs flat out
using the described way). It seems that the Brokers spend a lot of time
dispatching at this (quite high) rate to all the queues.
So I assumed that there are more clever ways (e.g. 1 queue/client) to do
this and get better results.
> If you post the problems you experienced trying to bind 400 keys and one
> queue that would be helpful.
>
Basically its when I try this code :
8-< -------------------------------
/* Declare an exclusive queue on the broker
*/
string queue = exchange + session.getId().getName();
session.queueDeclare(arg::queue=queue, arg::exclusive=true,
arg::autoDelete=true);
/* map the bindingkeys to my queue
*/
for (int i=0; i< subscriptions; i++){
string s = destName + convertInt(i);
listener.prepareQueue(queue, exchange, s);
}
listener.listen();
>-8 ------------------------------
When I try this I get the message :
# Subscribing to queue amq.topic6916d790-3d2e-42c0-a826-2fce0811a61c
# Subscribing to queue amq.topic6916d790-3d2e-42c0-a826-2fce0811a61c
2010-10-05 17:56:07 warning Exception received from broker: not-allowed:
not-allowed: Consumer tags must be unique
(qpid/broker/SessionAdapter.cpp:489) [caused by 8 \x04:\x07]
not-allowed: not-allowed: Consumer tags must be unique
(qpid/broker/SessionAdapter.cpp:489)
Cheers,
Felix
--
View this message in context: http://apache-qpid-users.2158936.n2.nabble.com/Qpid-messaging-design-tp5600245p5603791.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: Qpid messaging design
Posted by Steve Huston <sh...@riverace.com>.
Hi Felix,
> We're currently evaluating Qpid as a middleware solution and
> during this process I stumbled upon some questions which I
> hoped to get clarified with some of your help.
Sure, glad to help.
> For the evaluation I have to implement a typical scenario on
> our site. 400 Metrics (numbers) are published every second
> and need to be retrieved < 300ms by max 10 subscribers. It
> should still be possible to subscribe to individual values only.
Ok.
> In JMS words : 400 topics each subscribed by 10 Message listeners.
>
>
> Now, reading the documentation and the examples the way to go
> is to send the values via a topic exchange and set the
> routing key for each Message to the according value
> identifier: for (i=0;..){
> message.getDeliveryProperties().setRoutingKey(routing_key+i);
> async(session).messageTransfer(arg::content=message,
> arg::destination="amq.topic");
> }
> (I'm aware that packing them into one message is very obvious
> and faster. But for various reasons this is currently not
> possible in our environment.)
Ok. Looks good.
> The subscriber on the other hand will subscribe using the
> same exchange, but for each subscription a different routingKey.
> for (i=0..){
> session.queueDeclare(arg::queue=queue+i,
> arg::exclusive=false, arg::autoDelete=true);
> session.exchangeBind(arg::exchange="amq.topic",
> arg::queue=queue+i, arg::bindingKey=metric1); }
>
>
>
>
> 1. Is this the only way to realize the described scenario?
No. But it should work.
> 2. Why do I need an own queue per routing key per client (400 x 10)?
> I've tried to move the queueDeclare outside the loop to
> re-use the client-specific queue. I'd imagine that the server
> then dispatches messages with the according routing key to
> this client's queue. But apparently this is not allowed by
> the client library.
Do you mean you want each client to declare one queue and then bind the
amq.topic exchange to that queue using different routing keys? That
should be possible.
> It might well be that I've misunderstood some concepts and
> would therefore I'd appreciate any help on this.
If you post the problems you experienced trying to bind 400 keys and one
queue that would be helpful.
-Steve
--
Steve Huston, Riverace Corporation
Total Lifecycle Support for Your Networked Applications
http://www.riverace.com
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org