You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by gregory james marsh <ma...@cse.ohio-state.edu> on 2009/05/13 05:23:11 UTC
Technical Report on Qpid Broker Federation Experiments
Hi All,
A few other lab members and myself have just completed an Ohio State
University Technical Report dealing with Qpid broker federation entitled
"Scaling Advanced Message Queuing Protocol (AMQP) Architecture with Broker
Federation and InfiniBand". The pdf is publicly available at
http://nowlab.cse.ohio-state.edu/projects/amqp/ as well as
ftp://ftp.cse.ohio-state.edu/pub/tech-report/TRList.html
Basically this report tests federation with k-nomial tree topology. We
were unable to use Qpid rdma/InfiniBand (IB) due to problem documented on
JIRA (QPID-1855). So we performed our experiments using IPoIB via Qpid
TCP connector.
Based on what we saw in our experiments, each of us has written some
questions for the list about how federation is designed & implemented
internally. I've compiled the questions below. I make reference to
figures and tables from the report where appropriate.
1. In detail how does the connection of a destination broker to a source
broker with "qpid-route route add" differ from a consumer application
declaring and binding a queue on a source broker? Are there functional
differences in how messages are routed to a federated destination broker
as opposed to a consumer? What anticipated affect would these differences
have on host system performance?
2. Why does a chain of federated brokers to one consumer have a higher
bytes/sec rate on their interfaces (using SYSSTAT/sar to collect traffic
data on IPoIB interface) than a single broker to one consumer? This
experiment is explained in Sections 3.1, 4.2, Figure 8a.
3. Why does the activity at the root broker and producer increase as we
increase the federation tree? Our understanding was that the only work
root broker/producer does is send messages to broker(s) they are directly
connected to. (This is similar to question 2, but different experiment.
In such situations the net interfaces on the producers and consumers (as
well as brokers) of more complex trees showed elevatated bytes/sec rates
compared to less complex trees. Experiment is explained in Sections 3.3,
4.4, Figure 14a.)
4. We observed that the tree P-B-4B-16C was performing better than the
tree P-B-4B-4C. And I thought this was because more consumers being
attached keeps the leaf brokers busy thus reducing the contention at the
root broker. Can we confirm if this understanding is correct. (This is a
comparison of the experiments performed in Sections 3.3/4.4 and 5.
Compare numbers in Tables 5, 6, 7, 8, particularly the small message rate
numbers in Tables 5 & 8 for 64B and 256B.)
5. What is the advised measurement for performance in federated
broker-consumer design?
(We used the benchmarks we created for our SC08 workshop paper last
fall--also available at above sites. Delivered bandwidth in the current
report is our benchmark bandwidth multiplied by number of consumers).
6. Which of the tests included with Qpid release might help us get more
insights on our topology study?
Thanks in advance for an comments and insights.
Greg Marsh
Network Based Computing Lab
Ohio State University
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Technical Report on Qpid Broker Federation Experiments
Posted by gregory james marsh <ma...@cse.ohio-state.edu>.
Hi Ted,
Thanks for taking a look at the report and the comments.
I've included highlights of my M4 publisher, broker fed, and consumer codes
below. In brief my publishers are using async transfers, broker fed is using
qpid-route add (with tcp -t default to use ipoib), and consumers are using
LocalQueues. The consumer's subscription settings are below and we use these
because after trial and error we found they were giving us best peformance.
Regards,
Greg
/* IP ADDRS */
root_broker= ip address 1
level1_broker1= ip address 2
/* PUBLISHER CODE */
Connection connection;
ConnectionSettings conn_setngs;
Message message;
Session session;
// Setup the session
conn_setngs.host= root_broker;
conn_setngs.protocol= "tcp";
conn_setngs.tcpNoDelay= true;
connection.open(conn_setngs);
session = connection.newSession();
// Async msg xfer
async(session).messageTransfer(arg::content=message,
arg::destination="amq.fanout");
/* BROKER FEDERATION */
qpid-route route add $level1_broker1 $root_broker amq.fanout no_routing_key
.....and so on to form tree structures
/* CONSUMER CODE */
Connection connection;
ConnectionSettings conn_setngs;
Message message;
Session session;
LocalQueue local_my_queue;
SubscriptionSettings subscrptn_setngs;
// Setup the session
conn_setngs.host= level1_broker1;
conn_setngs.protocol= "tcp";
conn_setngs.tcpNoDelay= true;
connection.open(conn_setngs);
session = connection.newSession();
// Bind a queue
my_queue_name=session.getId().getName();
session.queueDeclare(arg::queue=my_queue_name, arg::exclusive=true,
arg::autoDelete=true);
session.exchangeBind(arg::exchange="amq.fanout",
arg::queue=my_queue_name,
arg::bindingKey="my-key");
// Set subscription settings
subscrptn_setngs.flowControl= FlowControl::unlimited();
subscrptn_setngs.autoAck= 1;
// Accept and acquire modes defined in ./src/gen/qpid/framing/enum.h
subscrptn_setngs.acceptMode= ACCEPT_MODE_EXPLICIT;
subscrptn_setngs.acquireMode= ACQUIRE_MODE_PRE_ACQUIRED;
// Subscribe to fanout via local queue
SubscriptionManager subscriptions(session);
subscriptions.subscribe(local_my_queue, my_queue_name,
subscrptn_setngs, my_queue_name);
// Get a message
producer_msg = local_my_queue.get();
// end codes
On Fri, 29 May 2009, Ted Ross wrote:
> Hi Greg,
>
> Thanks for sharing this report. Your detailed results are interesting,
> especially since this is a level of benchmarking that we haven't gotten to
> ourselves.
>
> I'll take a whack at some of your questions in-line below. Hopefully others
> will chime in as well.
>
> First or all, I'm curious as to how you configured your federation routes.
> The paper details the physical broker topologies but doesn't mention which
> qpid-route options were used to set up your test networks. This, of course,
> has performance implications.
>
> Regards,
>
> -Ted
>
> gregory james marsh wrote:
>>
>> Hi All,
>>
>> A few other lab members and myself have just completed an Ohio State
>> University Technical Report dealing with Qpid broker federation entitled
>> "Scaling Advanced Message Queuing Protocol (AMQP) Architecture with Broker
>> Federation and InfiniBand". The pdf is publicly available at
>> http://nowlab.cse.ohio-state.edu/projects/amqp/ as well as
>> ftp://ftp.cse.ohio-state.edu/pub/tech-report/TRList.html
>>
>> Basically this report tests federation with k-nomial tree topology. We
>> were unable to use Qpid rdma/InfiniBand (IB) due to problem documented on
>> JIRA (QPID-1855). So we performed our experiments using IPoIB via Qpid TCP
>> connector.
>>
>> Based on what we saw in our experiments, each of us has written some
>> questions for the list about how federation is designed & implemented
>> internally. I've compiled the questions below. I make reference to
>> figures and tables from the report where appropriate.
>>
>>
>> 1. In detail how does the connection of a destination broker to a source
>> broker with "qpid-route route add" differ from a consumer application
>> declaring and binding a queue on a source broker? Are there functional
>> differences in how messages are routed to a federated destination broker as
>> opposed to a consumer? What anticipated affect would these differences
>> have on host system performance?
> In case you haven't already seen it, there's a Wiki page on federation at
> http://qpid.apache.org/using-broker-federation.html.
>
> In Qpid federation, the destination broker behaves as a normal AMQP client to
> the source broker. With "qpid-route route add" and "qpid-route dynamic add",
> the destination broker declares a private, auto-delete queue on the source
> broker and creates a subscription to that queue. The "route add" command
> causes the queue to be bound explicitly to an exchange on the source broker.
> The "dynamic add" does not directly bind the queue but dynamically creates
> and deletes bindings as consumer applications add bindings to the destination
> exchange.
>
> Federation does not introduce any differences in the way messages are
> forwarded. It simply uses the normal AMQP mechanisms. The "internal"
> federation client exposes less capability than is offered by the Qpid client
> APIs. In particular, delivery policy and flow control. These are issues
> that will be addressed in future development.
>>
>> 2. Why does a chain of federated brokers to one consumer have a higher
>> bytes/sec rate on their interfaces (using SYSSTAT/sar to collect traffic
>> data on IPoIB interface) than a single broker to one consumer? This
>> experiment is explained in Sections 3.1, 4.2, Figure 8a.
> There are lots of variables here. The text and diagrams in the paper don't
> provide enough detail to definitively answer this question. It's possible
> that a different delivery policy on your clients (vs. the federation links)
> is a cause of the disparity. In M4, the federation links are synchronous
> (i.e. ack per message). In 0.5 (the next release), there's an option for
> batching acknowledgements on federation links.
>
> If your test clients use more efficient asynchronous transfers, the
> federation links will have higher acknowledgement traffic than will the
> normal client links (in M4).
>>
>> 3. Why does the activity at the root broker and producer increase as we
>> increase the federation tree? Our understanding was that the only work
>> root broker/producer does is send messages to broker(s) they are directly
>> connected to. (This is similar to question 2, but different experiment. In
>> such situations the net interfaces on the producers and consumers (as well
>> as brokers) of more complex trees showed elevatated bytes/sec rates
>> compared to less complex trees. Experiment is explained in Sections 3.3,
>> 4.4, Figure 14a.)
> I'd be interested to see exactly how you configured your federation routes in
> this scenario.
>>
>> 4. We observed that the tree P-B-4B-16C was performing better than the
>> tree P-B-4B-4C. And I thought this was because more consumers being
>> attached keeps the leaf brokers busy thus reducing the contention at the
>> root broker. Can we confirm if this understanding is correct. (This is a
>> comparison of the experiments performed in Sections 3.3/4.4 and 5. Compare
>> numbers in Tables 5, 6, 7, 8, particularly the small message rate numbers
>> in Tables 5 & 8 for 64B and 256B.)
>>
>> 5. What is the advised measurement for performance in federated
>> broker-consumer design?
>> (We used the benchmarks we created for our SC08 workshop paper last
>> fall--also available at above sites. Delivered bandwidth in the current
>> report is our benchmark bandwidth multiplied by number of consumers).
>>
>> 6. Which of the tests included with Qpid release might help us get more
>> insights on our topology study?
>>
>>
>> Thanks in advance for an comments and insights.
>>
>> Greg Marsh
>> Network Based Computing Lab
>> Ohio State University
>>
>>
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Technical Report on Qpid Broker Federation Experiments
Posted by gregory james marsh <ma...@cse.ohio-state.edu>.
Hi Ted,
Thanks for taking a look at the report and the comments.
I've included highlights of my M4 publisher, broker fed, and consumer codes
below. In brief my publishers are using async transfers, broker fed is using
qpid-route add (with tcp -t default to use ipoib), and consumers are using
LocalQueues. The consumer's subscription settings are below and we use these
because after trial and error we found they were giving us best peformance.
Regards,
Greg
/* IP ADDRS */
root_broker= ip address 1
level1_broker1= ip address 2
/* PUBLISHER CODE */
Connection connection;
ConnectionSettings conn_setngs;
Message message;
Session session;
// Setup the session
conn_setngs.host= root_broker;
conn_setngs.protocol= "tcp";
conn_setngs.tcpNoDelay= true;
connection.open(conn_setngs);
session = connection.newSession();
// Async msg xfer
async(session).messageTransfer(arg::content=message,
arg::destination="amq.fanout");
/* BROKER FEDERATION */
qpid-route route add $level1_broker1 $root_broker amq.fanout no_routing_key
.....and so on to form tree structures
/* CONSUMER CODE */
Connection connection;
ConnectionSettings conn_setngs;
Message message;
Session session;
LocalQueue local_my_queue;
SubscriptionSettings subscrptn_setngs;
// Setup the session
conn_setngs.host= level1_broker1;
conn_setngs.protocol= "tcp";
conn_setngs.tcpNoDelay= true;
connection.open(conn_setngs);
session = connection.newSession();
// Bind a queue
my_queue_name=session.getId().getName();
session.queueDeclare(arg::queue=my_queue_name, arg::exclusive=true,
arg::autoDelete=true);
session.exchangeBind(arg::exchange="amq.fanout",
arg::queue=my_queue_name,
arg::bindingKey="my-key");
// Set subscription settings
subscrptn_setngs.flowControl= FlowControl::unlimited();
subscrptn_setngs.autoAck= 1;
// Accept and acquire modes defined in ./src/gen/qpid/framing/enum.h
subscrptn_setngs.acceptMode= ACCEPT_MODE_EXPLICIT;
subscrptn_setngs.acquireMode= ACQUIRE_MODE_PRE_ACQUIRED;
// Subscribe to fanout via local queue
SubscriptionManager subscriptions(session);
subscriptions.subscribe(local_my_queue, my_queue_name,
subscrptn_setngs, my_queue_name);
// Get a message
producer_msg = local_my_queue.get();
// end codes
On Fri, 29 May 2009, Ted Ross wrote:
> Hi Greg,
>
> Thanks for sharing this report. Your detailed results are interesting,
> especially since this is a level of benchmarking that we haven't gotten to
> ourselves.
>
> I'll take a whack at some of your questions in-line below. Hopefully others
> will chime in as well.
>
> First or all, I'm curious as to how you configured your federation routes.
> The paper details the physical broker topologies but doesn't mention which
> qpid-route options were used to set up your test networks. This, of course,
> has performance implications.
>
> Regards,
>
> -Ted
>
> gregory james marsh wrote:
>>
>> Hi All,
>>
>> A few other lab members and myself have just completed an Ohio State
>> University Technical Report dealing with Qpid broker federation entitled
>> "Scaling Advanced Message Queuing Protocol (AMQP) Architecture with Broker
>> Federation and InfiniBand". The pdf is publicly available at
>> http://nowlab.cse.ohio-state.edu/projects/amqp/ as well as
>> ftp://ftp.cse.ohio-state.edu/pub/tech-report/TRList.html
>>
>> Basically this report tests federation with k-nomial tree topology. We
>> were unable to use Qpid rdma/InfiniBand (IB) due to problem documented on
>> JIRA (QPID-1855). So we performed our experiments using IPoIB via Qpid TCP
>> connector.
>>
>> Based on what we saw in our experiments, each of us has written some
>> questions for the list about how federation is designed & implemented
>> internally. I've compiled the questions below. I make reference to
>> figures and tables from the report where appropriate.
>>
>>
>> 1. In detail how does the connection of a destination broker to a source
>> broker with "qpid-route route add" differ from a consumer application
>> declaring and binding a queue on a source broker? Are there functional
>> differences in how messages are routed to a federated destination broker as
>> opposed to a consumer? What anticipated affect would these differences
>> have on host system performance?
> In case you haven't already seen it, there's a Wiki page on federation at
> http://qpid.apache.org/using-broker-federation.html.
>
> In Qpid federation, the destination broker behaves as a normal AMQP client to
> the source broker. With "qpid-route route add" and "qpid-route dynamic add",
> the destination broker declares a private, auto-delete queue on the source
> broker and creates a subscription to that queue. The "route add" command
> causes the queue to be bound explicitly to an exchange on the source broker.
> The "dynamic add" does not directly bind the queue but dynamically creates
> and deletes bindings as consumer applications add bindings to the destination
> exchange.
>
> Federation does not introduce any differences in the way messages are
> forwarded. It simply uses the normal AMQP mechanisms. The "internal"
> federation client exposes less capability than is offered by the Qpid client
> APIs. In particular, delivery policy and flow control. These are issues
> that will be addressed in future development.
>>
>> 2. Why does a chain of federated brokers to one consumer have a higher
>> bytes/sec rate on their interfaces (using SYSSTAT/sar to collect traffic
>> data on IPoIB interface) than a single broker to one consumer? This
>> experiment is explained in Sections 3.1, 4.2, Figure 8a.
> There are lots of variables here. The text and diagrams in the paper don't
> provide enough detail to definitively answer this question. It's possible
> that a different delivery policy on your clients (vs. the federation links)
> is a cause of the disparity. In M4, the federation links are synchronous
> (i.e. ack per message). In 0.5 (the next release), there's an option for
> batching acknowledgements on federation links.
>
> If your test clients use more efficient asynchronous transfers, the
> federation links will have higher acknowledgement traffic than will the
> normal client links (in M4).
>>
>> 3. Why does the activity at the root broker and producer increase as we
>> increase the federation tree? Our understanding was that the only work
>> root broker/producer does is send messages to broker(s) they are directly
>> connected to. (This is similar to question 2, but different experiment. In
>> such situations the net interfaces on the producers and consumers (as well
>> as brokers) of more complex trees showed elevatated bytes/sec rates
>> compared to less complex trees. Experiment is explained in Sections 3.3,
>> 4.4, Figure 14a.)
> I'd be interested to see exactly how you configured your federation routes in
> this scenario.
>>
>> 4. We observed that the tree P-B-4B-16C was performing better than the
>> tree P-B-4B-4C. And I thought this was because more consumers being
>> attached keeps the leaf brokers busy thus reducing the contention at the
>> root broker. Can we confirm if this understanding is correct. (This is a
>> comparison of the experiments performed in Sections 3.3/4.4 and 5. Compare
>> numbers in Tables 5, 6, 7, 8, particularly the small message rate numbers
>> in Tables 5 & 8 for 64B and 256B.)
>>
>> 5. What is the advised measurement for performance in federated
>> broker-consumer design?
>> (We used the benchmarks we created for our SC08 workshop paper last
>> fall--also available at above sites. Delivered bandwidth in the current
>> report is our benchmark bandwidth multiplied by number of consumers).
>>
>> 6. Which of the tests included with Qpid release might help us get more
>> insights on our topology study?
>>
>>
>> Thanks in advance for an comments and insights.
>>
>> Greg Marsh
>> Network Based Computing Lab
>> Ohio State University
>>
>>
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Technical Report on Qpid Broker Federation Experiments
Posted by Ted Ross <tr...@redhat.com>.
Hi Greg,
Thanks for sharing this report. Your detailed results are interesting,
especially since this is a level of benchmarking that we haven't gotten
to ourselves.
I'll take a whack at some of your questions in-line below. Hopefully
others will chime in as well.
First or all, I'm curious as to how you configured your federation
routes. The paper details the physical broker topologies but doesn't
mention which qpid-route options were used to set up your test
networks. This, of course, has performance implications.
Regards,
-Ted
gregory james marsh wrote:
>
> Hi All,
>
> A few other lab members and myself have just completed an Ohio State
> University Technical Report dealing with Qpid broker federation
> entitled "Scaling Advanced Message Queuing Protocol (AMQP)
> Architecture with Broker Federation and InfiniBand". The pdf is
> publicly available at http://nowlab.cse.ohio-state.edu/projects/amqp/
> as well as ftp://ftp.cse.ohio-state.edu/pub/tech-report/TRList.html
>
> Basically this report tests federation with k-nomial tree topology.
> We were unable to use Qpid rdma/InfiniBand (IB) due to problem
> documented on JIRA (QPID-1855). So we performed our experiments using
> IPoIB via Qpid TCP connector.
>
> Based on what we saw in our experiments, each of us has written some
> questions for the list about how federation is designed & implemented
> internally. I've compiled the questions below. I make reference to
> figures and tables from the report where appropriate.
>
>
> 1. In detail how does the connection of a destination broker to a
> source broker with "qpid-route route add" differ from a consumer
> application declaring and binding a queue on a source broker? Are
> there functional differences in how messages are routed to a federated
> destination broker as opposed to a consumer? What anticipated affect
> would these differences have on host system performance?
In case you haven't already seen it, there's a Wiki page on federation
at http://qpid.apache.org/using-broker-federation.html.
In Qpid federation, the destination broker behaves as a normal AMQP
client to the source broker. With "qpid-route route add" and
"qpid-route dynamic add", the destination broker declares a private,
auto-delete queue on the source broker and creates a subscription to
that queue. The "route add" command causes the queue to be bound
explicitly to an exchange on the source broker. The "dynamic add" does
not directly bind the queue but dynamically creates and deletes bindings
as consumer applications add bindings to the destination exchange.
Federation does not introduce any differences in the way messages are
forwarded. It simply uses the normal AMQP mechanisms. The "internal"
federation client exposes less capability than is offered by the Qpid
client APIs. In particular, delivery policy and flow control. These
are issues that will be addressed in future development.
>
> 2. Why does a chain of federated brokers to one consumer have a higher
> bytes/sec rate on their interfaces (using SYSSTAT/sar to collect
> traffic data on IPoIB interface) than a single broker to one
> consumer? This experiment is explained in Sections 3.1, 4.2, Figure 8a.
There are lots of variables here. The text and diagrams in the paper
don't provide enough detail to definitively answer this question. It's
possible that a different delivery policy on your clients (vs. the
federation links) is a cause of the disparity. In M4, the federation
links are synchronous (i.e. ack per message). In 0.5 (the next
release), there's an option for batching acknowledgements on federation
links.
If your test clients use more efficient asynchronous transfers, the
federation links will have higher acknowledgement traffic than will the
normal client links (in M4).
>
> 3. Why does the activity at the root broker and producer increase as
> we increase the federation tree? Our understanding was that the only
> work root broker/producer does is send messages to broker(s) they are
> directly connected to. (This is similar to question 2, but different
> experiment. In such situations the net interfaces on the producers and
> consumers (as well as brokers) of more complex trees showed elevatated
> bytes/sec rates compared to less complex trees. Experiment is
> explained in Sections 3.3, 4.4, Figure 14a.)
I'd be interested to see exactly how you configured your federation
routes in this scenario.
>
> 4. We observed that the tree P-B-4B-16C was performing better than
> the tree P-B-4B-4C. And I thought this was because more consumers
> being attached keeps the leaf brokers busy thus reducing the
> contention at the root broker. Can we confirm if this understanding is
> correct. (This is a comparison of the experiments performed in
> Sections 3.3/4.4 and 5. Compare numbers in Tables 5, 6, 7, 8,
> particularly the small message rate numbers in Tables 5 & 8 for 64B
> and 256B.)
>
> 5. What is the advised measurement for performance in federated
> broker-consumer design?
> (We used the benchmarks we created for our SC08 workshop paper last
> fall--also available at above sites. Delivered bandwidth in the
> current report is our benchmark bandwidth multiplied by number of
> consumers).
>
> 6. Which of the tests included with Qpid release might help us get
> more insights on our topology study?
>
>
> Thanks in advance for an comments and insights.
>
> Greg Marsh
> Network Based Computing Lab
> Ohio State University
>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Technical Report on Qpid Broker Federation Experiments
Posted by Ted Ross <tr...@redhat.com>.
Hi Greg,
Thanks for sharing this report. Your detailed results are interesting,
especially since this is a level of benchmarking that we haven't gotten
to ourselves.
I'll take a whack at some of your questions in-line below. Hopefully
others will chime in as well.
First or all, I'm curious as to how you configured your federation
routes. The paper details the physical broker topologies but doesn't
mention which qpid-route options were used to set up your test
networks. This, of course, has performance implications.
Regards,
-Ted
gregory james marsh wrote:
>
> Hi All,
>
> A few other lab members and myself have just completed an Ohio State
> University Technical Report dealing with Qpid broker federation
> entitled "Scaling Advanced Message Queuing Protocol (AMQP)
> Architecture with Broker Federation and InfiniBand". The pdf is
> publicly available at http://nowlab.cse.ohio-state.edu/projects/amqp/
> as well as ftp://ftp.cse.ohio-state.edu/pub/tech-report/TRList.html
>
> Basically this report tests federation with k-nomial tree topology.
> We were unable to use Qpid rdma/InfiniBand (IB) due to problem
> documented on JIRA (QPID-1855). So we performed our experiments using
> IPoIB via Qpid TCP connector.
>
> Based on what we saw in our experiments, each of us has written some
> questions for the list about how federation is designed & implemented
> internally. I've compiled the questions below. I make reference to
> figures and tables from the report where appropriate.
>
>
> 1. In detail how does the connection of a destination broker to a
> source broker with "qpid-route route add" differ from a consumer
> application declaring and binding a queue on a source broker? Are
> there functional differences in how messages are routed to a federated
> destination broker as opposed to a consumer? What anticipated affect
> would these differences have on host system performance?
In case you haven't already seen it, there's a Wiki page on federation
at http://qpid.apache.org/using-broker-federation.html.
In Qpid federation, the destination broker behaves as a normal AMQP
client to the source broker. With "qpid-route route add" and
"qpid-route dynamic add", the destination broker declares a private,
auto-delete queue on the source broker and creates a subscription to
that queue. The "route add" command causes the queue to be bound
explicitly to an exchange on the source broker. The "dynamic add" does
not directly bind the queue but dynamically creates and deletes bindings
as consumer applications add bindings to the destination exchange.
Federation does not introduce any differences in the way messages are
forwarded. It simply uses the normal AMQP mechanisms. The "internal"
federation client exposes less capability than is offered by the Qpid
client APIs. In particular, delivery policy and flow control. These
are issues that will be addressed in future development.
>
> 2. Why does a chain of federated brokers to one consumer have a higher
> bytes/sec rate on their interfaces (using SYSSTAT/sar to collect
> traffic data on IPoIB interface) than a single broker to one
> consumer? This experiment is explained in Sections 3.1, 4.2, Figure 8a.
There are lots of variables here. The text and diagrams in the paper
don't provide enough detail to definitively answer this question. It's
possible that a different delivery policy on your clients (vs. the
federation links) is a cause of the disparity. In M4, the federation
links are synchronous (i.e. ack per message). In 0.5 (the next
release), there's an option for batching acknowledgements on federation
links.
If your test clients use more efficient asynchronous transfers, the
federation links will have higher acknowledgement traffic than will the
normal client links (in M4).
>
> 3. Why does the activity at the root broker and producer increase as
> we increase the federation tree? Our understanding was that the only
> work root broker/producer does is send messages to broker(s) they are
> directly connected to. (This is similar to question 2, but different
> experiment. In such situations the net interfaces on the producers and
> consumers (as well as brokers) of more complex trees showed elevatated
> bytes/sec rates compared to less complex trees. Experiment is
> explained in Sections 3.3, 4.4, Figure 14a.)
I'd be interested to see exactly how you configured your federation
routes in this scenario.
>
> 4. We observed that the tree P-B-4B-16C was performing better than
> the tree P-B-4B-4C. And I thought this was because more consumers
> being attached keeps the leaf brokers busy thus reducing the
> contention at the root broker. Can we confirm if this understanding is
> correct. (This is a comparison of the experiments performed in
> Sections 3.3/4.4 and 5. Compare numbers in Tables 5, 6, 7, 8,
> particularly the small message rate numbers in Tables 5 & 8 for 64B
> and 256B.)
>
> 5. What is the advised measurement for performance in federated
> broker-consumer design?
> (We used the benchmarks we created for our SC08 workshop paper last
> fall--also available at above sites. Delivered bandwidth in the
> current report is our benchmark bandwidth multiplied by number of
> consumers).
>
> 6. Which of the tests included with Qpid release might help us get
> more insights on our topology study?
>
>
> Thanks in advance for an comments and insights.
>
> Greg Marsh
> Network Based Computing Lab
> Ohio State University
>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org