Re: Technical Report on Qpid Broker Federation Experiments

[gregory james marsh <ma...@cse.ohio-state.edu>]

Hi Ted,

Thanks for taking a look at the report and the comments.

I've included highlights of my M4 publisher, broker fed, and consumer codes
below.  In brief my publishers are using async transfers, broker fed is using
qpid-route add (with tcp -t default to use ipoib), and consumers are using
LocalQueues.  The consumer's subscription settings are below and we use these
because after trial and error we found they were giving us best peformance.

Regards,

Greg



/* IP ADDRS */
root_broker= ip address 1
level1_broker1= ip address 2




/* PUBLISHER CODE */
Connection connection;
ConnectionSettings conn_setngs;
Message message;
Session session;

// Setup the session
conn_setngs.host= root_broker;
conn_setngs.protocol= "tcp";
conn_setngs.tcpNoDelay= true;
connection.open(conn_setngs);
session = connection.newSession();

// Async msg xfer
async(session).messageTransfer(arg::content=message,
   arg::destination="amq.fanout");




/* BROKER FEDERATION */
qpid-route route add $level1_broker1 $root_broker amq.fanout no_routing_key
.....and so on to form tree structures




/* CONSUMER CODE */
Connection connection;
ConnectionSettings conn_setngs;
Message message;
Session session;
LocalQueue local_my_queue;
SubscriptionSettings subscrptn_setngs;

// Setup the session
conn_setngs.host= level1_broker1;
conn_setngs.protocol= "tcp";
conn_setngs.tcpNoDelay= true;
connection.open(conn_setngs);
session = connection.newSession();

// Bind a queue
my_queue_name=session.getId().getName();
session.queueDeclare(arg::queue=my_queue_name, arg::exclusive=true,
                     arg::autoDelete=true);
session.exchangeBind(arg::exchange="amq.fanout",
                     arg::queue=my_queue_name,
                     arg::bindingKey="my-key");

// Set subscription settings
subscrptn_setngs.flowControl= FlowControl::unlimited();
subscrptn_setngs.autoAck= 1;
// Accept and acquire modes defined in ./src/gen/qpid/framing/enum.h
subscrptn_setngs.acceptMode= ACCEPT_MODE_EXPLICIT;
subscrptn_setngs.acquireMode= ACQUIRE_MODE_PRE_ACQUIRED;

// Subscribe to fanout via local queue
SubscriptionManager subscriptions(session);
subscriptions.subscribe(local_my_queue, my_queue_name,
                         subscrptn_setngs, my_queue_name);

// Get a message
producer_msg = local_my_queue.get();

// end codes




On Fri, 29 May 2009, Ted Ross wrote:

> Hi Greg,
>
> Thanks for sharing this report.  Your detailed results are interesting, 
> especially since this is a level of benchmarking that we haven't gotten to 
> ourselves.
>
> I'll take a whack at some of your questions in-line below.  Hopefully others 
> will chime in as well.
>
> First or all, I'm curious as to how you configured your federation routes. 
> The paper details the physical broker topologies but doesn't mention which 
> qpid-route options were used to set up your test networks.  This, of course, 
> has performance implications.
>
> Regards,
>
> -Ted
>
> gregory james marsh wrote:
>> 
>> Hi All,
>> 
>> A few other lab members and myself have just completed an Ohio State 
>> University Technical Report dealing with Qpid broker federation entitled 
>> "Scaling Advanced Message Queuing Protocol (AMQP) Architecture with Broker 
>> Federation and InfiniBand".  The pdf is publicly available at 
>> http://nowlab.cse.ohio-state.edu/projects/amqp/ as well as 
>> ftp://ftp.cse.ohio-state.edu/pub/tech-report/TRList.html
>> 
>> Basically this report tests federation with k-nomial tree topology.  We 
>> were unable to use Qpid rdma/InfiniBand (IB) due to problem documented on 
>> JIRA (QPID-1855).  So we performed our experiments using IPoIB via Qpid TCP 
>> connector.
>> 
>> Based on what we saw in our experiments, each of us has written some 
>> questions for the list about how federation is designed & implemented 
>> internally.  I've compiled the questions below.  I make reference to 
>> figures and tables from the report where appropriate.
>> 
>> 
>> 1. In detail how does the connection of a destination broker to a source 
>> broker with "qpid-route route add" differ from a consumer application 
>> declaring and binding a queue on a source broker?  Are there functional 
>> differences in how messages are routed to a federated destination broker as 
>> opposed to a consumer?  What anticipated affect would these differences 
>> have on host system performance?
> In case you haven't already seen it, there's a Wiki page on federation at 
> http://qpid.apache.org/using-broker-federation.html.
>
> In Qpid federation, the destination broker behaves as a normal AMQP client to 
> the source broker.  With "qpid-route route add" and "qpid-route dynamic add", 
> the destination broker declares a private, auto-delete queue on the source 
> broker and creates a subscription to that queue.  The "route add" command 
> causes the queue to be bound explicitly to an exchange on the source broker. 
> The "dynamic add" does not directly bind the queue but dynamically creates 
> and deletes bindings as consumer applications add bindings to the destination 
> exchange.
>
> Federation does not introduce any differences in the way messages are 
> forwarded.  It simply uses the normal AMQP mechanisms.  The "internal" 
> federation client exposes less capability than is offered by the Qpid client 
> APIs.  In particular, delivery policy and flow control.  These are issues 
> that will be addressed in future development.
>> 
>> 2. Why does a chain of federated brokers to one consumer have a higher 
>> bytes/sec rate on their interfaces (using SYSSTAT/sar to collect traffic 
>> data on IPoIB interface) than a single broker to one consumer?  This 
>> experiment is explained in Sections 3.1, 4.2, Figure 8a.
> There are lots of variables here.  The text and diagrams in the paper don't 
> provide enough detail to definitively answer this question.  It's possible 
> that a different delivery policy on your clients (vs. the federation links) 
> is a cause of the disparity.  In M4, the federation links are synchronous 
> (i.e. ack per message).  In 0.5 (the next release), there's an option for 
> batching acknowledgements on federation links.
>
> If your test clients use more efficient asynchronous transfers, the 
> federation links will have higher acknowledgement traffic than will the 
> normal client links (in M4).
>> 
>> 3. Why does the activity at the root broker and producer increase as we 
>> increase the federation tree?  Our understanding was that the only work 
>> root broker/producer does is send messages to broker(s) they are directly 
>> connected to.  (This is similar to question 2, but different experiment. In 
>> such situations the net interfaces on the producers and consumers (as well 
>> as brokers) of more complex trees showed elevatated bytes/sec rates 
>> compared to less complex trees.  Experiment is explained in Sections 3.3, 
>> 4.4, Figure 14a.)
> I'd be interested to see exactly how you configured your federation routes in 
> this scenario.
>> 
>> 4.  We observed that the tree P-B-4B-16C was performing better than the 
>> tree P-B-4B-4C. And I thought this was because more consumers being 
>> attached keeps the leaf brokers busy thus reducing the contention at the 
>> root broker. Can we confirm if this understanding is correct.  (This is a 
>> comparison of the experiments performed in Sections 3.3/4.4 and 5. Compare 
>> numbers in Tables 5, 6, 7, 8, particularly the small message rate numbers 
>> in Tables 5 & 8 for 64B and 256B.)
>> 
>> 5.  What is the advised measurement for performance in federated 
>> broker-consumer design?
>> (We used the benchmarks we created for our SC08 workshop paper last 
>> fall--also available at above sites.  Delivered bandwidth in the current 
>> report is our benchmark bandwidth multiplied by number of consumers).
>> 
>> 6.  Which of the tests included with Qpid release might help us get more 
>> insights on our topology study?
>> 
>> 
>> Thanks in advance for an comments and insights.
>> 
>> Greg Marsh
>> Network Based Computing Lab
>> Ohio State University
>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project:      http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>> 
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org