You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Andy Li <an...@gmail.com> on 2009/10/29 19:06:33 UTC
Flow control behavior of fanout exchange
Hi,
We ran into strange differences of flow control behavior of fanout exchange.
We use configuration with one publisher and 10 subscribers.
1)On some machines, publisher throughput will be higher than subscriber
throughput, and eventually the queue will overflow and throw an exception.
This is, as I understand, the expected behavior - publisher has entire 1
Gbit bandwidth to itself, whereas subscribers share it, so they have lower
theoretical limit. We've coded a simple flow control scheme to manage this.
2)However, on other machines we installed the broker, publisher and each
subscriber throughput seem to move in sync. There is little buffering in the
queue, and the overflow never happens. While the lack of overflow is
convenient, bandwidth utilization is 2-4 times lower compared to broker
machines exibiting behavior (1). In (1), combined output bandwidth utilized
by subscribers is 80-90% of 1Gbit available. In (2), it's 20-40%, depending
on the machine and latency to client.
All broker machines use the same OS version and broker 0.5 binaries with
default broker config. We can't figure out what is causing this difference
in behavior. We did seem to induce behavior (2) on machines (1) by running
unrelated workload concurrently that strongly competed for bandwidth.
Has anyone else run into the same issue? Or has any idea about what might
cause this? Any help or suggestions would be greatly appreciated.
Thanks,
Andy
Client Details:
The publisher and subscriber clients are the same in both cases and run on a
separate machine. We're using asynchronous send and sync() every 1000
messages. Listeners use ACCEPT_MODE_NONE and FlowControl::unlimited().
Message size is random, linearly distributed between 500-1500 bytes.
RE: C++ broker setup help
Posted by Steve Huston <sh...@riverace.com>.
Hi Wielly,
> Wielly wrote:
> > Dear Developers,
> >
> > I am writing to enquiry information regarding how to setup
> C++ broker under
> > windows environment. I am looking for a contact person who
> can help me on
> > giving me some guidance regarding this matter. I will
> highly appreciate for
> > your help.
The users list is the place to ask. There are a number of people
familiar with Qpid on Windows here.
> > I have downloaded windows installer /0.5-windows/qpidc-0.5.msi
from
> > http://qpid.apache.org/download.html.
> >
> > My OS is Windows XP. After I install the software, the
> following folders
> > were created for me.
> >
> >
> > C:\Program Files\Apache\qpidc-0.5:
> >
> >
> >
> > \bin
> > \docs
> > \examples
> > \include
> > \srcdist
> >
> >
> > For my current project, I need to do 2 features:
> Persistence storage and
> > SSL.
Neither of those features are available for Windows in Qpid 0.5.
Persistence will be in 0.6. There's a chance that SSL for the Qpid
client will be in 0.6 but that's very uncertain.
If you want to try the persistence that will be in 0.6, you will need
to check out the current source from Qpid's svn repository, install
cmake, python, and ruby, as well as Visual Studio 2008. For
persistence you will need Microsoft SQL Express or SQL Server. Be
aware there are still some known bugs in the persistence code that are
being actively worked for 0.6.
Best regards,
-Steve
--
Steve Huston, Riverace Corporation
Total Lifecycle Support for Your Networked Applications
http://www.riverace.com
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: C++ broker setup help
Posted by Carl Trieloff <cc...@redhat.com>.
Wielly,
I have copied the user list, for details see :
http://qpid.apache.org/mailing-lists.html
Mailing the user list is the best way to get answers from the Qpid project.
Brief notes, is that you will need to use a trunk install to get
persistence support on Windows,
the 0.5 release on had a corresponding module for persistence for linux.
king regards
Carl.
Wielly wrote:
> Dear Developers,
>
> I am writing to enquiry information regarding how to setup C++ broker under
> windows environment. I am looking for a contact person who can help me on
> giving me some guidance regarding this matter. I will highly appreciate for
> your help.
>
> I have downloaded windows installer /0.5-windows/qpidc-0.5.msi from
> http://qpid.apache.org/download.html.
>
> My OS is Windows XP. After I install the software, the following folders
> were created for me.
>
>
> C:\Program Files\Apache\qpidc-0.5:
>
>
>
> \bin
> \docs
> \examples
> \include
> \srcdist
>
>
> For my current project, I need to do 2 features: Persistence storage and
> SSL. Since your source information provides more on how to setup those
> features under Linux environment. Based on information from the site you
> have provided, the persistence BDB and SSL can be set from a XML config
> file. I can not find this config file from my installation folder. Can you
> assist me on how to configure/ enable BDB persistence and SSL under windows
> envonment?
>
> Thanks and have a great day!
>
>
> Regards,
>
>
> Wielly
> Software Developer
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Flow control behavior of fanout exchange
Posted by Carl Trieloff <cc...@redhat.com>.
> Is there any particular reason why exception and session halt were chosen as
> the response to full buffer in QPID? We've used ActiveMQ before, and it has
> a host of correctness issues and lower performance, but they did seem to
> work out the publisher flow control pretty well for topics. I may be
> mistaken, but I think if the buffer is full in the broker they just rely on
> the TCP protocol itself in the OS to produce the backpressure and flow
> control in the publisher.
>
Such a scheme means that you need a large thread pool or the broker will
starve on CPU. Qpid has been
implemented to be async and process (not-block) and when it can't do so
return the thread. Thus there is
no thread to have back pressure on. this gives higher scalability,
better latency and CPU utilization. In some
of our test with the current trunk it can do about ~ 1.2 Million
msg/second on 4 modern cores, and client
to client latency of ~35us.
One thing to watch with your scheme is that session reject has a
sequence wrap around issue on 0.5, which is
fixed on trunk.
Carl.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Flow control behavior of fanout exchange
Posted by Andy Li <an...@gmail.com>.
Carl,
We are aware of the static global publisher flow control option. Thanks for
mentioning it to help us.
We were trying to implement a dynamic scheme that would not hold back the
best (and common) case scenario of single currently active publisher and
lower number of subscriptions, while preventing failure in a worse case
scenario. We pause publisher after overflow exception before restarting the
Session (but not long enough for the buffer to fully empty under any
conditions), and exponentially lower async send window size on repeated
overflows. We also utilize sequence number stamps per <sender, topic> to
discard the duplicate deliveries that inevitably occur in the receivers when
restarting from the last sync point before overflow. Those sequence stamps
could also help detect dropped messages, though we haven't seen any (cheers
QPID).
*Are there any known stability issues if there are recurring overflow
exceptions / publisher session restarts? Any memory/resource leaks?* So far,
broker seemed stable in our tests of this exception-driven flow control
scheme. We did run into OOM issues while running 10 concurrent fanout
exchanges, even though each one should only use 100MB buffer, and we had 4
GB of RAM. We haven't had a chance to investigate that in detail, but I
don't believe it was our flow control scheme, since the overflows were rare
in that configuration (single subscription per fanout exchange).
Is there any particular reason why exception and session halt were chosen as
the response to full buffer in QPID? We've used ActiveMQ before, and it has
a host of correctness issues and lower performance, but they did seem to
work out the publisher flow control pretty well for topics. I may be
mistaken, but I think if the buffer is full in the broker they just rely on
the TCP protocol itself in the OS to produce the backpressure and flow
control in the publisher.
Thanks,
Andy
Re: durable messaging
Posted by Ian kinkade <ki...@idi-middleware.com>.
Hi Bill,
Looks like the persistence store is implemented using a DB:
http://qpid.et.redhat.com/download.html
I am still looking for meaningful documentation.
Best Regards .............. Ian
--
Ian Kinkade
CEO
Information Design, Inc.
145 Durham Road, Suite 11
Madison, CT 06443 USA
URL: www.idi-middleware.com
Email: kinkade@idi-middleware.com
Work: 203-245-0772 Ext: 6212
Fax: 203-245-1885
Cell: 203-589-1192
On 3/7/2010 3:46 PM, Bill Whiting wrote:
> I'm trying to enable persistence. If I define an exchange and queue
> with the --durable option then the qpid-config command does not
> complain, but the objects are not re-created when I re-cycle the broker
>
> If I run the broker manually with
> qpidd --data-dir /var/lib/qpidd -t
> then the broker notes that persistence is not enabled. Is there a web
> page or documentation that describes enabling the store for persistent
> messaging?
>
> //Bill
>
> On 11/05/2009 08:31 AM, Carl Trieloff wrote:
>>
>>
>> Mike,
>>
>> The issue is that with some high core count machines with multi
>> socket, a few things can go wrong. It
>> starts with some of the value add the hardware may do by using a
>> feature called SMIs. These are hardware
>> interrupts that stop the CPUs, then load some code into the CPU to do
>> management stuff or ECC checks, power
>> (green computing etc...). The bad side is that they 'stop' all the
>> CPUs on the machine. we have plotted SMIs
>> up to 1-2ms on some machines. My employer has worked with quite of a
>> few hardware suppliers to certify
>> a bunch of machines (remove SMIs for Realtime). note in many cases
>> the SMI don't impact applications, i.e.
>> in Java the effects of the GC are larger.
>>
>> Few other things that go on, NUMA is on a per socket basis, if you
>> run multi socket with high core count
>> and the CPU load is not high enough for the scheduler to keep the CPU
>> locality then you can have expensive
>> memory access and cache effects come into play also getting less
>> effective locking. If you are RHEL5.4 I can
>> provide some setting which will give you NUMA aware memory
>> allocation, which can increase throughput up
>> to 75%, and improve latency about 25% for NUMA machines.
>>
>> Thus the quick experiment of setting the worker-threads = to the
>> cores on one socket, increases the CPU a little for
>> those threads, and the probability of scheduling off core going down.
>> This then 'removes some of the hardware effects'
>>
>> Obviously if it is run on a SMI free or SMI re profiled machine (the
>> fast ones you noted have little or no SMIs)
>> and numactl & things like cpuspeed are set then the more powerful
>> machine will beat the slower one. But in this case the
>> faster machine is getting in it's own way.
>>
>> Carl.
>>
>> Mike D.. wrote:
>>> Hi,
>>> This "matched flow" behavior is quite interesting and luckily we
>>> have not
>>> experienced it when prototyping on our developer machines.
>>>
>>> Would you mind explain a bit Carl why this would happen and what's your
>>> suggestion to user of QPID? Soon we will test the proof of concept
>>> on our
>>> servers as well. How can we have QPID utilizing both CPUs (8
>>> processors)?
>>>
>>>
>>> thanks,
>>> mike
>>>
>>>
>>>
>>>
>>> Carl Trieloff wrote:
>>>>
>>>> I mailed you the deck directly.
>>>>
>>>> Carl.
>>>>
>>>>
>>>> Andy Li wrote:
>>>>> Carl,
>>>>>
>>>>> Yes, reducing the number of worker threads from #cores + 1 to 4
>>>>> did switch the data center machines to behavior (1). Looks like
>>>>> you've diagnosed the issue!
>>>>>
>>>>> Unfortunately, I couldn't find a copy of your talk at HPC anywhere
>>>>> on the web. It's listed on their website, but no files posted.
>>>>>
>>>>> Thanks,
>>>>> Andy
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ok, I think I know what might be going on, I believe it is
>>>>> hardware related -- Take a look at the
>>>>> presentation I did with Lee Fisher at HPC on Wall Street.
>>>>>
>>>>> Anyway, try the following and lets see if we can alter the
>>>>> behaviour. On the data centre machines
>>>>> run qpidd with --workerthreads 4
>>>>>
>>>>> If that alters the results i'll expand my theory and how to
>>>>> resolve the hardware side.
>>>>> Carl.
>>>>>
>>>>
>>>
>>
>>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: durable messaging
Posted by Bill Whiting <te...@bellsouth.net>.
Kim,
Thanks, that fixed my problem.
//Bill
On 03/08/2010 07:20 AM, Kim van der Riet wrote:
> On Sun, 2010-03-07 at 15:46 -0500, Bill Whiting wrote:
>
>> I'm trying to enable persistence. If I define an exchange and queue
>> with the --durable option then the qpid-config command does not
>> complain, but the objects are not re-created when I re-cycle the broker
>>
>> If I run the broker manually with
>> qpidd --data-dir /var/lib/qpidd -t
>> then the broker notes that persistence is not enabled. Is there a web
>> page or documentation that describes enabling the store for persistent
>> messaging?
>>
>> //Bill
>>
>>
> Make sure you have the store module loaded, use the --load-module
> path/to/msgstore.so when starting the broker (if it is not already in
> your config file):
>
> $ ./qpidd --load-module /path/to/msgstore.so --auth no --log-enable notice+
> 2010-03-08 07:16:48 notice Journal "TplStore": Created
> 2010-03-08 07:16:48 notice Store module initialized; store-dir=/home/username/.qpidd
> 2010-03-08 07:16:48 notice SASL disabled: No Authentication Performed
> 2010-03-08 07:16:48 notice Listening on TCP port 5672
> 2010-03-08 07:16:48 notice Broker running
>
> Note the first two lines in the log.
>
> http://cwiki.apache.org/confluence/display/qpid/FAQ#FAQ-Persistence
>
> I agree that it would be helpful if the qpid-config program would warn
> if persistence is not loaded and you try to use persistence features.
>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: durable messaging
Posted by Kim van der Riet <ki...@redhat.com>.
On Sun, 2010-03-07 at 15:46 -0500, Bill Whiting wrote:
> I'm trying to enable persistence. If I define an exchange and queue
> with the --durable option then the qpid-config command does not
> complain, but the objects are not re-created when I re-cycle the broker
>
> If I run the broker manually with
> qpidd --data-dir /var/lib/qpidd -t
> then the broker notes that persistence is not enabled. Is there a web
> page or documentation that describes enabling the store for persistent
> messaging?
>
> //Bill
>
Make sure you have the store module loaded, use the --load-module
path/to/msgstore.so when starting the broker (if it is not already in
your config file):
$ ./qpidd --load-module /path/to/msgstore.so --auth no --log-enable notice+
2010-03-08 07:16:48 notice Journal "TplStore": Created
2010-03-08 07:16:48 notice Store module initialized; store-dir=/home/username/.qpidd
2010-03-08 07:16:48 notice SASL disabled: No Authentication Performed
2010-03-08 07:16:48 notice Listening on TCP port 5672
2010-03-08 07:16:48 notice Broker running
Note the first two lines in the log.
http://cwiki.apache.org/confluence/display/qpid/FAQ#FAQ-Persistence
I agree that it would be helpful if the qpid-config program would warn
if persistence is not loaded and you try to use persistence features.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
durable messaging
Posted by Bill Whiting <te...@bellsouth.net>.
I'm trying to enable persistence. If I define an exchange and queue
with the --durable option then the qpid-config command does not
complain, but the objects are not re-created when I re-cycle the broker
If I run the broker manually with
qpidd --data-dir /var/lib/qpidd -t
then the broker notes that persistence is not enabled. Is there a web
page or documentation that describes enabling the store for persistent
messaging?
//Bill
On 11/05/2009 08:31 AM, Carl Trieloff wrote:
>
>
> Mike,
>
> The issue is that with some high core count machines with multi
> socket, a few things can go wrong. It
> starts with some of the value add the hardware may do by using a
> feature called SMIs. These are hardware
> interrupts that stop the CPUs, then load some code into the CPU to do
> management stuff or ECC checks, power
> (green computing etc...). The bad side is that they 'stop' all the
> CPUs on the machine. we have plotted SMIs
> up to 1-2ms on some machines. My employer has worked with quite of a
> few hardware suppliers to certify
> a bunch of machines (remove SMIs for Realtime). note in many cases the
> SMI don't impact applications, i.e.
> in Java the effects of the GC are larger.
>
> Few other things that go on, NUMA is on a per socket basis, if you run
> multi socket with high core count
> and the CPU load is not high enough for the scheduler to keep the CPU
> locality then you can have expensive
> memory access and cache effects come into play also getting less
> effective locking. If you are RHEL5.4 I can
> provide some setting which will give you NUMA aware memory
> allocation, which can increase throughput up
> to 75%, and improve latency about 25% for NUMA machines.
>
> Thus the quick experiment of setting the worker-threads = to the cores
> on one socket, increases the CPU a little for
> those threads, and the probability of scheduling off core going down.
> This then 'removes some of the hardware effects'
>
> Obviously if it is run on a SMI free or SMI re profiled machine (the
> fast ones you noted have little or no SMIs)
> and numactl & things like cpuspeed are set then the more powerful
> machine will beat the slower one. But in this case the
> faster machine is getting in it's own way.
>
> Carl.
>
> Mike D.. wrote:
>> Hi,
>> This "matched flow" behavior is quite interesting and luckily we have
>> not
>> experienced it when prototyping on our developer machines.
>>
>> Would you mind explain a bit Carl why this would happen and what's your
>> suggestion to user of QPID? Soon we will test the proof of concept on
>> our
>> servers as well. How can we have QPID utilizing both CPUs (8
>> processors)?
>>
>>
>> thanks,
>> mike
>>
>>
>>
>>
>> Carl Trieloff wrote:
>>>
>>> I mailed you the deck directly.
>>>
>>> Carl.
>>>
>>>
>>> Andy Li wrote:
>>>> Carl,
>>>>
>>>> Yes, reducing the number of worker threads from #cores + 1 to 4 did
>>>> switch the data center machines to behavior (1). Looks like you've
>>>> diagnosed the issue!
>>>>
>>>> Unfortunately, I couldn't find a copy of your talk at HPC anywhere
>>>> on the web. It's listed on their website, but no files posted.
>>>>
>>>> Thanks,
>>>> Andy
>>>>
>>>>
>>>>
>>>>
>>>> ok, I think I know what might be going on, I believe it is
>>>> hardware related -- Take a look at the
>>>> presentation I did with Lee Fisher at HPC on Wall Street.
>>>>
>>>> Anyway, try the following and lets see if we can alter the
>>>> behaviour. On the data centre machines
>>>> run qpidd with --workerthreads 4
>>>>
>>>> If that alters the results i'll expand my theory and how to
>>>> resolve the hardware side.
>>>> Carl.
>>>>
>>>
>>
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Flow control behavior of fanout exchange
Posted by Carl Trieloff <cc...@redhat.com>.
Mike,
The issue is that with some high core count machines with multi socket,
a few things can go wrong. It
starts with some of the value add the hardware may do by using a feature
called SMIs. These are hardware
interrupts that stop the CPUs, then load some code into the CPU to do
management stuff or ECC checks, power
(green computing etc...). The bad side is that they 'stop' all the CPUs
on the machine. we have plotted SMIs
up to 1-2ms on some machines. My employer has worked with quite of a few
hardware suppliers to certify
a bunch of machines (remove SMIs for Realtime). note in many cases the
SMI don't impact applications, i.e.
in Java the effects of the GC are larger.
Few other things that go on, NUMA is on a per socket basis, if you run
multi socket with high core count
and the CPU load is not high enough for the scheduler to keep the CPU
locality then you can have expensive
memory access and cache effects come into play also getting less
effective locking. If you are RHEL5.4 I can
provide some setting which will give you NUMA aware memory allocation,
which can increase throughput up
to 75%, and improve latency about 25% for NUMA machines.
Thus the quick experiment of setting the worker-threads = to the cores
on one socket, increases the CPU a little for
those threads, and the probability of scheduling off core going down.
This then 'removes some of the hardware effects'
Obviously if it is run on a SMI free or SMI re profiled machine (the
fast ones you noted have little or no SMIs)
and numactl & things like cpuspeed are set then the more powerful
machine will beat the slower one. But in this case the
faster machine is getting in it's own way.
Carl.
Mike D.. wrote:
> Hi,
> This "matched flow" behavior is quite interesting and luckily we have not
> experienced it when prototyping on our developer machines.
>
> Would you mind explain a bit Carl why this would happen and what's your
> suggestion to user of QPID? Soon we will test the proof of concept on our
> servers as well. How can we have QPID utilizing both CPUs (8 processors)?
>
>
> thanks,
> mike
>
>
>
>
> Carl Trieloff wrote:
>
>>
>> I mailed you the deck directly.
>>
>> Carl.
>>
>>
>> Andy Li wrote:
>>
>>> Carl,
>>>
>>> Yes, reducing the number of worker threads from #cores + 1 to 4 did
>>> switch the data center machines to behavior (1). Looks like you've
>>> diagnosed the issue!
>>>
>>> Unfortunately, I couldn't find a copy of your talk at HPC anywhere on
>>> the web. It's listed on their website, but no files posted.
>>>
>>> Thanks,
>>> Andy
>>>
>>>
>>>
>>>
>>> ok, I think I know what might be going on, I believe it is
>>> hardware related -- Take a look at the
>>> presentation I did with Lee Fisher at HPC on Wall Street.
>>>
>>> Anyway, try the following and lets see if we can alter the
>>> behaviour. On the data centre machines
>>> run qpidd with --workerthreads 4
>>>
>>> If that alters the results i'll expand my theory and how to
>>> resolve the hardware side.
>>> Carl.
>>>
>>>
>>
>>
>
>
Re: Flow control behavior of fanout exchange
Posted by "Mike D.." <mi...@hotmail.com>.
Hi,
This "matched flow" behavior is quite interesting and luckily we have not
experienced it when prototyping on our developer machines.
Would you mind explain a bit Carl why this would happen and what's your
suggestion to user of QPID? Soon we will test the proof of concept on our
servers as well. How can we have QPID utilizing both CPUs (8 processors)?
thanks,
mike
Carl Trieloff wrote:
>
>
>
> I mailed you the deck directly.
>
> Carl.
>
>
> Andy Li wrote:
>> Carl,
>>
>> Yes, reducing the number of worker threads from #cores + 1 to 4 did
>> switch the data center machines to behavior (1). Looks like you've
>> diagnosed the issue!
>>
>> Unfortunately, I couldn't find a copy of your talk at HPC anywhere on
>> the web. It's listed on their website, but no files posted.
>>
>> Thanks,
>> Andy
>>
>>
>>
>>
>> ok, I think I know what might be going on, I believe it is
>> hardware related -- Take a look at the
>> presentation I did with Lee Fisher at HPC on Wall Street.
>>
>> Anyway, try the following and lets see if we can alter the
>> behaviour. On the data centre machines
>> run qpidd with --workerthreads 4
>>
>> If that alters the results i'll expand my theory and how to
>> resolve the hardware side.
>> Carl.
>>
>
>
>
--
View this message in context: http://n2.nabble.com/Flow-control-behavior-of-fanout-exchange-tp3914564p3948592.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Flow control behavior of fanout exchange
Posted by Carl Trieloff <cc...@redhat.com>.
I mailed you the deck directly.
Carl.
Andy Li wrote:
> Carl,
>
> Yes, reducing the number of worker threads from #cores + 1 to 4 did
> switch the data center machines to behavior (1). Looks like you've
> diagnosed the issue!
>
> Unfortunately, I couldn't find a copy of your talk at HPC anywhere on
> the web. It's listed on their website, but no files posted.
>
> Thanks,
> Andy
>
>
>
>
> ok, I think I know what might be going on, I believe it is
> hardware related -- Take a look at the
> presentation I did with Lee Fisher at HPC on Wall Street.
>
> Anyway, try the following and lets see if we can alter the
> behaviour. On the data centre machines
> run qpidd with --workerthreads 4
>
> If that alters the results i'll expand my theory and how to
> resolve the hardware side.
> Carl.
>
Re: Flow control behavior of fanout exchange
Posted by Andy Li <an...@gmail.com>.
Carl,
Yes, reducing the number of worker threads from #cores + 1 to 4 did switch
the data center machines to behavior (1). Looks like you've diagnosed the
issue!
Unfortunately, I couldn't find a copy of your talk at HPC anywhere on the
web. It's listed on their website, but no files posted.
Thanks,
Andy
>
> ok, I think I know what might be going on, I believe it is hardware related
> -- Take a look at the
> presentation I did with Lee Fisher at HPC on Wall Street.
>
> Anyway, try the following and lets see if we can alter the behaviour. On
> the data centre machines
> run qpidd with --workerthreads 4
>
> If that alters the results i'll expand my theory and how to resolve the
> hardware side.
> Carl.
>
>
Re: Flow control behavior of fanout exchange
Posted by Carl Trieloff <cc...@redhat.com>.
ok, I think I know what might be going on, I believe it is hardware
related -- Take a look at the
presentation I did with Lee Fisher at HPC on Wall Street.
Anyway, try the following and lets see if we can alter the behaviour. On
the data centre machines
run qpidd with --workerthreads 4
If that alters the results i'll expand my theory and how to resolve the
hardware side.
Carl.
Andy Li wrote:
> Hi Robert and Carl,
>
> Thank you for your response.
>
> The behavior does not seem to depend on the network topology, or the client
> machine (as long as it
> is distinct from the broker machine). It depends on the broker machine. In
> fact, "fast" (1)
> brokers show higher performance even over a higher latency link to the
> client, than the "slow" (2)
> brokers with client sitting on the same switch. Publisher and subscribers
> ran together on the same
> machine in most tests, but we've tried running them separately today and it
> didn't make a
> difference.
>
> Details:
> --------------
>
> We tested broker on 4 distinct physical machines (A,B,X,Y). Each has a
> single 1 Gbit Ethernet
> interface. Machines A and B are on the same switch with very low latency
> between them. Same for
> X,Y. Between (A,B) and (X,Y) there is an intra-city WAN link with capacity
> well in excess of 1 Gbit and
> latency of .5-1 ms (sync send throughput at about 700-800 msg/sec). For
> both types of links, the
> capacity is symmetrical. Machines A and B have fast (1) broker behavior; X
> and Y - slower (2). Let A->B mean "clients on A, broker on B".
>
> A->X and A->Y exhibit slow, synced pub/sub throughput (2). Even over low
> latency link, X->Y and
> Y->X exhibit (2). Vice versa, even over high latency link, X->A and X->B
> have fast, out-op-sync
> pub/sub behavior (1). A->B is a little bit faster than X->B and X->A, but
> the difference in
> throughput is nowhere near the big disparity between (1) and (2). Also, the
> qualitative behavior
> (publisher and subscriber throughput out of sync) is the same regardless of
> whether the client is
> separated by WAN or single switch.
>
> Machine B (one of the "fast" brokers") has OpenSuse 11.1. All other machines
> have have CentOS 5.
>
> The "fast" broker machines actually have weaker hardware:
>
> Machines X and Y ("slow" brokers, behavior (2))
> Datacenter app servers
> CPU: Dual Intel Xeon E5440 2.83 GHz (8 cores); FSB 1333MHz; L2 Cache 12MB
> RAM: 8GB
> Single 1Gbit NIC
>
>
> Machines A and B ("fast" brokers, behavior (1))
> Dell Optiplex 755 workstations
> CPU: Intel Core 2 Quad Q6600 @ 2.4 GHz (4 cores); FSB 1066MHz; L2 Cache 8MB
> RAM: 4GB
> Single 1Gbit NIC
>
> Thanks,
> Andy
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Flow control behavior of fanout exchange
Posted by Andy Li <an...@gmail.com>.
Hi Robert and Carl,
Thank you for your response.
The behavior does not seem to depend on the network topology, or the client
machine (as long as it
is distinct from the broker machine). It depends on the broker machine. In
fact, "fast" (1)
brokers show higher performance even over a higher latency link to the
client, than the "slow" (2)
brokers with client sitting on the same switch. Publisher and subscribers
ran together on the same
machine in most tests, but we've tried running them separately today and it
didn't make a
difference.
Details:
--------------
We tested broker on 4 distinct physical machines (A,B,X,Y). Each has a
single 1 Gbit Ethernet
interface. Machines A and B are on the same switch with very low latency
between them. Same for
X,Y. Between (A,B) and (X,Y) there is an intra-city WAN link with capacity
well in excess of 1 Gbit and
latency of .5-1 ms (sync send throughput at about 700-800 msg/sec). For
both types of links, the
capacity is symmetrical. Machines A and B have fast (1) broker behavior; X
and Y - slower (2). Let A->B mean "clients on A, broker on B".
A->X and A->Y exhibit slow, synced pub/sub throughput (2). Even over low
latency link, X->Y and
Y->X exhibit (2). Vice versa, even over high latency link, X->A and X->B
have fast, out-op-sync
pub/sub behavior (1). A->B is a little bit faster than X->B and X->A, but
the difference in
throughput is nowhere near the big disparity between (1) and (2). Also, the
qualitative behavior
(publisher and subscriber throughput out of sync) is the same regardless of
whether the client is
separated by WAN or single switch.
Machine B (one of the "fast" brokers") has OpenSuse 11.1. All other machines
have have CentOS 5.
The "fast" broker machines actually have weaker hardware:
Machines X and Y ("slow" brokers, behavior (2))
Datacenter app servers
CPU: Dual Intel Xeon E5440 2.83 GHz (8 cores); FSB 1333MHz; L2 Cache 12MB
RAM: 8GB
Single 1Gbit NIC
Machines A and B ("fast" brokers, behavior (1))
Dell Optiplex 755 workstations
CPU: Intel Core 2 Quad Q6600 @ 2.4 GHz (4 cores); FSB 1066MHz; L2 Cache 8MB
RAM: 4GB
Single 1Gbit NIC
Thanks,
Andy
Re: Flow control behavior of fanout exchange
Posted by Andy Li <an...@gmail.com>.
Aside from our particulars, I just wanted to confirm if anyone ever
experienced such "matched flow"
behavior in the broker? As I understand, there is nothing in QPID to enforce
the publisher to be as
slow as the subscribers. Of course, if the publisher is slow for some other
reason, the (faster)
subscriptions can do no better than match its speed. However, this doesn't
explain why publisher
speed drops down "naturally" in the "slow" brokers if we run the test with
higher number of
subscriptions.
Thanks,
Andy
Re: Flow control behavior of fanout exchange
Posted by Carl Trieloff <cc...@redhat.com>.
Andy Li wrote:
> Hi,
>
> We ran into strange differences of flow control behavior of fanout exchange.
> We use configuration with one publisher and 10 subscribers.
>
> 1)On some machines, publisher throughput will be higher than subscriber
> throughput, and eventually the queue will overflow and throw an exception.
> This is, as I understand, the expected behavior - publisher has entire 1
> Gbit bandwidth to itself, whereas subscribers share it, so they have lower
> theoretical limit. We've coded a simple flow control scheme to manage this.
>
FYI, the broker also has a prublisher max rate config options which may
help you.
> 2)However, on other machines we installed the broker, publisher and each
> subscriber throughput seem to move in sync. There is little buffering in the
> queue, and the overflow never happens. While the lack of overflow is
> convenient, bandwidth utilization is 2-4 times lower compared to broker
> machines exibiting behavior (1). In (1), combined output bandwidth utilized
> by subscribers is 80-90% of 1Gbit available. In (2), it's 20-40%, depending
> on the machine and latency to client.
>
> All broker machines use the same OS version and broker 0.5 binaries with
> default broker config. We can't figure out what is causing this difference
> in behavior. We did seem to induce behavior (2) on machines (1) by running
> unrelated workload concurrently that strongly competed for bandwidth.
Should like hardware or network, or it cold be things like cpuspeed
running on one machine and
not the other etc... Any more details on network, machines etc. That is
where I would start looking.
Carl.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Flow control behavior of fanout exchange
Posted by Robert Greig <ro...@gmail.com>.
It's quite an obvious point but I'll ask it anyway - are there
differences in the network topology between the nodes exhibiting the
different behaviour? e.g. switches, NICs?
Which OS are you using?
I'm not entirely clear about the topology you are describing - is the
publisher on a separate machine from the consumers?
Robert
2009/10/29 Andy Li <an...@gmail.com>:
> Hi,
>
> We ran into strange differences of flow control behavior of fanout exchange.
> We use configuration with one publisher and 10 subscribers.
>
> 1)On some machines, publisher throughput will be higher than subscriber
> throughput, and eventually the queue will overflow and throw an exception.
> This is, as I understand, the expected behavior - publisher has entire 1
> Gbit bandwidth to itself, whereas subscribers share it, so they have lower
> theoretical limit. We've coded a simple flow control scheme to manage this.
>
> 2)However, on other machines we installed the broker, publisher and each
> subscriber throughput seem to move in sync. There is little buffering in the
> queue, and the overflow never happens. While the lack of overflow is
> convenient, bandwidth utilization is 2-4 times lower compared to broker
> machines exibiting behavior (1). In (1), combined output bandwidth utilized
> by subscribers is 80-90% of 1Gbit available. In (2), it's 20-40%, depending
> on the machine and latency to client.
>
> All broker machines use the same OS version and broker 0.5 binaries with
> default broker config. We can't figure out what is causing this difference
> in behavior. We did seem to induce behavior (2) on machines (1) by running
> unrelated workload concurrently that strongly competed for bandwidth.
>
> Has anyone else run into the same issue? Or has any idea about what might
> cause this? Any help or suggestions would be greatly appreciated.
>
> Thanks,
> Andy
>
> Client Details:
>
> The publisher and subscriber clients are the same in both cases and run on a
> separate machine. We're using asynchronous send and sync() every 1000
> messages. Listeners use ACCEPT_MODE_NONE and FlowControl::unlimited().
> Message size is random, linearly distributed between 500-1500 bytes.
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org