Potentially related issues between QPID threads

[boundabout <bo...@hushmail.com>]

We have observed two issues, possibly with a common cause, with threading in
qpid 0.18.

Issue 1 - Using the C++ broker on an IBM M4, performance is very poor (and
CPU utilisation very high) unless the size of the thread pool is reduced by
setting worker-threads to a small number. I believe the default is to create
one more thread than the number of logical cores, which for this box is 24.
So the question here is why having a lot of threads could increase CPU
loading. I wondered if it might be that threads were being moved between
cores, causing a lot of cache invalidation, but setting hard affinity
indicates that this is not the case.

Issue 2 - A client process in C++ on Linux RHEL 5.8 creates a number of
listener threads, using the messaging interface, all listening to the same
on-box broker. Each listener has its own connection, session and receiver.
It is observed that each listener gets another thread created in the qpid
library. The listeners are configured, and messages sent to the broker, such
that some will receive data, and some will not. CPU loading for the active
and dormant qpid threads increases with the number of threads. This is not
the case when listeners in separate processes are used.

It seems, in both cases, as if threads somehow have a dependency on each
other, such that for each working thread it has to do some work relating to
each other thread, and these other threads somehow get some work to do due
to the first. This seems pretty weird to me, I'd be grateful if anyone could
shed any light and, even better, suggest a way of avoiding this effect.



--
View this message in context: http://qpid.2158936.n2.nabble.com/Potentially-related-issues-between-QPID-threads-tp7595682.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Potentially related issues between QPID threads

[Andrew Stitcher <as...@redhat.com>]

On Fri, 2013-07-19 at 00:29 -0700, boundabout wrote:
> We have observed two issues, possibly with a common cause, with threading in
> qpid 0.18.
> 
> Issue 1 - Using the C++ broker on an IBM M4, performance is very poor (and
> CPU utilisation very high) unless the size of the thread pool is reduced by
> setting worker-threads to a small number. I believe the default is to create
> one more thread than the number of logical cores, which for this box is 24.
> So the question here is why having a lot of threads could increase CPU
> loading. I wondered if it might be that threads were being moved between
> cores, causing a lot of cache invalidation, but setting hard affinity
> indicates that this is not the case.

One issue that we have observed is that running on multiple sockets
doesn't give such high performance as running on a single socket. Some
of the best performance can be got by setting the process/thread
affinity so that you are using the usual n+1 rule but limiting the
threads to run on the CPUs in a single socket.

My presumption here is that the memory hierarchy matters a lot and that
invalidating 2nd/3rd level caches messes up performance.

As to the client issue you are having - I don't think it is especially
related to the broker issue, but there may be a general resemblance if
you are using a lot of connections/Listeners.

Andrew



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Potentially related issues between QPID threads

[Fraser Adams <fr...@blueyonder.co.uk>]

Hello Mr boundabout,
I had a go at reproducing this. I decided to use the headers exchange 
'cause that's what I tend to use mostly myself.

I'm running with a single producer with a sender capacity of 100 sending 
1500 byte messages with headers set thus:
     message.setProperty("data-service", utf8("amqp-delivery"));
     message.setProperty("data-type", utf8("audio"));

I've got consumers with headers bindings that match either:
data-service: amqp-delivery, data-type: audio
for consumers that I want to actually match the messages I'm sending and:
data-service: amqp-delivery, data-type: video
For consumers that I want to run up, but ignore the messages being sent 
by the producer.

I've set link: {reliability: unreliable} in the Address and 
{tcp-nodelay: true} for connectionOptions to try to eke out extra 
performance.


I hope that's a reasonable facsimile of your test case? I've attached my 
test code (caveat it's quite hacky - my C++ is pretty hideously rusty :-()

I've tried using separate Processes (ItemProducer.cpp/ItemConsumer.cpp) 
and threads (Threads.cpp) The Threads programme takes command line 
arguments Usage: ./Threads [matchCount (default = 1)] [missCount 
(default = 0)] [shared]

So doing:
./Threads 3 2 shared
Will start up one producer and 5 consumers with 3 matching the sent 
items and 2 not matching in this case the consumers will share the same 
Connection (if shared is omitted each consumer will use its own Connection).



Unfortunately I've not been able to reproduce what you're seeing in the 
consumers at all, however I don't have the luxury of an M4 server with 
24 logical cores and all I've got to run things on is my Core 2 Duo 
laptop with two P7450 cores running at 2.13GHz (the vastly different 
number of cores *might well* be significant to the different observations).

Here's what I've been seeing:
Using separate processes:

1 Producer 1 Consumer:
5435 messages/second observed on the queue enqueue/dequeue rate, qpidd@87%

1 Producer 2 Consumers (both receiving):
3491 messages/second observed on each queue enqueue/dequeue rate, qpidd@92%

1 Producer 3 Consumers (all receiving):
2679 messages/second observed on each queue enqueue/dequeue rate, qpidd@95%

1 Producer 5 Consumers (3 receiving, 2 not matching):
2387 messages/second observed on each matching queue enqueue/dequeue 
rate, qpidd@99%
0 messages/second observed on each non-matching queue enqueue/dequeue rate



Using Threads (with separate connection for each consumer):

1 Producer 1 Consumer:
5667 messages/second observed on the queue enqueue/dequeue rate, 
qpidd@94%, Threads@97%

1 Producer 2 Consumers (both receiving):
3721 messages/second observed on each queue enqueue/dequeue rate, 
qpidd@100%, Threads@90%

1 Producer 3 Consumers (all receiving):
2735 messages/second observed on each queue enqueue/dequeue rate, 
qpidd@103%, Threads@84%

1 Producer 5 Consumers (3 receiving, 2 not matching):
2433 messages/second observed on each matching queue enqueue/dequeue 
rate, qpidd@110%, Threads@78%
0 messages/second observed on each non-matching queue enqueue/dequeue rate


Using Threads (with shared connection):

1 Producer 1 Consumer:
5675 messages/second observed on the queue enqueue/dequeue rate, 
qpidd@94%, Threads@98%

1 Producer 2 Consumers (both receiving):
3840 messages/second observed on each queue enqueue/dequeue rate, 
qpidd@97%, Threads@91%

1 Producer 3 Consumers (all receiving):
2987 messages/second observed on each queue enqueue/dequeue rate, 
qpidd@101%, Threads@88%

1 Producer 5 Consumers (3 receiving, 2 not matching):
2637 messages/second observed on each matching queue enqueue/dequeue 
rate, qpidd@104%, Threads@76%
0 messages/second observed on each non-matching queue enqueue/dequeue rate



So I'm seeing that using Threads is giving consistently higher figures 
than Processes and the shared Connection giving higher figures than 
using a separate Connection for each consumer.

I also tried
1 Producer 13 Consumers (3 receiving, 10 not matching):
1768 messages/second observed on each matching queue enqueue/dequeue 
rate, qpidd@108%, Threads@56%
0 messages/second observed on each non-matching queue enqueue/dequeue rate



This last figure seems slightly interesting, so as I increase the number 
of *non-matching* consumers but keep the number of matching consumers 
constant the overall CPU utilisation of the Threads program drops, so 
even though they aren't receiving data it looks like there's some form 
of thread "contention" (I get similar with shared and non-shared queues).

I don't know that C++ qpid::messaging code at all really, but my guess 
is that there's likely an internal thread handling the prefetch into the 
client runtime receiver queues (sized by setCapacity) and likely some 
mutex or other synchronisation between that and the receiver.get() call 
- the more things synchronising to that blocking queue the more wait 
cycles. But that's all just a guess I'm afraid.


One other potentially interesting observation, so you say "It is 
observed that each listener gets another thread created in the qpid 
library "

I don't see that at all. I ran up "top -H" to see the threads being 
created and I consistently see a number of Threads equal to 
num-receiving-consumers + 1 (for the producer thread) + 2 (probably 
internal qpid client threads)
e.g. running ./Threads 5 10 shared
I'm seeing 8 Threads in other words the threads for the non-matching 
consumers are essentially idle and not registered by top.


As I say I've only got a two core box and I don't know the 
qpid::messaging code at all really, but if you are indeed observing each 
listener getting another thread created on a 24 core box *I wonder* if 
there's a mechanism in the qpid client run-time that's allocating 
internal worker threads up to some limit based on the number of cores? 
The broker certainly does that....


Given that I'm observing the overall CPU utilisation dropping as I 
increase the number of non-matching threads I wonder if you are seeing 
lots of worker threads getting spawned which is creating even more 
synchronisation contention than I'm seeing, but that's *really* just a 
guess.


Although it's just a guess it's at least vaguely plausible given your 
observation for Issue 1 where you say reducing the broker thread pool 
size to something smaller that 24 actually increases throughput.


Unfortunately though if the client runtime is actually creating a worker 
thread pool based on the number of cores I've no idea how to change this 
size. I know with the broker it's a simple case of doing 
--worker-threads N but I've no idea at all with the client :-(


I'm not sure that I've really helped any, perhaps you'd be able to try 
my test programs to see if they are able to reproduce (or otherwise) the 
sort of issues you are seeing, at the very least if my test program 
yields the same results that you are seeing on your system it rules out 
an application issue.

Perhaps one of the qpid devs with access to a fancier box than me might 
be able to try some of this stuff against a few more cores.


My tests were run against a build off trunk (qpid 0.25) I did on 20th 
July I don't know how materially different that is to 0.18 in terms of 
thread behaviour.

Best regards,
Frase

On 19/07/13 08:29, boundabout wrote:
> We have observed two issues, possibly with a common cause, with threading in
> qpid 0.18.
>
> Issue 1 - Using the C++ broker on an IBM M4, performance is very poor (and
> CPU utilisation very high) unless the size of the thread pool is reduced by
> setting worker-threads to a small number. I believe the default is to create
> one more thread than the number of logical cores, which for this box is 24.
> So the question here is why having a lot of threads could increase CPU
> loading. I wondered if it might be that threads were being moved between
> cores, causing a lot of cache invalidation, but setting hard affinity
> indicates that this is not the case.
>
> Issue 2 - A client process in C++ on Linux RHEL 5.8 creates a number of
> listener threads, using the messaging interface, all listening to the same
> on-box broker. Each listener has its own connection, session and receiver.
> It is observed that each listener gets another thread created in the qpid
> library. The listeners are configured, and messages sent to the broker, such
> that some will receive data, and some will not. CPU loading for the active
> and dormant qpid threads increases with the number of threads. This is not
> the case when listeners in separate processes are used.
>
> It seems, in both cases, as if threads somehow have a dependency on each
> other, such that for each working thread it has to do some work relating to
> each other thread, and these other threads somehow get some work to do due
> to the first. This seems pretty weird to me, I'd be grateful if anyone could
> shed any light and, even better, suggest a way of avoiding this effect.
>
>
>
> --
> View this message in context: http://qpid.2158936.n2.nabble.com/Potentially-related-issues-between-QPID-threads-tp7595682.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>