You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by "GS.Chandra N" <gs...@gmail.com> on 2009/03/13 20:45:38 UTC
Subscriptions slows down broker (was Memory pile up on broker)
Gordon, Tedd
After the patches to the headersexchange.cpp was applied, i can now clearly
see the incoming rates with the help of the tool Tedd sent.
The rates are at about 90-100 odd messages per sec (a drop from 5200 odd
without subscriptions).
Sort of matches my earlier observations using qpid-tool. What next?
Do you think there might be any way to add the headers exchange performance
improvements to your list of priority fixes? I'm not on the dev forums so i
'm not sure if the issue is communicated out there.
I will try to test the multiple subscriptions XML exchange next to if that
improves matters in any respect.
Thanks for the great support in fully un-reavelling the issue.
Rgds
gs
ps : I can also still reproduce the memory pile up by blocking the broker
using qpid-tool and setting the mgmgt-pub interval to 1 which of-course i
can avoid by not letting qpid-tool stick around and increasing the mgmt-pub
interval to 10 or more.
On Thu, Mar 12, 2009 at 3:45 PM, GS.Chandra N <gs...@gmail.com>wrote:
> Hi,
>
> I found that if i kept the qpid-tool open long enough all my clients and
> publishers would throw exceptions and come out.
>
> Earlier when i was performing my tests i had 3 putty windows opened to all
> the 3 servers and it seems as if, i was keeping qpid-tool opened just long
> enough to cause build up but closing it to run top and so on a so forth such
> that the memory piled up but did not cause timeouts at the client either.
>
> Observations
>
> 1.When there are subscriptions and a qpid-tool around messages seem to pile
> up. Why is this so? *
> *
> 2. When there are no subscribers and no qpid-tool the publishing processes
> cause the CPu to go to 90% at the publishing box. But when i started up
> subscribers, the publishing box CPU fell to 0% and all the python processes
> piled up memory.
>
> Concern : I'm not able to find out here if the publishers are still able to
> send out messages at the speed that it should when subscriptions are around.
> Or perhaps the broker is not able to pull enough messages - iam not sure
> which of this is happeneing. Probably latter? How do i tell?
>
> If i use qpid-tool to connect to the broker at this point, every time i hit
> show exchange i get the same stats with the same time stamp. it looks like
> the broker is too busy trying to match subscriptions even to refresh the
> stats.
>
> 3.Concern : Once the subscriptions are made, the CPU at the broker box* *(high
> end dual core XEON with ht) goes to 90%. Even at this level, i'm not sure
> the broker is able to match and discard all the messages fast enough (due to
> 2nd observation). Or how do i tell?
>
>
> Thanks
> gs
>
> ps : Is there a sure shot way to find out the message rates ? (I currently
> use qpid-tool show exchange command to find the no of msgRecieves and divide
> by time-dfference from 2 times the command is run)
>
>
> On Thu, Mar 12, 2009 at 3:11 PM, GS.Chandra N <gs...@gmail.com>wrote:
>
>> I'm able to reproduce the issue, when i keep qpid-tool open all the time
>> with the mgmt switches.
>>
>> However i'm not able to do so if qpid-tool is not open (did not try with
>> qpid-tool and no mgmt switches).
>>
>> Thanks
>> gs
>>
>>
>> On Thu, Mar 12, 2009 at 1:31 AM, Gordon Sim <gs...@redhat.com> wrote:
>>
>>> GS.Chandra N wrote:
>>>
>>>> ps : Please find attached the scripts that create the issue
>>>>
>>>
>>> I'm afraid I wasn't able to recreate the issue with your tests. Memory
>>> stayed reasonable and constant as the test was running (5 servers, 5
>>> clients).
>>>
>>> I'm baffled at this point as to what it is you are seeing.
>>>
>>> broker - runs on a dedicated box
>>>> qpidd -d --port=5672 --mgmt-enable=yes --mgmt-pub-interval=1 --auth=no
>>>> --log-source=yes --log-to-file=/tmp/somename
>>>>
>>>
>>> Do you have qpid-tool or anything else running during the test? Does
>>> turning management off have any impact on the issue?
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> Apache Qpid - AMQP Messaging Implementation
>>> Project: http://qpid.apache.org
>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>
>>>
>>
>
Re: Subscriptions slows down broker (was Memory pile up on broker)
Posted by Carl Trieloff <cc...@redhat.com>.
attached, works with linux kernel with private-futex support (RHEL5.3 or
F10 etc). If you run
an older kernel without private-futexes shout and I will give you that
script.
Carl.
GS.Chandra N wrote:
> That ought to be interesting. I shall do the profiling and perhaps take a
> look at the code too. Please do send me the script.
>
> Thanks
> gs
>
> On Mon, Mar 16, 2009 at 6:09 AM, Carl Trieloff <cc...@redhat.com>wrote:
>
>
>> I don't believe any profiling has been done on the headers exchange. If you
>> are on Linux, run oprofile on it
>> and either create a patch to correct the hot spot or mail the profile to
>> the list.
>>
>> if needed I can also give mail you a stap (system tap) script for lock
>> analysis for the headers exchange.
>> regards,
>> Carl.
>>
>>
>>
>> GS.Chandra N wrote:
>>
>> Gordon, Tedd
>>
>> After the patches to the headersexchange.cpp was applied, i can now clearly
>> see the incoming rates with the help of the tool Tedd sent.
>>
>> The rates are at about 90-100 odd messages per sec (a drop from 5200 odd
>> without subscriptions).
>>
>> Sort of matches my earlier observations using qpid-tool. What next?
>>
>> Do you think there might be any way to add the headers exchange performance
>> improvements to your list of priority fixes? I'm not on the dev forums so i
>> 'm not sure if the issue is communicated out there.
>>
>> I will try to test the multiple subscriptions XML exchange next to if that
>> improves matters in any respect.
>>
>> Thanks for the great support in fully un-reavelling the issue.
>>
>> Rgds
>> gs
>>
>> ps : I can also still reproduce the memory pile up by blocking the broker
>> using qpid-tool and setting the mgmgt-pub interval to 1 which of-course i
>> can avoid by not letting qpid-tool stick around and increasing the mgmt-pub
>> interval to 10 or more.
>>
>> On Thu, Mar 12, 2009 at 3:45 PM, GS.Chandra N <gs...@gmail.com>wrote:
>>
>>
>>> Hi,
>>>
>>> I found that if i kept the qpid-tool open long enough all my clients and
>>> publishers would throw exceptions and come out.
>>>
>>> Earlier when i was performing my tests i had 3 putty windows opened to all
>>> the 3 servers and it seems as if, i was keeping qpid-tool opened just long
>>> enough to cause build up but closing it to run top and so on a so forth such
>>> that the memory piled up but did not cause timeouts at the client either.
>>>
>>> Observations
>>>
>>> 1.When there are subscriptions and a qpid-tool around messages seem to
>>> pile up. Why is this so? *
>>> *
>>> 2. When there are no subscribers and no qpid-tool the publishing processes
>>> cause the CPu to go to 90% at the publishing box. But when i started up
>>> subscribers, the publishing box CPU fell to 0% and all the python processes
>>> piled up memory.
>>>
>>> Concern : I'm not able to find out here if the publishers are still able
>>> to send out messages at the speed that it should when subscriptions are
>>> around. Or perhaps the broker is not able to pull enough messages - iam not
>>> sure which of this is happeneing. Probably latter? How do i tell?
>>>
>>> If i use qpid-tool to connect to the broker at this point, every time i
>>> hit show exchange i get the same stats with the same time stamp. it looks
>>> like the broker is too busy trying to match subscriptions even to refresh
>>> the stats.
>>>
>>> 3.Concern : Once the subscriptions are made, the CPU at the broker box* *(high
>>> end dual core XEON with ht) goes to 90%. Even at this level, i'm not sure
>>> the broker is able to match and discard all the messages fast enough (due to
>>> 2nd observation). Or how do i tell?
>>>
>>>
>>> Thanks
>>> gs
>>>
>>> ps : Is there a sure shot way to find out the message rates ? (I currently
>>> use qpid-tool show exchange command to find the no of msgRecieves and divide
>>> by time-dfference from 2 times the command is run)
>>>
>>> On Thu, Mar 12, 2009 at 3:11 PM, GS.Chandra N <gs...@gmail.com>wrote:
>>>
>>>
>>>> I'm able to reproduce the issue, when i keep qpid-tool open all the time
>>>> with the mgmt switches.
>>>>
>>>> However i'm not able to do so if qpid-tool is not open (did not try with
>>>> qpid-tool and no mgmt switches).
>>>>
>>>> Thanks
>>>> gs
>>>>
>>>> On Thu, Mar 12, 2009 at 1:31 AM, Gordon Sim <gs...@redhat.com> wrote:
>>>>
>>>>
>>>>> GS.Chandra N wrote:
>>>>>
>>>>>
>>>>>> ps : Please find attached the scripts that create the issue
>>>>>>
>>>>>>
>>>>> I'm afraid I wasn't able to recreate the issue with your tests. Memory
>>>>> stayed reasonable and constant as the test was running (5 servers, 5
>>>>> clients).
>>>>>
>>>>> I'm baffled at this point as to what it is you are seeing.
>>>>>
>>>>> broker - runs on a dedicated box
>>>>>
>>>>>> qpidd -d --port=5672 --mgmt-enable=yes --mgmt-pub-interval=1
>>>>>> --auth=no --log-source=yes --log-to-file=/tmp/somename
>>>>>>
>>>>>>
>>>>> Do you have qpid-tool or anything else running during the test? Does
>>>>> turning management off have any impact on the issue?
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> Apache Qpid - AMQP Messaging Implementation
>>>>> Project: http://qpid.apache.org
>>>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>>>
>>>>>
>>>>>
>>
>
>
Re: Subscriptions slows down broker (was Memory pile up on broker)
Posted by "GS.Chandra N" <gs...@gmail.com>.
That ought to be interesting. I shall do the profiling and perhaps take a
look at the code too. Please do send me the script.
Thanks
gs
On Mon, Mar 16, 2009 at 6:09 AM, Carl Trieloff <cc...@redhat.com>wrote:
>
> I don't believe any profiling has been done on the headers exchange. If you
> are on Linux, run oprofile on it
> and either create a patch to correct the hot spot or mail the profile to
> the list.
>
> if needed I can also give mail you a stap (system tap) script for lock
> analysis for the headers exchange.
> regards,
> Carl.
>
>
>
> GS.Chandra N wrote:
>
> Gordon, Tedd
>
> After the patches to the headersexchange.cpp was applied, i can now clearly
> see the incoming rates with the help of the tool Tedd sent.
>
> The rates are at about 90-100 odd messages per sec (a drop from 5200 odd
> without subscriptions).
>
> Sort of matches my earlier observations using qpid-tool. What next?
>
> Do you think there might be any way to add the headers exchange performance
> improvements to your list of priority fixes? I'm not on the dev forums so i
> 'm not sure if the issue is communicated out there.
>
> I will try to test the multiple subscriptions XML exchange next to if that
> improves matters in any respect.
>
> Thanks for the great support in fully un-reavelling the issue.
>
> Rgds
> gs
>
> ps : I can also still reproduce the memory pile up by blocking the broker
> using qpid-tool and setting the mgmgt-pub interval to 1 which of-course i
> can avoid by not letting qpid-tool stick around and increasing the mgmt-pub
> interval to 10 or more.
>
> On Thu, Mar 12, 2009 at 3:45 PM, GS.Chandra N <gs...@gmail.com>wrote:
>
>> Hi,
>>
>> I found that if i kept the qpid-tool open long enough all my clients and
>> publishers would throw exceptions and come out.
>>
>> Earlier when i was performing my tests i had 3 putty windows opened to all
>> the 3 servers and it seems as if, i was keeping qpid-tool opened just long
>> enough to cause build up but closing it to run top and so on a so forth such
>> that the memory piled up but did not cause timeouts at the client either.
>>
>> Observations
>>
>> 1.When there are subscriptions and a qpid-tool around messages seem to
>> pile up. Why is this so? *
>> *
>> 2. When there are no subscribers and no qpid-tool the publishing processes
>> cause the CPu to go to 90% at the publishing box. But when i started up
>> subscribers, the publishing box CPU fell to 0% and all the python processes
>> piled up memory.
>>
>> Concern : I'm not able to find out here if the publishers are still able
>> to send out messages at the speed that it should when subscriptions are
>> around. Or perhaps the broker is not able to pull enough messages - iam not
>> sure which of this is happeneing. Probably latter? How do i tell?
>>
>> If i use qpid-tool to connect to the broker at this point, every time i
>> hit show exchange i get the same stats with the same time stamp. it looks
>> like the broker is too busy trying to match subscriptions even to refresh
>> the stats.
>>
>> 3.Concern : Once the subscriptions are made, the CPU at the broker box* *(high
>> end dual core XEON with ht) goes to 90%. Even at this level, i'm not sure
>> the broker is able to match and discard all the messages fast enough (due to
>> 2nd observation). Or how do i tell?
>>
>>
>> Thanks
>> gs
>>
>> ps : Is there a sure shot way to find out the message rates ? (I currently
>> use qpid-tool show exchange command to find the no of msgRecieves and divide
>> by time-dfference from 2 times the command is run)
>>
>> On Thu, Mar 12, 2009 at 3:11 PM, GS.Chandra N <gs...@gmail.com>wrote:
>>
>>> I'm able to reproduce the issue, when i keep qpid-tool open all the time
>>> with the mgmt switches.
>>>
>>> However i'm not able to do so if qpid-tool is not open (did not try with
>>> qpid-tool and no mgmt switches).
>>>
>>> Thanks
>>> gs
>>>
>>> On Thu, Mar 12, 2009 at 1:31 AM, Gordon Sim <gs...@redhat.com> wrote:
>>>
>>>> GS.Chandra N wrote:
>>>>
>>>>> ps : Please find attached the scripts that create the issue
>>>>>
>>>>
>>>> I'm afraid I wasn't able to recreate the issue with your tests. Memory
>>>> stayed reasonable and constant as the test was running (5 servers, 5
>>>> clients).
>>>>
>>>> I'm baffled at this point as to what it is you are seeing.
>>>>
>>>> broker - runs on a dedicated box
>>>>> qpidd -d --port=5672 --mgmt-enable=yes --mgmt-pub-interval=1
>>>>> --auth=no --log-source=yes --log-to-file=/tmp/somename
>>>>>
>>>>
>>>> Do you have qpid-tool or anything else running during the test? Does
>>>> turning management off have any impact on the issue?
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> Apache Qpid - AMQP Messaging Implementation
>>>> Project: http://qpid.apache.org
>>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>>
>>>>
>>>
>>
>
>
Re: Subscriptions slows down broker (was Memory pile up on broker)
Posted by Carl Trieloff <cc...@redhat.com>.
I don't believe any profiling has been done on the headers exchange. If
you are on Linux, run oprofile on it
and either create a patch to correct the hot spot or mail the profile to
the list.
if needed I can also give mail you a stap (system tap) script for lock
analysis for the headers exchange.
regards,
Carl.
GS.Chandra N wrote:
> Gordon, Tedd
>
> After the patches to the headersexchange.cpp was applied, i can now
> clearly see the incoming rates with the help of the tool Tedd sent.
>
> The rates are at about 90-100 odd messages per sec (a drop from 5200
> odd without subscriptions).
>
> Sort of matches my earlier observations using qpid-tool. What next?
>
> Do you think there might be any way to add the headers exchange
> performance improvements to your list of priority fixes? I'm not on
> the dev forums so i 'm not sure if the issue is communicated out there.
>
> I will try to test the multiple subscriptions XML exchange next to if
> that improves matters in any respect.
>
> Thanks for the great support in fully un-reavelling the issue.
>
> Rgds
> gs
>
> ps : I can also still reproduce the memory pile up by blocking the
> broker using qpid-tool and setting the mgmgt-pub interval to 1 which
> of-course i can avoid by not letting qpid-tool stick around and
> increasing the mgmt-pub interval to 10 or more.
>
> On Thu, Mar 12, 2009 at 3:45 PM, GS.Chandra N <gs.chandran.n@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi,
>
> I found that if i kept the qpid-tool open long enough all my
> clients and publishers would throw exceptions and come out.
>
> Earlier when i was performing my tests i had 3 putty windows
> opened to all the 3 servers and it seems as if, i was keeping
> qpid-tool opened just long enough to cause build up but closing it
> to run top and so on a so forth such that the memory piled up but
> did not cause timeouts at the client either.
>
> Observations
>
> 1.When there are subscriptions and a qpid-tool around messages
> seem to pile up. Why is this so? *
> *
> 2. When there are no subscribers and no qpid-tool the publishing
> processes cause the CPu to go to 90% at the publishing box. But
> when i started up subscribers, the publishing box CPU fell to 0%
> and all the python processes piled up memory.
>
> Concern : I'm not able to find out here if the publishers are
> still able to send out messages at the speed that it should when
> subscriptions are around. Or perhaps the broker is not able to
> pull enough messages - iam not sure which of this is happeneing.
> Probably latter? How do i tell?
>
> If i use qpid-tool to connect to the broker at this point, every
> time i hit show exchange i get the same stats with the same time
> stamp. it looks like the broker is too busy trying to match
> subscriptions even to refresh the stats.
>
> 3.Concern : Once the subscriptions are made, the CPU at the broker
> box* *(high end dual core XEON with ht) goes to 90%. Even at this
> level, i'm not sure the broker is able to match and discard all
> the messages fast enough (due to 2nd observation). Or how do i tell?
>
>
> Thanks
> gs
>
> ps : Is there a sure shot way to find out the message rates ? (I
> currently use qpid-tool show exchange command to find the no of
> msgRecieves and divide by time-dfference from 2 times the command
> is run)
>
>
> On Thu, Mar 12, 2009 at 3:11 PM, GS.Chandra N
> <gs.chandran.n@gmail.com <ma...@gmail.com>> wrote:
>
> I'm able to reproduce the issue, when i keep qpid-tool open
> all the time with the mgmt switches.
>
> However i'm not able to do so if qpid-tool is not open (did
> not try with qpid-tool and no mgmt switches).
>
> Thanks
> gs
>
>
> On Thu, Mar 12, 2009 at 1:31 AM, Gordon Sim <gsim@redhat.com
> <ma...@redhat.com>> wrote:
>
> GS.Chandra N wrote:
>
> ps : Please find attached the scripts that create the
> issue
>
>
> I'm afraid I wasn't able to recreate the issue with your
> tests. Memory stayed reasonable and constant as the test
> was running (5 servers, 5 clients).
>
> I'm baffled at this point as to what it is you are seeing.
>
>
> broker - runs on a dedicated box
> qpidd -d --port=5672 --mgmt-enable=yes
> --mgmt-pub-interval=1 --auth=no --log-source=yes
> --log-to-file=/tmp/somename
>
>
> Do you have qpid-tool or anything else running during the
> test? Does turning management off have any impact on the
> issue?
>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
> <ma...@qpid.apache.org>
>
>
>
>