You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by mquestioner <wq...@avancetechnology.com> on 2009/10/19 02:17:30 UTC
Re: Slow consumers slow down fast consumer queue requests
Yeah... by reusing the same temporary queues, the problem goes away. (We used
1 temp queue per request because the requests were originated at different
threads. We changed to use a fixed set of temp queues, listeners, and local
dispatching to work around.)
But this seems just fix the symptoms. If supposedly, one has many clients
come and go and each of them would have to create at least some queues; the
system would still be slowed down. The bottomline is that creating temp
queues should not slow down others.
aortiz wrote:
>
> You should not create the session and temporary queue for every request.
> You should create them when you create the client [1] (or at least cache
> them). These operations are expensive and require remote calls to the
> broker.
>
> The slowdown of faster clients may be due to Producer Flow Control [2],
> which is enabled by default.
>
> -Andres
> [1]
> http://activemq.apache.org/how-should-i-implement-request-response-with-jms.html
> [2] http://activemq.apache.org/producer-flow-control.html
>
>
> mquestioner wrote:
>>
>> Thanks. Will try.
>>
>> More detail about the client is that when we send a queue request, we do
>>
>> - create a Session
>> - create a TemporaryQueue
>>
>> I wonder if these two will be broadcasting something to the other clients
>> and cause the backlog. Shouldn't be, but...
>>
>>
>>
>> rajdavies wrote:
>>>
>>> Does this behaviour change if you use the latest snapshot ?
>>>
>>> thanks,
>>>
>>> Rob
>>> On 20 Jul 2009, at 03:51, mquestioner wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> We have a case where a consumer connected through a slow connection
>>>> would
>>>> slow down the fast consumer. I have already
>>>> http://activemq.apache.org/slow-consumers.html and
>>>> http://activemq.apache.org/slow-consumer-handling.html However,
>>>> there are
>>>> specific advises do not seem applicable to our scenario.
>>>>
>>>> Out set up looks like:
>>>> The server
>>>> Constantly broadcasting non-persistent messages over a topic (a)
>>>> Listen on a number of persistent and non-persistent queues to
>>>> process
>>>> messages asynchronously (Listener) (b)
>>>> Reply to the temporary queue of the request (c)
>>>>
>>>> The client
>>>> Listen to the broadcast messages in background (d)
>>>> Make request to the queues (e)
>>>> Wait for reply (f)
>>>>
>>>> When the clients are on fast connection, everything is fine.
>>>> However, once a
>>>> slow connecting client joins, we observe:
>>>>
>>>> 1) (e) and (f) of the fast client become extremely sluggish instantly.
>>>>
>>>> 2) Timestamp logging of (b) and (c) shows that the server still
>>>> takes very
>>>> little time to process each request, ruling out problems such as DB
>>>> bottleneck. On the other hand, the messaging waiting time (time at
>>>> the entry
>>>> of the listner - msg.getJMStimestamp()) increases by 50-100X
>>>>
>>>> 3) Either killing the slow connecting client or stop broadcasting
>>>> message
>>>> will have things return to normal
>>>>
>>>> The strange thing is that, according to
>>>> http://activemq.apache.org/slow-consumers.html, ``A slow consumer is
>>>> not
>>>> really an issue with queues.'' And that the problem occurs almost
>>>> instantly
>>>> after the slow connection is established, it seems too short for any
>>>> memory
>>>> usage built-up; and indeed no such problem is observed. We also try
>>>> twisting
>>>> ActiveMQ broker and consumer settings like prefetchsize=2000,
>>>> dedicated task
>>>> runner = true/false, increase memoryLimit/memoryUsage,
>>>> optimizedDispatch=true. None seems to solve the problem.
>>>>
>>>> What would be the causes and solutions for these scenario?
>>>>
>>>> (The ActiveMQ site's suggestions of discarding old messages would
>>>> create
>>>> stale data problems for our application.)
>>>>
>>>> Thanks a lot!
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Slow-consumers-slow-down-fast-consumer-queue-requests-tp24563120p24563120.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Slow-consumers-slow-down-fast-consumer-queue-requests-tp24563120p25951782.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Slow consumers slow down fast consumer queue requests
Posted by Rob Davies <ra...@gmail.com>.
temporary destinations add additional overhead because to be compliant
with the JMS specification, a messaging system needs to validate that
the temporary destination exists before sending every message. To
avoid slowing delivery of messages to a crawl to a temporary queue,
ActiveMQ sends advisory messages to every client on creation/deletion
of a temporary destination. Hence if you are constantly creating/
destroying temporary destinations, you are creating a lot of
additional chatter.
Not ideal - but at least this should explain what is happening. You
can disable advisory messages - see http://activemq.apache.org/advisory-message.html
.
On 19 Oct 2009, at 01:17, mquestioner wrote:
>
> Yeah... by reusing the same temporary queues, the problem goes away.
> (We used
> 1 temp queue per request because the requests were originated at
> different
> threads. We changed to use a fixed set of temp queues, listeners,
> and local
> dispatching to work around.)
>
> But this seems just fix the symptoms. If supposedly, one has many
> clients
> come and go and each of them would have to create at least some
> queues; the
> system would still be slowed down. The bottomline is that creating
> temp
> queues should not slow down others.
>
>
> aortiz wrote:
>>
>> You should not create the session and temporary queue for every
>> request.
>> You should create them when you create the client [1] (or at least
>> cache
>> them). These operations are expensive and require remote calls to the
>> broker.
>>
>> The slowdown of faster clients may be due to Producer Flow Control
>> [2],
>> which is enabled by default.
>>
>> -Andres
>> [1]
>> http://activemq.apache.org/how-should-i-implement-request-response-with-jms.html
>> [2] http://activemq.apache.org/producer-flow-control.html
>>
>>
>> mquestioner wrote:
>>>
>>> Thanks. Will try.
>>>
>>> More detail about the client is that when we send a queue request,
>>> we do
>>>
>>> - create a Session
>>> - create a TemporaryQueue
>>>
>>> I wonder if these two will be broadcasting something to the other
>>> clients
>>> and cause the backlog. Shouldn't be, but...
>>>
>>>
>>>
>>> rajdavies wrote:
>>>>
>>>> Does this behaviour change if you use the latest snapshot ?
>>>>
>>>> thanks,
>>>>
>>>> Rob
>>>> On 20 Jul 2009, at 03:51, mquestioner wrote:
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> We have a case where a consumer connected through a slow
>>>>> connection
>>>>> would
>>>>> slow down the fast consumer. I have already
>>>>> http://activemq.apache.org/slow-consumers.html and
>>>>> http://activemq.apache.org/slow-consumer-handling.html However,
>>>>> there are
>>>>> specific advises do not seem applicable to our scenario.
>>>>>
>>>>> Out set up looks like:
>>>>> The server
>>>>> Constantly broadcasting non-persistent messages over a topic (a)
>>>>> Listen on a number of persistent and non-persistent queues to
>>>>> process
>>>>> messages asynchronously (Listener) (b)
>>>>> Reply to the temporary queue of the request (c)
>>>>>
>>>>> The client
>>>>> Listen to the broadcast messages in background (d)
>>>>> Make request to the queues (e)
>>>>> Wait for reply (f)
>>>>>
>>>>> When the clients are on fast connection, everything is fine.
>>>>> However, once a
>>>>> slow connecting client joins, we observe:
>>>>>
>>>>> 1) (e) and (f) of the fast client become extremely sluggish
>>>>> instantly.
>>>>>
>>>>> 2) Timestamp logging of (b) and (c) shows that the server still
>>>>> takes very
>>>>> little time to process each request, ruling out problems such as
>>>>> DB
>>>>> bottleneck. On the other hand, the messaging waiting time (time at
>>>>> the entry
>>>>> of the listner - msg.getJMStimestamp()) increases by 50-100X
>>>>>
>>>>> 3) Either killing the slow connecting client or stop broadcasting
>>>>> message
>>>>> will have things return to normal
>>>>>
>>>>> The strange thing is that, according to
>>>>> http://activemq.apache.org/slow-consumers.html, ``A slow
>>>>> consumer is
>>>>> not
>>>>> really an issue with queues.'' And that the problem occurs almost
>>>>> instantly
>>>>> after the slow connection is established, it seems too short for
>>>>> any
>>>>> memory
>>>>> usage built-up; and indeed no such problem is observed. We also
>>>>> try
>>>>> twisting
>>>>> ActiveMQ broker and consumer settings like prefetchsize=2000,
>>>>> dedicated task
>>>>> runner = true/false, increase memoryLimit/memoryUsage,
>>>>> optimizedDispatch=true. None seems to solve the problem.
>>>>>
>>>>> What would be the causes and solutions for these scenario?
>>>>>
>>>>> (The ActiveMQ site's suggestions of discarding old messages would
>>>>> create
>>>>> stale data problems for our application.)
>>>>>
>>>>> Thanks a lot!
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/Slow-consumers-slow-down-fast-consumer-queue-requests-tp24563120p24563120.html
>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Slow-consumers-slow-down-fast-consumer-queue-requests-tp24563120p25951782.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/