You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Franco Ng <fr...@ebshk.com> on 2019/05/02 01:51:12 UTC
ActiveMQ cannot handle too many topic with redeliveryPlugin
Our market data would publish over 20,000 msg/s within 30,000 topics,
redeliveryPlugin and socketBufferSize, however, ActiveMQ seems unable to
handle this volume that CPU usage % is 50% and cannot publish any message to
subscriber(s) after certain starting time. Even, I stop the market data
publishing, the CPU usage of ActiveMQ broker is still eat up 50%, cannot
drop for a long time and NO error/warning log recorded in activemq.log.
When I remove redeliveryPlugin and socketBufferSize from Broker
(activemq.xml), it resume normal.
Please help to solve. Also, how to show redelivery message in log when
occur, thanks.
Activemq.xml (part)
<redeliveryPlugin fallbackToDeadLetter="true"
sendToDlqIfMaxRetriesExceeded="true">
<redeliveryPolicyMap>
<redeliveryPolicyMap>
<defaultEntry>
<redeliveryPolicy useCollisionAvoidance="true" maximumRedeliveries="1"
initialRedeliveryDelay="1000" redeliveryDelay="1000"/>
</defaultEntry>
</redeliveryPolicyMap>
</redeliveryPolicyMap>
</redeliveryPlugin>
<transportConnector name="openwire"
uri="tcp://0.0.0.0:61616?maximumConnections=3000&wireFormat.maxFrameSize=104857600&transport.socketBufferSize=655360&transport.ioBufferSize=81920"
/>
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
答复: ActiveMQ cannot handle too many topic with redeliveryPlugin
Posted by 付 传淮 <fu...@outlook.com>.
Can you provide any dump file:
Just like threaddump or heapdump?
-------------
Personal Email
-----邮件原件-----
发件人: Franco Ng <fr...@ebshk.com>
发送时间: 2019年5月2日 9:51
收件人: users@activemq.apache.org
主题: ActiveMQ cannot handle too many topic with redeliveryPlugin
Our market data would publish over 20,000 msg/s within 30,000 topics, redeliveryPlugin and socketBufferSize, however, ActiveMQ seems unable to handle this volume that CPU usage % is 50% and cannot publish any message to
subscriber(s) after certain starting time. Even, I stop the market data publishing, the CPU usage of ActiveMQ broker is still eat up 50%, cannot drop for a long time and NO error/warning log recorded in activemq.log.
When I remove redeliveryPlugin and socketBufferSize from Broker (activemq.xml), it resume normal.
Please help to solve. Also, how to show redelivery message in log when occur, thanks.
Activemq.xml (part)
<redeliveryPlugin fallbackToDeadLetter="true"
sendToDlqIfMaxRetriesExceeded="true">
<redeliveryPolicyMap>
<redeliveryPolicyMap>
<defaultEntry>
<redeliveryPolicy useCollisionAvoidance="true" maximumRedeliveries="1"
initialRedeliveryDelay="1000" redeliveryDelay="1000"/> </defaultEntry> </redeliveryPolicyMap> </redeliveryPolicyMap> </redeliveryPlugin>
<transportConnector name="openwire"
uri="tcp://0.0.0.0:61616?maximumConnections=3000&wireFormat.maxFrameSize=104857600&transport.socketBufferSize=655360&transport.ioBufferSize=81920"
/>
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: ActiveMQ cannot handle too many topic with redeliveryPlugin
Posted by Tim Bain <tb...@alumni.duke.edu>.
Those two (profiler capture, or thread dump) should also be smaller and you
might be able to attach them directly. If not, just use one of the free
file-posting sites if you don't want to use your own Dropbox or Google
Drive or similar.
Tim
On Thu, May 2, 2019, 7:28 AM Tim Bain <tb...@alumni.duke.edu> wrote:
> A CPU sampling capture from a profiler such as JVisualVM will be far more
> effective than a thread dump, if it's an option.
>
> A heap dump will probably not be useful here, since it doesn't sound like
> you're running out of memory.
>
> Tim
>
> On Thu, May 2, 2019, 4:39 AM Christopher Shannon <
> christopher.l.shannon@gmail.com> wrote:
>
>> 20k messages and 30k topics is quite a bit so I'm not surprised the CPU
>> usage is very high. The reason the CPU is high after stopping the
>> publishing is probably because you have a backlog on the broker that the
>> consumers need to work off. You might also have a lot of messages being
>> processed for the DLQ.
>>
>> If you think the issue is related specifically to the redelivery plugin
>> then your best bet is to hook up a profiler to the broker and to see where
>> the hot spot is that is using the most CPU in the plugin. You can also
>> trying using a thread dump to figure it out.
>>
>> On Thu, May 2, 2019 at 5:53 AM Franco Ng <fr...@ebshk.com> wrote:
>>
>> > No problem, I can provide heap dump but the file size is over 1GB.
>> Please
>> > provide location /link to upload, thanks.
>> >
>> >
>> >
>> > --
>> > Sent from:
>> > http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>> >
>>
>
Re: ActiveMQ cannot handle too many topic with redeliveryPlugin
Posted by Tim Bain <tb...@alumni.duke.edu>.
A CPU sampling capture from a profiler such as JVisualVM will be far more
effective than a thread dump, if it's an option.
A heap dump will probably not be useful here, since it doesn't sound like
you're running out of memory.
Tim
On Thu, May 2, 2019, 4:39 AM Christopher Shannon <
christopher.l.shannon@gmail.com> wrote:
> 20k messages and 30k topics is quite a bit so I'm not surprised the CPU
> usage is very high. The reason the CPU is high after stopping the
> publishing is probably because you have a backlog on the broker that the
> consumers need to work off. You might also have a lot of messages being
> processed for the DLQ.
>
> If you think the issue is related specifically to the redelivery plugin
> then your best bet is to hook up a profiler to the broker and to see where
> the hot spot is that is using the most CPU in the plugin. You can also
> trying using a thread dump to figure it out.
>
> On Thu, May 2, 2019 at 5:53 AM Franco Ng <fr...@ebshk.com> wrote:
>
> > No problem, I can provide heap dump but the file size is over 1GB.
> Please
> > provide location /link to upload, thanks.
> >
> >
> >
> > --
> > Sent from:
> > http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
> >
>
Re: ActiveMQ cannot handle too many topic with redeliveryPlugin
Posted by Christopher Shannon <ch...@gmail.com>.
20k messages and 30k topics is quite a bit so I'm not surprised the CPU
usage is very high. The reason the CPU is high after stopping the
publishing is probably because you have a backlog on the broker that the
consumers need to work off. You might also have a lot of messages being
processed for the DLQ.
If you think the issue is related specifically to the redelivery plugin
then your best bet is to hook up a profiler to the broker and to see where
the hot spot is that is using the most CPU in the plugin. You can also
trying using a thread dump to figure it out.
On Thu, May 2, 2019 at 5:53 AM Franco Ng <fr...@ebshk.com> wrote:
> No problem, I can provide heap dump but the file size is over 1GB. Please
> provide location /link to upload, thanks.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: ActiveMQ cannot handle too many topic with redeliveryPlugin
Posted by Franco Ng <fr...@ebshk.com>.
No problem, I can provide heap dump but the file size is over 1GB. Please
provide location /link to upload, thanks.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html