You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by alisenberg <li...@gmail.com> on 2010/04/23 05:25:37 UTC
Broker hangs and cant restart it
This is the scenario:
I open three web admin consoles and send 10.000 messages from each console
at the same time. Each message is about 50k. I mark persistence. And all
messages go to the same queue. When the Queue receives 20.000 messages, the
broker appears to hang.
I can see that tmp_storage is consuming all the allocated space so the “Temp
percent used” reaches 100%.
My first question is, why tmp_storage is growing if all messages were marked
as persistent?
After the broker hangs, I try to shut it down, but the only way is to force
kill it.
Second question: even if the tmp_storage is full, why the broker is not
shutting down in a normal way?
After killing it, I restart it but it hangs and does not complete the
initialization.
But, if I configure the “tempUsage” a Little bit higher, then the broker
starts again and the problem gets solved.
So, my last question is, why I need to increase tempUsage in order to
restart the broker?
Thanks,
Adrian
http://old.nabble.com/file/p28337031/activemq.xml activemq.xml
--
View this message in context: http://old.nabble.com/Broker-hangs-and-cant-restart-it-tp28337031p28337031.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker hangs and cant restart it
Posted by alisenberg <li...@gmail.com>.
Gary Tully wrote:
>
> A thread dump of the broker would help to diagnose what is going on.
>
> On 23 April 2010 14:44, alisenberg <li...@gmail.com> wrote:
>
>>
>>
>> bsnyder wrote:
>> >
>> > On Thu, Apr 22, 2010 at 9:25 PM, alisenberg <li...@gmail.com>
>> wrote:
>> >>
>> >> This is the scenario:
>> >>
>> >> I open three web admin consoles and send 10.000 messages from each
>> >> console
>> >> at the same time. Each message is about 50k. I mark persistence. And
>> all
>> >> messages go to the same queue. When the Queue receives 20.000
>> messages,
>> >> the
>> >> broker appears to hang.
>> >> I can see that tmp_storage is consuming all the allocated space so the
>> >> “Temp
>> >> percent used” reaches 100%.
>> >>
>> >> My first question is, why tmp_storage is growing if all messages were
>> >> marked
>> >> as persistent?
>> >>
>> >> After the broker hangs, I try to shut it down, but the only way is to
>> >> force
>> >> kill it.
>> >>
>> >> Second question: even if the tmp_storage is full, why the broker is
>> not
>> >> shutting down in a normal way?
>> >>
>> >> After killing it, I restart it but it hangs and does not complete the
>> >> initialization.
>> >>
>> >> But, if I configure the “tempUsage” a Little bit higher, then the
>> broker
>> >> starts again and the problem gets solved.
>> >>
>> >> So, my last question is, why I need to increase tempUsage in order to
>> >> restart the broker?
>> >
>> > What version of ActiveMQ are you using? How have you configured the
>> > broker? Are you using the default configuration? What do you mean that
>> > the broker is hanging? Are you unable to send anymore messages to it?
>> >
>> > I think your pursuit of the tempUsage is a red herring. Instead, I
>> > think what you are experiencing is producer flow control:
>> >
>> > http://activemq.apache.org/producer-flow-control.html
>> >
>> > Trying disabling producer flow control via the conf/activemq.xml
>> > config and see what happens.
>> >
>> > Bruce
>> > --
>> > perl -e 'print
>> > unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
>> > );'
>> >
>> > ActiveMQ in Action: http://bit.ly/2je6cQ
>> > Blog: http://bruceblog.org/
>> > Twitter: http://twitter.com/brucesnyder
>> >
>> >
>>
>> I think I am not using flowcontrol. I´m using 5.3. The broker is not
>> receiving messages any more and producers are stopped but no error is
>> thrown.
>>
>> Also when I try to inspect that queue with jconsole, jconsole also hangs
>> displaying only a gray screen.
>>
>> CPU and RAM seems to be ok.
>>
>> This is my configuration:
>>
>> <broker xmlns="http://activemq.apache.org/schema/core"
>> brokerName="localhost" dataDirectory="/data/activemq53/" useJmx="true"
>> advisorySupport="false" useShutdownHook="true">
>>
>> <plugins>
>> <statisticsBrokerPlugin/>
>> </plugins>
>>
>> <!-- Destination specific policies using destination names or
>> wildcards -->
>> <destinationPolicy>
>> <policyMap>
>> <policyEntries>
>> <policyEntry queue=">"
>> producerFlowControl="false">
>> <pendingQueuePolicy>
>> <fileQueueCursor />
>> </pendingQueuePolicy>
>> </policyEntry>
>> <policyEntry topic=">" producerFlowControl="false">
>> <dispatchPolicy>
>> <strictOrderDispatchPolicy/>
>> </dispatchPolicy>
>> <subscriptionRecoveryPolicy>
>> <lastImageSubscriptionRecoveryPolicy/>
>> </subscriptionRecoveryPolicy>
>> </policyEntry>
>> </policyEntries>
>> </policyMap>
>> </destinationPolicy>
>>
>> <destinationInterceptors>
>> <virtualDestinationInterceptor>
>> <virtualDestinations>
>> <compositeQueue name="EVENTS.A">
>> <forwardTo>
>> <queue physicalName="EVENTS.A.STATISTICS" />
>> <queue physicalName="EVENTS.A.WAREHOUSE" />
>> </forwardTo>
>> </compositeQueue>
>> </virtualDestinations>
>> </virtualDestinationInterceptor>
>> </destinationInterceptors>
>>
>> <persistenceAdapter>
>> <kahaDB directory="/data/activemq53/kahadb"
>> enableIndexWriteAsync="true" journalMaxFileLength="20mb"/>
>> </persistenceAdapter>
>>
>> <systemUsage>
>> <systemUsage sendFailIfNoSpace="true">
>> <memoryUsage>
>> <memoryUsage limit="100 mb"/>
>> </memoryUsage>
>> <storeUsage>
>> <storeUsage limit="80 gb" name="foo"/>
>> </storeUsage>
>> <tempUsage>
>> <tempUsage limit="1 gb"/>
>> </tempUsage>
>> </systemUsage>
>> </systemUsage>
>>
>> <transportConnectors>
>> <transportConnector name="openwire" uri="tcp://0.0.0.0:61616
>> "/>
>> </transportConnectors>
>>
>> </broker>
>>
>> Thanks
>> --
>> View this message in context:
>> http://old.nabble.com/Broker-hangs-and-cant-restart-it-tp28337031p28341513.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>
>
Sorry but I am not able to get the thread dump :(
I found that if I change from filecursor to default cursor the problem is
solved. I guess that filecursor is writing many files
(data-TopicSubscription-#) to tmp_storage. With filecursor, when that
directory reaches temp size, the problem appears. With default cursor no
file is written, so that seems to fix the problem...
Any idea of problems using filecursor?
Thanks
--
View this message in context: http://old.nabble.com/Broker-hangs-and-cant-restart-it-tp28337031p28345933.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker hangs and cant restart it
Posted by Gary Tully <ga...@gmail.com>.
A thread dump of the broker would help to diagnose what is going on.
On 23 April 2010 14:44, alisenberg <li...@gmail.com> wrote:
>
>
> bsnyder wrote:
> >
> > On Thu, Apr 22, 2010 at 9:25 PM, alisenberg <li...@gmail.com> wrote:
> >>
> >> This is the scenario:
> >>
> >> I open three web admin consoles and send 10.000 messages from each
> >> console
> >> at the same time. Each message is about 50k. I mark persistence. And all
> >> messages go to the same queue. When the Queue receives 20.000 messages,
> >> the
> >> broker appears to hang.
> >> I can see that tmp_storage is consuming all the allocated space so the
> >> “Temp
> >> percent used” reaches 100%.
> >>
> >> My first question is, why tmp_storage is growing if all messages were
> >> marked
> >> as persistent?
> >>
> >> After the broker hangs, I try to shut it down, but the only way is to
> >> force
> >> kill it.
> >>
> >> Second question: even if the tmp_storage is full, why the broker is not
> >> shutting down in a normal way?
> >>
> >> After killing it, I restart it but it hangs and does not complete the
> >> initialization.
> >>
> >> But, if I configure the “tempUsage” a Little bit higher, then the broker
> >> starts again and the problem gets solved.
> >>
> >> So, my last question is, why I need to increase tempUsage in order to
> >> restart the broker?
> >
> > What version of ActiveMQ are you using? How have you configured the
> > broker? Are you using the default configuration? What do you mean that
> > the broker is hanging? Are you unable to send anymore messages to it?
> >
> > I think your pursuit of the tempUsage is a red herring. Instead, I
> > think what you are experiencing is producer flow control:
> >
> > http://activemq.apache.org/producer-flow-control.html
> >
> > Trying disabling producer flow control via the conf/activemq.xml
> > config and see what happens.
> >
> > Bruce
> > --
> > perl -e 'print
> > unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> > );'
> >
> > ActiveMQ in Action: http://bit.ly/2je6cQ
> > Blog: http://bruceblog.org/
> > Twitter: http://twitter.com/brucesnyder
> >
> >
>
> I think I am not using flowcontrol. I´m using 5.3. The broker is not
> receiving messages any more and producers are stopped but no error is
> thrown.
>
> Also when I try to inspect that queue with jconsole, jconsole also hangs
> displaying only a gray screen.
>
> CPU and RAM seems to be ok.
>
> This is my configuration:
>
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="localhost" dataDirectory="/data/activemq53/" useJmx="true"
> advisorySupport="false" useShutdownHook="true">
>
> <plugins>
> <statisticsBrokerPlugin/>
> </plugins>
>
> <!-- Destination specific policies using destination names or
> wildcards -->
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry queue=">" producerFlowControl="false">
> <pendingQueuePolicy>
> <fileQueueCursor />
> </pendingQueuePolicy>
> </policyEntry>
> <policyEntry topic=">" producerFlowControl="false">
> <dispatchPolicy>
> <strictOrderDispatchPolicy/>
> </dispatchPolicy>
> <subscriptionRecoveryPolicy>
> <lastImageSubscriptionRecoveryPolicy/>
> </subscriptionRecoveryPolicy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
> <destinationInterceptors>
> <virtualDestinationInterceptor>
> <virtualDestinations>
> <compositeQueue name="EVENTS.A">
> <forwardTo>
> <queue physicalName="EVENTS.A.STATISTICS" />
> <queue physicalName="EVENTS.A.WAREHOUSE" />
> </forwardTo>
> </compositeQueue>
> </virtualDestinations>
> </virtualDestinationInterceptor>
> </destinationInterceptors>
>
> <persistenceAdapter>
> <kahaDB directory="/data/activemq53/kahadb"
> enableIndexWriteAsync="true" journalMaxFileLength="20mb"/>
> </persistenceAdapter>
>
> <systemUsage>
> <systemUsage sendFailIfNoSpace="true">
> <memoryUsage>
> <memoryUsage limit="100 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="80 gb" name="foo"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="1 gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
> <transportConnectors>
> <transportConnector name="openwire" uri="tcp://0.0.0.0:61616
> "/>
> </transportConnectors>
>
> </broker>
>
> Thanks
> --
> View this message in context:
> http://old.nabble.com/Broker-hangs-and-cant-restart-it-tp28337031p28341513.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com
Re: Broker hangs and cant restart it
Posted by alisenberg <li...@gmail.com>.
bsnyder wrote:
>
> On Thu, Apr 22, 2010 at 9:25 PM, alisenberg <li...@gmail.com> wrote:
>>
>> This is the scenario:
>>
>> I open three web admin consoles and send 10.000 messages from each
>> console
>> at the same time. Each message is about 50k. I mark persistence. And all
>> messages go to the same queue. When the Queue receives 20.000 messages,
>> the
>> broker appears to hang.
>> I can see that tmp_storage is consuming all the allocated space so the
>> “Temp
>> percent used” reaches 100%.
>>
>> My first question is, why tmp_storage is growing if all messages were
>> marked
>> as persistent?
>>
>> After the broker hangs, I try to shut it down, but the only way is to
>> force
>> kill it.
>>
>> Second question: even if the tmp_storage is full, why the broker is not
>> shutting down in a normal way?
>>
>> After killing it, I restart it but it hangs and does not complete the
>> initialization.
>>
>> But, if I configure the “tempUsage” a Little bit higher, then the broker
>> starts again and the problem gets solved.
>>
>> So, my last question is, why I need to increase tempUsage in order to
>> restart the broker?
>
> What version of ActiveMQ are you using? How have you configured the
> broker? Are you using the default configuration? What do you mean that
> the broker is hanging? Are you unable to send anymore messages to it?
>
> I think your pursuit of the tempUsage is a red herring. Instead, I
> think what you are experiencing is producer flow control:
>
> http://activemq.apache.org/producer-flow-control.html
>
> Trying disabling producer flow control via the conf/activemq.xml
> config and see what happens.
>
> Bruce
> --
> perl -e 'print
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> ActiveMQ in Action: http://bit.ly/2je6cQ
> Blog: http://bruceblog.org/
> Twitter: http://twitter.com/brucesnyder
>
>
I think I am not using flowcontrol. I´m using 5.3. The broker is not
receiving messages any more and producers are stopped but no error is
thrown.
Also when I try to inspect that queue with jconsole, jconsole also hangs
displaying only a gray screen.
CPU and RAM seems to be ok.
This is my configuration:
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost" dataDirectory="/data/activemq53/" useJmx="true"
advisorySupport="false" useShutdownHook="true">
<plugins>
<statisticsBrokerPlugin/>
</plugins>
<!-- Destination specific policies using destination names or
wildcards -->
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" producerFlowControl="false">
<pendingQueuePolicy>
<fileQueueCursor />
</pendingQueuePolicy>
</policyEntry>
<policyEntry topic=">" producerFlowControl="false">
<dispatchPolicy>
<strictOrderDispatchPolicy/>
</dispatchPolicy>
<subscriptionRecoveryPolicy>
<lastImageSubscriptionRecoveryPolicy/>
</subscriptionRecoveryPolicy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<destinationInterceptors>
<virtualDestinationInterceptor>
<virtualDestinations>
<compositeQueue name="EVENTS.A">
<forwardTo>
<queue physicalName="EVENTS.A.STATISTICS" />
<queue physicalName="EVENTS.A.WAREHOUSE" />
</forwardTo>
</compositeQueue>
</virtualDestinations>
</virtualDestinationInterceptor>
</destinationInterceptors>
<persistenceAdapter>
<kahaDB directory="/data/activemq53/kahadb"
enableIndexWriteAsync="true" journalMaxFileLength="20mb"/>
</persistenceAdapter>
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
<memoryUsage>
<memoryUsage limit="100 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="80 gb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="1 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
</transportConnectors>
</broker>
Thanks
--
View this message in context: http://old.nabble.com/Broker-hangs-and-cant-restart-it-tp28337031p28341513.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker hangs and cant restart it
Posted by Bruce Snyder <br...@gmail.com>.
On Thu, Apr 22, 2010 at 9:25 PM, alisenberg <li...@gmail.com> wrote:
>
> This is the scenario:
>
> I open three web admin consoles and send 10.000 messages from each console
> at the same time. Each message is about 50k. I mark persistence. And all
> messages go to the same queue. When the Queue receives 20.000 messages, the
> broker appears to hang.
> I can see that tmp_storage is consuming all the allocated space so the “Temp
> percent used” reaches 100%.
>
> My first question is, why tmp_storage is growing if all messages were marked
> as persistent?
>
> After the broker hangs, I try to shut it down, but the only way is to force
> kill it.
>
> Second question: even if the tmp_storage is full, why the broker is not
> shutting down in a normal way?
>
> After killing it, I restart it but it hangs and does not complete the
> initialization.
>
> But, if I configure the “tempUsage” a Little bit higher, then the broker
> starts again and the problem gets solved.
>
> So, my last question is, why I need to increase tempUsage in order to
> restart the broker?
What version of ActiveMQ are you using? How have you configured the
broker? Are you using the default configuration? What do you mean that
the broker is hanging? Are you unable to send anymore messages to it?
I think your pursuit of the tempUsage is a red herring. Instead, I
think what you are experiencing is producer flow control:
http://activemq.apache.org/producer-flow-control.html
Trying disabling producer flow control via the conf/activemq.xml
config and see what happens.
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
ActiveMQ in Action: http://bit.ly/2je6cQ
Blog: http://bruceblog.org/
Twitter: http://twitter.com/brucesnyder