You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by SelvarajAM <se...@hcl.in> on 2009/11/10 14:26:28 UTC
Activemq broker hangs when stomp client disconnects unexpectedly
Hi,
I am using embedded Activemq broker (5.3 Version), Flex client and STOMP
protocol for the communication.
The problem is,
When unexpected error occurs in the Flex client, Activemq broker memory
usage increases gradually and then it hangs giving the following message in
the log file
Memory usage change from: 1% of available memory, to: 2% of available memory
.
.
.
Memory usage change from: 99% of available memory, to: 100% of available
memory
Usage Manager memory limit reached for topic://a. Producers will be
throttled to the rate at which messages are removed from this destination to
prevent flooding it. See
http://activemq.apache.org/producer-flow-control.html for more info
Below is the xml configuration we use
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">
<!--
The <broker> element is used to configure the ActiveMQ broker.
-->
<broker xmlns="http://activemq.apache.org/schema/core"
persistent="false" useJmx="true" brokerName="PRI" >
<transportConnectors>
<transportConnector name="Stomp" uri="stomp://0.0.0.0:8084"/>
<transportConnector name="Nio" uri="nio://0.0.0.0:8086"/>
<transportConnector name="Tcp" uri="tcp://0.0.0.0:8087"/>
</transportConnectors>
</broker>
</beans>
We use NON-PERSISTENT messages and send to topic. The client listens to the
topic and process the message.
The jconsole shows the broker MemoryPercentUsage as 100% when activemq
hangs.
Why does an error in client crashes the Activemq server??? Do anyone face
the same issue??
Only restarting the broker helps.
I haved changed a lot of modification in the xml configuration but could not
find the root cause.
Kindly help me on this as this is a production issue.
--
View this message in context: http://old.nabble.com/Activemq-broker-hangs-when-stomp-client-disconnects-unexpectedly-tp26283384p26283384.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Activemq broker hangs when stomp client disconnects
unexpectedly
Posted by SelvarajAM <se...@hcl.in>.
Could not find the root cause. So as a workaround i am restarting my broker
programatically when the memory limit reaches above 80%
[broker.getSystemUsage().getMemoryUsage().getPercentUsage() ]
My client (browser) refreshs automatically if the server doesnt send data
for 15 secs.
The above workaround works fine for me. :-)
I shall update once i find the right fix ....
SelvarajAM wrote:
>
> Below is the o/p of netstat
>
> tcp 0 0 *:8084 *:*
> LISTEN
> tcp 0 92531 pri:8084
> ::ffff:10.161.:boinc-client FIN_WAIT1
> tcp 1 0 pri:48292 pri:8084
> ESTABLISHED
> tcp 0 0 pri:8084
> ::ffff:10.161.:availant-mgr ESTABLISHED
> tcp 0 0 pri:8084 pri:48292
> ESTABLISHED
> warning, got duplicate tcp line.
> warning, got duplicate tcp line.
>
>
> The o/p is same both the client was fine and when it encountered error.
>
> Can you give an example of xml configuartion on how to configure for
> prefetch limit for consumer.
>
> I tried the following. But did not work.
>
> <transportConnector name="StompNio"
> uri="stomp://0.0.0.0:8084?jms.dispatchAsync=true&jms.prefetchSize=10"/>
>
> Kindly see the jconsole for subscription attributes.
> Once the client fails, the MessageCountAwaitingAcknowledge starts
> increasing.
>
> Dont why it waits for acknowledgement. Kindly shed some light on it
>
>
> http://old.nabble.com/file/p26302014/Subscription.jpg
>
:-D:-D:-D:-D:-D:-D:-D:-D:-D:-D:-D
--
View this message in context: http://old.nabble.com/Activemq-broker-hangs-when-stomp-client-disconnects-unexpectedly-tp26283384p26335328.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Activemq broker hangs when stomp client disconnects
unexpectedly
Posted by SelvarajAM <se...@hcl.in>.
Below is the o/p of netstat
tcp 0 0 *:8084 *:*
LISTEN
tcp 0 92531 pri:8084
::ffff:10.161.:boinc-client FIN_WAIT1
tcp 1 0 pri:48292 pri:8084
ESTABLISHED
tcp 0 0 pri:8084
::ffff:10.161.:availant-mgr ESTABLISHED
tcp 0 0 pri:8084 pri:48292
ESTABLISHED
warning, got duplicate tcp line.
warning, got duplicate tcp line.
The o/p is same both the client was fine and when it encountered error.
Can you give an example of xml configuartion on how to configure for
prefetch limit for consumer.
I tried the following. But did not work.
<transportConnector name="StompNio"
uri="stomp://0.0.0.0:8084?jms.dispatchAsync=true&jms.prefetchSize=10"/>
Kindly see the jconsole for subscription attributes.
Once the client fails, the MessageCountAwaitingAcknowledge starts
increasing.
Dont why it waits for acknowledgement. Kindly shed some light on it
http://old.nabble.com/file/p26302014/Subscription.jpg
--
View this message in context: http://old.nabble.com/Activemq-broker-hangs-when-stomp-client-disconnects-unexpectedly-tp26283384p26302014.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Activemq broker hangs when stomp client disconnects unexpectedly
Posted by Gary Tully <ga...@gmail.com>.
what does the os say about that connection. Can you use netstat to get
the tcp connection status?
with that information it may be possible to determine what timeouts
are relevant.
Also, the prefetch for non persistent topic consumers is short-max, so
you need to configure a value that will limit the amount of pending
messages that are queued for that consumer while it is 'disconnected'
but still present to the broker.
see: http://activemq.apache.org/what-is-the-prefetch-limit-for.html
2009/11/11 SelvarajAM <se...@hcl.in>:
>
> Hi Gary,
>
> Thanks for your reply.
> I have attached my jconsole screen shot. Kindly have a look at it.
>
> Though the client connection is lost (or has got error), jconsole says it
> is NOT BLOCKED and NOT SLOW.
>
> Also i tried all the configuaration for discarding the messages as in
> http://activemq.apache.org/slow-consumer-handling.html
> But it did not work. Just the memory increases and activemq stops.
> I dont know why the configuration is not working.
>
> Can you please tell me where to configure the timeouts????
>
> One thing i noticed is, in jconsole->Connection->Stomp->[Diconnected
> Client]->Operations->stop()
> when i send the stop command, activemq started sending messages to other
> clients without any problem.
>
> But i dont know how to identify the disonnected client...and close it
> programatically...
>
> Another strange thing i see in jconsole is, the TotalDequeue count is less
> compared to TotalEnqueue count.
>
> Below is the xml configuration which i use to reproduce the issue. The
> system memory is very less (1mb) inorder to reproduce it faster.
>
> <beans
> xmlns="http://www.springframework.org/schema/beans"
> xmlns:amq="http://activemq.apache.org/schema/core"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd">
>
>
> <broker xmlns="http://activemq.apache.org/schema/core"
> persistent="false" useJmx="true" brokerName="PRI" >
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic="a" producerFlowControl="true"
> memoryLimit="1 mb">
> <pendingMessageLimitStrategy>
> <constantPendingMessageLimitStrategy limit="10"/>
> </pendingMessageLimitStrategy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="1 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="1 mb" name="foo"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="1 mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
> <transportConnectors>
> <transportConnector name="StompNio"
> uri="stomp://0.0.0.0:8084?jms.dispatchAsync=true"/>
> <transportConnector name="Nio" uri="nio://0.0.0.0:8086"/>
> <transportConnector name="Tcp" uri="tcp://0.0.0.0:8087"/>
> </transportConnectors>
> </broker>
> </beans>
> http://old.nabble.com/file/p26299914/Broker.jpg
> http://old.nabble.com/file/p26299914/ClientDisconnect.jpg
> --
> View this message in context: http://old.nabble.com/Activemq-broker-hangs-when-stomp-client-disconnects-unexpectedly-tp26283384p26299914.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com
Re: Activemq broker hangs when stomp client disconnects
unexpectedly
Posted by SelvarajAM <se...@hcl.in>.
Hi Gary,
Thanks for your reply.
I have attached my jconsole screen shot. Kindly have a look at it.
Though the client connection is lost (or has got error), jconsole says it
is NOT BLOCKED and NOT SLOW.
Also i tried all the configuaration for discarding the messages as in
http://activemq.apache.org/slow-consumer-handling.html
But it did not work. Just the memory increases and activemq stops.
I dont know why the configuration is not working.
Can you please tell me where to configure the timeouts????
One thing i noticed is, in jconsole->Connection->Stomp->[Diconnected
Client]->Operations->stop()
when i send the stop command, activemq started sending messages to other
clients without any problem.
But i dont know how to identify the disonnected client...and close it
programatically...
Another strange thing i see in jconsole is, the TotalDequeue count is less
compared to TotalEnqueue count.
Below is the xml configuration which i use to reproduce the issue. The
system memory is very less (1mb) inorder to reproduce it faster.
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">
<broker xmlns="http://activemq.apache.org/schema/core"
persistent="false" useJmx="true" brokerName="PRI" >
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic="a" producerFlowControl="true"
memoryLimit="1 mb">
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="10"/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="1 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 mb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="1 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="StompNio"
uri="stomp://0.0.0.0:8084?jms.dispatchAsync=true"/>
<transportConnector name="Nio" uri="nio://0.0.0.0:8086"/>
<transportConnector name="Tcp" uri="tcp://0.0.0.0:8087"/>
</transportConnectors>
</broker>
</beans>
http://old.nabble.com/file/p26299914/Broker.jpg
http://old.nabble.com/file/p26299914/ClientDisconnect.jpg
--
View this message in context: http://old.nabble.com/Activemq-broker-hangs-when-stomp-client-disconnects-unexpectedly-tp26283384p26299914.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Activemq broker hangs when stomp client disconnects unexpectedly
Posted by Gary Tully <ga...@gmail.com>.
have a look at http://activemq.apache.org/slow-consumer-handling.html
with persistence=false, no disk spooling will occur so all messages
will remain in memory unless you add some discard policy.
what is a little odd is that the broker still thinks there is a
consumer, you may need to configure some timeouts. What does netstat
or the console say about the connection from your client?
2009/11/10 SelvarajAM <se...@hcl.in>:
>
> Hi,
>
> I am using embedded Activemq broker (5.3 Version), Flex client and STOMP
> protocol for the communication.
>
> The problem is,
>
> When unexpected error occurs in the Flex client, Activemq broker memory
> usage increases gradually and then it hangs giving the following message in
> the log file
>
> Memory usage change from: 1% of available memory, to: 2% of available memory
> .
> .
> .
> Memory usage change from: 99% of available memory, to: 100% of available
> memory
>
> Usage Manager memory limit reached for topic://a. Producers will be
> throttled to the rate at which messages are removed from this destination to
> prevent flooding it. See
> http://activemq.apache.org/producer-flow-control.html for more info
>
> Below is the xml configuration we use
>
> <beans
> xmlns="http://www.springframework.org/schema/beans"
> xmlns:amq="http://activemq.apache.org/schema/core"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd">
>
>
> <!--
> The <broker> element is used to configure the ActiveMQ broker.
> -->
> <broker xmlns="http://activemq.apache.org/schema/core"
> persistent="false" useJmx="true" brokerName="PRI" >
> <transportConnectors>
> <transportConnector name="Stomp" uri="stomp://0.0.0.0:8084"/>
> <transportConnector name="Nio" uri="nio://0.0.0.0:8086"/>
> <transportConnector name="Tcp" uri="tcp://0.0.0.0:8087"/>
> </transportConnectors>
> </broker>
> </beans>
>
> We use NON-PERSISTENT messages and send to topic. The client listens to the
> topic and process the message.
>
> The jconsole shows the broker MemoryPercentUsage as 100% when activemq
> hangs.
>
> Why does an error in client crashes the Activemq server??? Do anyone face
> the same issue??
>
> Only restarting the broker helps.
> I haved changed a lot of modification in the xml configuration but could not
> find the root cause.
>
> Kindly help me on this as this is a production issue.
>
> --
> View this message in context: http://old.nabble.com/Activemq-broker-hangs-when-stomp-client-disconnects-unexpectedly-tp26283384p26283384.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com