You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Hayate <ze...@gmail.com> on 2016/03/26 10:27:22 UTC
ActiveMQ hangs and i don't understand why
My problem:
ActiveMQ hangs yesterday, i could not open web-interface, consumers could
not connect to broker.
After restarting ActiveMQ all works well.
We use ActiveMQ Version 5.10.1. Uptime before this problem was more than 60
days. 60 days ago was last settings changing and we don't have such problems
before.
We use written in php producers and consumers. Some producers and consumers
work with persistence messages, and some with non-persistence (this messages
contains log-info).
Consumers and producers connect to ActiveMQ via stomp protocol implemented
by this library: https://github.com/centraldesktop/php-stomp (By the way,
what is the best tool for producing and consuming messages from php? With
keep-alive and etc?)
And in ActiveMQ logs i see such messages:
2016-03-26 02:16:19,552 | ERROR | Could not accept connection :
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space |
org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server
Thread Handler:
stomp://127.0.0.1:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2016-03-26 02:17:37,002 | ERROR | Could not accept connection :
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space |
org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server
Thread Handler:
stomp://127.0.0.1:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2016-03-26 02:17:37,794 | WARN | Failed to register MBean
org.apache.activemq:type=Broker,brokerName=localhost,endpoint=dynamicProducer,clientId=ID_host-4265-1453350583818-4_795037,producerId=ID_host-4265-1453350583818-4_795037_-1_1
| org.apache.activemq.broker.jmx.ManagedRegionBroker | ActiveMQ Transport:
tcp:///127.0.0.1:32482@61613
2016-03-26 02:18:30,999 | ERROR | Could not accept connection :
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space |
org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server
Thread Handler:
stomp://127.0.0.1:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2016-03-26 02:18:37,485 | WARN | Exception occurred processing:
CONNECT
host:127.0.0.1
accept-version:1.0,1.1,1.2
passcode:*****
login:login
: java.lang.OutOfMemoryError: Java heap space |
org.apache.activemq.transport.stomp.ProtocolConverter | ActiveMQ Transport:
tcp:///127.0.0.1:32482@61613
2016-03-26 02:21:10,769 | ERROR | Could not accept connection :
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space |
org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server
Thread Handler:
stomp://127.0.0.1:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2016-03-26 02:21:10,769 | WARN | Transport Connection to:
tcp://127.0.0.1:32482 failed: java.io.IOException: Unexpected error occured:
java.lang.OutOfMemoryError: Java heap space |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
Transport: tcp:///127.0.0.1:32482@61613
2016-03-26 02:21:50,985 | ERROR | Could not accept connection :
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space |
org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server
Thread Handler:
stomp://127.0.0.1:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600
Can you please guide me, what could happen, why and what i should do that
this does not happen again?
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-hangs-and-i-don-t-understand-why-tp4709963.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ hangs and i don't understand why
Posted by Hayate <ze...@gmail.com>.
Well.
Our admin was unable configure server for connecting from jconsole. But i
read that 5.13.2 since 5.10.1 have many memory and cpu consuming fixes, we
update ActiveMQ and problem no longer appeared.
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-hangs-and-i-don-t-understand-why-tp4709963p4710647.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ hangs and i don't understand why
Posted by Tim Bain <tb...@alumni.duke.edu>.
Keep in mind that large numbers unconsumed messages stuck in the DLQ count
as a slow consumer and will use broker memory.
Is Producer Flow Control enabled? If so, what limits are you using?
What's your heap size? And how much of it is being used right after the
broker restarts?
On Mar 27, 2016 7:58 PM, "artnaseef" <ar...@artnaseef.com> wrote:
> Is the AMQ broker embedded? If so, perhaps another application is leaking
> memory.
>
> Or, are there any custom plugins?
>
> Bottom line here - there is something taking up all of the memory, causing
> the OOM condition. Taking heap dumps and/or histograms may be the only
> quick way to find out just what's consuming that memory.
>
> The heap charts in jconsole and visualvm can give an indication of whether
> this is a gradual problem over time (such as a slow memory leak), or a
> sudden event (such as an application with an oversize transaction flooding
> the broker).
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-hangs-and-i-don-t-understand-why-tp4709963p4709980.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: ActiveMQ hangs and i don't understand why
Posted by artnaseef <ar...@amlinv.com>.
For the JMX: there is a newer setting in the activemq configuration file
that may override the command-line settings; here's a fairly stock example:
<managementContext>
<managementContext createConnector="true"/>
</managementContext>
Check for these and any output during broker starting indicating the
management context failed to start -- if both try to start (via command
line and the config file), typically one fails. The following can help
track down the state of JMX as well:
- netstat -an | grep 1616
- netstat -an | grep LISTEN
- ps axu | grep java
The first two will show info on the ports open for listening on the
server. The third shows the actual command-line used to start ActiveMQ.
Note that JMX can be tricky to start behind a firewall because it tends to
allocate a second port dynamically. Adding a line like the following to
the ACTIVEMQ_SUNJMX_START may fix that issue:
-Dcom.sun.management.jmxremote.rmi.port=1615
Then make sure the same is open through the firewall. Note that JMX uses a
different method of connection for local processes, so it's entirely
possible that the JVM is not properly configured to allow network
connections.
On Mon, Mar 28, 2016 at 2:23 AM, Hayate [via ActiveMQ] <
ml-node+s2283324n4709982h90@n4.nabble.com> wrote:
> We use standalone broker, because we write our applications in PHP. And we
> don't use custom plugins.
>
> I try configure jmx for jconsole, on localhost all works well, but i can't
> connect to server over vpn.
> I use this manual: http://activemq.apache.org/jmx.html
>
> And here ACTIVEMQ_SUNJMX_START variable in startup script:
>
> ACTIVEMQ_SUNJMX_START=" -Dcom.sun.management.jmxremote.port=1616
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_BASE}/conf/jmx.password
>
> -Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_BASE}/conf/jmx.access
>
> -Djava.rmi.server.hostname=vpn_id_address
> -Dcom.sun.management.jmxremote.local.only=false"
>
> What i do wrong?
> Thanks for advices.
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://activemq.2283324.n4.nabble.com/ActiveMQ-hangs-and-i-don-t-understand-why-tp4709963p4709982.html
> To start a new topic under ActiveMQ - User, email
> ml-node+s2283324n2341805h3@n4.nabble.com
> To unsubscribe from ActiveMQ - User, click here
> <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFtbGludi5jb218MjM0MTgwNXwyMDc3NjQwODU5>
> .
> NAML
> <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-hangs-and-i-don-t-understand-why-tp4709963p4709987.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ hangs and i don't understand why
Posted by Hayate <ze...@gmail.com>.
We use standalone broker, because we write our applications in PHP. And we
don't use custom plugins.
I try configure jmx for jconsole, on localhost all works well, but i can't
connect to server over vpn.
I use this manual: http://activemq.apache.org/jmx.html
And here ACTIVEMQ_SUNJMX_START variable in startup script:
ACTIVEMQ_SUNJMX_START=" -Dcom.sun.management.jmxremote.port=1616
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_BASE}/conf/jmx.password
-Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_BASE}/conf/jmx.access
-Djava.rmi.server.hostname=vpn_id_address
-Dcom.sun.management.jmxremote.local.only=false"
What i do wrong?
Thanks for advices.
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-hangs-and-i-don-t-understand-why-tp4709963p4709982.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ hangs and i don't understand why
Posted by artnaseef <ar...@artnaseef.com>.
Is the AMQ broker embedded? If so, perhaps another application is leaking
memory.
Or, are there any custom plugins?
Bottom line here - there is something taking up all of the memory, causing
the OOM condition. Taking heap dumps and/or histograms may be the only
quick way to find out just what's consuming that memory.
The heap charts in jconsole and visualvm can give an indication of whether
this is a gradual problem over time (such as a slow memory leak), or a
sudden event (such as an application with an oversize transaction flooding
the broker).
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-hangs-and-i-don-t-understand-why-tp4709963p4709980.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ hangs and i don't understand why
Posted by Hayate <ze...@gmail.com>.
Messages consumed fast, never seen more than 30 messages in queue. 90% of
time i see 0 messages in all queues and only consumed counter inreases.
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-hangs-and-i-don-t-understand-why-tp4709963p4709978.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ hangs and i don't understand why
Posted by artnaseef <ar...@artnaseef.com>.
BTW, once the JVM reaches the out-of-memory condition, it can become
impossible to perform any operations on it - such as obtaining stack dumps
and heap dumps. With that said, a heap dump, or at least histogram, can
help determine the precise cause of the out-of-memory condition.
The jmap program provides one means to obtain heap dumps and histograms.
Again, though, it may be unable to do so once the broker JVM runs out of
memory, so it may be necessary to run it before hitting that condition in
the hopes of capturing the cause earlier. Watching heap usage with jconsole
or jvisualvm can also help here to determine if memory is "leaking" (i.e.
being consumed and not released more and more over time).
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-hangs-and-i-don-t-understand-why-tp4709963p4709975.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ hangs and i don't understand why
Posted by artnaseef <ar...@artnaseef.com>.
OutOfMemory most commonly occurs with slow consumption.
The broker can run out of memory if messages are produced to it too quickly
and consumers don't keep up. It will also run out of memory if massive
transactions are used and not committed nor rolled back before the broker's
memory is exhausted.
If this is the cause of problems, changing the JDK used won't eliminate it.
A good next diagnostic step is to watch Queues, and Topic subscriptions, for
slow consumption and see if that's related to the problem.
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-hangs-and-i-don-t-understand-why-tp4709963p4709974.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ hangs and i don't understand why
Posted by Hayate <ze...@gmail.com>.
Catch this situation again. Even web-console hangs. But this time no info
even in log, only startup log info.
Maybe openjdk is not the best choice?
java -version
openjdk version "1.7.0_80"
OpenJDK Runtime Environment (build 1.7.0_80-b15)
OpenJDK 64-Bit Server VM (build 24.80-b11, mixed mode)
Or ActiveMQ version 5.10.1 have some problems?
How i can analize problem., if i don't see any problem messages in logs?
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-hangs-and-i-don-t-understand-why-tp4709963p4709973.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.