Out of memory (Permgen?) + JMX attribute Health/CurrentStatus?

[jlindwall <jl...@yahoo.com>]

I have activemq configured with a durable subscriber to a topic backed by
oracle. I left a publisher running at a rate of 100 msgs/sec; each msg is
1K.  The msgs expire in 30 secs.  The durable subscriber disconnected so, as
expected, messages queued up in the db.  After about 25 minutes I got an
OutOfMemory error in my log; eventually the broker became unresponsive.

According to jstat permgen space usage is at 96% (!).  There are no permgen
messages in the logs though.

My questions:

1. I had lowered the max heap space to 512M as a test -- the jstat output
makes me thing this is ok, but maybe I need to boost permgen space?
2. How reliable/useful is this Health/CurrentStatus mbean attribute?  (see
below)
3. I know it is a longshot given the details I can provide, but any clues or
tips on what went wrong for me?

I'll see if I can reproduce the problem.

== More details

There are 320k messages loitering in the database; I suspect that expiration
is not working properly.  Any special tricks to enabling expiration? 

There are no ERROR messages in the logs, but there are WARN level msgs.

activemq.log:2014-02-06 22:58:41,527 | WARN  | JDBC Failure: Protocol
violation: [ 0, ] | org.apache.activemq.store.jdbc.JDBCPersistenceAdapter |
ActiveMQ Broker[localhost] Scheduler
...
activemq.log:2014-02-06 22:58:12,295 | WARN  | Failed to browse Topic:
fooTopic | org.apache.activemq.broker.region.Topic | ActiveMQ
Broker[localhost] Scheduler

There are some messages in the log but they do not have the log4j format so
I'm not sure the application generated them:

 - java.lang.OutOfMemoryError: GC overhead limit exceeded
 - java.lang.OutOfMemoryError: Java heap space

"jstat -gc 1000" shows a static situation and (to me) a healthy looking
heap, but permgen space usage is at 96% (!).  There are no permgen messages
in the logs though.

 S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC    
PU    YGC     YGCT    FGC    FGCT     GCT   
33536.0 32768.0  0.0    0.0   108544.0 16046.8   348160.0   286152.0 
24576.0 23536.2   1943   71.415 7421  8187.680 8259.095

I was expecting to use the JMX Health MBean attribute "CurrentStatus" as a
healthcheck datapoint for our monitoring system.  Today though I managed to
inadvertently induce an out of memory condition in activemq that seems to
have also led to a JDBC failure.  Connection attempts to the broker result
in a "javax.jms.JMSException: Wire format negotiation timeout: peer did not
send his wire format".  Persistent messages are lingering in the jdbc
datastore even though the messages are all expired.  The RESTful endpoint is
unresponsive.

I used jmxterm to gain access to the activemq mbeans.  Checking the "Health"
mbean I was expecting to see something like a "Critical" status in the
CurrentStatus attribute, but instead it says "Good".

How reliable/useful is this Health/CurrentStatus mbean attribute? 

Here is my jmxterm output:

$>bean org.apache.activemq:brokerName=localhost,service=Health,type=Broker
#bean is set to
org.apache.activemq:brokerName=localhost,service=Health,type=Broker
$>info
#mbean = org.apache.activemq:brokerName=localhost,service=Health,type=Broker
#class name = org.apache.activemq.broker.jmx.HealthView
# attributes
  %0   - CurrentStatus (java.lang.String, r)
# operations
  %0   - javax.management.openmbean.TabularData health()
  %1   - java.util.List healthList()
#there's no notifications
$>get CurrentStatus
#mbean =
org.apache.activemq:brokerName=localhost,service=Health,type=Broker:
CurrentStatus = Good;




--
View this message in context: http://activemq.2283324.n4.nabble.com/Out-of-memory-Permgen-JMX-attribute-Health-CurrentStatus-tp4677636.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.