You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by tejeswara <te...@gmail.com> on 2009/10/13 00:41:47 UTC
java/lang/OutOfMemoryError
Our broker is running on Linux (RHEL4 box) and after 1 or 2 days I see below
error in logs.
Note that, below values are set in bin/activemq in out test lab:
if [ -z "$ACTIVEMQ_OPTS" ] ; then
ACTIVEMQ_OPTS="-Xmx512M -Dorg.apache.activemq.UseDedicatedTaskRunner=true"
fi
I tried to stress test broker by creating 200 queues and put 5000 messages
(each about 1500 bytes size)
in each of these queues. However I could not see any error. Below error
occured when broker is idle.
>From logs, only activity during this idle period I see is 'DEBUG
InactivityMonitor - 30177 ms elapsed since last read check'
Can someone help in pointing out what is going wrong?
Note that, OutOfMemoryError is seen in our production even when heap size is
bumped up to 1024 and
dedicated task runner is set to false. This error usvally occurs after 2
days broker running without restart.
if [ -z "$ACTIVEMQ_OPTS" ] ; then
ACTIVEMQ_OPTS="-Xmx1024M
-Dorg.apache.activemq.UseDedicatedTaskRunner=false"
fi
Stack Trace:->
JVMDUMP006I Processing Dump Event "uncaught", detail
"java/lang/OutOfMemoryError" - Please Wait.
JVMDUMP007I JVM Requesting Snap Dump using
'/x/web/STAGE2SC5464/idimqbroker/Snap0001.20091010.015439.16581.trc'
JVMDUMP010I Snap Dump written to
/x/web/STAGE2SC5464/idimqbroker/Snap0001.20091010.015439.16581.trc
JVMDUMP007I JVM Requesting Heap Dump using
'/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.015439.16581.phd'
JVMDUMP010I Heap Dump written to
/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.015439.16581.phd
JVMDUMP007I JVM Requesting Java Dump using
'/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.015439.16581.txt'
JVMDUMP010I Java Dump written to
/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.015439.16581.txt
JVMDUMP013I Processed Dump Event "uncaught", detail
"java/lang/OutOfMemoryError".
Exception in thread "MulticastDiscovery: null" java.lang.OutOfMemoryError:
heap allocation failed
at java.net.PlainDatagramSocketImpl.receive0(Native Method)
at
java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:181)
at java.net.DatagramSocket.receive(DatagramSocket.java:724)
at
org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent.run(MulticastDiscoveryAgent.java:338)
at java.lang.Thread.run(Thread.java:803)
JVMDUMP006I Processing Dump Event "uncaught", detail
"java/lang/OutOfMemoryError" - Please Wait.
JVMDUMP007I JVM Requesting Snap Dump using
'/x/web/STAGE2SC5464/idimqbroker/Snap0002.20091010.034616.16581.trc'
JVMDUMP010I Snap Dump written to
/x/web/STAGE2SC5464/idimqbroker/Snap0002.20091010.034616.16581.trc
JVMDUMP007I JVM Requesting Heap Dump using
'/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034616.16581.phd'
JVMDUMP016E Aborting: Cannot create file
(/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034616.16581.phd)
JVMDUMP012E Error in Heap Dump: binary dump file cannot be written
JVMDUMP007I JVM Requesting Java Dump using
'/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034616.16581.txt'
UTE503: utsGetTracePointIterator cannot allocate iterator's buffer
JVMDUMP010I Java Dump written to
/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034616.16581.txt
JVMDUMP013I Processed Dump Event "uncaught", detail
"java/lang/OutOfMemoryError".
Exception in thread "MulticastDiscovery: tcp://stage2sc5464.:61616"
java.lang.OutOfMemoryError: heap allocation failed
at java.net.PlainDatagramSocketImpl.receive0(Native Method)
at
java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:181)
at java.net.DatagramSocket.receive(DatagramSocket.java:724)
at
org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent.run(MulticastDiscoveryAgent.java:338)
at java.lang.Thread.run(Thread.java:803)
JVMDUMP006I Processing Dump Event "uncaught", detail
"java/lang/OutOfMemoryError" - Please Wait.
JVMDUMP007I JVM Requesting Snap Dump using
'/x/web/STAGE2SC5464/idimqbroker/Snap0003.20091010.034640.16581.trc'
JVMDUMP010I Snap Dump written to
/x/web/STAGE2SC5464/idimqbroker/Snap0003.20091010.034640.16581.trc
JVMDUMP007I JVM Requesting Heap Dump using
'/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034640.16581.phd'
JVMDUMP016E Aborting: Cannot create file
(/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034640.16581.phd)
JVMDUMP012E Error in Heap Dump: binary dump file cannot be written
JVMDUMP007I JVM Requesting Java Dump using
'/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034640.16581.txt'
UTE503: utsGetTracePointIterator cannot allocate iterator's buffer
JVMDUMP010I Java Dump written to
/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034640.16581.txt
JVMDUMP013I Processed Dump Event "uncaught", detail
"java/lang/OutOfMemoryError".
Exception in thread "InactivityMonitor WriteCheck"
java.lang.OutOfMemoryError
at java.security.AccessController.getProtectionDomains(Native
Method)
at
java.security.AccessController.getContext(AccessController.java:132)
at java.lang.Thread.initialize(Thread.java:347)
at java.lang.Thread.<init>(Thread.java:278)
at java.lang.Thread.<init>(Thread.java:170)
at
org.apache.activemq.transport.InactivityMonitor$5.newThread(InactivityMonitor.java:303)
at
java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:411)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:463)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:878)
at
org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:128)
at
org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:103)
at
org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
at java.util.TimerThread.mainLoop(Timer.java:527)
at java.util.TimerThread.run(Timer.java:477)
Exception in thread "RMI TCP Connection(454)-10.15.26.210"
java.lang.NoClassDefFoundError: java.io.StreamCorruptedException
at
sun.rmi.transport.StreamRemoteCall.getResultStream(StreamRemoteCall.java:177)
at
sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:335)
at sun.rmi.transport.Transport$1.run(Transport.java:168)
at
java.security.AccessController.doPrivileged(AccessController.java:275)
java.lang.NoClassDefFoundError: java.rmi.UnmarshalException
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:506)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.handleRequest(TCPTransport.java:838)
at
sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:912)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:358)
at java.lang.Thread.run(Thread.java:803)
at sun.rmi.transport.DGCImpl_Stub.dirty(DGCImpl_Stub.java:67)
at
sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:344)
at
sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:160)
at
sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:555)
at java.lang.Thread.run(Thread.java:803)
Exception in thread "RMI LeaseChecker" java.lang.NoClassDefFoundError:
java.rmi.server.Unreferenced
at sun.rmi.transport.Target.refSetRemove(Target.java:320)
at sun.rmi.transport.Target.vmidDead(Target.java:444)
at sun.rmi.transport.DGCImpl.checkLeases(DGCImpl.java:265)
at sun.rmi.transport.DGCImpl.access$400(DGCImpl.java:47)
at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:329)
at java.lang.Thread.run(Thread.java:803)
Unhandled exception
Type=Segmentation error vmState=0x00053fff
J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000
Signal_Code=00000001
Handler1=008F64F0 Handler2=00142477 InaccessibleAddress=00000000
EDI=00000000 ESI=27E112E0 EAX=00000000 EBX=00000000
ECX=08B7DF40 EDX=00000000
EIP=00F97DB4 ES=0000007B DS=C02D007B ESP=00855BB8
EFlags=00210206 CS=00000073 SS=0000007B EBP=00855BB8
Module=/x/web/STAGE2SC5464/idimqbroker/java/jre/bin/libj9jit23.so
Module_base_address=00F09000
Method_being_compiled=org/springframework/jms/listener/DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run()V
Target=2_30_20070524_12771_lHdSMR (Linux 2.6.9-55.ELsmp)
CPU=x86 (8 logical CPUs) (0x47621d000 RAM)
JVMDUMP006I Processing Dump Event "gpf", detail "" - Please Wait.
JVMDUMP007I JVM Requesting System Dump using
'/x/web/STAGE2SC5464/idimqbroker/core.20091011.220932.16581.dmp'
JVMDUMP012E Error in System Dump: cannot find core file, check "ulimit -c"
is set high enough
JVMDUMP007I JVM Requesting Snap Dump using
'/x/web/STAGE2SC5464/idimqbroker/Snap0004.20091011.220932.16581.trc'
JVMDUMP010I Snap Dump written to
/x/web/STAGE2SC5464/idimqbroker/Snap0004.20091011.220932.16581.trc
JVMDUMP007I JVM Requesting Java Dump using
'/tmp/javacore.20091011.220932.16581.txt'
NULL
------------------------------------------------------------------------
0SECTION TITLE subcomponent dump routine
NULL ===============================
1TISIGINFO Dump Event "gpf" (00002000) received
1TIDATETIME Date: 2009/10/11 at 22:10:17
1TIFILENAME Javacore filename: /tmp/javacore.20091011.220932.16581.txt
NULL
------------------------------------------------------------------------
0SECTION GPINFO subcomponent dump routine
NULL ================================
2XHOSLEVEL OS Level : Linux 2.6.9-55.ELsmp
2XHCPUS Processors -
3XHCPUARCH Architecture : x86
3XHNUMCPUS How Many : 8
NULL
--
View this message in context: http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25864372.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: java/lang/OutOfMemoryError
Posted by tejeswara <te...@gmail.com>.
I shall test with disable multicast dicovery. Thanks.
On production we are seeing below exception. Is it also related multicast
discovery or Inactity monitor?
I am also turning off inactivity monitor thread with
maxInactivityDuration=0:
<transportConnector name="openwire"
uri="tcp://localhost:61616?wireFormat.maxInactivityDuration=0"
JVMDUMP013I Processed Dump Event "uncaught", detail
"java/lang/OutOfMemoryError".
Exception in thread "DefaultMessageListenerContainer-359522" Exception in
thread "InactivityMonitor WriteCheck" java.lang.OutOfMemoryError: Failed to
fork OS thread
java.lang.OutOfMemoryError: Failed to fork OS thread
at java.lang.Thread.startImpl(Native Method)
at java.lang.Thread.startImpl(Native Method)
at java.lang.Thread.start(Thread.java:970)
at java.lang.Thread.start(Thread.java:970)
at
org.springframework.core.task.SimpleAsyncTaskExecutor.doExecute(SimpleAsyncTaskExecutor.java:157)
Gary Tully wrote:
>
> The networkConnector is using multicast discovery - seeking brokers to
> network with.
>
> The transportConnector is publishing its endpoint over multicast so making
> it available to other brokers who wish to connect with it.
>
> AMQ-2283 is related to the networkConnector, so you should use a static
> url
> if you still want to create a network.
>
> 2009/10/17 tejeswara <te...@gmail.com>
>
>>
>> Gary,
>>
>> In my config file I have multicast in both these entries. Which one you
>> think possibly causing this memory leak?. We plan to have failover broker
>> so
>> we may need multicast discovery?.
>>
>> <networkConnectors>
>> <networkConnector name="default" uri="multicast://default"/>
>> </networkConnectors>
>>
>> <transportConnectors>
>> <transportConnector name="default" uri="tcp://localhost:61616"
>> discoveryUri="multicast://default"/>
>> </transportConnectors>
>>
>>
>>
>>
>>
>> Gary Tully wrote:
>> >
>> > the config looks ok, just one thought, do you need multicast discovery
>> on
>> > your transport connector? I wonder if it is possible you are seeing
>> > https://issues.apache.org/activemq/browse/AMQ-2283 which resulted in an
>> > leak
>> > of discovery threads? Removing the discoveryUri will work around it.
>> >
>> > 2009/10/13 tejeswara <te...@gmail.com>
>> >
>> >>
>> >> Here is our activemq.xml. As you can see it is pretty much same as the
>> >> template version we get from activemq package:
>> >> <beans
>> >> xmlns="http://www.springframework.org/schema/beans"
>> >> xmlns:amq="http://activemq.apache.org/schema/core"
>> >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> >> xsi:schemaLocation="http://www.springframework.org/schema/beans
>> >> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>> >> http://activemq.apache.org/schema/core
>> >> http://activemq.apache.org/schema/core/activemq-core.xsd
>> >> http://activemq.apache.org/camel/schema/spring
>> >> http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">
>> >> <bean
>> >>
>> >>
>> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>> >> <property name="locations">
>> >>
>> >> <value>file:///${activemq.base}/conf/credentials.properties</value>
>> >> </property>
>> >> </bean>
>> >> <broker xmlns="http://activemq.apache.org/schema/core"
>> >> brokerName="localhost" dataDirectory="${activemq.base}/data">
>> >> <destinationPolicy>
>> >> <policyMap>
>> >> <policyEntries>
>> >> <policyEntry queue=">" memoryLimit="50mb"/>
>> >> <policyEntry topic=">" memoryLimit="5mb">
>> >> </policyEntry>
>> >> </policyEntries>
>> >> </policyMap>
>> >> </destinationPolicy>
>> >> <managementContext>
>> >> <managementContext createConnector="true"
>> >> connectorPort="11099"/>
>> >> </managementContext>
>> >> <networkConnectors>
>> >> <networkConnector name="default-nc"
>> >> uri="multicast://default"/>
>> >> </networkConnectors>
>> >> <persistenceAdapter>
>> >> <amqPersistenceAdapter syncOnWrite="false"
>> >> directory="${activemq.base}/data" maxFileLength="20 mb"/>
>> >> </persistenceAdapter>
>> >> <systemUsage>
>> >> <systemUsage>
>> >> <memoryUsage>
>> >> <memoryUsage limit="200 mb"/>
>> >> </memoryUsage>
>> >> <storeUsage>
>> >> <storeUsage limit="1 gb" name="foo"/>
>> >> </storeUsage>
>> >> <tempUsage>
>> >> <tempUsage limit="100 mb"/>
>> >> </tempUsage>
>> >> </systemUsage>
>> >> </systemUsage>
>> >> <transportConnectors>
>> >> <transportConnector name="openwire"
>> >> uri="tcp://localhost:61616"
>> >> discoveryUri="multicast://default"/>
>> >> </transportConnectors>
>> >> </broker>
>> >> <camelContext id="camel"
>> >> xmlns="http://activemq.apache.org/camel/schema/spring">
>> >> <package>org.foo.bar</package>
>> >> <route>
>> >> <from uri="activemq:example.A"/>
>> >> <to uri="activemq:example.B"/>
>> >> </route>
>> >> </camelContext>
>> >> <bean id="activemq"
>> >> class="org.apache.activemq.camel.component.ActiveMQComponent" >
>> >> <property name="connectionFactory">
>> >> <bean class="org.apache.activemq.ActiveMQConnectionFactory">
>> >> <property name="brokerURL"
>> >> value="vm://localhost?create=false&waitForStart=10000" />
>> >> <property name="userName" value="${activemq.username}"/>
>> >> <property name="password" value="${activemq.password}"/>
>> >> </bean>
>> >> </property>
>> >> </bean>
>> >> <jetty xmlns="http://mortbay.com/schemas/jetty/1.0">
>> >> <connectors>
>> >> <nioConnector port="11098"/>
>> >> </connectors>
>> >> <handlers>
>> >> <webAppContext contextPath="/admin"
>> >> resourceBase="${activemq.base}/webapps/admin" logUrlOnStart="true"/>
>> >> <webAppContext contextPath="/demo"
>> >> resourceBase="${activemq.base}/webapps/demo" logUrlOnStart="true"/>
>> >> <webAppContext contextPath="/fileserver"
>> >> resourceBase="${activemq.base}/webapps/fileserver"
>> logUrlOnStart="true"/>
>> >> </handlers>
>> >> </jetty>
>> >> </beans>
>> >>
>> >>
>> >>
>> >> Gary Tully wrote:
>> >> >
>> >> > Can you post your activemq.xml? It looks like you need to restrict
>> the
>> >> > memory usage of the broker and have it spool messages to disk when
>> the
>> >> > memory limits are reached. See some additional detail at
>> >> >
>> >>
>> http://activemq.apache.org/javalangoutofmemory.html#java.lang.OutOfMemory-SpoolingMessagestoDisk
>> >> > *
>> >> >
>> >> > *
>> >> > 2009/10/12 tejeswara <te...@gmail.com>
>> >> >
>> >> >>
>> >> >> Our broker is running on Linux (RHEL4 box) and after 1 or 2 days I
>> see
>> >> >> below
>> >> >> error in logs.
>> >> >> Note that, below values are set in bin/activemq in out test lab:
>> >> >> if [ -z "$ACTIVEMQ_OPTS" ] ; then
>> >> >> ACTIVEMQ_OPTS="-Xmx512M
>> >> >> -Dorg.apache.activemq.UseDedicatedTaskRunner=true"
>> >> >> fi
>> >> >>
>> >> >> I tried to stress test broker by creating 200 queues and put 5000
>> >> >> messages
>> >> >> (each about 1500 bytes size)
>> >> >> in each of these queues. However I could not see any error. Below
>> >> error
>> >> >> occured when broker is idle.
>> >> >> From logs, only activity during this idle period I see is 'DEBUG
>> >> >> InactivityMonitor - 30177 ms elapsed since last read
>> >> check'
>> >> >>
>> >> >> Can someone help in pointing out what is going wrong?
>> >> >>
>> >> >> Note that, OutOfMemoryError is seen in our production even when
>> heap
>> >> size
>> >> >> is
>> >> >> bumped up to 1024 and
>> >> >> dedicated task runner is set to false. This error usvally occurs
>> after
>> >> 2
>> >> >> days broker running without restart.
>> >> >> if [ -z "$ACTIVEMQ_OPTS" ] ; then
>> >> >> ACTIVEMQ_OPTS="-Xmx1024M
>> >> >> -Dorg.apache.activemq.UseDedicatedTaskRunner=false"
>> >> >> fi
>> >> >>
>> >> >>
>> >> >> Stack Trace:->
>> >> >>
>> >> >> JVMDUMP006I Processing Dump Event "uncaught", detail
>> >> >> "java/lang/OutOfMemoryError" - Please Wait.
>> >> >> JVMDUMP007I JVM Requesting Snap Dump using
>> >> >>
>> '/x/web/STAGE2SC5464/idimqbroker/Snap0001.20091010.015439.16581.trc'
>> >> >> JVMDUMP010I Snap Dump written to
>> >> >> /x/web/STAGE2SC5464/idimqbroker/Snap0001.20091010.015439.16581.trc
>> >> >> JVMDUMP007I JVM Requesting Heap Dump using
>> >> >>
>> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.015439.16581.phd'
>> >> >> JVMDUMP010I Heap Dump written to
>> >> >> /x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.015439.16581.phd
>> >> >> JVMDUMP007I JVM Requesting Java Dump using
>> >> >>
>> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.015439.16581.txt'
>> >> >> JVMDUMP010I Java Dump written to
>> >> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.015439.16581.txt
>> >> >> JVMDUMP013I Processed Dump Event "uncaught", detail
>> >> >> "java/lang/OutOfMemoryError".
>> >> >> Exception in thread "MulticastDiscovery: null"
>> >> >> java.lang.OutOfMemoryError:
>> >> >> heap allocation failed
>> >> >> at java.net.PlainDatagramSocketImpl.receive0(Native Method)
>> >> >> at
>> >> >>
>> >>
>> java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:181)
>> >> >> at java.net.DatagramSocket.receive(DatagramSocket.java:724)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent.run(MulticastDiscoveryAgent.java:338)
>> >> >> at java.lang.Thread.run(Thread.java:803)
>> >> >> JVMDUMP006I Processing Dump Event "uncaught", detail
>> >> >> "java/lang/OutOfMemoryError" - Please Wait.
>> >> >> JVMDUMP007I JVM Requesting Snap Dump using
>> >> >>
>> '/x/web/STAGE2SC5464/idimqbroker/Snap0002.20091010.034616.16581.trc'
>> >> >> JVMDUMP010I Snap Dump written to
>> >> >> /x/web/STAGE2SC5464/idimqbroker/Snap0002.20091010.034616.16581.trc
>> >> >> JVMDUMP007I JVM Requesting Heap Dump using
>> >> >>
>> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034616.16581.phd'
>> >> >> JVMDUMP016E Aborting: Cannot create file
>> >> >>
>> (/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034616.16581.phd)
>> >> >> JVMDUMP012E Error in Heap Dump: binary dump file cannot be written
>> >> >> JVMDUMP007I JVM Requesting Java Dump using
>> >> >>
>> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034616.16581.txt'
>> >> >> UTE503: utsGetTracePointIterator cannot allocate iterator's buffer
>> >> >> JVMDUMP010I Java Dump written to
>> >> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034616.16581.txt
>> >> >> JVMDUMP013I Processed Dump Event "uncaught", detail
>> >> >> "java/lang/OutOfMemoryError".
>> >> >> Exception in thread "MulticastDiscovery: tcp://stage2sc5464.:61616"
>> >> >> java.lang.OutOfMemoryError: heap allocation failed
>> >> >> at java.net.PlainDatagramSocketImpl.receive0(Native Method)
>> >> >> at
>> >> >>
>> >>
>> java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:181)
>> >> >> at java.net.DatagramSocket.receive(DatagramSocket.java:724)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent.run(MulticastDiscoveryAgent.java:338)
>> >> >> at java.lang.Thread.run(Thread.java:803)
>> >> >> JVMDUMP006I Processing Dump Event "uncaught", detail
>> >> >> "java/lang/OutOfMemoryError" - Please Wait.
>> >> >> JVMDUMP007I JVM Requesting Snap Dump using
>> >> >>
>> '/x/web/STAGE2SC5464/idimqbroker/Snap0003.20091010.034640.16581.trc'
>> >> >> JVMDUMP010I Snap Dump written to
>> >> >> /x/web/STAGE2SC5464/idimqbroker/Snap0003.20091010.034640.16581.trc
>> >> >> JVMDUMP007I JVM Requesting Heap Dump using
>> >> >>
>> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034640.16581.phd'
>> >> >> JVMDUMP016E Aborting: Cannot create file
>> >> >>
>> (/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034640.16581.phd)
>> >> >> JVMDUMP012E Error in Heap Dump: binary dump file cannot be written
>> >> >> JVMDUMP007I JVM Requesting Java Dump using
>> >> >>
>> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034640.16581.txt'
>> >> >> UTE503: utsGetTracePointIterator cannot allocate iterator's buffer
>> >> >> JVMDUMP010I Java Dump written to
>> >> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034640.16581.txt
>> >> >> JVMDUMP013I Processed Dump Event "uncaught", detail
>> >> >> "java/lang/OutOfMemoryError".
>> >> >> Exception in thread "InactivityMonitor WriteCheck"
>> >> >> java.lang.OutOfMemoryError
>> >> >> at
>> java.security.AccessController.getProtectionDomains(Native
>> >> >> Method)
>> >> >> at
>> >> >>
>> java.security.AccessController.getContext(AccessController.java:132)
>> >> >> at java.lang.Thread.initialize(Thread.java:347)
>> >> >> at java.lang.Thread.<init>(Thread.java:278)
>> >> >> at java.lang.Thread.<init>(Thread.java:170)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.activemq.transport.InactivityMonitor$5.newThread(InactivityMonitor.java:303)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:411)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:463)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:878)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:128)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:103)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
>> >> >> at java.util.TimerThread.mainLoop(Timer.java:527)
>> >> >> at java.util.TimerThread.run(Timer.java:477)
>> >> >> Exception in thread "RMI TCP Connection(454)-10.15.26.210"
>> >> >> java.lang.NoClassDefFoundError: java.io.StreamCorruptedException
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> sun.rmi.transport.StreamRemoteCall.getResultStream(StreamRemoteCall.java:177)
>> >> >> at
>> >> >> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:335)
>> >> >> at sun.rmi.transport.Transport$1.run(Transport.java:168)
>> >> >> at
>> >> >>
>> java.security.AccessController.doPrivileged(AccessController.java:275)
>> >> >> java.lang.NoClassDefFoundError: java.rmi.UnmarshalException
>> >> >> at
>> >> >>
>> >>
>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:506)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.handleRequest(TCPTransport.java:838)
>> >> >> at
>> >> >>
>> >>
>> sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:912)
>> >> >> at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:358)
>> >> >> at java.lang.Thread.run(Thread.java:803)
>> >> >> at
>> sun.rmi.transport.DGCImpl_Stub.dirty(DGCImpl_Stub.java:67)
>> >> >> at
>> >> >>
>> >>
>> sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:344)
>> >> >> at
>> >> >>
>> >>
>> sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:160)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:555)
>> >> >> at java.lang.Thread.run(Thread.java:803)
>> >> >> Exception in thread "RMI LeaseChecker"
>> java.lang.NoClassDefFoundError:
>> >> >> java.rmi.server.Unreferenced
>> >> >> at sun.rmi.transport.Target.refSetRemove(Target.java:320)
>> >> >> at sun.rmi.transport.Target.vmidDead(Target.java:444)
>> >> >> at sun.rmi.transport.DGCImpl.checkLeases(DGCImpl.java:265)
>> >> >> at sun.rmi.transport.DGCImpl.access$400(DGCImpl.java:47)
>> >> >> at
>> sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:329)
>> >> >> at java.lang.Thread.run(Thread.java:803)
>> >> >> Unhandled exception
>> >> >> Type=Segmentation error vmState=0x00053fff
>> >> >> J9Generic_Signal_Number=00000004 Signal_Number=0000000b
>> >> >> Error_Value=00000000
>> >> >> Signal_Code=00000001
>> >> >> Handler1=008F64F0 Handler2=00142477 InaccessibleAddress=00000000
>> >> >> EDI=00000000 ESI=27E112E0 EAX=00000000 EBX=00000000
>> >> >> ECX=08B7DF40 EDX=00000000
>> >> >> EIP=00F97DB4 ES=0000007B DS=C02D007B ESP=00855BB8
>> >> >> EFlags=00210206 CS=00000073 SS=0000007B EBP=00855BB8
>> >> >> Module=/x/web/STAGE2SC5464/idimqbroker/java/jre/bin/libj9jit23.so
>> >> >> Module_base_address=00F09000
>> >> >>
>> >> >>
>> >> >>
>> >>
>> Method_being_compiled=org/springframework/jms/listener/DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run()V
>> >> >> Target=2_30_20070524_12771_lHdSMR (Linux 2.6.9-55.ELsmp)
>> >> >> CPU=x86 (8 logical CPUs) (0x47621d000 RAM)
>> >> >> JVMDUMP006I Processing Dump Event "gpf", detail "" - Please Wait.
>> >> >> JVMDUMP007I JVM Requesting System Dump using
>> >> >> '/x/web/STAGE2SC5464/idimqbroker/core.20091011.220932.16581.dmp'
>> >> >> JVMDUMP012E Error in System Dump: cannot find core file, check
>> "ulimit
>> >> >> -c"
>> >> >> is set high enough
>> >> >> JVMDUMP007I JVM Requesting Snap Dump using
>> >> >>
>> '/x/web/STAGE2SC5464/idimqbroker/Snap0004.20091011.220932.16581.trc'
>> >> >> JVMDUMP010I Snap Dump written to
>> >> >> /x/web/STAGE2SC5464/idimqbroker/Snap0004.20091011.220932.16581.trc
>> >> >> JVMDUMP007I JVM Requesting Java Dump using
>> >> >> '/tmp/javacore.20091011.220932.16581.txt'
>> >> >> NULL
>> >> >>
>> >>
>> ------------------------------------------------------------------------
>> >> >> 0SECTION TITLE subcomponent dump routine
>> >> >> NULL ===============================
>> >> >> 1TISIGINFO Dump Event "gpf" (00002000) received
>> >> >> 1TIDATETIME Date: 2009/10/11 at 22:10:17
>> >> >> 1TIFILENAME Javacore filename:
>> >> >> /tmp/javacore.20091011.220932.16581.txt
>> >> >> NULL
>> >> >>
>> >>
>> ------------------------------------------------------------------------
>> >> >> 0SECTION GPINFO subcomponent dump routine
>> >> >> NULL ================================
>> >> >> 2XHOSLEVEL OS Level : Linux 2.6.9-55.ELsmp
>> >> >> 2XHCPUS Processors -
>> >> >> 3XHCPUARCH Architecture : x86
>> >> >> 3XHNUMCPUS How Many : 8
>> >> >> NULL
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> View this message in context:
>> >> >>
>> >>
>> http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25864372.html
>> >> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >> > --
>> >> > http://blog.garytully.com
>> >> >
>> >> > Open Source Integration
>> >> > http://fusesource.com
>> >> >
>> >> >
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25881255.html
>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >>
>> >>
>> >
>> >
>> > --
>> > http://blog.garytully.com
>> >
>> > Open Source Integration
>> > http://fusesource.com
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25935327.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>
>
--
View this message in context: http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25959853.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: java/lang/OutOfMemoryError
Posted by Gary Tully <ga...@gmail.com>.
The networkConnector is using multicast discovery - seeking brokers to
network with.
The transportConnector is publishing its endpoint over multicast so making
it available to other brokers who wish to connect with it.
AMQ-2283 is related to the networkConnector, so you should use a static url
if you still want to create a network.
2009/10/17 tejeswara <te...@gmail.com>
>
> Gary,
>
> In my config file I have multicast in both these entries. Which one you
> think possibly causing this memory leak?. We plan to have failover broker
> so
> we may need multicast discovery?.
>
> <networkConnectors>
> <networkConnector name="default" uri="multicast://default"/>
> </networkConnectors>
>
> <transportConnectors>
> <transportConnector name="default" uri="tcp://localhost:61616"
> discoveryUri="multicast://default"/>
> </transportConnectors>
>
>
>
>
>
> Gary Tully wrote:
> >
> > the config looks ok, just one thought, do you need multicast discovery
> on
> > your transport connector? I wonder if it is possible you are seeing
> > https://issues.apache.org/activemq/browse/AMQ-2283 which resulted in an
> > leak
> > of discovery threads? Removing the discoveryUri will work around it.
> >
> > 2009/10/13 tejeswara <te...@gmail.com>
> >
> >>
> >> Here is our activemq.xml. As you can see it is pretty much same as the
> >> template version we get from activemq package:
> >> <beans
> >> xmlns="http://www.springframework.org/schema/beans"
> >> xmlns:amq="http://activemq.apache.org/schema/core"
> >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> >> xsi:schemaLocation="http://www.springframework.org/schema/beans
> >> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> >> http://activemq.apache.org/schema/core
> >> http://activemq.apache.org/schema/core/activemq-core.xsd
> >> http://activemq.apache.org/camel/schema/spring
> >> http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">
> >> <bean
> >>
> >>
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> >> <property name="locations">
> >>
> >> <value>file:///${activemq.base}/conf/credentials.properties</value>
> >> </property>
> >> </bean>
> >> <broker xmlns="http://activemq.apache.org/schema/core"
> >> brokerName="localhost" dataDirectory="${activemq.base}/data">
> >> <destinationPolicy>
> >> <policyMap>
> >> <policyEntries>
> >> <policyEntry queue=">" memoryLimit="50mb"/>
> >> <policyEntry topic=">" memoryLimit="5mb">
> >> </policyEntry>
> >> </policyEntries>
> >> </policyMap>
> >> </destinationPolicy>
> >> <managementContext>
> >> <managementContext createConnector="true"
> >> connectorPort="11099"/>
> >> </managementContext>
> >> <networkConnectors>
> >> <networkConnector name="default-nc"
> >> uri="multicast://default"/>
> >> </networkConnectors>
> >> <persistenceAdapter>
> >> <amqPersistenceAdapter syncOnWrite="false"
> >> directory="${activemq.base}/data" maxFileLength="20 mb"/>
> >> </persistenceAdapter>
> >> <systemUsage>
> >> <systemUsage>
> >> <memoryUsage>
> >> <memoryUsage limit="200 mb"/>
> >> </memoryUsage>
> >> <storeUsage>
> >> <storeUsage limit="1 gb" name="foo"/>
> >> </storeUsage>
> >> <tempUsage>
> >> <tempUsage limit="100 mb"/>
> >> </tempUsage>
> >> </systemUsage>
> >> </systemUsage>
> >> <transportConnectors>
> >> <transportConnector name="openwire"
> >> uri="tcp://localhost:61616"
> >> discoveryUri="multicast://default"/>
> >> </transportConnectors>
> >> </broker>
> >> <camelContext id="camel"
> >> xmlns="http://activemq.apache.org/camel/schema/spring">
> >> <package>org.foo.bar</package>
> >> <route>
> >> <from uri="activemq:example.A"/>
> >> <to uri="activemq:example.B"/>
> >> </route>
> >> </camelContext>
> >> <bean id="activemq"
> >> class="org.apache.activemq.camel.component.ActiveMQComponent" >
> >> <property name="connectionFactory">
> >> <bean class="org.apache.activemq.ActiveMQConnectionFactory">
> >> <property name="brokerURL"
> >> value="vm://localhost?create=false&waitForStart=10000" />
> >> <property name="userName" value="${activemq.username}"/>
> >> <property name="password" value="${activemq.password}"/>
> >> </bean>
> >> </property>
> >> </bean>
> >> <jetty xmlns="http://mortbay.com/schemas/jetty/1.0">
> >> <connectors>
> >> <nioConnector port="11098"/>
> >> </connectors>
> >> <handlers>
> >> <webAppContext contextPath="/admin"
> >> resourceBase="${activemq.base}/webapps/admin" logUrlOnStart="true"/>
> >> <webAppContext contextPath="/demo"
> >> resourceBase="${activemq.base}/webapps/demo" logUrlOnStart="true"/>
> >> <webAppContext contextPath="/fileserver"
> >> resourceBase="${activemq.base}/webapps/fileserver"
> logUrlOnStart="true"/>
> >> </handlers>
> >> </jetty>
> >> </beans>
> >>
> >>
> >>
> >> Gary Tully wrote:
> >> >
> >> > Can you post your activemq.xml? It looks like you need to restrict the
> >> > memory usage of the broker and have it spool messages to disk when the
> >> > memory limits are reached. See some additional detail at
> >> >
> >>
> http://activemq.apache.org/javalangoutofmemory.html#java.lang.OutOfMemory-SpoolingMessagestoDisk
> >> > *
> >> >
> >> > *
> >> > 2009/10/12 tejeswara <te...@gmail.com>
> >> >
> >> >>
> >> >> Our broker is running on Linux (RHEL4 box) and after 1 or 2 days I
> see
> >> >> below
> >> >> error in logs.
> >> >> Note that, below values are set in bin/activemq in out test lab:
> >> >> if [ -z "$ACTIVEMQ_OPTS" ] ; then
> >> >> ACTIVEMQ_OPTS="-Xmx512M
> >> >> -Dorg.apache.activemq.UseDedicatedTaskRunner=true"
> >> >> fi
> >> >>
> >> >> I tried to stress test broker by creating 200 queues and put 5000
> >> >> messages
> >> >> (each about 1500 bytes size)
> >> >> in each of these queues. However I could not see any error. Below
> >> error
> >> >> occured when broker is idle.
> >> >> From logs, only activity during this idle period I see is 'DEBUG
> >> >> InactivityMonitor - 30177 ms elapsed since last read
> >> check'
> >> >>
> >> >> Can someone help in pointing out what is going wrong?
> >> >>
> >> >> Note that, OutOfMemoryError is seen in our production even when heap
> >> size
> >> >> is
> >> >> bumped up to 1024 and
> >> >> dedicated task runner is set to false. This error usvally occurs
> after
> >> 2
> >> >> days broker running without restart.
> >> >> if [ -z "$ACTIVEMQ_OPTS" ] ; then
> >> >> ACTIVEMQ_OPTS="-Xmx1024M
> >> >> -Dorg.apache.activemq.UseDedicatedTaskRunner=false"
> >> >> fi
> >> >>
> >> >>
> >> >> Stack Trace:->
> >> >>
> >> >> JVMDUMP006I Processing Dump Event "uncaught", detail
> >> >> "java/lang/OutOfMemoryError" - Please Wait.
> >> >> JVMDUMP007I JVM Requesting Snap Dump using
> >> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0001.20091010.015439.16581.trc'
> >> >> JVMDUMP010I Snap Dump written to
> >> >> /x/web/STAGE2SC5464/idimqbroker/Snap0001.20091010.015439.16581.trc
> >> >> JVMDUMP007I JVM Requesting Heap Dump using
> >> >> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.015439.16581.phd'
> >> >> JVMDUMP010I Heap Dump written to
> >> >> /x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.015439.16581.phd
> >> >> JVMDUMP007I JVM Requesting Java Dump using
> >> >> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.015439.16581.txt'
> >> >> JVMDUMP010I Java Dump written to
> >> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.015439.16581.txt
> >> >> JVMDUMP013I Processed Dump Event "uncaught", detail
> >> >> "java/lang/OutOfMemoryError".
> >> >> Exception in thread "MulticastDiscovery: null"
> >> >> java.lang.OutOfMemoryError:
> >> >> heap allocation failed
> >> >> at java.net.PlainDatagramSocketImpl.receive0(Native Method)
> >> >> at
> >> >>
> >>
> java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:181)
> >> >> at java.net.DatagramSocket.receive(DatagramSocket.java:724)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent.run(MulticastDiscoveryAgent.java:338)
> >> >> at java.lang.Thread.run(Thread.java:803)
> >> >> JVMDUMP006I Processing Dump Event "uncaught", detail
> >> >> "java/lang/OutOfMemoryError" - Please Wait.
> >> >> JVMDUMP007I JVM Requesting Snap Dump using
> >> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0002.20091010.034616.16581.trc'
> >> >> JVMDUMP010I Snap Dump written to
> >> >> /x/web/STAGE2SC5464/idimqbroker/Snap0002.20091010.034616.16581.trc
> >> >> JVMDUMP007I JVM Requesting Heap Dump using
> >> >> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034616.16581.phd'
> >> >> JVMDUMP016E Aborting: Cannot create file
> >> >> (/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034616.16581.phd)
> >> >> JVMDUMP012E Error in Heap Dump: binary dump file cannot be written
> >> >> JVMDUMP007I JVM Requesting Java Dump using
> >> >> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034616.16581.txt'
> >> >> UTE503: utsGetTracePointIterator cannot allocate iterator's buffer
> >> >> JVMDUMP010I Java Dump written to
> >> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034616.16581.txt
> >> >> JVMDUMP013I Processed Dump Event "uncaught", detail
> >> >> "java/lang/OutOfMemoryError".
> >> >> Exception in thread "MulticastDiscovery: tcp://stage2sc5464.:61616"
> >> >> java.lang.OutOfMemoryError: heap allocation failed
> >> >> at java.net.PlainDatagramSocketImpl.receive0(Native Method)
> >> >> at
> >> >>
> >>
> java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:181)
> >> >> at java.net.DatagramSocket.receive(DatagramSocket.java:724)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent.run(MulticastDiscoveryAgent.java:338)
> >> >> at java.lang.Thread.run(Thread.java:803)
> >> >> JVMDUMP006I Processing Dump Event "uncaught", detail
> >> >> "java/lang/OutOfMemoryError" - Please Wait.
> >> >> JVMDUMP007I JVM Requesting Snap Dump using
> >> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0003.20091010.034640.16581.trc'
> >> >> JVMDUMP010I Snap Dump written to
> >> >> /x/web/STAGE2SC5464/idimqbroker/Snap0003.20091010.034640.16581.trc
> >> >> JVMDUMP007I JVM Requesting Heap Dump using
> >> >> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034640.16581.phd'
> >> >> JVMDUMP016E Aborting: Cannot create file
> >> >> (/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034640.16581.phd)
> >> >> JVMDUMP012E Error in Heap Dump: binary dump file cannot be written
> >> >> JVMDUMP007I JVM Requesting Java Dump using
> >> >> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034640.16581.txt'
> >> >> UTE503: utsGetTracePointIterator cannot allocate iterator's buffer
> >> >> JVMDUMP010I Java Dump written to
> >> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034640.16581.txt
> >> >> JVMDUMP013I Processed Dump Event "uncaught", detail
> >> >> "java/lang/OutOfMemoryError".
> >> >> Exception in thread "InactivityMonitor WriteCheck"
> >> >> java.lang.OutOfMemoryError
> >> >> at java.security.AccessController.getProtectionDomains(Native
> >> >> Method)
> >> >> at
> >> >> java.security.AccessController.getContext(AccessController.java:132)
> >> >> at java.lang.Thread.initialize(Thread.java:347)
> >> >> at java.lang.Thread.<init>(Thread.java:278)
> >> >> at java.lang.Thread.<init>(Thread.java:170)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.transport.InactivityMonitor$5.newThread(InactivityMonitor.java:303)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:411)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:463)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:878)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:128)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:103)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
> >> >> at java.util.TimerThread.mainLoop(Timer.java:527)
> >> >> at java.util.TimerThread.run(Timer.java:477)
> >> >> Exception in thread "RMI TCP Connection(454)-10.15.26.210"
> >> >> java.lang.NoClassDefFoundError: java.io.StreamCorruptedException
> >> >> at
> >> >>
> >> >>
> >>
> sun.rmi.transport.StreamRemoteCall.getResultStream(StreamRemoteCall.java:177)
> >> >> at
> >> >> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:335)
> >> >> at sun.rmi.transport.Transport$1.run(Transport.java:168)
> >> >> at
> >> >>
> java.security.AccessController.doPrivileged(AccessController.java:275)
> >> >> java.lang.NoClassDefFoundError: java.rmi.UnmarshalException
> >> >> at
> >> >>
> >> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:506)
> >> >> at
> >> >>
> >> >>
> >>
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.handleRequest(TCPTransport.java:838)
> >> >> at
> >> >>
> >>
> sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
> >> >> at
> >> >>
> >> >>
> >>
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:912)
> >> >> at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:358)
> >> >> at java.lang.Thread.run(Thread.java:803)
> >> >> at sun.rmi.transport.DGCImpl_Stub.dirty(DGCImpl_Stub.java:67)
> >> >> at
> >> >>
> >>
> sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:344)
> >> >> at
> >> >>
> >>
> sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:160)
> >> >> at
> >> >>
> >> >>
> >>
> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:555)
> >> >> at java.lang.Thread.run(Thread.java:803)
> >> >> Exception in thread "RMI LeaseChecker"
> java.lang.NoClassDefFoundError:
> >> >> java.rmi.server.Unreferenced
> >> >> at sun.rmi.transport.Target.refSetRemove(Target.java:320)
> >> >> at sun.rmi.transport.Target.vmidDead(Target.java:444)
> >> >> at sun.rmi.transport.DGCImpl.checkLeases(DGCImpl.java:265)
> >> >> at sun.rmi.transport.DGCImpl.access$400(DGCImpl.java:47)
> >> >> at
> sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:329)
> >> >> at java.lang.Thread.run(Thread.java:803)
> >> >> Unhandled exception
> >> >> Type=Segmentation error vmState=0x00053fff
> >> >> J9Generic_Signal_Number=00000004 Signal_Number=0000000b
> >> >> Error_Value=00000000
> >> >> Signal_Code=00000001
> >> >> Handler1=008F64F0 Handler2=00142477 InaccessibleAddress=00000000
> >> >> EDI=00000000 ESI=27E112E0 EAX=00000000 EBX=00000000
> >> >> ECX=08B7DF40 EDX=00000000
> >> >> EIP=00F97DB4 ES=0000007B DS=C02D007B ESP=00855BB8
> >> >> EFlags=00210206 CS=00000073 SS=0000007B EBP=00855BB8
> >> >> Module=/x/web/STAGE2SC5464/idimqbroker/java/jre/bin/libj9jit23.so
> >> >> Module_base_address=00F09000
> >> >>
> >> >>
> >> >>
> >>
> Method_being_compiled=org/springframework/jms/listener/DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run()V
> >> >> Target=2_30_20070524_12771_lHdSMR (Linux 2.6.9-55.ELsmp)
> >> >> CPU=x86 (8 logical CPUs) (0x47621d000 RAM)
> >> >> JVMDUMP006I Processing Dump Event "gpf", detail "" - Please Wait.
> >> >> JVMDUMP007I JVM Requesting System Dump using
> >> >> '/x/web/STAGE2SC5464/idimqbroker/core.20091011.220932.16581.dmp'
> >> >> JVMDUMP012E Error in System Dump: cannot find core file, check
> "ulimit
> >> >> -c"
> >> >> is set high enough
> >> >> JVMDUMP007I JVM Requesting Snap Dump using
> >> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0004.20091011.220932.16581.trc'
> >> >> JVMDUMP010I Snap Dump written to
> >> >> /x/web/STAGE2SC5464/idimqbroker/Snap0004.20091011.220932.16581.trc
> >> >> JVMDUMP007I JVM Requesting Java Dump using
> >> >> '/tmp/javacore.20091011.220932.16581.txt'
> >> >> NULL
> >> >>
> >> ------------------------------------------------------------------------
> >> >> 0SECTION TITLE subcomponent dump routine
> >> >> NULL ===============================
> >> >> 1TISIGINFO Dump Event "gpf" (00002000) received
> >> >> 1TIDATETIME Date: 2009/10/11 at 22:10:17
> >> >> 1TIFILENAME Javacore filename:
> >> >> /tmp/javacore.20091011.220932.16581.txt
> >> >> NULL
> >> >>
> >> ------------------------------------------------------------------------
> >> >> 0SECTION GPINFO subcomponent dump routine
> >> >> NULL ================================
> >> >> 2XHOSLEVEL OS Level : Linux 2.6.9-55.ELsmp
> >> >> 2XHCPUS Processors -
> >> >> 3XHCPUARCH Architecture : x86
> >> >> 3XHNUMCPUS How Many : 8
> >> >> NULL
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> View this message in context:
> >> >>
> >>
> http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25864372.html
> >> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >> >>
> >> >>
> >> >
> >> >
> >> > --
> >> > http://blog.garytully.com
> >> >
> >> > Open Source Integration
> >> > http://fusesource.com
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25881255.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > http://blog.garytully.com
> >
> > Open Source Integration
> > http://fusesource.com
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25935327.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com
Re: java/lang/OutOfMemoryError
Posted by tejeswara <te...@gmail.com>.
Gary,
In my config file I have multicast in both these entries. Which one you
think possibly causing this memory leak?. We plan to have failover broker so
we may need multicast discovery?.
<networkConnectors>
<networkConnector name="default" uri="multicast://default"/>
</networkConnectors>
<transportConnectors>
<transportConnector name="default" uri="tcp://localhost:61616"
discoveryUri="multicast://default"/>
</transportConnectors>
Gary Tully wrote:
>
> the config looks ok, just one thought, do you need multicast discovery on
> your transport connector? I wonder if it is possible you are seeing
> https://issues.apache.org/activemq/browse/AMQ-2283 which resulted in an
> leak
> of discovery threads? Removing the discoveryUri will work around it.
>
> 2009/10/13 tejeswara <te...@gmail.com>
>
>>
>> Here is our activemq.xml. As you can see it is pretty much same as the
>> template version we get from activemq package:
>> <beans
>> xmlns="http://www.springframework.org/schema/beans"
>> xmlns:amq="http://activemq.apache.org/schema/core"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>> http://activemq.apache.org/schema/core
>> http://activemq.apache.org/schema/core/activemq-core.xsd
>> http://activemq.apache.org/camel/schema/spring
>> http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">
>> <bean
>>
>> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>> <property name="locations">
>>
>> <value>file:///${activemq.base}/conf/credentials.properties</value>
>> </property>
>> </bean>
>> <broker xmlns="http://activemq.apache.org/schema/core"
>> brokerName="localhost" dataDirectory="${activemq.base}/data">
>> <destinationPolicy>
>> <policyMap>
>> <policyEntries>
>> <policyEntry queue=">" memoryLimit="50mb"/>
>> <policyEntry topic=">" memoryLimit="5mb">
>> </policyEntry>
>> </policyEntries>
>> </policyMap>
>> </destinationPolicy>
>> <managementContext>
>> <managementContext createConnector="true"
>> connectorPort="11099"/>
>> </managementContext>
>> <networkConnectors>
>> <networkConnector name="default-nc"
>> uri="multicast://default"/>
>> </networkConnectors>
>> <persistenceAdapter>
>> <amqPersistenceAdapter syncOnWrite="false"
>> directory="${activemq.base}/data" maxFileLength="20 mb"/>
>> </persistenceAdapter>
>> <systemUsage>
>> <systemUsage>
>> <memoryUsage>
>> <memoryUsage limit="200 mb"/>
>> </memoryUsage>
>> <storeUsage>
>> <storeUsage limit="1 gb" name="foo"/>
>> </storeUsage>
>> <tempUsage>
>> <tempUsage limit="100 mb"/>
>> </tempUsage>
>> </systemUsage>
>> </systemUsage>
>> <transportConnectors>
>> <transportConnector name="openwire"
>> uri="tcp://localhost:61616"
>> discoveryUri="multicast://default"/>
>> </transportConnectors>
>> </broker>
>> <camelContext id="camel"
>> xmlns="http://activemq.apache.org/camel/schema/spring">
>> <package>org.foo.bar</package>
>> <route>
>> <from uri="activemq:example.A"/>
>> <to uri="activemq:example.B"/>
>> </route>
>> </camelContext>
>> <bean id="activemq"
>> class="org.apache.activemq.camel.component.ActiveMQComponent" >
>> <property name="connectionFactory">
>> <bean class="org.apache.activemq.ActiveMQConnectionFactory">
>> <property name="brokerURL"
>> value="vm://localhost?create=false&waitForStart=10000" />
>> <property name="userName" value="${activemq.username}"/>
>> <property name="password" value="${activemq.password}"/>
>> </bean>
>> </property>
>> </bean>
>> <jetty xmlns="http://mortbay.com/schemas/jetty/1.0">
>> <connectors>
>> <nioConnector port="11098"/>
>> </connectors>
>> <handlers>
>> <webAppContext contextPath="/admin"
>> resourceBase="${activemq.base}/webapps/admin" logUrlOnStart="true"/>
>> <webAppContext contextPath="/demo"
>> resourceBase="${activemq.base}/webapps/demo" logUrlOnStart="true"/>
>> <webAppContext contextPath="/fileserver"
>> resourceBase="${activemq.base}/webapps/fileserver" logUrlOnStart="true"/>
>> </handlers>
>> </jetty>
>> </beans>
>>
>>
>>
>> Gary Tully wrote:
>> >
>> > Can you post your activemq.xml? It looks like you need to restrict the
>> > memory usage of the broker and have it spool messages to disk when the
>> > memory limits are reached. See some additional detail at
>> >
>> http://activemq.apache.org/javalangoutofmemory.html#java.lang.OutOfMemory-SpoolingMessagestoDisk
>> > *
>> >
>> > *
>> > 2009/10/12 tejeswara <te...@gmail.com>
>> >
>> >>
>> >> Our broker is running on Linux (RHEL4 box) and after 1 or 2 days I see
>> >> below
>> >> error in logs.
>> >> Note that, below values are set in bin/activemq in out test lab:
>> >> if [ -z "$ACTIVEMQ_OPTS" ] ; then
>> >> ACTIVEMQ_OPTS="-Xmx512M
>> >> -Dorg.apache.activemq.UseDedicatedTaskRunner=true"
>> >> fi
>> >>
>> >> I tried to stress test broker by creating 200 queues and put 5000
>> >> messages
>> >> (each about 1500 bytes size)
>> >> in each of these queues. However I could not see any error. Below
>> error
>> >> occured when broker is idle.
>> >> From logs, only activity during this idle period I see is 'DEBUG
>> >> InactivityMonitor - 30177 ms elapsed since last read
>> check'
>> >>
>> >> Can someone help in pointing out what is going wrong?
>> >>
>> >> Note that, OutOfMemoryError is seen in our production even when heap
>> size
>> >> is
>> >> bumped up to 1024 and
>> >> dedicated task runner is set to false. This error usvally occurs after
>> 2
>> >> days broker running without restart.
>> >> if [ -z "$ACTIVEMQ_OPTS" ] ; then
>> >> ACTIVEMQ_OPTS="-Xmx1024M
>> >> -Dorg.apache.activemq.UseDedicatedTaskRunner=false"
>> >> fi
>> >>
>> >>
>> >> Stack Trace:->
>> >>
>> >> JVMDUMP006I Processing Dump Event "uncaught", detail
>> >> "java/lang/OutOfMemoryError" - Please Wait.
>> >> JVMDUMP007I JVM Requesting Snap Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0001.20091010.015439.16581.trc'
>> >> JVMDUMP010I Snap Dump written to
>> >> /x/web/STAGE2SC5464/idimqbroker/Snap0001.20091010.015439.16581.trc
>> >> JVMDUMP007I JVM Requesting Heap Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.015439.16581.phd'
>> >> JVMDUMP010I Heap Dump written to
>> >> /x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.015439.16581.phd
>> >> JVMDUMP007I JVM Requesting Java Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.015439.16581.txt'
>> >> JVMDUMP010I Java Dump written to
>> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.015439.16581.txt
>> >> JVMDUMP013I Processed Dump Event "uncaught", detail
>> >> "java/lang/OutOfMemoryError".
>> >> Exception in thread "MulticastDiscovery: null"
>> >> java.lang.OutOfMemoryError:
>> >> heap allocation failed
>> >> at java.net.PlainDatagramSocketImpl.receive0(Native Method)
>> >> at
>> >>
>> java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:181)
>> >> at java.net.DatagramSocket.receive(DatagramSocket.java:724)
>> >> at
>> >>
>> >>
>> org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent.run(MulticastDiscoveryAgent.java:338)
>> >> at java.lang.Thread.run(Thread.java:803)
>> >> JVMDUMP006I Processing Dump Event "uncaught", detail
>> >> "java/lang/OutOfMemoryError" - Please Wait.
>> >> JVMDUMP007I JVM Requesting Snap Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0002.20091010.034616.16581.trc'
>> >> JVMDUMP010I Snap Dump written to
>> >> /x/web/STAGE2SC5464/idimqbroker/Snap0002.20091010.034616.16581.trc
>> >> JVMDUMP007I JVM Requesting Heap Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034616.16581.phd'
>> >> JVMDUMP016E Aborting: Cannot create file
>> >> (/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034616.16581.phd)
>> >> JVMDUMP012E Error in Heap Dump: binary dump file cannot be written
>> >> JVMDUMP007I JVM Requesting Java Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034616.16581.txt'
>> >> UTE503: utsGetTracePointIterator cannot allocate iterator's buffer
>> >> JVMDUMP010I Java Dump written to
>> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034616.16581.txt
>> >> JVMDUMP013I Processed Dump Event "uncaught", detail
>> >> "java/lang/OutOfMemoryError".
>> >> Exception in thread "MulticastDiscovery: tcp://stage2sc5464.:61616"
>> >> java.lang.OutOfMemoryError: heap allocation failed
>> >> at java.net.PlainDatagramSocketImpl.receive0(Native Method)
>> >> at
>> >>
>> java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:181)
>> >> at java.net.DatagramSocket.receive(DatagramSocket.java:724)
>> >> at
>> >>
>> >>
>> org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent.run(MulticastDiscoveryAgent.java:338)
>> >> at java.lang.Thread.run(Thread.java:803)
>> >> JVMDUMP006I Processing Dump Event "uncaught", detail
>> >> "java/lang/OutOfMemoryError" - Please Wait.
>> >> JVMDUMP007I JVM Requesting Snap Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0003.20091010.034640.16581.trc'
>> >> JVMDUMP010I Snap Dump written to
>> >> /x/web/STAGE2SC5464/idimqbroker/Snap0003.20091010.034640.16581.trc
>> >> JVMDUMP007I JVM Requesting Heap Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034640.16581.phd'
>> >> JVMDUMP016E Aborting: Cannot create file
>> >> (/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034640.16581.phd)
>> >> JVMDUMP012E Error in Heap Dump: binary dump file cannot be written
>> >> JVMDUMP007I JVM Requesting Java Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034640.16581.txt'
>> >> UTE503: utsGetTracePointIterator cannot allocate iterator's buffer
>> >> JVMDUMP010I Java Dump written to
>> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034640.16581.txt
>> >> JVMDUMP013I Processed Dump Event "uncaught", detail
>> >> "java/lang/OutOfMemoryError".
>> >> Exception in thread "InactivityMonitor WriteCheck"
>> >> java.lang.OutOfMemoryError
>> >> at java.security.AccessController.getProtectionDomains(Native
>> >> Method)
>> >> at
>> >> java.security.AccessController.getContext(AccessController.java:132)
>> >> at java.lang.Thread.initialize(Thread.java:347)
>> >> at java.lang.Thread.<init>(Thread.java:278)
>> >> at java.lang.Thread.<init>(Thread.java:170)
>> >> at
>> >>
>> >>
>> org.apache.activemq.transport.InactivityMonitor$5.newThread(InactivityMonitor.java:303)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:411)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:463)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:878)
>> >> at
>> >>
>> >>
>> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:128)
>> >> at
>> >>
>> >>
>> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:103)
>> >> at
>> >>
>> >>
>> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
>> >> at java.util.TimerThread.mainLoop(Timer.java:527)
>> >> at java.util.TimerThread.run(Timer.java:477)
>> >> Exception in thread "RMI TCP Connection(454)-10.15.26.210"
>> >> java.lang.NoClassDefFoundError: java.io.StreamCorruptedException
>> >> at
>> >>
>> >>
>> sun.rmi.transport.StreamRemoteCall.getResultStream(StreamRemoteCall.java:177)
>> >> at
>> >> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:335)
>> >> at sun.rmi.transport.Transport$1.run(Transport.java:168)
>> >> at
>> >> java.security.AccessController.doPrivileged(AccessController.java:275)
>> >> java.lang.NoClassDefFoundError: java.rmi.UnmarshalException
>> >> at
>> >>
>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:506)
>> >> at
>> >>
>> >>
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.handleRequest(TCPTransport.java:838)
>> >> at
>> >>
>> sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
>> >> at
>> >>
>> >>
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:912)
>> >> at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:358)
>> >> at java.lang.Thread.run(Thread.java:803)
>> >> at sun.rmi.transport.DGCImpl_Stub.dirty(DGCImpl_Stub.java:67)
>> >> at
>> >>
>> sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:344)
>> >> at
>> >>
>> sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:160)
>> >> at
>> >>
>> >>
>> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:555)
>> >> at java.lang.Thread.run(Thread.java:803)
>> >> Exception in thread "RMI LeaseChecker" java.lang.NoClassDefFoundError:
>> >> java.rmi.server.Unreferenced
>> >> at sun.rmi.transport.Target.refSetRemove(Target.java:320)
>> >> at sun.rmi.transport.Target.vmidDead(Target.java:444)
>> >> at sun.rmi.transport.DGCImpl.checkLeases(DGCImpl.java:265)
>> >> at sun.rmi.transport.DGCImpl.access$400(DGCImpl.java:47)
>> >> at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:329)
>> >> at java.lang.Thread.run(Thread.java:803)
>> >> Unhandled exception
>> >> Type=Segmentation error vmState=0x00053fff
>> >> J9Generic_Signal_Number=00000004 Signal_Number=0000000b
>> >> Error_Value=00000000
>> >> Signal_Code=00000001
>> >> Handler1=008F64F0 Handler2=00142477 InaccessibleAddress=00000000
>> >> EDI=00000000 ESI=27E112E0 EAX=00000000 EBX=00000000
>> >> ECX=08B7DF40 EDX=00000000
>> >> EIP=00F97DB4 ES=0000007B DS=C02D007B ESP=00855BB8
>> >> EFlags=00210206 CS=00000073 SS=0000007B EBP=00855BB8
>> >> Module=/x/web/STAGE2SC5464/idimqbroker/java/jre/bin/libj9jit23.so
>> >> Module_base_address=00F09000
>> >>
>> >>
>> >>
>> Method_being_compiled=org/springframework/jms/listener/DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run()V
>> >> Target=2_30_20070524_12771_lHdSMR (Linux 2.6.9-55.ELsmp)
>> >> CPU=x86 (8 logical CPUs) (0x47621d000 RAM)
>> >> JVMDUMP006I Processing Dump Event "gpf", detail "" - Please Wait.
>> >> JVMDUMP007I JVM Requesting System Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/core.20091011.220932.16581.dmp'
>> >> JVMDUMP012E Error in System Dump: cannot find core file, check "ulimit
>> >> -c"
>> >> is set high enough
>> >> JVMDUMP007I JVM Requesting Snap Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0004.20091011.220932.16581.trc'
>> >> JVMDUMP010I Snap Dump written to
>> >> /x/web/STAGE2SC5464/idimqbroker/Snap0004.20091011.220932.16581.trc
>> >> JVMDUMP007I JVM Requesting Java Dump using
>> >> '/tmp/javacore.20091011.220932.16581.txt'
>> >> NULL
>> >>
>> ------------------------------------------------------------------------
>> >> 0SECTION TITLE subcomponent dump routine
>> >> NULL ===============================
>> >> 1TISIGINFO Dump Event "gpf" (00002000) received
>> >> 1TIDATETIME Date: 2009/10/11 at 22:10:17
>> >> 1TIFILENAME Javacore filename:
>> >> /tmp/javacore.20091011.220932.16581.txt
>> >> NULL
>> >>
>> ------------------------------------------------------------------------
>> >> 0SECTION GPINFO subcomponent dump routine
>> >> NULL ================================
>> >> 2XHOSLEVEL OS Level : Linux 2.6.9-55.ELsmp
>> >> 2XHCPUS Processors -
>> >> 3XHCPUARCH Architecture : x86
>> >> 3XHNUMCPUS How Many : 8
>> >> NULL
>> >>
>> >>
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25864372.html
>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >>
>> >>
>> >
>> >
>> > --
>> > http://blog.garytully.com
>> >
>> > Open Source Integration
>> > http://fusesource.com
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25881255.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>
>
--
View this message in context: http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25935327.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: java/lang/OutOfMemoryError
Posted by Gary Tully <ga...@gmail.com>.
the config looks ok, just one thought, do you need multicast discovery on
your transport connector? I wonder if it is possible you are seeing
https://issues.apache.org/activemq/browse/AMQ-2283 which resulted in an leak
of discovery threads? Removing the discoveryUri will work around it.
2009/10/13 tejeswara <te...@gmail.com>
>
> Here is our activemq.xml. As you can see it is pretty much same as the
> template version we get from activemq package:
> <beans
> xmlns="http://www.springframework.org/schema/beans"
> xmlns:amq="http://activemq.apache.org/schema/core"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd
> http://activemq.apache.org/camel/schema/spring
> http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">
> <bean
>
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> <property name="locations">
>
> <value>file:///${activemq.base}/conf/credentials.properties</value>
> </property>
> </bean>
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="localhost" dataDirectory="${activemq.base}/data">
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry queue=">" memoryLimit="50mb"/>
> <policyEntry topic=">" memoryLimit="5mb">
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> <managementContext>
> <managementContext createConnector="true"
> connectorPort="11099"/>
> </managementContext>
> <networkConnectors>
> <networkConnector name="default-nc" uri="multicast://default"/>
> </networkConnectors>
> <persistenceAdapter>
> <amqPersistenceAdapter syncOnWrite="false"
> directory="${activemq.base}/data" maxFileLength="20 mb"/>
> </persistenceAdapter>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="200 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="1 gb" name="foo"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="100 mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
> <transportConnectors>
> <transportConnector name="openwire" uri="tcp://localhost:61616"
> discoveryUri="multicast://default"/>
> </transportConnectors>
> </broker>
> <camelContext id="camel"
> xmlns="http://activemq.apache.org/camel/schema/spring">
> <package>org.foo.bar</package>
> <route>
> <from uri="activemq:example.A"/>
> <to uri="activemq:example.B"/>
> </route>
> </camelContext>
> <bean id="activemq"
> class="org.apache.activemq.camel.component.ActiveMQComponent" >
> <property name="connectionFactory">
> <bean class="org.apache.activemq.ActiveMQConnectionFactory">
> <property name="brokerURL"
> value="vm://localhost?create=false&waitForStart=10000" />
> <property name="userName" value="${activemq.username}"/>
> <property name="password" value="${activemq.password}"/>
> </bean>
> </property>
> </bean>
> <jetty xmlns="http://mortbay.com/schemas/jetty/1.0">
> <connectors>
> <nioConnector port="11098"/>
> </connectors>
> <handlers>
> <webAppContext contextPath="/admin"
> resourceBase="${activemq.base}/webapps/admin" logUrlOnStart="true"/>
> <webAppContext contextPath="/demo"
> resourceBase="${activemq.base}/webapps/demo" logUrlOnStart="true"/>
> <webAppContext contextPath="/fileserver"
> resourceBase="${activemq.base}/webapps/fileserver" logUrlOnStart="true"/>
> </handlers>
> </jetty>
> </beans>
>
>
>
> Gary Tully wrote:
> >
> > Can you post your activemq.xml? It looks like you need to restrict the
> > memory usage of the broker and have it spool messages to disk when the
> > memory limits are reached. See some additional detail at
> >
> http://activemq.apache.org/javalangoutofmemory.html#java.lang.OutOfMemory-SpoolingMessagestoDisk
> > *
> >
> > *
> > 2009/10/12 tejeswara <te...@gmail.com>
> >
> >>
> >> Our broker is running on Linux (RHEL4 box) and after 1 or 2 days I see
> >> below
> >> error in logs.
> >> Note that, below values are set in bin/activemq in out test lab:
> >> if [ -z "$ACTIVEMQ_OPTS" ] ; then
> >> ACTIVEMQ_OPTS="-Xmx512M
> >> -Dorg.apache.activemq.UseDedicatedTaskRunner=true"
> >> fi
> >>
> >> I tried to stress test broker by creating 200 queues and put 5000
> >> messages
> >> (each about 1500 bytes size)
> >> in each of these queues. However I could not see any error. Below error
> >> occured when broker is idle.
> >> From logs, only activity during this idle period I see is 'DEBUG
> >> InactivityMonitor - 30177 ms elapsed since last read check'
> >>
> >> Can someone help in pointing out what is going wrong?
> >>
> >> Note that, OutOfMemoryError is seen in our production even when heap
> size
> >> is
> >> bumped up to 1024 and
> >> dedicated task runner is set to false. This error usvally occurs after 2
> >> days broker running without restart.
> >> if [ -z "$ACTIVEMQ_OPTS" ] ; then
> >> ACTIVEMQ_OPTS="-Xmx1024M
> >> -Dorg.apache.activemq.UseDedicatedTaskRunner=false"
> >> fi
> >>
> >>
> >> Stack Trace:->
> >>
> >> JVMDUMP006I Processing Dump Event "uncaught", detail
> >> "java/lang/OutOfMemoryError" - Please Wait.
> >> JVMDUMP007I JVM Requesting Snap Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0001.20091010.015439.16581.trc'
> >> JVMDUMP010I Snap Dump written to
> >> /x/web/STAGE2SC5464/idimqbroker/Snap0001.20091010.015439.16581.trc
> >> JVMDUMP007I JVM Requesting Heap Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.015439.16581.phd'
> >> JVMDUMP010I Heap Dump written to
> >> /x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.015439.16581.phd
> >> JVMDUMP007I JVM Requesting Java Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.015439.16581.txt'
> >> JVMDUMP010I Java Dump written to
> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.015439.16581.txt
> >> JVMDUMP013I Processed Dump Event "uncaught", detail
> >> "java/lang/OutOfMemoryError".
> >> Exception in thread "MulticastDiscovery: null"
> >> java.lang.OutOfMemoryError:
> >> heap allocation failed
> >> at java.net.PlainDatagramSocketImpl.receive0(Native Method)
> >> at
> >>
> java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:181)
> >> at java.net.DatagramSocket.receive(DatagramSocket.java:724)
> >> at
> >>
> >>
> org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent.run(MulticastDiscoveryAgent.java:338)
> >> at java.lang.Thread.run(Thread.java:803)
> >> JVMDUMP006I Processing Dump Event "uncaught", detail
> >> "java/lang/OutOfMemoryError" - Please Wait.
> >> JVMDUMP007I JVM Requesting Snap Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0002.20091010.034616.16581.trc'
> >> JVMDUMP010I Snap Dump written to
> >> /x/web/STAGE2SC5464/idimqbroker/Snap0002.20091010.034616.16581.trc
> >> JVMDUMP007I JVM Requesting Heap Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034616.16581.phd'
> >> JVMDUMP016E Aborting: Cannot create file
> >> (/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034616.16581.phd)
> >> JVMDUMP012E Error in Heap Dump: binary dump file cannot be written
> >> JVMDUMP007I JVM Requesting Java Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034616.16581.txt'
> >> UTE503: utsGetTracePointIterator cannot allocate iterator's buffer
> >> JVMDUMP010I Java Dump written to
> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034616.16581.txt
> >> JVMDUMP013I Processed Dump Event "uncaught", detail
> >> "java/lang/OutOfMemoryError".
> >> Exception in thread "MulticastDiscovery: tcp://stage2sc5464.:61616"
> >> java.lang.OutOfMemoryError: heap allocation failed
> >> at java.net.PlainDatagramSocketImpl.receive0(Native Method)
> >> at
> >>
> java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:181)
> >> at java.net.DatagramSocket.receive(DatagramSocket.java:724)
> >> at
> >>
> >>
> org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent.run(MulticastDiscoveryAgent.java:338)
> >> at java.lang.Thread.run(Thread.java:803)
> >> JVMDUMP006I Processing Dump Event "uncaught", detail
> >> "java/lang/OutOfMemoryError" - Please Wait.
> >> JVMDUMP007I JVM Requesting Snap Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0003.20091010.034640.16581.trc'
> >> JVMDUMP010I Snap Dump written to
> >> /x/web/STAGE2SC5464/idimqbroker/Snap0003.20091010.034640.16581.trc
> >> JVMDUMP007I JVM Requesting Heap Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034640.16581.phd'
> >> JVMDUMP016E Aborting: Cannot create file
> >> (/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034640.16581.phd)
> >> JVMDUMP012E Error in Heap Dump: binary dump file cannot be written
> >> JVMDUMP007I JVM Requesting Java Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034640.16581.txt'
> >> UTE503: utsGetTracePointIterator cannot allocate iterator's buffer
> >> JVMDUMP010I Java Dump written to
> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034640.16581.txt
> >> JVMDUMP013I Processed Dump Event "uncaught", detail
> >> "java/lang/OutOfMemoryError".
> >> Exception in thread "InactivityMonitor WriteCheck"
> >> java.lang.OutOfMemoryError
> >> at java.security.AccessController.getProtectionDomains(Native
> >> Method)
> >> at
> >> java.security.AccessController.getContext(AccessController.java:132)
> >> at java.lang.Thread.initialize(Thread.java:347)
> >> at java.lang.Thread.<init>(Thread.java:278)
> >> at java.lang.Thread.<init>(Thread.java:170)
> >> at
> >>
> >>
> org.apache.activemq.transport.InactivityMonitor$5.newThread(InactivityMonitor.java:303)
> >> at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:411)
> >> at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:463)
> >> at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:878)
> >> at
> >>
> >>
> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:128)
> >> at
> >>
> >>
> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:103)
> >> at
> >>
> >>
> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
> >> at java.util.TimerThread.mainLoop(Timer.java:527)
> >> at java.util.TimerThread.run(Timer.java:477)
> >> Exception in thread "RMI TCP Connection(454)-10.15.26.210"
> >> java.lang.NoClassDefFoundError: java.io.StreamCorruptedException
> >> at
> >>
> >>
> sun.rmi.transport.StreamRemoteCall.getResultStream(StreamRemoteCall.java:177)
> >> at
> >> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:335)
> >> at sun.rmi.transport.Transport$1.run(Transport.java:168)
> >> at
> >> java.security.AccessController.doPrivileged(AccessController.java:275)
> >> java.lang.NoClassDefFoundError: java.rmi.UnmarshalException
> >> at
> >> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:506)
> >> at
> >>
> >>
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.handleRequest(TCPTransport.java:838)
> >> at
> >>
> sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
> >> at
> >>
> >>
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:912)
> >> at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:358)
> >> at java.lang.Thread.run(Thread.java:803)
> >> at sun.rmi.transport.DGCImpl_Stub.dirty(DGCImpl_Stub.java:67)
> >> at
> >>
> sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:344)
> >> at
> >>
> sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:160)
> >> at
> >>
> >>
> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:555)
> >> at java.lang.Thread.run(Thread.java:803)
> >> Exception in thread "RMI LeaseChecker" java.lang.NoClassDefFoundError:
> >> java.rmi.server.Unreferenced
> >> at sun.rmi.transport.Target.refSetRemove(Target.java:320)
> >> at sun.rmi.transport.Target.vmidDead(Target.java:444)
> >> at sun.rmi.transport.DGCImpl.checkLeases(DGCImpl.java:265)
> >> at sun.rmi.transport.DGCImpl.access$400(DGCImpl.java:47)
> >> at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:329)
> >> at java.lang.Thread.run(Thread.java:803)
> >> Unhandled exception
> >> Type=Segmentation error vmState=0x00053fff
> >> J9Generic_Signal_Number=00000004 Signal_Number=0000000b
> >> Error_Value=00000000
> >> Signal_Code=00000001
> >> Handler1=008F64F0 Handler2=00142477 InaccessibleAddress=00000000
> >> EDI=00000000 ESI=27E112E0 EAX=00000000 EBX=00000000
> >> ECX=08B7DF40 EDX=00000000
> >> EIP=00F97DB4 ES=0000007B DS=C02D007B ESP=00855BB8
> >> EFlags=00210206 CS=00000073 SS=0000007B EBP=00855BB8
> >> Module=/x/web/STAGE2SC5464/idimqbroker/java/jre/bin/libj9jit23.so
> >> Module_base_address=00F09000
> >>
> >>
> >>
> Method_being_compiled=org/springframework/jms/listener/DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run()V
> >> Target=2_30_20070524_12771_lHdSMR (Linux 2.6.9-55.ELsmp)
> >> CPU=x86 (8 logical CPUs) (0x47621d000 RAM)
> >> JVMDUMP006I Processing Dump Event "gpf", detail "" - Please Wait.
> >> JVMDUMP007I JVM Requesting System Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/core.20091011.220932.16581.dmp'
> >> JVMDUMP012E Error in System Dump: cannot find core file, check "ulimit
> >> -c"
> >> is set high enough
> >> JVMDUMP007I JVM Requesting Snap Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0004.20091011.220932.16581.trc'
> >> JVMDUMP010I Snap Dump written to
> >> /x/web/STAGE2SC5464/idimqbroker/Snap0004.20091011.220932.16581.trc
> >> JVMDUMP007I JVM Requesting Java Dump using
> >> '/tmp/javacore.20091011.220932.16581.txt'
> >> NULL
> >> ------------------------------------------------------------------------
> >> 0SECTION TITLE subcomponent dump routine
> >> NULL ===============================
> >> 1TISIGINFO Dump Event "gpf" (00002000) received
> >> 1TIDATETIME Date: 2009/10/11 at 22:10:17
> >> 1TIFILENAME Javacore filename:
> >> /tmp/javacore.20091011.220932.16581.txt
> >> NULL
> >> ------------------------------------------------------------------------
> >> 0SECTION GPINFO subcomponent dump routine
> >> NULL ================================
> >> 2XHOSLEVEL OS Level : Linux 2.6.9-55.ELsmp
> >> 2XHCPUS Processors -
> >> 3XHCPUARCH Architecture : x86
> >> 3XHNUMCPUS How Many : 8
> >> NULL
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25864372.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > http://blog.garytully.com
> >
> > Open Source Integration
> > http://fusesource.com
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25881255.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com
Re: java/lang/OutOfMemoryError
Posted by tejeswara <te...@gmail.com>.
Here is our activemq.xml. As you can see it is pretty much same as the
template version we get from activemq package:
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd
http://activemq.apache.org/camel/schema/spring
http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:///${activemq.base}/conf/credentials.properties</value>
</property>
</bean>
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost" dataDirectory="${activemq.base}/data">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" memoryLimit="50mb"/>
<policyEntry topic=">" memoryLimit="5mb">
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext createConnector="true"
connectorPort="11099"/>
</managementContext>
<networkConnectors>
<networkConnector name="default-nc" uri="multicast://default"/>
</networkConnectors>
<persistenceAdapter>
<amqPersistenceAdapter syncOnWrite="false"
directory="${activemq.base}/data" maxFileLength="20 mb"/>
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="200 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="100 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire" uri="tcp://localhost:61616"
discoveryUri="multicast://default"/>
</transportConnectors>
</broker>
<camelContext id="camel"
xmlns="http://activemq.apache.org/camel/schema/spring">
<package>org.foo.bar</package>
<route>
<from uri="activemq:example.A"/>
<to uri="activemq:example.B"/>
</route>
</camelContext>
<bean id="activemq"
class="org.apache.activemq.camel.component.ActiveMQComponent" >
<property name="connectionFactory">
<bean class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL"
value="vm://localhost?create=false&waitForStart=10000" />
<property name="userName" value="${activemq.username}"/>
<property name="password" value="${activemq.password}"/>
</bean>
</property>
</bean>
<jetty xmlns="http://mortbay.com/schemas/jetty/1.0">
<connectors>
<nioConnector port="11098"/>
</connectors>
<handlers>
<webAppContext contextPath="/admin"
resourceBase="${activemq.base}/webapps/admin" logUrlOnStart="true"/>
<webAppContext contextPath="/demo"
resourceBase="${activemq.base}/webapps/demo" logUrlOnStart="true"/>
<webAppContext contextPath="/fileserver"
resourceBase="${activemq.base}/webapps/fileserver" logUrlOnStart="true"/>
</handlers>
</jetty>
</beans>
Gary Tully wrote:
>
> Can you post your activemq.xml? It looks like you need to restrict the
> memory usage of the broker and have it spool messages to disk when the
> memory limits are reached. See some additional detail at
> http://activemq.apache.org/javalangoutofmemory.html#java.lang.OutOfMemory-SpoolingMessagestoDisk
> *
>
> *
> 2009/10/12 tejeswara <te...@gmail.com>
>
>>
>> Our broker is running on Linux (RHEL4 box) and after 1 or 2 days I see
>> below
>> error in logs.
>> Note that, below values are set in bin/activemq in out test lab:
>> if [ -z "$ACTIVEMQ_OPTS" ] ; then
>> ACTIVEMQ_OPTS="-Xmx512M
>> -Dorg.apache.activemq.UseDedicatedTaskRunner=true"
>> fi
>>
>> I tried to stress test broker by creating 200 queues and put 5000
>> messages
>> (each about 1500 bytes size)
>> in each of these queues. However I could not see any error. Below error
>> occured when broker is idle.
>> From logs, only activity during this idle period I see is 'DEBUG
>> InactivityMonitor - 30177 ms elapsed since last read check'
>>
>> Can someone help in pointing out what is going wrong?
>>
>> Note that, OutOfMemoryError is seen in our production even when heap size
>> is
>> bumped up to 1024 and
>> dedicated task runner is set to false. This error usvally occurs after 2
>> days broker running without restart.
>> if [ -z "$ACTIVEMQ_OPTS" ] ; then
>> ACTIVEMQ_OPTS="-Xmx1024M
>> -Dorg.apache.activemq.UseDedicatedTaskRunner=false"
>> fi
>>
>>
>> Stack Trace:->
>>
>> JVMDUMP006I Processing Dump Event "uncaught", detail
>> "java/lang/OutOfMemoryError" - Please Wait.
>> JVMDUMP007I JVM Requesting Snap Dump using
>> '/x/web/STAGE2SC5464/idimqbroker/Snap0001.20091010.015439.16581.trc'
>> JVMDUMP010I Snap Dump written to
>> /x/web/STAGE2SC5464/idimqbroker/Snap0001.20091010.015439.16581.trc
>> JVMDUMP007I JVM Requesting Heap Dump using
>> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.015439.16581.phd'
>> JVMDUMP010I Heap Dump written to
>> /x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.015439.16581.phd
>> JVMDUMP007I JVM Requesting Java Dump using
>> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.015439.16581.txt'
>> JVMDUMP010I Java Dump written to
>> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.015439.16581.txt
>> JVMDUMP013I Processed Dump Event "uncaught", detail
>> "java/lang/OutOfMemoryError".
>> Exception in thread "MulticastDiscovery: null"
>> java.lang.OutOfMemoryError:
>> heap allocation failed
>> at java.net.PlainDatagramSocketImpl.receive0(Native Method)
>> at
>> java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:181)
>> at java.net.DatagramSocket.receive(DatagramSocket.java:724)
>> at
>>
>> org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent.run(MulticastDiscoveryAgent.java:338)
>> at java.lang.Thread.run(Thread.java:803)
>> JVMDUMP006I Processing Dump Event "uncaught", detail
>> "java/lang/OutOfMemoryError" - Please Wait.
>> JVMDUMP007I JVM Requesting Snap Dump using
>> '/x/web/STAGE2SC5464/idimqbroker/Snap0002.20091010.034616.16581.trc'
>> JVMDUMP010I Snap Dump written to
>> /x/web/STAGE2SC5464/idimqbroker/Snap0002.20091010.034616.16581.trc
>> JVMDUMP007I JVM Requesting Heap Dump using
>> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034616.16581.phd'
>> JVMDUMP016E Aborting: Cannot create file
>> (/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034616.16581.phd)
>> JVMDUMP012E Error in Heap Dump: binary dump file cannot be written
>> JVMDUMP007I JVM Requesting Java Dump using
>> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034616.16581.txt'
>> UTE503: utsGetTracePointIterator cannot allocate iterator's buffer
>> JVMDUMP010I Java Dump written to
>> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034616.16581.txt
>> JVMDUMP013I Processed Dump Event "uncaught", detail
>> "java/lang/OutOfMemoryError".
>> Exception in thread "MulticastDiscovery: tcp://stage2sc5464.:61616"
>> java.lang.OutOfMemoryError: heap allocation failed
>> at java.net.PlainDatagramSocketImpl.receive0(Native Method)
>> at
>> java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:181)
>> at java.net.DatagramSocket.receive(DatagramSocket.java:724)
>> at
>>
>> org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent.run(MulticastDiscoveryAgent.java:338)
>> at java.lang.Thread.run(Thread.java:803)
>> JVMDUMP006I Processing Dump Event "uncaught", detail
>> "java/lang/OutOfMemoryError" - Please Wait.
>> JVMDUMP007I JVM Requesting Snap Dump using
>> '/x/web/STAGE2SC5464/idimqbroker/Snap0003.20091010.034640.16581.trc'
>> JVMDUMP010I Snap Dump written to
>> /x/web/STAGE2SC5464/idimqbroker/Snap0003.20091010.034640.16581.trc
>> JVMDUMP007I JVM Requesting Heap Dump using
>> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034640.16581.phd'
>> JVMDUMP016E Aborting: Cannot create file
>> (/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034640.16581.phd)
>> JVMDUMP012E Error in Heap Dump: binary dump file cannot be written
>> JVMDUMP007I JVM Requesting Java Dump using
>> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034640.16581.txt'
>> UTE503: utsGetTracePointIterator cannot allocate iterator's buffer
>> JVMDUMP010I Java Dump written to
>> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034640.16581.txt
>> JVMDUMP013I Processed Dump Event "uncaught", detail
>> "java/lang/OutOfMemoryError".
>> Exception in thread "InactivityMonitor WriteCheck"
>> java.lang.OutOfMemoryError
>> at java.security.AccessController.getProtectionDomains(Native
>> Method)
>> at
>> java.security.AccessController.getContext(AccessController.java:132)
>> at java.lang.Thread.initialize(Thread.java:347)
>> at java.lang.Thread.<init>(Thread.java:278)
>> at java.lang.Thread.<init>(Thread.java:170)
>> at
>>
>> org.apache.activemq.transport.InactivityMonitor$5.newThread(InactivityMonitor.java:303)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:411)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:463)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:878)
>> at
>>
>> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:128)
>> at
>>
>> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:103)
>> at
>>
>> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
>> at java.util.TimerThread.mainLoop(Timer.java:527)
>> at java.util.TimerThread.run(Timer.java:477)
>> Exception in thread "RMI TCP Connection(454)-10.15.26.210"
>> java.lang.NoClassDefFoundError: java.io.StreamCorruptedException
>> at
>>
>> sun.rmi.transport.StreamRemoteCall.getResultStream(StreamRemoteCall.java:177)
>> at
>> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:335)
>> at sun.rmi.transport.Transport$1.run(Transport.java:168)
>> at
>> java.security.AccessController.doPrivileged(AccessController.java:275)
>> java.lang.NoClassDefFoundError: java.rmi.UnmarshalException
>> at
>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:506)
>> at
>>
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.handleRequest(TCPTransport.java:838)
>> at
>> sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
>> at
>>
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:912)
>> at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:358)
>> at java.lang.Thread.run(Thread.java:803)
>> at sun.rmi.transport.DGCImpl_Stub.dirty(DGCImpl_Stub.java:67)
>> at
>> sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:344)
>> at
>> sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:160)
>> at
>>
>> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:555)
>> at java.lang.Thread.run(Thread.java:803)
>> Exception in thread "RMI LeaseChecker" java.lang.NoClassDefFoundError:
>> java.rmi.server.Unreferenced
>> at sun.rmi.transport.Target.refSetRemove(Target.java:320)
>> at sun.rmi.transport.Target.vmidDead(Target.java:444)
>> at sun.rmi.transport.DGCImpl.checkLeases(DGCImpl.java:265)
>> at sun.rmi.transport.DGCImpl.access$400(DGCImpl.java:47)
>> at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:329)
>> at java.lang.Thread.run(Thread.java:803)
>> Unhandled exception
>> Type=Segmentation error vmState=0x00053fff
>> J9Generic_Signal_Number=00000004 Signal_Number=0000000b
>> Error_Value=00000000
>> Signal_Code=00000001
>> Handler1=008F64F0 Handler2=00142477 InaccessibleAddress=00000000
>> EDI=00000000 ESI=27E112E0 EAX=00000000 EBX=00000000
>> ECX=08B7DF40 EDX=00000000
>> EIP=00F97DB4 ES=0000007B DS=C02D007B ESP=00855BB8
>> EFlags=00210206 CS=00000073 SS=0000007B EBP=00855BB8
>> Module=/x/web/STAGE2SC5464/idimqbroker/java/jre/bin/libj9jit23.so
>> Module_base_address=00F09000
>>
>>
>> Method_being_compiled=org/springframework/jms/listener/DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run()V
>> Target=2_30_20070524_12771_lHdSMR (Linux 2.6.9-55.ELsmp)
>> CPU=x86 (8 logical CPUs) (0x47621d000 RAM)
>> JVMDUMP006I Processing Dump Event "gpf", detail "" - Please Wait.
>> JVMDUMP007I JVM Requesting System Dump using
>> '/x/web/STAGE2SC5464/idimqbroker/core.20091011.220932.16581.dmp'
>> JVMDUMP012E Error in System Dump: cannot find core file, check "ulimit
>> -c"
>> is set high enough
>> JVMDUMP007I JVM Requesting Snap Dump using
>> '/x/web/STAGE2SC5464/idimqbroker/Snap0004.20091011.220932.16581.trc'
>> JVMDUMP010I Snap Dump written to
>> /x/web/STAGE2SC5464/idimqbroker/Snap0004.20091011.220932.16581.trc
>> JVMDUMP007I JVM Requesting Java Dump using
>> '/tmp/javacore.20091011.220932.16581.txt'
>> NULL
>> ------------------------------------------------------------------------
>> 0SECTION TITLE subcomponent dump routine
>> NULL ===============================
>> 1TISIGINFO Dump Event "gpf" (00002000) received
>> 1TIDATETIME Date: 2009/10/11 at 22:10:17
>> 1TIFILENAME Javacore filename:
>> /tmp/javacore.20091011.220932.16581.txt
>> NULL
>> ------------------------------------------------------------------------
>> 0SECTION GPINFO subcomponent dump routine
>> NULL ================================
>> 2XHOSLEVEL OS Level : Linux 2.6.9-55.ELsmp
>> 2XHCPUS Processors -
>> 3XHCPUARCH Architecture : x86
>> 3XHNUMCPUS How Many : 8
>> NULL
>>
>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25864372.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>
>
--
View this message in context: http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25881255.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: java/lang/OutOfMemoryError
Posted by jimmymartin <ja...@paypal.com>.
Gary,
Thanks for the reply. It appears that the broker is throwing the exception
when running the inactivity monitor. It seems to be able to handle message
traffic fine but idleness seems to kill it (there's a moral here somewhere).
We have seen the broker core dump after long periods of no activity but with
clients still connected to it.
Perhaps we should be killing the connections on the client side after
extended periods of idleness?
-Jimmy
Gary Tully wrote:
>
> Can you post your activemq.xml? It looks like you need to restrict the
> memory usage of the broker and have it spool messages to disk when the
> memory limits are reached. See some additional detail at
> http://activemq.apache.org/javalangoutofmemory.html#java.lang.OutOfMemory-SpoolingMessagestoDisk
> *
>
> *
> 2009/10/12 tejeswara <te...@gmail.com>
>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>
>
--
View this message in context: http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25874223.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: java/lang/OutOfMemoryError
Posted by Gary Tully <ga...@gmail.com>.
Can you post your activemq.xml? It looks like you need to restrict the
memory usage of the broker and have it spool messages to disk when the
memory limits are reached. See some additional detail at
http://activemq.apache.org/javalangoutofmemory.html#java.lang.OutOfMemory-SpoolingMessagestoDisk
*
*
2009/10/12 tejeswara <te...@gmail.com>
>
> Our broker is running on Linux (RHEL4 box) and after 1 or 2 days I see
> below
> error in logs.
> Note that, below values are set in bin/activemq in out test lab:
> if [ -z "$ACTIVEMQ_OPTS" ] ; then
> ACTIVEMQ_OPTS="-Xmx512M -Dorg.apache.activemq.UseDedicatedTaskRunner=true"
> fi
>
> I tried to stress test broker by creating 200 queues and put 5000 messages
> (each about 1500 bytes size)
> in each of these queues. However I could not see any error. Below error
> occured when broker is idle.
> From logs, only activity during this idle period I see is 'DEBUG
> InactivityMonitor - 30177 ms elapsed since last read check'
>
> Can someone help in pointing out what is going wrong?
>
> Note that, OutOfMemoryError is seen in our production even when heap size
> is
> bumped up to 1024 and
> dedicated task runner is set to false. This error usvally occurs after 2
> days broker running without restart.
> if [ -z "$ACTIVEMQ_OPTS" ] ; then
> ACTIVEMQ_OPTS="-Xmx1024M
> -Dorg.apache.activemq.UseDedicatedTaskRunner=false"
> fi
>
>
> Stack Trace:->
>
> JVMDUMP006I Processing Dump Event "uncaught", detail
> "java/lang/OutOfMemoryError" - Please Wait.
> JVMDUMP007I JVM Requesting Snap Dump using
> '/x/web/STAGE2SC5464/idimqbroker/Snap0001.20091010.015439.16581.trc'
> JVMDUMP010I Snap Dump written to
> /x/web/STAGE2SC5464/idimqbroker/Snap0001.20091010.015439.16581.trc
> JVMDUMP007I JVM Requesting Heap Dump using
> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.015439.16581.phd'
> JVMDUMP010I Heap Dump written to
> /x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.015439.16581.phd
> JVMDUMP007I JVM Requesting Java Dump using
> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.015439.16581.txt'
> JVMDUMP010I Java Dump written to
> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.015439.16581.txt
> JVMDUMP013I Processed Dump Event "uncaught", detail
> "java/lang/OutOfMemoryError".
> Exception in thread "MulticastDiscovery: null" java.lang.OutOfMemoryError:
> heap allocation failed
> at java.net.PlainDatagramSocketImpl.receive0(Native Method)
> at
> java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:181)
> at java.net.DatagramSocket.receive(DatagramSocket.java:724)
> at
>
> org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent.run(MulticastDiscoveryAgent.java:338)
> at java.lang.Thread.run(Thread.java:803)
> JVMDUMP006I Processing Dump Event "uncaught", detail
> "java/lang/OutOfMemoryError" - Please Wait.
> JVMDUMP007I JVM Requesting Snap Dump using
> '/x/web/STAGE2SC5464/idimqbroker/Snap0002.20091010.034616.16581.trc'
> JVMDUMP010I Snap Dump written to
> /x/web/STAGE2SC5464/idimqbroker/Snap0002.20091010.034616.16581.trc
> JVMDUMP007I JVM Requesting Heap Dump using
> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034616.16581.phd'
> JVMDUMP016E Aborting: Cannot create file
> (/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034616.16581.phd)
> JVMDUMP012E Error in Heap Dump: binary dump file cannot be written
> JVMDUMP007I JVM Requesting Java Dump using
> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034616.16581.txt'
> UTE503: utsGetTracePointIterator cannot allocate iterator's buffer
> JVMDUMP010I Java Dump written to
> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034616.16581.txt
> JVMDUMP013I Processed Dump Event "uncaught", detail
> "java/lang/OutOfMemoryError".
> Exception in thread "MulticastDiscovery: tcp://stage2sc5464.:61616"
> java.lang.OutOfMemoryError: heap allocation failed
> at java.net.PlainDatagramSocketImpl.receive0(Native Method)
> at
> java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:181)
> at java.net.DatagramSocket.receive(DatagramSocket.java:724)
> at
>
> org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent.run(MulticastDiscoveryAgent.java:338)
> at java.lang.Thread.run(Thread.java:803)
> JVMDUMP006I Processing Dump Event "uncaught", detail
> "java/lang/OutOfMemoryError" - Please Wait.
> JVMDUMP007I JVM Requesting Snap Dump using
> '/x/web/STAGE2SC5464/idimqbroker/Snap0003.20091010.034640.16581.trc'
> JVMDUMP010I Snap Dump written to
> /x/web/STAGE2SC5464/idimqbroker/Snap0003.20091010.034640.16581.trc
> JVMDUMP007I JVM Requesting Heap Dump using
> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034640.16581.phd'
> JVMDUMP016E Aborting: Cannot create file
> (/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034640.16581.phd)
> JVMDUMP012E Error in Heap Dump: binary dump file cannot be written
> JVMDUMP007I JVM Requesting Java Dump using
> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034640.16581.txt'
> UTE503: utsGetTracePointIterator cannot allocate iterator's buffer
> JVMDUMP010I Java Dump written to
> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034640.16581.txt
> JVMDUMP013I Processed Dump Event "uncaught", detail
> "java/lang/OutOfMemoryError".
> Exception in thread "InactivityMonitor WriteCheck"
> java.lang.OutOfMemoryError
> at java.security.AccessController.getProtectionDomains(Native
> Method)
> at
> java.security.AccessController.getContext(AccessController.java:132)
> at java.lang.Thread.initialize(Thread.java:347)
> at java.lang.Thread.<init>(Thread.java:278)
> at java.lang.Thread.<init>(Thread.java:170)
> at
>
> org.apache.activemq.transport.InactivityMonitor$5.newThread(InactivityMonitor.java:303)
> at
>
> java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:411)
> at
>
> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:463)
> at
>
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:878)
> at
>
> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:128)
> at
>
> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:103)
> at
>
> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
> at java.util.TimerThread.mainLoop(Timer.java:527)
> at java.util.TimerThread.run(Timer.java:477)
> Exception in thread "RMI TCP Connection(454)-10.15.26.210"
> java.lang.NoClassDefFoundError: java.io.StreamCorruptedException
> at
>
> sun.rmi.transport.StreamRemoteCall.getResultStream(StreamRemoteCall.java:177)
> at
> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:335)
> at sun.rmi.transport.Transport$1.run(Transport.java:168)
> at
> java.security.AccessController.doPrivileged(AccessController.java:275)
> java.lang.NoClassDefFoundError: java.rmi.UnmarshalException
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:506)
> at
>
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.handleRequest(TCPTransport.java:838)
> at
> sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
> at
>
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:912)
> at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:358)
> at java.lang.Thread.run(Thread.java:803)
> at sun.rmi.transport.DGCImpl_Stub.dirty(DGCImpl_Stub.java:67)
> at
> sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:344)
> at
> sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:160)
> at
>
> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:555)
> at java.lang.Thread.run(Thread.java:803)
> Exception in thread "RMI LeaseChecker" java.lang.NoClassDefFoundError:
> java.rmi.server.Unreferenced
> at sun.rmi.transport.Target.refSetRemove(Target.java:320)
> at sun.rmi.transport.Target.vmidDead(Target.java:444)
> at sun.rmi.transport.DGCImpl.checkLeases(DGCImpl.java:265)
> at sun.rmi.transport.DGCImpl.access$400(DGCImpl.java:47)
> at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:329)
> at java.lang.Thread.run(Thread.java:803)
> Unhandled exception
> Type=Segmentation error vmState=0x00053fff
> J9Generic_Signal_Number=00000004 Signal_Number=0000000b
> Error_Value=00000000
> Signal_Code=00000001
> Handler1=008F64F0 Handler2=00142477 InaccessibleAddress=00000000
> EDI=00000000 ESI=27E112E0 EAX=00000000 EBX=00000000
> ECX=08B7DF40 EDX=00000000
> EIP=00F97DB4 ES=0000007B DS=C02D007B ESP=00855BB8
> EFlags=00210206 CS=00000073 SS=0000007B EBP=00855BB8
> Module=/x/web/STAGE2SC5464/idimqbroker/java/jre/bin/libj9jit23.so
> Module_base_address=00F09000
>
>
> Method_being_compiled=org/springframework/jms/listener/DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run()V
> Target=2_30_20070524_12771_lHdSMR (Linux 2.6.9-55.ELsmp)
> CPU=x86 (8 logical CPUs) (0x47621d000 RAM)
> JVMDUMP006I Processing Dump Event "gpf", detail "" - Please Wait.
> JVMDUMP007I JVM Requesting System Dump using
> '/x/web/STAGE2SC5464/idimqbroker/core.20091011.220932.16581.dmp'
> JVMDUMP012E Error in System Dump: cannot find core file, check "ulimit -c"
> is set high enough
> JVMDUMP007I JVM Requesting Snap Dump using
> '/x/web/STAGE2SC5464/idimqbroker/Snap0004.20091011.220932.16581.trc'
> JVMDUMP010I Snap Dump written to
> /x/web/STAGE2SC5464/idimqbroker/Snap0004.20091011.220932.16581.trc
> JVMDUMP007I JVM Requesting Java Dump using
> '/tmp/javacore.20091011.220932.16581.txt'
> NULL
> ------------------------------------------------------------------------
> 0SECTION TITLE subcomponent dump routine
> NULL ===============================
> 1TISIGINFO Dump Event "gpf" (00002000) received
> 1TIDATETIME Date: 2009/10/11 at 22:10:17
> 1TIFILENAME Javacore filename:
> /tmp/javacore.20091011.220932.16581.txt
> NULL
> ------------------------------------------------------------------------
> 0SECTION GPINFO subcomponent dump routine
> NULL ================================
> 2XHOSLEVEL OS Level : Linux 2.6.9-55.ELsmp
> 2XHCPUS Processors -
> 3XHCPUARCH Architecture : x86
> 3XHNUMCPUS How Many : 8
> NULL
>
>
>
> --
> View this message in context:
> http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25864372.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com