You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@felix.apache.org by Charles Moulliard <cm...@gmail.com> on 2009/06/30 13:25:44 UTC

Apache Karaf & Spring DM : consider taking a snapshot and then shutdown the VM in case the thread still hangs

Hi,

When Camel tries to stop one of its context, the following message is
displayed in the log:

13:10:59,213 | INFO  | Timer-2          | DefaultCamelContext              |
e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
(CamelContext:InToOut) is stopping
13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext              |
e.camel.impl.DefaultCamelContext  975 | Apache Camel 2.0-SNAPSHOT
(CamelContext:InToOut) stopped
13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext              |
e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
(CamelContext:OutToIn) is stopping
*13:11:09,181 | ERROR | Thread-47        | RunnableTimedExecution
| oncurrent.RunnableTimedExecution  109 | Closing runnable for context
OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-core,
config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
consider taking a snapshot and then shutdown the VM in case the thread still
hangs
13:11:19,181 | ERROR | Thread-47        | RunnableTimedExecution           |
oncurrent.RunnableTimedExecution  109 | Closing runnable for context
OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-service,
config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
consider taking a snapshot and then shutdown the VM in case the thread still
hangs*

It seems to be related to the following Spring DM error :
http://jira.springframework.org/browse/OSGI-611 but I use Spring OSGI 1.2
and the error is still there

Does anybody knows how to solve this ?

Regards,

Charles Moulliard
Senior Enterprise Architect
Apache Camel Committer

*****************************
blog : http://cmoulliard.blogspot.com

Re: Apache Karaf & Spring DM : consider taking a snapshot and then shutdown the VM in case the thread still hangs

Posted by Charles Moulliard <cm...@gmail.com>.
I tought that there was a specific option in apache karaf ;-)

Charles Moulliard
Senior Enterprise Architect
Apache Camel Committer

*****************************
blog : http://cmoulliard.blogspot.com


On Tue, Jun 30, 2009 at 2:33 PM, Guillaume Nodet <gn...@gmail.com> wrote:

> Try googling for "java thread dump" and you'll find the answer.
>
> On Tue, Jun 30, 2009 at 13:50, Charles Moulliard<cm...@gmail.com>
> wrote:
> > How can I achieve this ?
> >
> > Charles Moulliard
> > Senior Enterprise Architect
> > Apache Camel Committer
> >
> > *****************************
> > blog : http://cmoulliard.blogspot.com
> >
> >
> > On Tue, Jun 30, 2009 at 1:42 PM, Guillaume Nodet <gn...@gmail.com>
> wrote:
> >
> >> Could you grab a thread dump of the vm when the shutdown is waiting ?
> >> There must be a deadlock somewhere but without any thread dump, it's
> >> kinda impossible to see what happen.
> >>
> >> On Tue, Jun 30, 2009 at 13:25, Charles Moulliard<cm...@gmail.com>
> >> wrote:
> >> > Hi,
> >> >
> >> > When Camel tries to stop one of its context, the following message is
> >> > displayed in the log:
> >> >
> >> > 13:10:59,213 | INFO  | Timer-2          | DefaultCamelContext
> >>  |
> >> > e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
> >> > (CamelContext:InToOut) is stopping
> >> > 13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext
> >>  |
> >> > e.camel.impl.DefaultCamelContext  975 | Apache Camel 2.0-SNAPSHOT
> >> > (CamelContext:InToOut) stopped
> >> > 13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext
> >>  |
> >> > e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
> >> > (CamelContext:OutToIn) is stopping
> >> > *13:11:09,181 | ERROR | Thread-47        | RunnableTimedExecution
> >> > | oncurrent.RunnableTimedExecution  109 | Closing runnable for context
> >> > OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-core,
> >> > config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
> >> > consider taking a snapshot and then shutdown the VM in case the thread
> >> still
> >> > hangs
> >> > 13:11:19,181 | ERROR | Thread-47        | RunnableTimedExecution
> >>   |
> >> > oncurrent.RunnableTimedExecution  109 | Closing runnable for context
> >> > OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-service,
> >> > config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
> >> > consider taking a snapshot and then shutdown the VM in case the thread
> >> still
> >> > hangs*
> >> >
> >> > It seems to be related to the following Spring DM error :
> >> > http://jira.springframework.org/browse/OSGI-611 but I use Spring OSGI
> >> 1.2
> >> > and the error is still there
> >> >
> >> > Does anybody knows how to solve this ?
> >> >
> >> > Regards,
> >> >
> >> > Charles Moulliard
> >> > Senior Enterprise Architect
> >> > Apache Camel Committer
> >> >
> >> > *****************************
> >> > blog : http://cmoulliard.blogspot.com
> >> >
> >>
> >>
> >>
> >> --
> >> Cheers,
> >> Guillaume Nodet
> >> ------------------------
> >> Blog: http://gnodet.blogspot.com/
> >> ------------------------
> >> Open Source SOA
> >> http://fusesource.com
> >>
> >
>
>
>
> --
> Cheers,
> Guillaume Nodet
> ------------------------
> Blog: http://gnodet.blogspot.com/
> ------------------------
> Open Source SOA
> http://fusesource.com
>

Re: Apache Karaf & Spring DM : consider taking a snapshot and then shutdown the VM in case the thread still hangs

Posted by Guillaume Nodet <gn...@gmail.com>.
Try googling for "java thread dump" and you'll find the answer.

On Tue, Jun 30, 2009 at 13:50, Charles Moulliard<cm...@gmail.com> wrote:
> How can I achieve this ?
>
> Charles Moulliard
> Senior Enterprise Architect
> Apache Camel Committer
>
> *****************************
> blog : http://cmoulliard.blogspot.com
>
>
> On Tue, Jun 30, 2009 at 1:42 PM, Guillaume Nodet <gn...@gmail.com> wrote:
>
>> Could you grab a thread dump of the vm when the shutdown is waiting ?
>> There must be a deadlock somewhere but without any thread dump, it's
>> kinda impossible to see what happen.
>>
>> On Tue, Jun 30, 2009 at 13:25, Charles Moulliard<cm...@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > When Camel tries to stop one of its context, the following message is
>> > displayed in the log:
>> >
>> > 13:10:59,213 | INFO  | Timer-2          | DefaultCamelContext
>>  |
>> > e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
>> > (CamelContext:InToOut) is stopping
>> > 13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext
>>  |
>> > e.camel.impl.DefaultCamelContext  975 | Apache Camel 2.0-SNAPSHOT
>> > (CamelContext:InToOut) stopped
>> > 13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext
>>  |
>> > e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
>> > (CamelContext:OutToIn) is stopping
>> > *13:11:09,181 | ERROR | Thread-47        | RunnableTimedExecution
>> > | oncurrent.RunnableTimedExecution  109 | Closing runnable for context
>> > OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-core,
>> > config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
>> > consider taking a snapshot and then shutdown the VM in case the thread
>> still
>> > hangs
>> > 13:11:19,181 | ERROR | Thread-47        | RunnableTimedExecution
>>   |
>> > oncurrent.RunnableTimedExecution  109 | Closing runnable for context
>> > OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-service,
>> > config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
>> > consider taking a snapshot and then shutdown the VM in case the thread
>> still
>> > hangs*
>> >
>> > It seems to be related to the following Spring DM error :
>> > http://jira.springframework.org/browse/OSGI-611 but I use Spring OSGI
>> 1.2
>> > and the error is still there
>> >
>> > Does anybody knows how to solve this ?
>> >
>> > Regards,
>> >
>> > Charles Moulliard
>> > Senior Enterprise Architect
>> > Apache Camel Committer
>> >
>> > *****************************
>> > blog : http://cmoulliard.blogspot.com
>> >
>>
>>
>>
>> --
>> Cheers,
>> Guillaume Nodet
>> ------------------------
>> Blog: http://gnodet.blogspot.com/
>> ------------------------
>> Open Source SOA
>> http://fusesource.com
>>
>



-- 
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
Open Source SOA
http://fusesource.com

Re: Apache Karaf & Spring DM : consider taking a snapshot and then shutdown the VM in case the thread still hangs

Posted by Charles Moulliard <cm...@gmail.com>.
How can I achieve this ?

Charles Moulliard
Senior Enterprise Architect
Apache Camel Committer

*****************************
blog : http://cmoulliard.blogspot.com


On Tue, Jun 30, 2009 at 1:42 PM, Guillaume Nodet <gn...@gmail.com> wrote:

> Could you grab a thread dump of the vm when the shutdown is waiting ?
> There must be a deadlock somewhere but without any thread dump, it's
> kinda impossible to see what happen.
>
> On Tue, Jun 30, 2009 at 13:25, Charles Moulliard<cm...@gmail.com>
> wrote:
> > Hi,
> >
> > When Camel tries to stop one of its context, the following message is
> > displayed in the log:
> >
> > 13:10:59,213 | INFO  | Timer-2          | DefaultCamelContext
>  |
> > e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
> > (CamelContext:InToOut) is stopping
> > 13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext
>  |
> > e.camel.impl.DefaultCamelContext  975 | Apache Camel 2.0-SNAPSHOT
> > (CamelContext:InToOut) stopped
> > 13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext
>  |
> > e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
> > (CamelContext:OutToIn) is stopping
> > *13:11:09,181 | ERROR | Thread-47        | RunnableTimedExecution
> > | oncurrent.RunnableTimedExecution  109 | Closing runnable for context
> > OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-core,
> > config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
> > consider taking a snapshot and then shutdown the VM in case the thread
> still
> > hangs
> > 13:11:19,181 | ERROR | Thread-47        | RunnableTimedExecution
>   |
> > oncurrent.RunnableTimedExecution  109 | Closing runnable for context
> > OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-service,
> > config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
> > consider taking a snapshot and then shutdown the VM in case the thread
> still
> > hangs*
> >
> > It seems to be related to the following Spring DM error :
> > http://jira.springframework.org/browse/OSGI-611 but I use Spring OSGI
> 1.2
> > and the error is still there
> >
> > Does anybody knows how to solve this ?
> >
> > Regards,
> >
> > Charles Moulliard
> > Senior Enterprise Architect
> > Apache Camel Committer
> >
> > *****************************
> > blog : http://cmoulliard.blogspot.com
> >
>
>
>
> --
> Cheers,
> Guillaume Nodet
> ------------------------
> Blog: http://gnodet.blogspot.com/
> ------------------------
> Open Source SOA
> http://fusesource.com
>

Re: Apache Karaf & Spring DM : consider taking a snapshot and then shutdown the VM in case the thread still hangs

Posted by "Richard S. Hall" <he...@ungoverned.org>.
Well, it looks like the refresh starts to happen on the PackageAdmin 
thread, which grabs the global lock to do the refresh. As part of an 
synchronous bundle event callback into Spring, it takes the PackageAdmin 
thread and uses it to do a timed wait for some reason. As a result, when 
the Spring extender thread tries to resolve a bundle, which also needs 
the global lock, it is unable to get it. This likely causes the other 
waiting thread waiting in Spring to get a timeout exception.

Makes sense, but I am not sure what we can do about it.

-> richard

On 7/1/09 11:25 AM, Charles Moulliard wrote:
> Here is the full dump :
>
> 2009-07-01 17:22:09
> Full thread dump Java HotSpot(TM) Server VM (11.2-b01 mixed mode):
>
> "Keep-Alive-Timer" daemon prio=8 tid=0x309f5c00 nid=0xd6c waiting on
> condition [0x3294f000..0x3294fc14]
>     java.lang.Thread.State: TIMED_WAITING (sleeping)
>          at java.lang.Thread.sleep(Native Method)
>          at sun.net.www.http.KeepAliveCache.run(KeepAliveCache.java:149)
>          at java.lang.Thread.run(Thread.java:619)
>
> "SpringOsgiExtenderThread-49" prio=6 tid=0x3250c800 nid=0xe84 in
> Object.wait() [0x3361f000..0x3361fc94]
>     java.lang.Thread.State: WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.lang.Object.wait(Object.java:485)
>          at
> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>          - locked<0x0fea0670>  (a [Ljava.lang.Object;)
>          at org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>          at
> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>          at
> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>          at
> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>          at
> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>          at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>          at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>          at
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>          at
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>          at
> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
> )
>          at
> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
> xt.java:69)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
> va:269)
>          - locked<0x2cf47890>  (a java.lang.Object)
>          at
> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
> text.java:247)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
> tionContextExecutor.java:222)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
> ionContextExecutor.java:175)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
> java:175)
>          at
> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:3084" daemon prio=4 tid=0x31ae5800
> nid=0xae4 runnable [0x3692f000..0x3692fc94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x31ae5400
> nid=0x5d4 runnable [0x3679f000..0x3679fd94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:3083" daemon prio=4 tid=0x31828c00
> nid=0x920 runnable [0x365bf000..0x365bfc14]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x30f00400
> nid=0x334 runnable [0x3633f000..0x3633fd94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "InactivityMonitor Async Task:
> java.util.concurrent.ThreadPoolExecutor$Worker@137854f" daemon prio=6
> tid=0x32608800 nid=0xa48 waiting on condition [0x
> 3357f000..0x3357fd94]
>     java.lang.Thread.State: TIMED_WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for<0x10cba430>  (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>          at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>          at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>          at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>          at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:2934" daemon prio=4 tid=0x31fbb400
> nid=0x6c0 runnable [0x3683f000..0x3683fa94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x32138400
> nid=0x29c runnable [0x367ef000..0x367efb94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:2933" daemon prio=4 tid=0x30a7b400
> nid=0x214 runnable [0x3665f000..0x3665fd14]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x32232800
> nid=0xf64 runnable [0x3660f000..0x3660fa14]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "SocketConnector-5" daemon prio=6 tid=0x31fe4400 nid=0x5b0 runnable
> [0x332af000..0x332afa94]
>     java.lang.Thread.State: RUNNABLE
>          at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
>          at
> sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:274)
>          at
> sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:256)
>          at
> sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:137)
>          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>          - locked<0x11a0c020>  (a sun.nio.ch.Util$1)
>          - locked<0x11a0c010>  (a java.util.Collections$UnmodifiableSet)
>          - locked<0x11a0b7d8>  (a sun.nio.ch.WindowsSelectorImpl)
>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>          at
> org.apache.mina.transport.socket.nio.SocketConnector$Worker.run(SocketConnector.java:378)
>          at
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
>          at java.lang.Thread.run(Thread.java:619)
>
> "InactivityMonitor Async Task:
> java.util.concurrent.ThreadPoolExecutor$Worker@13144f4" daemon prio=6
> tid=0x30d25c00 nid=0xc6c waiting on condition [0x
> 334df000..0x334dfa14]
>     java.lang.Thread.State: TIMED_WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for<0x10cba430>  (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>          at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>          at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>          at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>          at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:2638" daemon prio=4 tid=0x31246400
> nid=0xab8 runnable [0x36abf000..0x36abfa14]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x31afe400
> nid=0x784 runnable [0x36a1f000..0x36a1fb14]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:2637" daemon prio=4 tid=0x340e0800
> nid=0x4dc runnable [0x3697f000..0x3697fc94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x3224ec00
> nid=0x404 runnable [0x368df000..0x368dfd94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:2632" daemon prio=4 tid=0x31fe4800
> nid=0x558 runnable [0x3674f000..0x3674fd94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x32072c00
> nid=0xf74 runnable [0x366af000..0x366afa94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:2631" daemon prio=4 tid=0x32750400
> nid=0xc74 runnable [0x3656f000..0x3656fd14]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x308f3400
> nid=0x3d8 runnable [0x364cf000..0x364cfd94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:2593" daemon prio=4 tid=0x32140c00
> nid=0xc58 runnable [0x3651f000..0x3651fd14]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x3225e800
> nid=0x694 runnable [0x3647f000..0x3647fa14]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:2592" daemon prio=4 tid=0x322e5400
> nid=0xfb8 runnable [0x363df000..0x363dfb94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x31fecc00
> nid=0x42c runnable [0x3601f000..0x3601fc94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:2589" daemon prio=4 tid=0x32056800
> nid=0x9b8 runnable [0x35fcf000..0x35fcfb94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x3258ac00
> nid=0xa5c runnable [0x33caf000..0x33cafc94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:2588" daemon prio=4 tid=0x31c49800
> nid=0x3cc runnable [0x3606f000..0x3606fa94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x32127400
> nid=0x488 runnable [0x32c6f000..0x32c6fb94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:2582" daemon prio=4 tid=0x31f53400
> nid=0xfb4 runnable [0x33edf000..0x33edfb14]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x322e8400
> nid=0xff8 runnable [0x33e8f000..0x33e8fc14]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:2581" daemon prio=4 tid=0x321e4c00
> nid=0xcec runnable [0x33cff000..0x33cffa94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x31136c00
> nid=0xa54 runnable [0x33a2f000..0x33a2fb94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:2309" daemon prio=4 tid=0x32524800
> nid=0x584 runnable [0x361ff000..0x361ffc94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x31067400
> nid=0x9c8 runnable [0x3615f000..0x3615fd94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "Camel thread 6: ActiveMQComponent" daemon prio=6 tid=0x31d3fc00 nid=0xb48
> runnable [0x3610f000..0x3610fc94]
>     java.lang.Thread.State: TIMED_WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for<0x1133f570>  (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
>          at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>          at java.lang.Thread.run(Thread.java:619)
>
> "Camel thread 5: ActiveMQComponent" daemon prio=6 tid=0x31c51c00 nid=0xe34
> runnable [0x360bf000..0x360bfc14]
>     java.lang.Thread.State: TIMED_WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for<0x1133f570>  (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
>          at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>          at java.lang.Thread.run(Thread.java:619)
>
> "Thread-23" daemon prio=6 tid=0x31cab000 nid=0xde8 waiting on condition
> [0x3339f000..0x3339fa14]
>     java.lang.Thread.State: TIMED_WAITING (sleeping)
>          at java.lang.Thread.sleep(Native Method)
>          at
> org.apache.geronimo.transaction.manager.TransactionTimer$CurrentTime.run(TransactionTimer.java:38)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:2308" daemon prio=4 tid=0x3084c800
> nid=0xa40 runnable [0x33c5f000..0x33c5fa14]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x31cd0800
> nid=0xe54 runnable [0x3398f000..0x3398fb14]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "Camel thread 4: ActiveMQComponent" daemon prio=6 tid=0x30d6f400 nid=0x3b4
> runnable [0x3393f000..0x3393fb94]
>     java.lang.Thread.State: TIMED_WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for<0x1133f570>  (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
>          at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>          at java.lang.Thread.run(Thread.java:619)
>
> "Camel thread 3: ActiveMQComponent" daemon prio=6 tid=0x325bdc00 nid=0x5e0
> runnable [0x3389f000..0x3389fc14]
>     java.lang.Thread.State: TIMED_WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for<0x1133f570>  (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
>          at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>          at java.lang.Thread.run(Thread.java:619)
>
> "Camel thread 2: ActiveMQComponent" daemon prio=6 tid=0x3084a000 nid=0x9ac
> runnable [0x332ff000..0x332ffc94]
>     java.lang.Thread.State: TIMED_WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for<0x1133f570>  (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
>          at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:2281" daemon prio=4 tid=0x31c38400
> nid=0x578 runnable [0x33c0f000..0x33c0fc94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x308f2400
> nid=0xa80 runnable [0x339df000..0x339dfd94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:2280" daemon prio=4 tid=0x317f8400
> nid=0xce4 runnable [0x337af000..0x337afc14]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x31830800
> nid=0x460 runnable [0x3370f000..0x3370fc94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "org.apache.commons.vfs.cache.SoftRefFilesCache$SoftRefReleaseThread" daemon
> prio=6 tid=0x32143800 nid=0x9a4 in Object.wait() [0x32d0f000..0x32d0fa94]
>
>     java.lang.Thread.State: TIMED_WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>          - locked<0x106fa908>  (a java.lang.ref.ReferenceQueue$Lock)
>          at
> org.apache.commons.vfs.cache.SoftRefFilesCache$SoftRefReleaseThread.run(SoftRefFilesCache.java:79)
>
> "QueueThread:queue://x3s:notification:message" daemon prio=6 tid=0x32143400
> nid=0xf58 waiting on condition [0x33bbf000..0x33bbfb94]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for<0x10a5c850>  (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>          at java.lang.Thread.run(Thread.java:619)
>
> "QueueThread:queue://x3s:notification" daemon prio=6 tid=0x31273800
> nid=0x2d0 waiting on condition [0x33b6f000..0x33b6fc94]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for<0x10a3f248>  (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:2230" daemon prio=4 tid=0x31273400
> nid=0x7c4 runnable [0x33b1f000..0x33b1fd94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x313b4800
> nid=0xe3c runnable [0x3343f000..0x3343fa94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:2229" daemon prio=4 tid=0x310ad800
> nid=0xa08 runnable [0x33acf000..0x33acfb94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x310ad400
> nid=0xd3c runnable [0x33a7f000..0x33a7fc94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "QueueThread:queue://x3s:requestmessage" daemon prio=6 tid=0x3120a800
> nid=0xd28 waiting on condition [0x338ef000..0x338efc94]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for<0x10a53e88>  (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>          at java.lang.Thread.run(Thread.java:619)
>
> "QueueThread:queue://x3s:request" daemon prio=6 tid=0x3086fc00 nid=0xf04
> waiting on condition [0x3384f000..0x3384fa14]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for<0x10a5e9d0>  (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:2225" daemon prio=4 tid=0x31bf5400
> nid=0x13c runnable [0x337ff000..0x337ffa94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x31f78c00
> nid=0x354 runnable [0x3375f000..0x3375fb94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "InactivityMonitor WriteCheck" daemon prio=6 tid=0x3084d400 nid=0x860 in
> Object.wait() [0x336bf000..0x336bfd14]
>     java.lang.Thread.State: TIMED_WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.util.TimerThread.mainLoop(Timer.java:509)
>          - locked<0x112506f0>  (a java.util.TaskQueue)
>          at java.util.TimerThread.run(Timer.java:462)
>
> "InactivityMonitor ReadCheck" daemon prio=6 tid=0x31014400 nid=0x69c in
> Object.wait() [0x3366f000..0x3366fd94]
>     java.lang.Thread.State: TIMED_WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.util.TimerThread.mainLoop(Timer.java:509)
>          - locked<0x1125e188>  (a java.util.TaskQueue)
>          at java.util.TimerThread.run(Timer.java:462)
>
> "ActiveMQ Transport: tcp:///127.0.0.1:2224" daemon prio=4 tid=0x3138e400
> nid=0xb74 runnable [0x335cf000..0x335cfa94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x30fdcc00
> nid=0x158 runnable [0x3352f000..0x3352fb94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>          at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>          at java.io.DataInputStream.readInt(DataInputStream.java:370)
>          at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>          at java.lang.Thread.run(Thread.java:619)
>
> "QFJ Timer" daemon prio=6 tid=0x3118b400 nid=0xa64 waiting on condition
> [0x32c1f000..0x32c1fa14]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for<0x111e03c0>  (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>          at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>          at java.lang.Thread.run(Thread.java:619)
>
> "MySQL Statement Cancellation Timer" daemon prio=6 tid=0x30fc5800 nid=0x50c
> in Object.wait() [0x3348f000..0x3348fa94]
>     java.lang.Thread.State: WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          - waiting on<0x10c61a88>  (a java.util.TaskQueue)
>          at java.lang.Object.wait(Object.java:485)
>          at java.util.TimerThread.mainLoop(Timer.java:483)
>          - locked<0x10c61a88>  (a java.util.TaskQueue)
>          at java.util.TimerThread.run(Timer.java:462)
>
> "ActiveMQ Transport Server: stomp://localhost:61613" daemon prio=10
> tid=0x312ea800 nid=0x39c runnable [0x3325f000..0x3325fd94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.PlainSocketImpl.socketAccept(Native Method)
>          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>          - locked<0x10c61fc8>  (a java.net.SocksSocketImpl)
>          at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>          at java.net.ServerSocket.accept(ServerSocket.java:421)
>          at
> org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:274)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport Server Thread Handler: stomp://localhost:61613" daemon
> prio=8 tid=0x30605800 nid=0xd5c runnable [0x329ef000..0x329efa14]
>     java.lang.Thread.State: TIMED_WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for<0x10c62138>  (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
>          at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>          at
> org.apache.activemq.transport.tcp.TcpTransportServer$1.run(TcpTransportServer.java:339)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport Server: tcp://localhost:61616" daemon prio=10
> tid=0x31d7c400 nid=0x574 runnable [0x3320f000..0x3320fa94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.PlainSocketImpl.socketAccept(Native Method)
>          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>          - locked<0x10a65180>  (a java.net.SocksSocketImpl)
>          at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>          at java.net.ServerSocket.accept(ServerSocket.java:421)
>          at
> org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:274)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Transport Server Thread Handler: tcp://localhost:61616" daemon
> prio=8 tid=0x31c62800 nid=0xed8 runnable [0x331bf000..0x331bfb14]
>     java.lang.Thread.State: TIMED_WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for<0x10a65300>  (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
>          at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>          at
> org.apache.activemq.transport.tcp.TcpTransportServer$1.run(TcpTransportServer.java:339)
>          at java.lang.Thread.run(Thread.java:619)
>
> "NioSocketAcceptor-2" prio=6 tid=0x31a4bc00 nid=0x9a8 runnable
> [0x32cbf000..0x32cbfb94]
>     java.lang.Thread.State: RUNNABLE
>          at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
>          at
> sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:274)
>          at
> sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:256)
>          at
> sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:137)
>          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>          - locked<0x10b07508>  (a sun.nio.ch.Util$1)
>          - locked<0x10b074f8>  (a java.util.Collections$UnmodifiableSet)
>          - locked<0x10b070c8>  (a sun.nio.ch.WindowsSelectorImpl)
>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>          at
> org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:288)
>          at
> org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:402)
>          at
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>          at java.lang.Thread.run(Thread.java:619)
>
> "ActiveMQ Scheduler" daemon prio=6 tid=0x30f1cc00 nid=0x5c8 in Object.wait()
> [0x3316f000..0x3316fc94]
>     java.lang.Thread.State: TIMED_WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.util.TimerThread.mainLoop(Timer.java:509)
>          - locked<0x10a75348>  (a java.util.TaskQueue)
>          at java.util.TimerThread.run(Timer.java:462)
>
> "ActiveMQ Data File Writer" daemon prio=10 tid=0x3051dc00 nid=0x978 in
> Object.wait() [0x3280f000..0x3280fd14]
>     java.lang.Thread.State: WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.lang.Object.wait(Object.java:485)
>          at
> org.apache.activemq.kaha.impl.async.NIODataFileAppender.processQueue(NIODataFileAppender.java:79)
>          - locked<0x10a771e8>  (a
> org.apache.activemq.kaha.impl.async.DataFileAppender$1)
>          at
> org.apache.activemq.kaha.impl.async.DataFileAppender$2.run(DataFileAppender.java:227)
>
> "localShell" prio=6 tid=0x31856c00 nid=0xaec runnable
> [0x32a8f000..0x32a8fa94]
>     java.lang.Thread.State: RUNNABLE
>          at jline.WindowsTerminal.readByte(Native Method)
>          at jline.WindowsTerminal.readCharacter(WindowsTerminal.java:233)
>          at jline.WindowsTerminal.readVirtualKey(WindowsTerminal.java:319)
>          at jline.ConsoleReader.readVirtualKey(ConsoleReader.java:1453)
>          at jline.ConsoleReader.readBinding(ConsoleReader.java:654)
>          at jline.ConsoleReader.readLine(ConsoleReader.java:494)
>          at jline.ConsoleReader.readLine(ConsoleReader.java:448)
>          at
> org.apache.geronimo.gshell.console.JLineConsole.readLine(JLineConsole.java:95)
>          at org.apache.geronimo.gshell.console.Console.work(Console.java:153)
>          at org.apache.geronimo.gshell.console.Console.run(Console.java:128)
>          at
> org.apache.geronimo.gshell.wisdom.shell.ShellImpl.run(ShellImpl.java:252)
>          at
> org.apache.felix.karaf.gshell.core.ShellWrapper.run(ShellWrapper.java:81)
>          at
> org.apache.felix.karaf.gshell.core.LocalConsole.run(LocalConsole.java:119)
>          at java.lang.Thread.run(Thread.java:619)
>
> "RMI RenewClean-[172.16.10.128:2163]" daemon prio=6 tid=0x3196c400 nid=0xb50
> in Object.wait() [0x328ff000..0x328ffb94]
>     java.lang.Thread.State: TIMED_WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>          - locked<0x10836d70>  (a java.lang.ref.ReferenceQueue$Lock)
>          at
> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
>          at java.lang.Thread.run(Thread.java:619)
>
> "RMI Scheduler(0)" daemon prio=6 tid=0x30a63800 nid=0x138 waiting on
> condition [0x3299f000..0x3299fc94]
>     java.lang.Thread.State: TIMED_WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for<0x100e7d90>  (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
>          at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>          at java.lang.Thread.run(Thread.java:619)
>
> "GC Daemon" daemon prio=2 tid=0x30a46400 nid=0xd64 in Object.wait()
> [0x32bcf000..0x32bcfa94]
>     java.lang.Thread.State: TIMED_WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at sun.misc.GC$Daemon.run(GC.java:100)
>          - locked<0x105eddf0>  (a sun.misc.GC$LatencyLock)
>
> "RMI Reaper" prio=6 tid=0x30f38400 nid=0x628 in Object.wait()
> [0x32b7f000..0x32b7fb14]
>     java.lang.Thread.State: WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          - waiting on<0x10124788>  (a java.lang.ref.ReferenceQueue$Lock)
>          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>          - locked<0x10124788>  (a java.lang.ref.ReferenceQueue$Lock)
>          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>          at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
>          at java.lang.Thread.run(Thread.java:619)
>
> "RMI TCP Accept-0" daemon prio=6 tid=0x3214a800 nid=0xc0 runnable
> [0x32b2f000..0x32b2fb94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.PlainSocketImpl.socketAccept(Native Method)
>          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>          - locked<0x1053b880>  (a java.net.SocksSocketImpl)
>          at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>          at java.net.ServerSocket.accept(ServerSocket.java:421)
>          at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
>          at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>          at java.lang.Thread.run(Thread.java:619)
>
> "RMI TCP Accept-1099" daemon prio=6 tid=0x30f50800 nid=0xb2c runnable
> [0x3285f000..0x3285fd14]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.PlainSocketImpl.socketAccept(Native Method)
>          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>          - locked<0x1053b9f8>  (a java.net.SocksSocketImpl)
>          at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>          at java.net.ServerSocket.accept(ServerSocket.java:421)
>          at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
>          at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>          at java.lang.Thread.run(Thread.java:619)
>
> "Timer-2" daemon prio=6 tid=0x30d0b800 nid=0x9ec waiting for monitor entry
> [0x317bf000..0x317bfa94]
>     java.lang.Thread.State: BLOCKED (on object monitor)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.close(DependencyWaiterApplicatio
> nContextExecutor.java:345)
>          - waiting to lock<0x2cf47890>  (a java.lang.Object)
>          - locked<0x2cf4bc48>  (a java.lang.Object)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.doClose(AbstractDelegatedExecutionApplicationContext.
> java:236)
>          at
> org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:794)
>          - locked<0x2cf47820>  (a java.lang.Object)
>          at
> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$3.run(ContextLoaderListener.java:807)
>          at
> org.springframework.osgi.extender.internal.util.concurrent.RunnableTimedExecution$MonitoredRunnable.run(RunnableTimedExecution.java:60)
>          at
> org.springframework.scheduling.timer.DelegatingTimerTask.run(DelegatingTimerTask.java:66)
>          at java.util.TimerThread.mainLoop(Timer.java:512)
>          at java.util.TimerThread.run(Timer.java:462)
>
> "Timer-1" daemon prio=6 tid=0x30cb6400 nid=0x520 in Object.wait()
> [0x3176f000..0x3176fb14]
>     java.lang.Thread.State: WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.lang.Object.wait(Object.java:485)
>          at java.util.TimerThread.mainLoop(Timer.java:483)
>          - locked<0x101b9568>  (a java.util.TaskQueue)
>          at java.util.TimerThread.run(Timer.java:462)
>
> "pool-2-thread-1" prio=6 tid=0x30974800 nid=0x70c waiting on condition
> [0x3171f000..0x3171fb94]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for<0x10144650>  (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>          at java.lang.Thread.run(Thread.java:619)
>
> "pool-3-thread-1" prio=6 tid=0x3063d800 nid=0xa00 waiting on condition
> [0x316cf000..0x316cfc14]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for<0x10143a80>  (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>          at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>          at java.lang.Thread.run(Thread.java:619)
>
> "Timer-0" daemon prio=6 tid=0x30668800 nid=0x728 in Object.wait()
> [0x3167f000..0x3167fc94]
>     java.lang.Thread.State: TIMED_WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.util.TimerThread.mainLoop(Timer.java:509)
>          - locked<0x1014a870>  (a java.util.TaskQueue)
>          at java.util.TimerThread.run(Timer.java:462)
>
> "pool-1-thread-1" prio=6 tid=0x30942400 nid=0x554 waiting on condition
> [0x3162f000..0x3162fd14]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for<0x1014af60>  (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>          at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>          at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>          at java.lang.Thread.run(Thread.java:619)
>
> "Configuration Updater" daemon prio=6 tid=0x30101c00 nid=0xc64 in
> Object.wait() [0x315df000..0x315dfd94]
>     java.lang.Thread.State: WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          - waiting on<0x1014beb8>  (a java.util.LinkedList)
>          at java.lang.Object.wait(Object.java:485)
>          at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:69)
>          - locked<0x1014beb8>  (a java.util.LinkedList)
>
> "Thread-1" prio=6 tid=0x30b04c00 nid=0xa14 waiting on condition
> [0x3158f000..0x3158fa14]
>     java.lang.Thread.State: TIMED_WAITING (sleeping)
>          at java.lang.Thread.sleep(Native Method)
>          at org.apache.felix.karaf.main.Main.lock(Main.java:1011)
>          at org.apache.felix.karaf.main.Main$1.run(Main.java:188)
>
> "FelixPackageAdmin" daemon prio=6 tid=0x30b88800 nid=0xef4 in Object.wait()
> [0x3153f000..0x3153fa94]
>     java.lang.Thread.State: TIMED_WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at
> org.springframework.osgi.extender.internal.util.concurrent.Counter.waitFor(Counter.java:162)
>          - locked<0x2d186df0>  (a
> org.springframework.osgi.extender.internal.util.concurrent.Counter)
>          at
> org.springframework.osgi.extender.internal.util.concurrent.Counter.waitForZero(Counter.java:138)
>          - locked<0x2d186df0>  (a
> org.springframework.osgi.extender.internal.util.concurrent.Counter)
>          at
> org.springframework.osgi.extender.internal.util.concurrent.RunnableTimedExecution.execute(RunnableTimedExecution.java:108)
>          at
> org.springframework.osgi.extender.internal.activator.ContextLoaderListener.maybeCloseApplicationContextFor(ContextLoaderListener.java:800)
>          at
> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$ContextBundleListener.handleEvent(ContextLoaderListener.java:244
> )
>          at
> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$BaseListener.bundleChanged(ContextLoaderListener.java:174)
>          at
> org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:771)
>          at
> org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:700)
>          at
> org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:597)
>          at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:3409)
>          at org.apache.felix.framework.Felix.stopBundle(Felix.java:1807)
>          at
> org.apache.felix.framework.Felix$RefreshHelper.stop(Felix.java:4061)
>          at org.apache.felix.framework.Felix.refreshPackages(Felix.java:3238)
>          at
> org.apache.felix.framework.PackageAdminImpl.run(PackageAdminImpl.java:329)
>          at java.lang.Thread.run(Thread.java:619)
>
> "FelixStartLevel" daemon prio=6 tid=0x308da400 nid=0xe28 in Object.wait()
> [0x314ef000..0x314efb14]
>     java.lang.Thread.State: WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          - waiting on<0x101568e0>  (a java.util.ArrayList)
>          at java.lang.Object.wait(Object.java:485)
>          at
> org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:241)
>          - locked<0x101568e0>  (a java.util.ArrayList)
>          at java.lang.Thread.run(Thread.java:619)
>
> "FelixDispatchQueue" prio=6 tid=0x305fec00 nid=0x900 in Object.wait()
> [0x30c8f000..0x30c8fb94]
>     java.lang.Thread.State: WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.lang.Object.wait(Object.java:485)
>          at
> org.apache.felix.framework.util.EventDispatcher.run(EventDispatcher.java:875)
>          - locked<0x100f6538>  (a java.util.ArrayList)
>          at
> org.apache.felix.framework.util.EventDispatcher.access$000(EventDispatcher.java:50)
>          at
> org.apache.felix.framework.util.EventDispatcher$1.run(EventDispatcher.java:102)
>          at java.lang.Thread.run(Thread.java:619)
>
> "RMI TCP Accept-0" daemon prio=6 tid=0x3088ec00 nid=0xb14 runnable
> [0x30bef000..0x30befc94]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.PlainSocketImpl.socketAccept(Native Method)
>          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>          - locked<0x0feb8eb8>  (a java.net.SocksSocketImpl)
>          at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>          at java.net.ServerSocket.accept(ServerSocket.java:421)
>          at
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
>          at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
>          at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>          at java.lang.Thread.run(Thread.java:619)
>
> "Low Memory Detector" daemon prio=6 tid=0x301ad800 nid=0xf48 runnable
> [0x00000000..0x00000000]
>     java.lang.Thread.State: RUNNABLE
>
> "CompilerThread1" daemon prio=10 tid=0x301aac00 nid=0x770 waiting on
> condition [0x00000000..0x304af47c]
>     java.lang.Thread.State: RUNNABLE
>
> "CompilerThread0" daemon prio=10 tid=0x301a8c00 nid=0x8f8 waiting on
> condition [0x00000000..0x3045f108]
>     java.lang.Thread.State: RUNNABLE
>
> "Attach Listener" daemon prio=10 tid=0x301a3c00 nid=0xa10 runnable
> [0x00000000..0x00000000]
>     java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x301a2800 nid=0x7a8 waiting on
> condition [0x00000000..0x00000000]
>     java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=8 tid=0x3018f400 nid=0xab4 in Object.wait()
> [0x3036f000..0x3036fb94]
>     java.lang.Thread.State: WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>          - locked<0x0fea3bf8>  (a java.lang.ref.ReferenceQueue$Lock)
>          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>          at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x3018dc00 nid=0xbe0 in Object.wait()
> [0x3031f000..0x3031fc14]
>     java.lang.Thread.State: WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.lang.Object.wait(Object.java:485)
>          at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>          - locked<0x0fe754e8>  (a java.lang.ref.Reference$Lock)
>
> "main" prio=6 tid=0x002a8c00 nid=0x320 in Object.wait()
> [0x0088f000..0x0088fe54]
>     java.lang.Thread.State: WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          - waiting on<0x0fea3608>  (a
> org.apache.felix.framework.util.ThreadGate)
>          at
> org.apache.felix.framework.util.ThreadGate.await(ThreadGate.java:48)
>          - locked<0x0fea3608>  (a org.apache.felix.framework.util.ThreadGate)
>          at org.apache.felix.framework.Felix.waitForStop(Felix.java:785)
>          at org.apache.felix.karaf.main.Main.destroy(Main.java:201)
>          at org.apache.felix.karaf.main.Main.main(Main.java:293)
>          at org.apache.felix.karaf.main.Bootstrap.main(Bootstrap.java:66)
>
> "VM Thread" prio=10 tid=0x3018a800 nid=0xd54 runnable
>
> "GC task thread#0 (ParallelGC)" prio=6 tid=0x002af800 nid=0xc08 runnable
>
> "GC task thread#1 (ParallelGC)" prio=6 tid=0x300e0c00 nid=0xacc runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x30836400 nid=0xa1c waiting on
> condition
>
> JNI global references: 1705
>
> Heap
>   PSYoungGen      total 25280K, used 14234K [0x2c530000, 0x2e1a0000,
> 0x2fe10000)
>    eden space 21248K, 60% used [0x2c530000,0x2d1d43b0,0x2d9f0000)
>    from space 4032K, 31% used [0x2ddb0000,0x2def2830,0x2e1a0000)
>    to   space 3840K, 0% used [0x2d9f0000,0x2d9f0000,0x2ddb0000)
>   PSOldGen        total 233024K, used 34880K [0x0fe10000, 0x1e1a0000,
> 0x2c530000)
>    object space 233024K, 14% used [0x0fe10000,0x12020368,0x1e1a0000)
>   PSPermGen       total 141568K, used 73711K [0x03a10000, 0x0c450000,
> 0x0fe10000)
>    object space 141568K, 52% used [0x03a10000,0x0820bce8,0x0c450000)
>
> Exception in thread "SpringOsgiExtenderThread-49"
> java.lang.IllegalStateException: BeanFactory not initialized or already
> closed - call 'refresh' befo
> re accessing beans via the ApplicationContext
>          at
> org.springframework.context.support.AbstractRefreshableApplicationContext.getBeanFactory(AbstractRefreshableApplicationContext.java:153)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.close(DependencyWaiterApplicatio
> nContextExecutor.java:345)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.fail(DependencyWaiterApplication
> ContextExecutor.java:401)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
> tionContextExecutor.java:287)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
> ionContextExecutor.java:175)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
> java:175)
>          at
> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>          at java.lang.Thread.run(Thread.java:619)
> Exception in thread "SpringOsgiExtenderThread-51"
> java.lang.IllegalStateException: BeanFactory not initialized or already
> closed - call 'refresh' befo
> re accessing beans via the ApplicationContext
>          at
> org.springframework.context.support.AbstractRefreshableApplicationContext.getBeanFactory(AbstractRefreshableApplicationContext.java:153)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.close(DependencyWaiterApplicatio
> nContextExecutor.java:345)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.fail(DependencyWaiterApplication
> ContextExecutor.java:401)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
> tionContextExecutor.java:287)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
> ionContextExecutor.java:175)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
> java:175)
>          at
> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>          at java.lang.Thread.run(Thread.java:619)
> Exception in thread "SpringOsgiExtenderThread-50"
> java.lang.IllegalStateException: BeanFactory not initialized or already
> closed - call 'refresh' befo
> re accessing beans via the ApplicationContext
>          at
> org.springframework.context.support.AbstractRefreshableApplicationContext.getBeanFactory(AbstractRefreshableApplicationContext.java:153)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.close(DependencyWaiterApplicatio
> nContextExecutor.java:345)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.fail(DependencyWaiterApplication
> ContextExecutor.java:401)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
> tionContextExecutor.java:287)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
> ionContextExecutor.java:175)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
> java:175)
>          at
> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>          at java.lang.Thread.run(Thread.java:619)
>
> Charles Moulliard
> Senior Enterprise Architect
> Apache Camel Committer
>
> *****************************
> blog : http://cmoulliard.blogspot.com
>
>
> On Wed, Jul 1, 2009 at 5:05 PM, Guillaume Nodet<gn...@gmail.com>  wrote:
>
>    
>> I guess the extender use multiple threads to manage the spring-dm
>> powered bundles.
>> The question is which thread is holding the felix lock ? I can't find it
>> ...
>>
>> On Wed, Jul 1, 2009 at 16:48, Richard S. Hall<he...@ungoverned.org>  wrote:
>>      
>>> This is odd looking. All threads are blocked doing the exact same thing.
>>>        
>> Are
>>      
>>> they trying to parallelize this?
>>>
>>> The trace doesn't demonstrate why they can't continue since it doesn't
>>> include all of the threads.
>>>
>>> ->  richard
>>>
>>> On 7/1/09 10:22 AM, Charles Moulliard wrote:
>>>        
>>>> I have been able to reproduce the problem. This error is generated when
>>>>          
>> I
>>      
>>>> launch consecutively two refresh commands of the same bundle without
>>>> waiting
>>>> that the first refresh has finished. The bundle refreshed contains
>>>>          
>> spring
>>      
>>>> XML config files + services which are exposed as OSGI service
>>>>
>>>> Here is a trace of the thread dump :
>>>>
>>>> 2009-07-01 16:12:52
>>>> Full thread dump Java HotSpot(TM) Server VM (11.2-b01 mixed mode):
>>>>
>>>> "SpringOsgiExtenderThread-31" prio=6 tid=0x31e42400 nid=0xa18 in
>>>> Object.wait() [0x33d4f000..0x33d4fd14]
>>>>     java.lang.Thread.State: WAITING (on object monitor)
>>>>          at java.lang.Object.wait(Native Method)
>>>>          at java.lang.Object.wait(Object.java:485)
>>>>          at
>>>> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>>>>          - locked<0x0fea0670>   (a [Ljava.lang.Object;)
>>>>          at
>>>> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>>>>          at
>>>> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>>>>          at
>>>> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
>>      
>>>> )
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
>>      
>>>> xt.java:69)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
>>      
>>>> va:269)
>>>>          - locked<0x2c998110>   (a java.lang.Object)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
>>      
>>>> text.java:247)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
>>      
>>>> tionContextExecutor.java:222)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
>>      
>>>> ionContextExecutor.java:175)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
>>      
>>>> java:175)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>>      
>>>>          at java.lang.Thread.run(Thread.java:619)
>>>>
>>>> "SpringOsgiExtenderThread-30" prio=6 tid=0x32196c00 nid=0x988 in
>>>> Object.wait() [0x3357f000..0x3357fd94]
>>>>     java.lang.Thread.State: BLOCKED (on object monitor)
>>>>          at java.lang.Object.wait(Native Method)
>>>>          at java.lang.Object.wait(Object.java:485)
>>>>          at
>>>> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>>>>          - locked<0x0fea0670>   (a [Ljava.lang.Object;)
>>>>          at
>>>> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>>>>          at
>>>> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>>>>          at
>>>> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
>>      
>>>> )
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
>>      
>>>> xt.java:69)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
>>      
>>>> va:269)
>>>>          - locked<0x2c97e368>   (a java.lang.Object)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
>>      
>>>> text.java:247)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
>>      
>>>> tionContextExecutor.java:222)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
>>      
>>>> ionContextExecutor.java:175)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
>>      
>>>> java:175)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>>      
>>>>          at java.lang.Thread.run(Thread.java:619)
>>>>
>>>> "RMI TCP Connection(7)-172.16.10.128" daemon prio=6 tid=0x3198c400
>>>> nid=0xe98
>>>> runnable [0x3294f000..0x3294fa14]
>>>>     java.lang.Thread.State: RUNNABLE
>>>>          at java.net.SocketInputStream.socketRead0(Native Method)
>>>>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>>>>          at
>>>>          
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>      
>>>>          at
>>>>          
>> java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>>      
>>>>          - locked<0x2c6ad740>   (a java.io.BufferedInputStream)
>>>>          at java.io.FilterInputStream.read(FilterInputStream.java:66)
>>>>          at
>>>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
>>>>          at
>>>>
>>>>
>>>>          
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>      
>>>>          at java.lang.Thread.run(Thread.java:619)
>>>>
>>>> "InactivityMonitor Async Task:
>>>> java.util.concurrent.ThreadPoolExecutor$Worker@57c459" daemon prio=6
>>>> tid=0x324cd400 nid=0xbf0 waiting on condition [0x3
>>>> 34df000..0x334dfb14]
>>>>     java.lang.Thread.State: TIMED_WAITING (parking)
>>>>          at sun.misc.Unsafe.park(Native Method)
>>>>          - parking to wait for<0x10cba430>   (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>          at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>          at
>>>>
>>>>
>>>>          
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>      
>>>>          at
>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>          at
>>>>
>>>>
>>>>          
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>      
>>>>          at java.lang.Thread.run(Thread.java:619)
>>>>
>>>> "SpringOsgiExtenderThread-29" prio=6 tid=0x324e6c00 nid=0x8e4 in
>>>> Object.wait() [0x332af000..0x332afb94]
>>>>     java.lang.Thread.State: BLOCKED (on object monitor)
>>>>          at java.lang.Object.wait(Native Method)
>>>>          at java.lang.Object.wait(Object.java:485)
>>>>          at
>>>> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>>>>          - locked<0x0fea0670>   (a [Ljava.lang.Object;)
>>>>          at
>>>> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>>>>          at
>>>> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>>>>          at
>>>> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
>>      
>>>> )
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
>>      
>>>> xt.java:69)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
>>      
>>>> va:269)
>>>>          - locked<0x2e5d6ba8>   (a java.lang.Object)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
>>      
>>>> text.java:247)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
>>      
>>>> tionContextExecutor.java:222)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
>>      
>>>> ionContextExecutor.java:175)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
>>      
>>>> java:175)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>>      
>>>>          at java.lang.Thread.run(Thread.java:619)
>>>>
>>>> "SpringOsgiExtenderThread-28" prio=6 tid=0x3207ac00 nid=0xfe4 in
>>>> Object.wait() [0x333ef000..0x333efc14]
>>>>     java.lang.Thread.State: BLOCKED (on object monitor)
>>>>          at java.lang.Object.wait(Native Method)
>>>>          at java.lang.Object.wait(Object.java:485)
>>>>          at
>>>> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>>>>          - locked<0x0fea0670>   (a [Ljava.lang.Object;)
>>>>          at
>>>> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>>>>          at
>>>> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>>>>          at
>>>> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
>>      
>>>> )
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
>>      
>>>> xt.java:69)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
>>      
>>>> va:269)
>>>>          - locked<0x2e5d7e70>   (a java.lang.Object)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
>>      
>>>> text.java:247)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
>>      
>>>> tionContextExecutor.java:222)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
>>      
>>>> ionContextExecutor.java:175)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
>>      
>>>> java:175)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>>      
>>>>          at java.lang.Thread.run(Thread.java:619)
>>>>
>>>> "SpringOsgiExtenderThread-27" prio=6 tid=0x312cc400 nid=0xa74 in
>>>> Object.wait() [0x328af000..0x328afd14]
>>>>     java.lang.Thread.State: BLOCKED (on object monitor)
>>>>          at java.lang.Object.wait(Native Method)
>>>>          at java.lang.Object.wait(Object.java:485)
>>>>          at
>>>> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>>>>          - locked<0x0fea0670>   (a [Ljava.lang.Object;)
>>>>          at
>>>> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>>>>          at
>>>> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>>>>          at
>>>> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
>>      
>>>> )
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
>>      
>>>> xt.java:69)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
>>      
>>>> va:269)
>>>>          - locked<0x2e5dbbf0>   (a java.lang.Object)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
>>      
>>>> text.java:247)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
>>      
>>>> tionContextExecutor.java:222)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
>>      
>>>> ionContextExecutor.java:175)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
>>      
>>>> java:175)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>>      
>>>>          at java.lang.Thread.run(Thread.java:619)
>>>>
>>>> "SpringOsgiExtenderThread-26" prio=6 tid=0x3256a400 nid=0x6b8 in
>>>> Object.wait() [0x32a3f000..0x32a3fd94]
>>>>     java.lang.Thread.State: BLOCKED (on object monitor)
>>>>          at java.lang.Object.wait(Native Method)
>>>>          at java.lang.Object.wait(Object.java:485)
>>>>          at
>>>> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>>>>          - locked<0x0fea0670>   (a [Ljava.lang.Object;)
>>>>          at
>>>> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>>>>          at
>>>> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>>>>          at
>>>> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
>>      
>>>> )
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
>>      
>>>> xt.java:69)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
>>      
>>>> va:269)
>>>>          - locked<0x2e5e0520>   (a java.lang.Object)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>>      
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
>>      
>>>> text.java:247)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
>>      
>>>> tionContextExecutor.java:222)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
>>      
>>>> ionContextExecutor.java:175)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
>>      
>>>> java:175)
>>>>          at
>>>>
>>>>
>>>>          
>> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>>      
>>>>          at java.lang.Thread.run(Thread.java:619)
>>>>
>>>> Regards,
>>>>
>>>> Charles Moulliard
>>>> Senior Enterprise Architect
>>>> Apache Camel Committer
>>>>
>>>> *****************************
>>>> blog : http://cmoulliard.blogspot.com
>>>>
>>>>
>>>> On Tue, Jun 30, 2009 at 1:42 PM, Guillaume Nodet<gn...@gmail.com>
>>>>          
>>   wrote:
>>      
>>>>          
>>>>> Could you grab a thread dump of the vm when the shutdown is waiting ?
>>>>> There must be a deadlock somewhere but without any thread dump, it's
>>>>> kinda impossible to see what happen.
>>>>>
>>>>> On Tue, Jun 30, 2009 at 13:25, Charles Moulliard<cm...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>            
>>>>>> Hi,
>>>>>>
>>>>>> When Camel tries to stop one of its context, the following message is
>>>>>> displayed in the log:
>>>>>>
>>>>>> 13:10:59,213 | INFO  | Timer-2          | DefaultCamelContext
>>>>>>
>>>>>>              
>>>>>   |
>>>>>
>>>>>            
>>>>>> e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
>>>>>> (CamelContext:InToOut) is stopping
>>>>>> 13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext
>>>>>>
>>>>>>              
>>>>>   |
>>>>>
>>>>>            
>>>>>> e.camel.impl.DefaultCamelContext  975 | Apache Camel 2.0-SNAPSHOT
>>>>>> (CamelContext:InToOut) stopped
>>>>>> 13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext
>>>>>>
>>>>>>              
>>>>>   |
>>>>>
>>>>>            
>>>>>> e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
>>>>>> (CamelContext:OutToIn) is stopping
>>>>>> *13:11:09,181 | ERROR | Thread-47        | RunnableTimedExecution
>>>>>> | oncurrent.RunnableTimedExecution  109 | Closing runnable for context
>>>>>> OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-core,
>>>>>> config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
>>>>>> consider taking a snapshot and then shutdown the VM in case the thread
>>>>>>
>>>>>>              
>>>>> still
>>>>>
>>>>>            
>>>>>> hangs
>>>>>> 13:11:19,181 | ERROR | Thread-47        | RunnableTimedExecution
>>>>>>
>>>>>>              
>>>>>    |
>>>>>
>>>>>            
>>>>>> oncurrent.RunnableTimedExecution  109 | Closing runnable for context
>>>>>> OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-service,
>>>>>> config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
>>>>>> consider taking a snapshot and then shutdown the VM in case the thread
>>>>>>
>>>>>>              
>>>>> still
>>>>>
>>>>>            
>>>>>> hangs*
>>>>>>
>>>>>> It seems to be related to the following Spring DM error :
>>>>>> http://jira.springframework.org/browse/OSGI-611 but I use Spring OSGI
>>>>>>
>>>>>>              
>>>>> 1.2
>>>>>
>>>>>            
>>>>>> and the error is still there
>>>>>>
>>>>>> Does anybody knows how to solve this ?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Charles Moulliard
>>>>>> Senior Enterprise Architect
>>>>>> Apache Camel Committer
>>>>>>
>>>>>> *****************************
>>>>>> blog : http://cmoulliard.blogspot.com
>>>>>>
>>>>>>
>>>>>>              
>>>>> --
>>>>> Cheers,
>>>>> Guillaume Nodet
>>>>> ------------------------
>>>>> Blog: http://gnodet.blogspot.com/
>>>>> ------------------------
>>>>> Open Source SOA
>>>>> http://fusesource.com
>>>>>
>>>>>
>>>>>            
>>>>          
>>
>> --
>> Cheers,
>> Guillaume Nodet
>> ------------------------
>> Blog: http://gnodet.blogspot.com/
>> ------------------------
>> Open Source SOA
>> http://fusesource.com
>>
>>      
>
>    

Re: Apache Karaf & Spring DM : consider taking a snapshot and then shutdown the VM in case the thread still hangs

Posted by Charles Moulliard <cm...@gmail.com>.
Here is the full dump :

2009-07-01 17:22:09
Full thread dump Java HotSpot(TM) Server VM (11.2-b01 mixed mode):

"Keep-Alive-Timer" daemon prio=8 tid=0x309f5c00 nid=0xd6c waiting on
condition [0x3294f000..0x3294fc14]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at sun.net.www.http.KeepAliveCache.run(KeepAliveCache.java:149)
        at java.lang.Thread.run(Thread.java:619)

"SpringOsgiExtenderThread-49" prio=6 tid=0x3250c800 nid=0xe84 in
Object.wait() [0x3361f000..0x3361fc94]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at
org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
        - locked <0x0fea0670> (a [Ljava.lang.Object;)
        at org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
        at
org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
        at
org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
        at
org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
        at
org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
        at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
        at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
        at
org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
        at
org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
        at
org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
)
        at
org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
xt.java:69)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
va:269)
        - locked <0x2cf47890> (a java.lang.Object)
        at
org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
text.java:247)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
tionContextExecutor.java:222)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
ionContextExecutor.java:175)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
java:175)
        at
org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp:///127.0.0.1:3084" daemon prio=4 tid=0x31ae5800
nid=0xae4 runnable [0x3692f000..0x3692fc94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x31ae5400
nid=0x5d4 runnable [0x3679f000..0x3679fd94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp:///127.0.0.1:3083" daemon prio=4 tid=0x31828c00
nid=0x920 runnable [0x365bf000..0x365bfc14]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x30f00400
nid=0x334 runnable [0x3633f000..0x3633fd94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"InactivityMonitor Async Task:
java.util.concurrent.ThreadPoolExecutor$Worker@137854f" daemon prio=6
tid=0x32608800 nid=0xa48 waiting on condition [0x
3357f000..0x3357fd94]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x10cba430> (a
java.util.concurrent.SynchronousQueue$TransferStack)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
        at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
        at
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp:///127.0.0.1:2934" daemon prio=4 tid=0x31fbb400
nid=0x6c0 runnable [0x3683f000..0x3683fa94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x32138400
nid=0x29c runnable [0x367ef000..0x367efb94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp:///127.0.0.1:2933" daemon prio=4 tid=0x30a7b400
nid=0x214 runnable [0x3665f000..0x3665fd14]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x32232800
nid=0xf64 runnable [0x3660f000..0x3660fa14]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"SocketConnector-5" daemon prio=6 tid=0x31fe4400 nid=0x5b0 runnable
[0x332af000..0x332afa94]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:274)
        at
sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:256)
        at
sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:137)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <0x11a0c020> (a sun.nio.ch.Util$1)
        - locked <0x11a0c010> (a java.util.Collections$UnmodifiableSet)
        - locked <0x11a0b7d8> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at
org.apache.mina.transport.socket.nio.SocketConnector$Worker.run(SocketConnector.java:378)
        at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
        at java.lang.Thread.run(Thread.java:619)

"InactivityMonitor Async Task:
java.util.concurrent.ThreadPoolExecutor$Worker@13144f4" daemon prio=6
tid=0x30d25c00 nid=0xc6c waiting on condition [0x
334df000..0x334dfa14]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x10cba430> (a
java.util.concurrent.SynchronousQueue$TransferStack)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
        at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
        at
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp:///127.0.0.1:2638" daemon prio=4 tid=0x31246400
nid=0xab8 runnable [0x36abf000..0x36abfa14]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x31afe400
nid=0x784 runnable [0x36a1f000..0x36a1fb14]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp:///127.0.0.1:2637" daemon prio=4 tid=0x340e0800
nid=0x4dc runnable [0x3697f000..0x3697fc94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x3224ec00
nid=0x404 runnable [0x368df000..0x368dfd94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp:///127.0.0.1:2632" daemon prio=4 tid=0x31fe4800
nid=0x558 runnable [0x3674f000..0x3674fd94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x32072c00
nid=0xf74 runnable [0x366af000..0x366afa94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp:///127.0.0.1:2631" daemon prio=4 tid=0x32750400
nid=0xc74 runnable [0x3656f000..0x3656fd14]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x308f3400
nid=0x3d8 runnable [0x364cf000..0x364cfd94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp:///127.0.0.1:2593" daemon prio=4 tid=0x32140c00
nid=0xc58 runnable [0x3651f000..0x3651fd14]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x3225e800
nid=0x694 runnable [0x3647f000..0x3647fa14]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp:///127.0.0.1:2592" daemon prio=4 tid=0x322e5400
nid=0xfb8 runnable [0x363df000..0x363dfb94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x31fecc00
nid=0x42c runnable [0x3601f000..0x3601fc94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp:///127.0.0.1:2589" daemon prio=4 tid=0x32056800
nid=0x9b8 runnable [0x35fcf000..0x35fcfb94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x3258ac00
nid=0xa5c runnable [0x33caf000..0x33cafc94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp:///127.0.0.1:2588" daemon prio=4 tid=0x31c49800
nid=0x3cc runnable [0x3606f000..0x3606fa94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x32127400
nid=0x488 runnable [0x32c6f000..0x32c6fb94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp:///127.0.0.1:2582" daemon prio=4 tid=0x31f53400
nid=0xfb4 runnable [0x33edf000..0x33edfb14]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x322e8400
nid=0xff8 runnable [0x33e8f000..0x33e8fc14]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp:///127.0.0.1:2581" daemon prio=4 tid=0x321e4c00
nid=0xcec runnable [0x33cff000..0x33cffa94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x31136c00
nid=0xa54 runnable [0x33a2f000..0x33a2fb94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp:///127.0.0.1:2309" daemon prio=4 tid=0x32524800
nid=0x584 runnable [0x361ff000..0x361ffc94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x31067400
nid=0x9c8 runnable [0x3615f000..0x3615fd94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"Camel thread 6: ActiveMQComponent" daemon prio=6 tid=0x31d3fc00 nid=0xb48
runnable [0x3610f000..0x3610fc94]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x1133f570> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"Camel thread 5: ActiveMQComponent" daemon prio=6 tid=0x31c51c00 nid=0xe34
runnable [0x360bf000..0x360bfc14]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x1133f570> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"Thread-23" daemon prio=6 tid=0x31cab000 nid=0xde8 waiting on condition
[0x3339f000..0x3339fa14]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.geronimo.transaction.manager.TransactionTimer$CurrentTime.run(TransactionTimer.java:38)

"ActiveMQ Transport: tcp:///127.0.0.1:2308" daemon prio=4 tid=0x3084c800
nid=0xa40 runnable [0x33c5f000..0x33c5fa14]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x31cd0800
nid=0xe54 runnable [0x3398f000..0x3398fb14]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"Camel thread 4: ActiveMQComponent" daemon prio=6 tid=0x30d6f400 nid=0x3b4
runnable [0x3393f000..0x3393fb94]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x1133f570> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"Camel thread 3: ActiveMQComponent" daemon prio=6 tid=0x325bdc00 nid=0x5e0
runnable [0x3389f000..0x3389fc14]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x1133f570> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"Camel thread 2: ActiveMQComponent" daemon prio=6 tid=0x3084a000 nid=0x9ac
runnable [0x332ff000..0x332ffc94]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x1133f570> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp:///127.0.0.1:2281" daemon prio=4 tid=0x31c38400
nid=0x578 runnable [0x33c0f000..0x33c0fc94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x308f2400
nid=0xa80 runnable [0x339df000..0x339dfd94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp:///127.0.0.1:2280" daemon prio=4 tid=0x317f8400
nid=0xce4 runnable [0x337af000..0x337afc14]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x31830800
nid=0x460 runnable [0x3370f000..0x3370fc94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"org.apache.commons.vfs.cache.SoftRefFilesCache$SoftRefReleaseThread" daemon
prio=6 tid=0x32143800 nid=0x9a4 in Object.wait() [0x32d0f000..0x32d0fa94]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0x106fa908> (a java.lang.ref.ReferenceQueue$Lock)
        at
org.apache.commons.vfs.cache.SoftRefFilesCache$SoftRefReleaseThread.run(SoftRefFilesCache.java:79)

"QueueThread:queue://x3s:notification:message" daemon prio=6 tid=0x32143400
nid=0xf58 waiting on condition [0x33bbf000..0x33bbfb94]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x10a5c850> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"QueueThread:queue://x3s:notification" daemon prio=6 tid=0x31273800
nid=0x2d0 waiting on condition [0x33b6f000..0x33b6fc94]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x10a3f248> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp:///127.0.0.1:2230" daemon prio=4 tid=0x31273400
nid=0x7c4 runnable [0x33b1f000..0x33b1fd94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x313b4800
nid=0xe3c runnable [0x3343f000..0x3343fa94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp:///127.0.0.1:2229" daemon prio=4 tid=0x310ad800
nid=0xa08 runnable [0x33acf000..0x33acfb94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x310ad400
nid=0xd3c runnable [0x33a7f000..0x33a7fc94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"QueueThread:queue://x3s:requestmessage" daemon prio=6 tid=0x3120a800
nid=0xd28 waiting on condition [0x338ef000..0x338efc94]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x10a53e88> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"QueueThread:queue://x3s:request" daemon prio=6 tid=0x3086fc00 nid=0xf04
waiting on condition [0x3384f000..0x3384fa14]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x10a5e9d0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp:///127.0.0.1:2225" daemon prio=4 tid=0x31bf5400
nid=0x13c runnable [0x337ff000..0x337ffa94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x31f78c00
nid=0x354 runnable [0x3375f000..0x3375fb94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"InactivityMonitor WriteCheck" daemon prio=6 tid=0x3084d400 nid=0x860 in
Object.wait() [0x336bf000..0x336bfd14]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x112506f0> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"InactivityMonitor ReadCheck" daemon prio=6 tid=0x31014400 nid=0x69c in
Object.wait() [0x3366f000..0x3366fd94]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x1125e188> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"ActiveMQ Transport: tcp:///127.0.0.1:2224" daemon prio=4 tid=0x3138e400
nid=0xb74 runnable [0x335cf000..0x335cfa94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x30fdcc00
nid=0x158 runnable [0x3352f000..0x3352fb94]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"QFJ Timer" daemon prio=6 tid=0x3118b400 nid=0xa64 waiting on condition
[0x32c1f000..0x32c1fa14]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x111e03c0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"MySQL Statement Cancellation Timer" daemon prio=6 tid=0x30fc5800 nid=0x50c
in Object.wait() [0x3348f000..0x3348fa94]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x10c61a88> (a java.util.TaskQueue)
        at java.lang.Object.wait(Object.java:485)
        at java.util.TimerThread.mainLoop(Timer.java:483)
        - locked <0x10c61a88> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"ActiveMQ Transport Server: stomp://localhost:61613" daemon prio=10
tid=0x312ea800 nid=0x39c runnable [0x3325f000..0x3325fd94]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0x10c61fc8> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at
org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:274)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport Server Thread Handler: stomp://localhost:61613" daemon
prio=8 tid=0x30605800 nid=0xd5c runnable [0x329ef000..0x329efa14]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x10c62138> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
        at
org.apache.activemq.transport.tcp.TcpTransportServer$1.run(TcpTransportServer.java:339)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport Server: tcp://localhost:61616" daemon prio=10
tid=0x31d7c400 nid=0x574 runnable [0x3320f000..0x3320fa94]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0x10a65180> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at
org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:274)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport Server Thread Handler: tcp://localhost:61616" daemon
prio=8 tid=0x31c62800 nid=0xed8 runnable [0x331bf000..0x331bfb14]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x10a65300> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
        at
org.apache.activemq.transport.tcp.TcpTransportServer$1.run(TcpTransportServer.java:339)
        at java.lang.Thread.run(Thread.java:619)

"NioSocketAcceptor-2" prio=6 tid=0x31a4bc00 nid=0x9a8 runnable
[0x32cbf000..0x32cbfb94]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:274)
        at
sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:256)
        at
sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:137)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <0x10b07508> (a sun.nio.ch.Util$1)
        - locked <0x10b074f8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x10b070c8> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
        at
org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:288)
        at
org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:402)
        at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Scheduler" daemon prio=6 tid=0x30f1cc00 nid=0x5c8 in Object.wait()
[0x3316f000..0x3316fc94]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x10a75348> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"ActiveMQ Data File Writer" daemon prio=10 tid=0x3051dc00 nid=0x978 in
Object.wait() [0x3280f000..0x3280fd14]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at
org.apache.activemq.kaha.impl.async.NIODataFileAppender.processQueue(NIODataFileAppender.java:79)
        - locked <0x10a771e8> (a
org.apache.activemq.kaha.impl.async.DataFileAppender$1)
        at
org.apache.activemq.kaha.impl.async.DataFileAppender$2.run(DataFileAppender.java:227)

"localShell" prio=6 tid=0x31856c00 nid=0xaec runnable
[0x32a8f000..0x32a8fa94]
   java.lang.Thread.State: RUNNABLE
        at jline.WindowsTerminal.readByte(Native Method)
        at jline.WindowsTerminal.readCharacter(WindowsTerminal.java:233)
        at jline.WindowsTerminal.readVirtualKey(WindowsTerminal.java:319)
        at jline.ConsoleReader.readVirtualKey(ConsoleReader.java:1453)
        at jline.ConsoleReader.readBinding(ConsoleReader.java:654)
        at jline.ConsoleReader.readLine(ConsoleReader.java:494)
        at jline.ConsoleReader.readLine(ConsoleReader.java:448)
        at
org.apache.geronimo.gshell.console.JLineConsole.readLine(JLineConsole.java:95)
        at org.apache.geronimo.gshell.console.Console.work(Console.java:153)
        at org.apache.geronimo.gshell.console.Console.run(Console.java:128)
        at
org.apache.geronimo.gshell.wisdom.shell.ShellImpl.run(ShellImpl.java:252)
        at
org.apache.felix.karaf.gshell.core.ShellWrapper.run(ShellWrapper.java:81)
        at
org.apache.felix.karaf.gshell.core.LocalConsole.run(LocalConsole.java:119)
        at java.lang.Thread.run(Thread.java:619)

"RMI RenewClean-[172.16.10.128:2163]" daemon prio=6 tid=0x3196c400 nid=0xb50
in Object.wait() [0x328ff000..0x328ffb94]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0x10836d70> (a java.lang.ref.ReferenceQueue$Lock)
        at
sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
        at java.lang.Thread.run(Thread.java:619)

"RMI Scheduler(0)" daemon prio=6 tid=0x30a63800 nid=0x138 waiting on
condition [0x3299f000..0x3299fc94]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x100e7d90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"GC Daemon" daemon prio=2 tid=0x30a46400 nid=0xd64 in Object.wait()
[0x32bcf000..0x32bcfa94]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at sun.misc.GC$Daemon.run(GC.java:100)
        - locked <0x105eddf0> (a sun.misc.GC$LatencyLock)

"RMI Reaper" prio=6 tid=0x30f38400 nid=0x628 in Object.wait()
[0x32b7f000..0x32b7fb14]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x10124788> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0x10124788> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
        at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
        at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-0" daemon prio=6 tid=0x3214a800 nid=0xc0 runnable
[0x32b2f000..0x32b2fb94]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0x1053b880> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
        at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
        at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-1099" daemon prio=6 tid=0x30f50800 nid=0xb2c runnable
[0x3285f000..0x3285fd14]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0x1053b9f8> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
        at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
        at java.lang.Thread.run(Thread.java:619)

"Timer-2" daemon prio=6 tid=0x30d0b800 nid=0x9ec waiting for monitor entry
[0x317bf000..0x317bfa94]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.close(DependencyWaiterApplicatio
nContextExecutor.java:345)
        - waiting to lock <0x2cf47890> (a java.lang.Object)
        - locked <0x2cf4bc48> (a java.lang.Object)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.doClose(AbstractDelegatedExecutionApplicationContext.
java:236)
        at
org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:794)
        - locked <0x2cf47820> (a java.lang.Object)
        at
org.springframework.osgi.extender.internal.activator.ContextLoaderListener$3.run(ContextLoaderListener.java:807)
        at
org.springframework.osgi.extender.internal.util.concurrent.RunnableTimedExecution$MonitoredRunnable.run(RunnableTimedExecution.java:60)
        at
org.springframework.scheduling.timer.DelegatingTimerTask.run(DelegatingTimerTask.java:66)
        at java.util.TimerThread.mainLoop(Timer.java:512)
        at java.util.TimerThread.run(Timer.java:462)

"Timer-1" daemon prio=6 tid=0x30cb6400 nid=0x520 in Object.wait()
[0x3176f000..0x3176fb14]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at java.util.TimerThread.mainLoop(Timer.java:483)
        - locked <0x101b9568> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"pool-2-thread-1" prio=6 tid=0x30974800 nid=0x70c waiting on condition
[0x3171f000..0x3171fb94]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x10144650> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"pool-3-thread-1" prio=6 tid=0x3063d800 nid=0xa00 waiting on condition
[0x316cf000..0x316cfc14]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x10143a80> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"Timer-0" daemon prio=6 tid=0x30668800 nid=0x728 in Object.wait()
[0x3167f000..0x3167fc94]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x1014a870> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"pool-1-thread-1" prio=6 tid=0x30942400 nid=0x554 waiting on condition
[0x3162f000..0x3162fd14]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x1014af60> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"Configuration Updater" daemon prio=6 tid=0x30101c00 nid=0xc64 in
Object.wait() [0x315df000..0x315dfd94]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x1014beb8> (a java.util.LinkedList)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:69)
        - locked <0x1014beb8> (a java.util.LinkedList)

"Thread-1" prio=6 tid=0x30b04c00 nid=0xa14 waiting on condition
[0x3158f000..0x3158fa14]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.felix.karaf.main.Main.lock(Main.java:1011)
        at org.apache.felix.karaf.main.Main$1.run(Main.java:188)

"FelixPackageAdmin" daemon prio=6 tid=0x30b88800 nid=0xef4 in Object.wait()
[0x3153f000..0x3153fa94]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at
org.springframework.osgi.extender.internal.util.concurrent.Counter.waitFor(Counter.java:162)
        - locked <0x2d186df0> (a
org.springframework.osgi.extender.internal.util.concurrent.Counter)
        at
org.springframework.osgi.extender.internal.util.concurrent.Counter.waitForZero(Counter.java:138)
        - locked <0x2d186df0> (a
org.springframework.osgi.extender.internal.util.concurrent.Counter)
        at
org.springframework.osgi.extender.internal.util.concurrent.RunnableTimedExecution.execute(RunnableTimedExecution.java:108)
        at
org.springframework.osgi.extender.internal.activator.ContextLoaderListener.maybeCloseApplicationContextFor(ContextLoaderListener.java:800)
        at
org.springframework.osgi.extender.internal.activator.ContextLoaderListener$ContextBundleListener.handleEvent(ContextLoaderListener.java:244
)
        at
org.springframework.osgi.extender.internal.activator.ContextLoaderListener$BaseListener.bundleChanged(ContextLoaderListener.java:174)
        at
org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:771)
        at
org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:700)
        at
org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:597)
        at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:3409)
        at org.apache.felix.framework.Felix.stopBundle(Felix.java:1807)
        at
org.apache.felix.framework.Felix$RefreshHelper.stop(Felix.java:4061)
        at org.apache.felix.framework.Felix.refreshPackages(Felix.java:3238)
        at
org.apache.felix.framework.PackageAdminImpl.run(PackageAdminImpl.java:329)
        at java.lang.Thread.run(Thread.java:619)

"FelixStartLevel" daemon prio=6 tid=0x308da400 nid=0xe28 in Object.wait()
[0x314ef000..0x314efb14]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x101568e0> (a java.util.ArrayList)
        at java.lang.Object.wait(Object.java:485)
        at
org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:241)
        - locked <0x101568e0> (a java.util.ArrayList)
        at java.lang.Thread.run(Thread.java:619)

"FelixDispatchQueue" prio=6 tid=0x305fec00 nid=0x900 in Object.wait()
[0x30c8f000..0x30c8fb94]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at
org.apache.felix.framework.util.EventDispatcher.run(EventDispatcher.java:875)
        - locked <0x100f6538> (a java.util.ArrayList)
        at
org.apache.felix.framework.util.EventDispatcher.access$000(EventDispatcher.java:50)
        at
org.apache.felix.framework.util.EventDispatcher$1.run(EventDispatcher.java:102)
        at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-0" daemon prio=6 tid=0x3088ec00 nid=0xb14 runnable
[0x30bef000..0x30befc94]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0x0feb8eb8> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
        at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
        at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
        at java.lang.Thread.run(Thread.java:619)

"Low Memory Detector" daemon prio=6 tid=0x301ad800 nid=0xf48 runnable
[0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x301aac00 nid=0x770 waiting on
condition [0x00000000..0x304af47c]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x301a8c00 nid=0x8f8 waiting on
condition [0x00000000..0x3045f108]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" daemon prio=10 tid=0x301a3c00 nid=0xa10 runnable
[0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x301a2800 nid=0x7a8 waiting on
condition [0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=8 tid=0x3018f400 nid=0xab4 in Object.wait()
[0x3036f000..0x3036fb94]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0x0fea3bf8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x3018dc00 nid=0xbe0 in Object.wait()
[0x3031f000..0x3031fc14]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        - locked <0x0fe754e8> (a java.lang.ref.Reference$Lock)

"main" prio=6 tid=0x002a8c00 nid=0x320 in Object.wait()
[0x0088f000..0x0088fe54]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x0fea3608> (a
org.apache.felix.framework.util.ThreadGate)
        at
org.apache.felix.framework.util.ThreadGate.await(ThreadGate.java:48)
        - locked <0x0fea3608> (a org.apache.felix.framework.util.ThreadGate)
        at org.apache.felix.framework.Felix.waitForStop(Felix.java:785)
        at org.apache.felix.karaf.main.Main.destroy(Main.java:201)
        at org.apache.felix.karaf.main.Main.main(Main.java:293)
        at org.apache.felix.karaf.main.Bootstrap.main(Bootstrap.java:66)

"VM Thread" prio=10 tid=0x3018a800 nid=0xd54 runnable

"GC task thread#0 (ParallelGC)" prio=6 tid=0x002af800 nid=0xc08 runnable

"GC task thread#1 (ParallelGC)" prio=6 tid=0x300e0c00 nid=0xacc runnable

"VM Periodic Task Thread" prio=10 tid=0x30836400 nid=0xa1c waiting on
condition

JNI global references: 1705

Heap
 PSYoungGen      total 25280K, used 14234K [0x2c530000, 0x2e1a0000,
0x2fe10000)
  eden space 21248K, 60% used [0x2c530000,0x2d1d43b0,0x2d9f0000)
  from space 4032K, 31% used [0x2ddb0000,0x2def2830,0x2e1a0000)
  to   space 3840K, 0% used [0x2d9f0000,0x2d9f0000,0x2ddb0000)
 PSOldGen        total 233024K, used 34880K [0x0fe10000, 0x1e1a0000,
0x2c530000)
  object space 233024K, 14% used [0x0fe10000,0x12020368,0x1e1a0000)
 PSPermGen       total 141568K, used 73711K [0x03a10000, 0x0c450000,
0x0fe10000)
  object space 141568K, 52% used [0x03a10000,0x0820bce8,0x0c450000)

Exception in thread "SpringOsgiExtenderThread-49"
java.lang.IllegalStateException: BeanFactory not initialized or already
closed - call 'refresh' befo
re accessing beans via the ApplicationContext
        at
org.springframework.context.support.AbstractRefreshableApplicationContext.getBeanFactory(AbstractRefreshableApplicationContext.java:153)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.close(DependencyWaiterApplicatio
nContextExecutor.java:345)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.fail(DependencyWaiterApplication
ContextExecutor.java:401)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
tionContextExecutor.java:287)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
ionContextExecutor.java:175)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
java:175)
        at
org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
        at java.lang.Thread.run(Thread.java:619)
Exception in thread "SpringOsgiExtenderThread-51"
java.lang.IllegalStateException: BeanFactory not initialized or already
closed - call 'refresh' befo
re accessing beans via the ApplicationContext
        at
org.springframework.context.support.AbstractRefreshableApplicationContext.getBeanFactory(AbstractRefreshableApplicationContext.java:153)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.close(DependencyWaiterApplicatio
nContextExecutor.java:345)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.fail(DependencyWaiterApplication
ContextExecutor.java:401)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
tionContextExecutor.java:287)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
ionContextExecutor.java:175)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
java:175)
        at
org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
        at java.lang.Thread.run(Thread.java:619)
Exception in thread "SpringOsgiExtenderThread-50"
java.lang.IllegalStateException: BeanFactory not initialized or already
closed - call 'refresh' befo
re accessing beans via the ApplicationContext
        at
org.springframework.context.support.AbstractRefreshableApplicationContext.getBeanFactory(AbstractRefreshableApplicationContext.java:153)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.close(DependencyWaiterApplicatio
nContextExecutor.java:345)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.fail(DependencyWaiterApplication
ContextExecutor.java:401)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
tionContextExecutor.java:287)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
ionContextExecutor.java:175)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
java:175)
        at
org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
        at java.lang.Thread.run(Thread.java:619)

Charles Moulliard
Senior Enterprise Architect
Apache Camel Committer

*****************************
blog : http://cmoulliard.blogspot.com


On Wed, Jul 1, 2009 at 5:05 PM, Guillaume Nodet <gn...@gmail.com> wrote:

> I guess the extender use multiple threads to manage the spring-dm
> powered bundles.
> The question is which thread is holding the felix lock ? I can't find it
> ...
>
> On Wed, Jul 1, 2009 at 16:48, Richard S. Hall<he...@ungoverned.org> wrote:
> > This is odd looking. All threads are blocked doing the exact same thing.
> Are
> > they trying to parallelize this?
> >
> > The trace doesn't demonstrate why they can't continue since it doesn't
> > include all of the threads.
> >
> > -> richard
> >
> > On 7/1/09 10:22 AM, Charles Moulliard wrote:
> >>
> >> I have been able to reproduce the problem. This error is generated when
> I
> >> launch consecutively two refresh commands of the same bundle without
> >> waiting
> >> that the first refresh has finished. The bundle refreshed contains
> spring
> >> XML config files + services which are exposed as OSGI service
> >>
> >> Here is a trace of the thread dump :
> >>
> >> 2009-07-01 16:12:52
> >> Full thread dump Java HotSpot(TM) Server VM (11.2-b01 mixed mode):
> >>
> >> "SpringOsgiExtenderThread-31" prio=6 tid=0x31e42400 nid=0xa18 in
> >> Object.wait() [0x33d4f000..0x33d4fd14]
> >>    java.lang.Thread.State: WAITING (on object monitor)
> >>         at java.lang.Object.wait(Native Method)
> >>         at java.lang.Object.wait(Object.java:485)
> >>         at
> >> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
> >>         - locked<0x0fea0670>  (a [Ljava.lang.Object;)
> >>         at
> >> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
> >>         at
> >> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
> >>         at
> >> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
> >>         at
> >>
> >>
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
> >>         at
> >>
> >>
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
> >>         at
> >>
> >>
> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
> >> )
> >>         at
> >>
> >>
> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
> >> xt.java:69)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
> >> va:269)
> >>         - locked<0x2c998110>  (a java.lang.Object)
> >>         at
> >>
> >>
> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
> >> text.java:247)
> >>         at
> >>
> >>
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
> >> tionContextExecutor.java:222)
> >>         at
> >>
> >>
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
> >> ionContextExecutor.java:175)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
> >> java:175)
> >>         at
> >>
> >>
> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
> >>         at java.lang.Thread.run(Thread.java:619)
> >>
> >> "SpringOsgiExtenderThread-30" prio=6 tid=0x32196c00 nid=0x988 in
> >> Object.wait() [0x3357f000..0x3357fd94]
> >>    java.lang.Thread.State: BLOCKED (on object monitor)
> >>         at java.lang.Object.wait(Native Method)
> >>         at java.lang.Object.wait(Object.java:485)
> >>         at
> >> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
> >>         - locked<0x0fea0670>  (a [Ljava.lang.Object;)
> >>         at
> >> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
> >>         at
> >> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
> >>         at
> >> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
> >>         at
> >>
> >>
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
> >>         at
> >>
> >>
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
> >>         at
> >>
> >>
> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
> >> )
> >>         at
> >>
> >>
> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
> >> xt.java:69)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
> >> va:269)
> >>         - locked<0x2c97e368>  (a java.lang.Object)
> >>         at
> >>
> >>
> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
> >> text.java:247)
> >>         at
> >>
> >>
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
> >> tionContextExecutor.java:222)
> >>         at
> >>
> >>
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
> >> ionContextExecutor.java:175)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
> >> java:175)
> >>         at
> >>
> >>
> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
> >>         at java.lang.Thread.run(Thread.java:619)
> >>
> >> "RMI TCP Connection(7)-172.16.10.128" daemon prio=6 tid=0x3198c400
> >> nid=0xe98
> >> runnable [0x3294f000..0x3294fa14]
> >>    java.lang.Thread.State: RUNNABLE
> >>         at java.net.SocketInputStream.socketRead0(Native Method)
> >>         at java.net.SocketInputStream.read(SocketInputStream.java:129)
> >>         at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> >>         at
> java.io.BufferedInputStream.read(BufferedInputStream.java:237)
> >>         - locked<0x2c6ad740>  (a java.io.BufferedInputStream)
> >>         at java.io.FilterInputStream.read(FilterInputStream.java:66)
> >>         at
> >> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
> >>         at
> >>
> >>
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
> >>         at
> >>
> >>
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
> >>         at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >>         at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>         at java.lang.Thread.run(Thread.java:619)
> >>
> >> "InactivityMonitor Async Task:
> >> java.util.concurrent.ThreadPoolExecutor$Worker@57c459" daemon prio=6
> >> tid=0x324cd400 nid=0xbf0 waiting on condition [0x3
> >> 34df000..0x334dfb14]
> >>    java.lang.Thread.State: TIMED_WAITING (parking)
> >>         at sun.misc.Unsafe.park(Native Method)
> >>         - parking to wait for<0x10cba430>  (a
> >> java.util.concurrent.SynchronousQueue$TransferStack)
> >>         at
> >> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> >>         at
> >>
> >>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> >>         at
> >>
> >>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> >>         at
> >> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> >>         at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> >>         at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> >>         at java.lang.Thread.run(Thread.java:619)
> >>
> >> "SpringOsgiExtenderThread-29" prio=6 tid=0x324e6c00 nid=0x8e4 in
> >> Object.wait() [0x332af000..0x332afb94]
> >>    java.lang.Thread.State: BLOCKED (on object monitor)
> >>         at java.lang.Object.wait(Native Method)
> >>         at java.lang.Object.wait(Object.java:485)
> >>         at
> >> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
> >>         - locked<0x0fea0670>  (a [Ljava.lang.Object;)
> >>         at
> >> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
> >>         at
> >> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
> >>         at
> >> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
> >>         at
> >>
> >>
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
> >>         at
> >>
> >>
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
> >>         at
> >>
> >>
> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
> >> )
> >>         at
> >>
> >>
> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
> >> xt.java:69)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
> >> va:269)
> >>         - locked<0x2e5d6ba8>  (a java.lang.Object)
> >>         at
> >>
> >>
> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
> >> text.java:247)
> >>         at
> >>
> >>
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
> >> tionContextExecutor.java:222)
> >>         at
> >>
> >>
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
> >> ionContextExecutor.java:175)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
> >> java:175)
> >>         at
> >>
> >>
> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
> >>         at java.lang.Thread.run(Thread.java:619)
> >>
> >> "SpringOsgiExtenderThread-28" prio=6 tid=0x3207ac00 nid=0xfe4 in
> >> Object.wait() [0x333ef000..0x333efc14]
> >>    java.lang.Thread.State: BLOCKED (on object monitor)
> >>         at java.lang.Object.wait(Native Method)
> >>         at java.lang.Object.wait(Object.java:485)
> >>         at
> >> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
> >>         - locked<0x0fea0670>  (a [Ljava.lang.Object;)
> >>         at
> >> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
> >>         at
> >> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
> >>         at
> >> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
> >>         at
> >>
> >>
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
> >>         at
> >>
> >>
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
> >>         at
> >>
> >>
> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
> >> )
> >>         at
> >>
> >>
> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
> >> xt.java:69)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
> >> va:269)
> >>         - locked<0x2e5d7e70>  (a java.lang.Object)
> >>         at
> >>
> >>
> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
> >> text.java:247)
> >>         at
> >>
> >>
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
> >> tionContextExecutor.java:222)
> >>         at
> >>
> >>
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
> >> ionContextExecutor.java:175)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
> >> java:175)
> >>         at
> >>
> >>
> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
> >>         at java.lang.Thread.run(Thread.java:619)
> >>
> >> "SpringOsgiExtenderThread-27" prio=6 tid=0x312cc400 nid=0xa74 in
> >> Object.wait() [0x328af000..0x328afd14]
> >>    java.lang.Thread.State: BLOCKED (on object monitor)
> >>         at java.lang.Object.wait(Native Method)
> >>         at java.lang.Object.wait(Object.java:485)
> >>         at
> >> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
> >>         - locked<0x0fea0670>  (a [Ljava.lang.Object;)
> >>         at
> >> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
> >>         at
> >> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
> >>         at
> >> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
> >>         at
> >>
> >>
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
> >>         at
> >>
> >>
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
> >>         at
> >>
> >>
> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
> >> )
> >>         at
> >>
> >>
> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
> >> xt.java:69)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
> >> va:269)
> >>         - locked<0x2e5dbbf0>  (a java.lang.Object)
> >>         at
> >>
> >>
> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
> >> text.java:247)
> >>         at
> >>
> >>
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
> >> tionContextExecutor.java:222)
> >>         at
> >>
> >>
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
> >> ionContextExecutor.java:175)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
> >> java:175)
> >>         at
> >>
> >>
> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
> >>         at java.lang.Thread.run(Thread.java:619)
> >>
> >> "SpringOsgiExtenderThread-26" prio=6 tid=0x3256a400 nid=0x6b8 in
> >> Object.wait() [0x32a3f000..0x32a3fd94]
> >>    java.lang.Thread.State: BLOCKED (on object monitor)
> >>         at java.lang.Object.wait(Native Method)
> >>         at java.lang.Object.wait(Object.java:485)
> >>         at
> >> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
> >>         - locked<0x0fea0670>  (a [Ljava.lang.Object;)
> >>         at
> >> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
> >>         at
> >> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
> >>         at
> >> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
> >>         at
> >>
> >>
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
> >>         at
> >>
> >>
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
> >>         at
> >>
> >>
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
> >>         at
> >>
> >>
> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
> >> )
> >>         at
> >>
> >>
> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
> >> xt.java:69)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
> >> va:269)
> >>         - locked<0x2e5e0520>  (a java.lang.Object)
> >>         at
> >>
> >>
> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
> >> text.java:247)
> >>         at
> >>
> >>
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
> >> tionContextExecutor.java:222)
> >>         at
> >>
> >>
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
> >> ionContextExecutor.java:175)
> >>         at
> >>
> >>
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
> >> java:175)
> >>         at
> >>
> >>
> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
> >>         at java.lang.Thread.run(Thread.java:619)
> >>
> >> Regards,
> >>
> >> Charles Moulliard
> >> Senior Enterprise Architect
> >> Apache Camel Committer
> >>
> >> *****************************
> >> blog : http://cmoulliard.blogspot.com
> >>
> >>
> >> On Tue, Jun 30, 2009 at 1:42 PM, Guillaume Nodet<gn...@gmail.com>
>  wrote:
> >>
> >>
> >>>
> >>> Could you grab a thread dump of the vm when the shutdown is waiting ?
> >>> There must be a deadlock somewhere but without any thread dump, it's
> >>> kinda impossible to see what happen.
> >>>
> >>> On Tue, Jun 30, 2009 at 13:25, Charles Moulliard<cm...@gmail.com>
> >>> wrote:
> >>>
> >>>>
> >>>> Hi,
> >>>>
> >>>> When Camel tries to stop one of its context, the following message is
> >>>> displayed in the log:
> >>>>
> >>>> 13:10:59,213 | INFO  | Timer-2          | DefaultCamelContext
> >>>>
> >>>
> >>>  |
> >>>
> >>>>
> >>>> e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
> >>>> (CamelContext:InToOut) is stopping
> >>>> 13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext
> >>>>
> >>>
> >>>  |
> >>>
> >>>>
> >>>> e.camel.impl.DefaultCamelContext  975 | Apache Camel 2.0-SNAPSHOT
> >>>> (CamelContext:InToOut) stopped
> >>>> 13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext
> >>>>
> >>>
> >>>  |
> >>>
> >>>>
> >>>> e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
> >>>> (CamelContext:OutToIn) is stopping
> >>>> *13:11:09,181 | ERROR | Thread-47        | RunnableTimedExecution
> >>>> | oncurrent.RunnableTimedExecution  109 | Closing runnable for context
> >>>> OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-core,
> >>>> config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
> >>>> consider taking a snapshot and then shutdown the VM in case the thread
> >>>>
> >>>
> >>> still
> >>>
> >>>>
> >>>> hangs
> >>>> 13:11:19,181 | ERROR | Thread-47        | RunnableTimedExecution
> >>>>
> >>>
> >>>   |
> >>>
> >>>>
> >>>> oncurrent.RunnableTimedExecution  109 | Closing runnable for context
> >>>> OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-service,
> >>>> config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
> >>>> consider taking a snapshot and then shutdown the VM in case the thread
> >>>>
> >>>
> >>> still
> >>>
> >>>>
> >>>> hangs*
> >>>>
> >>>> It seems to be related to the following Spring DM error :
> >>>> http://jira.springframework.org/browse/OSGI-611 but I use Spring OSGI
> >>>>
> >>>
> >>> 1.2
> >>>
> >>>>
> >>>> and the error is still there
> >>>>
> >>>> Does anybody knows how to solve this ?
> >>>>
> >>>> Regards,
> >>>>
> >>>> Charles Moulliard
> >>>> Senior Enterprise Architect
> >>>> Apache Camel Committer
> >>>>
> >>>> *****************************
> >>>> blog : http://cmoulliard.blogspot.com
> >>>>
> >>>>
> >>>
> >>> --
> >>> Cheers,
> >>> Guillaume Nodet
> >>> ------------------------
> >>> Blog: http://gnodet.blogspot.com/
> >>> ------------------------
> >>> Open Source SOA
> >>> http://fusesource.com
> >>>
> >>>
> >>
> >>
> >
>
>
>
> --
> Cheers,
> Guillaume Nodet
> ------------------------
> Blog: http://gnodet.blogspot.com/
> ------------------------
> Open Source SOA
> http://fusesource.com
>

Re: Apache Karaf & Spring DM : consider taking a snapshot and then shutdown the VM in case the thread still hangs

Posted by "Richard S. Hall" <he...@ungoverned.org>.
On 7/1/09 11:05 AM, Guillaume Nodet wrote:
> I guess the extender use multiple threads to manage the spring-dm
> powered bundles.
> The question is which thread is holding the felix lock ? I can't find it ...
>    

As I mentioned, not all threads are shown. For example, the the 
PackageAdmin thread for refreshing is not shown, nor is the Start Level 
thread.

My guess is the PackageAdmin thread has the global lock, but it is not 
clear why it isn't making progress.

-> richard

> On Wed, Jul 1, 2009 at 16:48, Richard S. Hall<he...@ungoverned.org>  wrote:
>    
>> This is odd looking. All threads are blocked doing the exact same thing. Are
>> they trying to parallelize this?
>>
>> The trace doesn't demonstrate why they can't continue since it doesn't
>> include all of the threads.
>>
>> ->  richard
>>
>> On 7/1/09 10:22 AM, Charles Moulliard wrote:
>>      
>>> I have been able to reproduce the problem. This error is generated when I
>>> launch consecutively two refresh commands of the same bundle without
>>> waiting
>>> that the first refresh has finished. The bundle refreshed contains spring
>>> XML config files + services which are exposed as OSGI service
>>>
>>> Here is a trace of the thread dump :
>>>
>>> 2009-07-01 16:12:52
>>> Full thread dump Java HotSpot(TM) Server VM (11.2-b01 mixed mode):
>>>
>>> "SpringOsgiExtenderThread-31" prio=6 tid=0x31e42400 nid=0xa18 in
>>> Object.wait() [0x33d4f000..0x33d4fd14]
>>>     java.lang.Thread.State: WAITING (on object monitor)
>>>          at java.lang.Object.wait(Native Method)
>>>          at java.lang.Object.wait(Object.java:485)
>>>          at
>>> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>>>          - locked<0x0fea0670>    (a [Ljava.lang.Object;)
>>>          at
>>> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>>>          at
>>> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>>>          at
>>> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>>>          at
>>>
>>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>>>          at
>>>
>>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>>>          at
>>>
>>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>>>          at
>>>
>>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>>>          at
>>>
>>> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
>>> )
>>>          at
>>>
>>> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
>>> xt.java:69)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
>>> va:269)
>>>          - locked<0x2c998110>    (a java.lang.Object)
>>>          at
>>>
>>> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
>>> text.java:247)
>>>          at
>>>
>>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
>>> tionContextExecutor.java:222)
>>>          at
>>>
>>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
>>> ionContextExecutor.java:175)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
>>> java:175)
>>>          at
>>>
>>> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>>>          at java.lang.Thread.run(Thread.java:619)
>>>
>>> "SpringOsgiExtenderThread-30" prio=6 tid=0x32196c00 nid=0x988 in
>>> Object.wait() [0x3357f000..0x3357fd94]
>>>     java.lang.Thread.State: BLOCKED (on object monitor)
>>>          at java.lang.Object.wait(Native Method)
>>>          at java.lang.Object.wait(Object.java:485)
>>>          at
>>> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>>>          - locked<0x0fea0670>    (a [Ljava.lang.Object;)
>>>          at
>>> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>>>          at
>>> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>>>          at
>>> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>>>          at
>>>
>>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>>>          at
>>>
>>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>>>          at
>>>
>>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>>>          at
>>>
>>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>>>          at
>>>
>>> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
>>> )
>>>          at
>>>
>>> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
>>> xt.java:69)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
>>> va:269)
>>>          - locked<0x2c97e368>    (a java.lang.Object)
>>>          at
>>>
>>> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
>>> text.java:247)
>>>          at
>>>
>>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
>>> tionContextExecutor.java:222)
>>>          at
>>>
>>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
>>> ionContextExecutor.java:175)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
>>> java:175)
>>>          at
>>>
>>> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>>>          at java.lang.Thread.run(Thread.java:619)
>>>
>>> "RMI TCP Connection(7)-172.16.10.128" daemon prio=6 tid=0x3198c400
>>> nid=0xe98
>>> runnable [0x3294f000..0x3294fa14]
>>>     java.lang.Thread.State: RUNNABLE
>>>          at java.net.SocketInputStream.socketRead0(Native Method)
>>>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>>>          at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>          at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>>>          - locked<0x2c6ad740>    (a java.io.BufferedInputStream)
>>>          at java.io.FilterInputStream.read(FilterInputStream.java:66)
>>>          at
>>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
>>>          at
>>>
>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>>>          at
>>>
>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>>>          at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>          at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>          at java.lang.Thread.run(Thread.java:619)
>>>
>>> "InactivityMonitor Async Task:
>>> java.util.concurrent.ThreadPoolExecutor$Worker@57c459" daemon prio=6
>>> tid=0x324cd400 nid=0xbf0 waiting on condition [0x3
>>> 34df000..0x334dfb14]
>>>     java.lang.Thread.State: TIMED_WAITING (parking)
>>>          at sun.misc.Unsafe.park(Native Method)
>>>          - parking to wait for<0x10cba430>    (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>          at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>          at
>>>
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>          at
>>>
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>          at
>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>          at
>>>
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>          at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>          at java.lang.Thread.run(Thread.java:619)
>>>
>>> "SpringOsgiExtenderThread-29" prio=6 tid=0x324e6c00 nid=0x8e4 in
>>> Object.wait() [0x332af000..0x332afb94]
>>>     java.lang.Thread.State: BLOCKED (on object monitor)
>>>          at java.lang.Object.wait(Native Method)
>>>          at java.lang.Object.wait(Object.java:485)
>>>          at
>>> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>>>          - locked<0x0fea0670>    (a [Ljava.lang.Object;)
>>>          at
>>> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>>>          at
>>> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>>>          at
>>> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>>>          at
>>>
>>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>>>          at
>>>
>>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>>>          at
>>>
>>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>>>          at
>>>
>>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>>>          at
>>>
>>> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
>>> )
>>>          at
>>>
>>> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
>>> xt.java:69)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
>>> va:269)
>>>          - locked<0x2e5d6ba8>    (a java.lang.Object)
>>>          at
>>>
>>> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
>>> text.java:247)
>>>          at
>>>
>>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
>>> tionContextExecutor.java:222)
>>>          at
>>>
>>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
>>> ionContextExecutor.java:175)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
>>> java:175)
>>>          at
>>>
>>> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>>>          at java.lang.Thread.run(Thread.java:619)
>>>
>>> "SpringOsgiExtenderThread-28" prio=6 tid=0x3207ac00 nid=0xfe4 in
>>> Object.wait() [0x333ef000..0x333efc14]
>>>     java.lang.Thread.State: BLOCKED (on object monitor)
>>>          at java.lang.Object.wait(Native Method)
>>>          at java.lang.Object.wait(Object.java:485)
>>>          at
>>> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>>>          - locked<0x0fea0670>    (a [Ljava.lang.Object;)
>>>          at
>>> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>>>          at
>>> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>>>          at
>>> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>>>          at
>>>
>>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>>>          at
>>>
>>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>>>          at
>>>
>>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>>>          at
>>>
>>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>>>          at
>>>
>>> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
>>> )
>>>          at
>>>
>>> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
>>> xt.java:69)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
>>> va:269)
>>>          - locked<0x2e5d7e70>    (a java.lang.Object)
>>>          at
>>>
>>> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
>>> text.java:247)
>>>          at
>>>
>>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
>>> tionContextExecutor.java:222)
>>>          at
>>>
>>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
>>> ionContextExecutor.java:175)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
>>> java:175)
>>>          at
>>>
>>> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>>>          at java.lang.Thread.run(Thread.java:619)
>>>
>>> "SpringOsgiExtenderThread-27" prio=6 tid=0x312cc400 nid=0xa74 in
>>> Object.wait() [0x328af000..0x328afd14]
>>>     java.lang.Thread.State: BLOCKED (on object monitor)
>>>          at java.lang.Object.wait(Native Method)
>>>          at java.lang.Object.wait(Object.java:485)
>>>          at
>>> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>>>          - locked<0x0fea0670>    (a [Ljava.lang.Object;)
>>>          at
>>> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>>>          at
>>> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>>>          at
>>> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>>>          at
>>>
>>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>>>          at
>>>
>>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>>>          at
>>>
>>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>>>          at
>>>
>>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>>>          at
>>>
>>> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
>>> )
>>>          at
>>>
>>> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
>>> xt.java:69)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
>>> va:269)
>>>          - locked<0x2e5dbbf0>    (a java.lang.Object)
>>>          at
>>>
>>> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
>>> text.java:247)
>>>          at
>>>
>>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
>>> tionContextExecutor.java:222)
>>>          at
>>>
>>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
>>> ionContextExecutor.java:175)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
>>> java:175)
>>>          at
>>>
>>> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>>>          at java.lang.Thread.run(Thread.java:619)
>>>
>>> "SpringOsgiExtenderThread-26" prio=6 tid=0x3256a400 nid=0x6b8 in
>>> Object.wait() [0x32a3f000..0x32a3fd94]
>>>     java.lang.Thread.State: BLOCKED (on object monitor)
>>>          at java.lang.Object.wait(Native Method)
>>>          at java.lang.Object.wait(Object.java:485)
>>>          at
>>> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>>>          - locked<0x0fea0670>    (a [Ljava.lang.Object;)
>>>          at
>>> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>>>          at
>>> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>>>          at
>>> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>>>          at
>>>
>>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>>>          at
>>>
>>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>>>          at
>>>
>>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>>>          at
>>>
>>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>>>          at
>>>
>>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>>>          at
>>>
>>> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
>>> )
>>>          at
>>>
>>> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
>>> xt.java:69)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
>>> va:269)
>>>          - locked<0x2e5e0520>    (a java.lang.Object)
>>>          at
>>>
>>> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
>>> text.java:247)
>>>          at
>>>
>>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
>>> tionContextExecutor.java:222)
>>>          at
>>>
>>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
>>> ionContextExecutor.java:175)
>>>          at
>>>
>>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
>>> java:175)
>>>          at
>>>
>>> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>>>          at java.lang.Thread.run(Thread.java:619)
>>>
>>> Regards,
>>>
>>> Charles Moulliard
>>> Senior Enterprise Architect
>>> Apache Camel Committer
>>>
>>> *****************************
>>> blog : http://cmoulliard.blogspot.com
>>>
>>>
>>> On Tue, Jun 30, 2009 at 1:42 PM, Guillaume Nodet<gn...@gmail.com>    wrote:
>>>
>>>
>>>        
>>>> Could you grab a thread dump of the vm when the shutdown is waiting ?
>>>> There must be a deadlock somewhere but without any thread dump, it's
>>>> kinda impossible to see what happen.
>>>>
>>>> On Tue, Jun 30, 2009 at 13:25, Charles Moulliard<cm...@gmail.com>
>>>> wrote:
>>>>
>>>>          
>>>>> Hi,
>>>>>
>>>>> When Camel tries to stop one of its context, the following message is
>>>>> displayed in the log:
>>>>>
>>>>> 13:10:59,213 | INFO  | Timer-2          | DefaultCamelContext
>>>>>
>>>>>            
>>>>   |
>>>>
>>>>          
>>>>> e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
>>>>> (CamelContext:InToOut) is stopping
>>>>> 13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext
>>>>>
>>>>>            
>>>>   |
>>>>
>>>>          
>>>>> e.camel.impl.DefaultCamelContext  975 | Apache Camel 2.0-SNAPSHOT
>>>>> (CamelContext:InToOut) stopped
>>>>> 13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext
>>>>>
>>>>>            
>>>>   |
>>>>
>>>>          
>>>>> e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
>>>>> (CamelContext:OutToIn) is stopping
>>>>> *13:11:09,181 | ERROR | Thread-47        | RunnableTimedExecution
>>>>> | oncurrent.RunnableTimedExecution  109 | Closing runnable for context
>>>>> OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-core,
>>>>> config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
>>>>> consider taking a snapshot and then shutdown the VM in case the thread
>>>>>
>>>>>            
>>>> still
>>>>
>>>>          
>>>>> hangs
>>>>> 13:11:19,181 | ERROR | Thread-47        | RunnableTimedExecution
>>>>>
>>>>>            
>>>>    |
>>>>
>>>>          
>>>>> oncurrent.RunnableTimedExecution  109 | Closing runnable for context
>>>>> OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-service,
>>>>> config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
>>>>> consider taking a snapshot and then shutdown the VM in case the thread
>>>>>
>>>>>            
>>>> still
>>>>
>>>>          
>>>>> hangs*
>>>>>
>>>>> It seems to be related to the following Spring DM error :
>>>>> http://jira.springframework.org/browse/OSGI-611 but I use Spring OSGI
>>>>>
>>>>>            
>>>> 1.2
>>>>
>>>>          
>>>>> and the error is still there
>>>>>
>>>>> Does anybody knows how to solve this ?
>>>>>
>>>>> Regards,
>>>>>
>>>>> Charles Moulliard
>>>>> Senior Enterprise Architect
>>>>> Apache Camel Committer
>>>>>
>>>>> *****************************
>>>>> blog : http://cmoulliard.blogspot.com
>>>>>
>>>>>
>>>>>            
>>>> --
>>>> Cheers,
>>>> Guillaume Nodet
>>>> ------------------------
>>>> Blog: http://gnodet.blogspot.com/
>>>> ------------------------
>>>> Open Source SOA
>>>> http://fusesource.com
>>>>
>>>>
>>>>          
>>>        
>
>
>
>    

Re: Apache Karaf & Spring DM : consider taking a snapshot and then shutdown the VM in case the thread still hangs

Posted by Guillaume Nodet <gn...@gmail.com>.
I guess the extender use multiple threads to manage the spring-dm
powered bundles.
The question is which thread is holding the felix lock ? I can't find it ...

On Wed, Jul 1, 2009 at 16:48, Richard S. Hall<he...@ungoverned.org> wrote:
> This is odd looking. All threads are blocked doing the exact same thing. Are
> they trying to parallelize this?
>
> The trace doesn't demonstrate why they can't continue since it doesn't
> include all of the threads.
>
> -> richard
>
> On 7/1/09 10:22 AM, Charles Moulliard wrote:
>>
>> I have been able to reproduce the problem. This error is generated when I
>> launch consecutively two refresh commands of the same bundle without
>> waiting
>> that the first refresh has finished. The bundle refreshed contains spring
>> XML config files + services which are exposed as OSGI service
>>
>> Here is a trace of the thread dump :
>>
>> 2009-07-01 16:12:52
>> Full thread dump Java HotSpot(TM) Server VM (11.2-b01 mixed mode):
>>
>> "SpringOsgiExtenderThread-31" prio=6 tid=0x31e42400 nid=0xa18 in
>> Object.wait() [0x33d4f000..0x33d4fd14]
>>    java.lang.Thread.State: WAITING (on object monitor)
>>         at java.lang.Object.wait(Native Method)
>>         at java.lang.Object.wait(Object.java:485)
>>         at
>> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>>         - locked<0x0fea0670>  (a [Ljava.lang.Object;)
>>         at
>> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>>         at
>> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>>         at
>> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>>         at
>>
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>>         at
>>
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>>         at
>>
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>>         at
>>
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>>         at
>>
>> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
>> )
>>         at
>>
>> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
>> xt.java:69)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
>> va:269)
>>         - locked<0x2c998110>  (a java.lang.Object)
>>         at
>>
>> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
>> text.java:247)
>>         at
>>
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
>> tionContextExecutor.java:222)
>>         at
>>
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
>> ionContextExecutor.java:175)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
>> java:175)
>>         at
>>
>> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>>         at java.lang.Thread.run(Thread.java:619)
>>
>> "SpringOsgiExtenderThread-30" prio=6 tid=0x32196c00 nid=0x988 in
>> Object.wait() [0x3357f000..0x3357fd94]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>         at java.lang.Object.wait(Native Method)
>>         at java.lang.Object.wait(Object.java:485)
>>         at
>> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>>         - locked<0x0fea0670>  (a [Ljava.lang.Object;)
>>         at
>> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>>         at
>> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>>         at
>> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>>         at
>>
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>>         at
>>
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>>         at
>>
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>>         at
>>
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>>         at
>>
>> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
>> )
>>         at
>>
>> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
>> xt.java:69)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
>> va:269)
>>         - locked<0x2c97e368>  (a java.lang.Object)
>>         at
>>
>> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
>> text.java:247)
>>         at
>>
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
>> tionContextExecutor.java:222)
>>         at
>>
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
>> ionContextExecutor.java:175)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
>> java:175)
>>         at
>>
>> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>>         at java.lang.Thread.run(Thread.java:619)
>>
>> "RMI TCP Connection(7)-172.16.10.128" daemon prio=6 tid=0x3198c400
>> nid=0xe98
>> runnable [0x3294f000..0x3294fa14]
>>    java.lang.Thread.State: RUNNABLE
>>         at java.net.SocketInputStream.socketRead0(Native Method)
>>         at java.net.SocketInputStream.read(SocketInputStream.java:129)
>>         at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>         at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>>         - locked<0x2c6ad740>  (a java.io.BufferedInputStream)
>>         at java.io.FilterInputStream.read(FilterInputStream.java:66)
>>         at
>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
>>         at
>>
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>>         at
>>
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>>         at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>         at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>         at java.lang.Thread.run(Thread.java:619)
>>
>> "InactivityMonitor Async Task:
>> java.util.concurrent.ThreadPoolExecutor$Worker@57c459" daemon prio=6
>> tid=0x324cd400 nid=0xbf0 waiting on condition [0x3
>> 34df000..0x334dfb14]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for<0x10cba430>  (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>         at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>         at
>>
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>         at
>>
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>         at
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>         at
>>
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>         at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>         at java.lang.Thread.run(Thread.java:619)
>>
>> "SpringOsgiExtenderThread-29" prio=6 tid=0x324e6c00 nid=0x8e4 in
>> Object.wait() [0x332af000..0x332afb94]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>         at java.lang.Object.wait(Native Method)
>>         at java.lang.Object.wait(Object.java:485)
>>         at
>> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>>         - locked<0x0fea0670>  (a [Ljava.lang.Object;)
>>         at
>> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>>         at
>> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>>         at
>> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>>         at
>>
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>>         at
>>
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>>         at
>>
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>>         at
>>
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>>         at
>>
>> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
>> )
>>         at
>>
>> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
>> xt.java:69)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
>> va:269)
>>         - locked<0x2e5d6ba8>  (a java.lang.Object)
>>         at
>>
>> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
>> text.java:247)
>>         at
>>
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
>> tionContextExecutor.java:222)
>>         at
>>
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
>> ionContextExecutor.java:175)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
>> java:175)
>>         at
>>
>> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>>         at java.lang.Thread.run(Thread.java:619)
>>
>> "SpringOsgiExtenderThread-28" prio=6 tid=0x3207ac00 nid=0xfe4 in
>> Object.wait() [0x333ef000..0x333efc14]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>         at java.lang.Object.wait(Native Method)
>>         at java.lang.Object.wait(Object.java:485)
>>         at
>> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>>         - locked<0x0fea0670>  (a [Ljava.lang.Object;)
>>         at
>> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>>         at
>> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>>         at
>> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>>         at
>>
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>>         at
>>
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>>         at
>>
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>>         at
>>
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>>         at
>>
>> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
>> )
>>         at
>>
>> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
>> xt.java:69)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
>> va:269)
>>         - locked<0x2e5d7e70>  (a java.lang.Object)
>>         at
>>
>> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
>> text.java:247)
>>         at
>>
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
>> tionContextExecutor.java:222)
>>         at
>>
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
>> ionContextExecutor.java:175)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
>> java:175)
>>         at
>>
>> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>>         at java.lang.Thread.run(Thread.java:619)
>>
>> "SpringOsgiExtenderThread-27" prio=6 tid=0x312cc400 nid=0xa74 in
>> Object.wait() [0x328af000..0x328afd14]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>         at java.lang.Object.wait(Native Method)
>>         at java.lang.Object.wait(Object.java:485)
>>         at
>> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>>         - locked<0x0fea0670>  (a [Ljava.lang.Object;)
>>         at
>> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>>         at
>> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>>         at
>> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>>         at
>>
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>>         at
>>
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>>         at
>>
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>>         at
>>
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>>         at
>>
>> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
>> )
>>         at
>>
>> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
>> xt.java:69)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
>> va:269)
>>         - locked<0x2e5dbbf0>  (a java.lang.Object)
>>         at
>>
>> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
>> text.java:247)
>>         at
>>
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
>> tionContextExecutor.java:222)
>>         at
>>
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
>> ionContextExecutor.java:175)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
>> java:175)
>>         at
>>
>> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>>         at java.lang.Thread.run(Thread.java:619)
>>
>> "SpringOsgiExtenderThread-26" prio=6 tid=0x3256a400 nid=0x6b8 in
>> Object.wait() [0x32a3f000..0x32a3fd94]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>         at java.lang.Object.wait(Native Method)
>>         at java.lang.Object.wait(Object.java:485)
>>         at
>> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>>         - locked<0x0fea0670>  (a [Ljava.lang.Object;)
>>         at
>> org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>>         at
>> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>>         at
>> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>>         at
>>
>> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>>         at
>>
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>>         at
>>
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>>         at
>>
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>>         at
>>
>> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>>         at
>>
>> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
>> )
>>         at
>>
>> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
>> xt.java:69)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
>> va:269)
>>         - locked<0x2e5e0520>  (a java.lang.Object)
>>         at
>>
>> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
>> text.java:247)
>>         at
>>
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
>> tionContextExecutor.java:222)
>>         at
>>
>> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
>> ionContextExecutor.java:175)
>>         at
>>
>> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
>> java:175)
>>         at
>>
>> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>>         at java.lang.Thread.run(Thread.java:619)
>>
>> Regards,
>>
>> Charles Moulliard
>> Senior Enterprise Architect
>> Apache Camel Committer
>>
>> *****************************
>> blog : http://cmoulliard.blogspot.com
>>
>>
>> On Tue, Jun 30, 2009 at 1:42 PM, Guillaume Nodet<gn...@gmail.com>  wrote:
>>
>>
>>>
>>> Could you grab a thread dump of the vm when the shutdown is waiting ?
>>> There must be a deadlock somewhere but without any thread dump, it's
>>> kinda impossible to see what happen.
>>>
>>> On Tue, Jun 30, 2009 at 13:25, Charles Moulliard<cm...@gmail.com>
>>> wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> When Camel tries to stop one of its context, the following message is
>>>> displayed in the log:
>>>>
>>>> 13:10:59,213 | INFO  | Timer-2          | DefaultCamelContext
>>>>
>>>
>>>  |
>>>
>>>>
>>>> e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
>>>> (CamelContext:InToOut) is stopping
>>>> 13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext
>>>>
>>>
>>>  |
>>>
>>>>
>>>> e.camel.impl.DefaultCamelContext  975 | Apache Camel 2.0-SNAPSHOT
>>>> (CamelContext:InToOut) stopped
>>>> 13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext
>>>>
>>>
>>>  |
>>>
>>>>
>>>> e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
>>>> (CamelContext:OutToIn) is stopping
>>>> *13:11:09,181 | ERROR | Thread-47        | RunnableTimedExecution
>>>> | oncurrent.RunnableTimedExecution  109 | Closing runnable for context
>>>> OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-core,
>>>> config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
>>>> consider taking a snapshot and then shutdown the VM in case the thread
>>>>
>>>
>>> still
>>>
>>>>
>>>> hangs
>>>> 13:11:19,181 | ERROR | Thread-47        | RunnableTimedExecution
>>>>
>>>
>>>   |
>>>
>>>>
>>>> oncurrent.RunnableTimedExecution  109 | Closing runnable for context
>>>> OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-service,
>>>> config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
>>>> consider taking a snapshot and then shutdown the VM in case the thread
>>>>
>>>
>>> still
>>>
>>>>
>>>> hangs*
>>>>
>>>> It seems to be related to the following Spring DM error :
>>>> http://jira.springframework.org/browse/OSGI-611 but I use Spring OSGI
>>>>
>>>
>>> 1.2
>>>
>>>>
>>>> and the error is still there
>>>>
>>>> Does anybody knows how to solve this ?
>>>>
>>>> Regards,
>>>>
>>>> Charles Moulliard
>>>> Senior Enterprise Architect
>>>> Apache Camel Committer
>>>>
>>>> *****************************
>>>> blog : http://cmoulliard.blogspot.com
>>>>
>>>>
>>>
>>> --
>>> Cheers,
>>> Guillaume Nodet
>>> ------------------------
>>> Blog: http://gnodet.blogspot.com/
>>> ------------------------
>>> Open Source SOA
>>> http://fusesource.com
>>>
>>>
>>
>>
>



-- 
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
Open Source SOA
http://fusesource.com

Re: Apache Karaf & Spring DM : consider taking a snapshot and then shutdown the VM in case the thread still hangs

Posted by "Richard S. Hall" <he...@ungoverned.org>.
This is odd looking. All threads are blocked doing the exact same thing. 
Are they trying to parallelize this?

The trace doesn't demonstrate why they can't continue since it doesn't 
include all of the threads.

-> richard

On 7/1/09 10:22 AM, Charles Moulliard wrote:
> I have been able to reproduce the problem. This error is generated when I
> launch consecutively two refresh commands of the same bundle without waiting
> that the first refresh has finished. The bundle refreshed contains spring
> XML config files + services which are exposed as OSGI service
>
> Here is a trace of the thread dump :
>
> 2009-07-01 16:12:52
> Full thread dump Java HotSpot(TM) Server VM (11.2-b01 mixed mode):
>
> "SpringOsgiExtenderThread-31" prio=6 tid=0x31e42400 nid=0xa18 in
> Object.wait() [0x33d4f000..0x33d4fd14]
>     java.lang.Thread.State: WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.lang.Object.wait(Object.java:485)
>          at
> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>          - locked<0x0fea0670>  (a [Ljava.lang.Object;)
>          at org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>          at
> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>          at
> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>          at
> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>          at
> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>          at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>          at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>          at
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>          at
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>          at
> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
> )
>          at
> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
> xt.java:69)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
> va:269)
>          - locked<0x2c998110>  (a java.lang.Object)
>          at
> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
> text.java:247)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
> tionContextExecutor.java:222)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
> ionContextExecutor.java:175)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
> java:175)
>          at
> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>          at java.lang.Thread.run(Thread.java:619)
>
> "SpringOsgiExtenderThread-30" prio=6 tid=0x32196c00 nid=0x988 in
> Object.wait() [0x3357f000..0x3357fd94]
>     java.lang.Thread.State: BLOCKED (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.lang.Object.wait(Object.java:485)
>          at
> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>          - locked<0x0fea0670>  (a [Ljava.lang.Object;)
>          at org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>          at
> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>          at
> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>          at
> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>          at
> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>          at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>          at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>          at
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>          at
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>          at
> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
> )
>          at
> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
> xt.java:69)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
> va:269)
>          - locked<0x2c97e368>  (a java.lang.Object)
>          at
> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
> text.java:247)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
> tionContextExecutor.java:222)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
> ionContextExecutor.java:175)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
> java:175)
>          at
> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>          at java.lang.Thread.run(Thread.java:619)
>
> "RMI TCP Connection(7)-172.16.10.128" daemon prio=6 tid=0x3198c400 nid=0xe98
> runnable [0x3294f000..0x3294fa14]
>     java.lang.Thread.State: RUNNABLE
>          at java.net.SocketInputStream.socketRead0(Native Method)
>          at java.net.SocketInputStream.read(SocketInputStream.java:129)
>          at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>          at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>          - locked<0x2c6ad740>  (a java.io.BufferedInputStream)
>          at java.io.FilterInputStream.read(FilterInputStream.java:66)
>          at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
>          at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>          at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>          at java.lang.Thread.run(Thread.java:619)
>
> "InactivityMonitor Async Task:
> java.util.concurrent.ThreadPoolExecutor$Worker@57c459" daemon prio=6
> tid=0x324cd400 nid=0xbf0 waiting on condition [0x3
> 34df000..0x334dfb14]
>     java.lang.Thread.State: TIMED_WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for<0x10cba430>  (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>          at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>          at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>          at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>          at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>          at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>          at java.lang.Thread.run(Thread.java:619)
>
> "SpringOsgiExtenderThread-29" prio=6 tid=0x324e6c00 nid=0x8e4 in
> Object.wait() [0x332af000..0x332afb94]
>     java.lang.Thread.State: BLOCKED (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.lang.Object.wait(Object.java:485)
>          at
> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>          - locked<0x0fea0670>  (a [Ljava.lang.Object;)
>          at org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>          at
> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>          at
> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>          at
> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>          at
> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>          at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>          at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>          at
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>          at
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>          at
> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
> )
>          at
> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
> xt.java:69)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
> va:269)
>          - locked<0x2e5d6ba8>  (a java.lang.Object)
>          at
> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
> text.java:247)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
> tionContextExecutor.java:222)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
> ionContextExecutor.java:175)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
> java:175)
>          at
> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>          at java.lang.Thread.run(Thread.java:619)
>
> "SpringOsgiExtenderThread-28" prio=6 tid=0x3207ac00 nid=0xfe4 in
> Object.wait() [0x333ef000..0x333efc14]
>     java.lang.Thread.State: BLOCKED (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.lang.Object.wait(Object.java:485)
>          at
> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>          - locked<0x0fea0670>  (a [Ljava.lang.Object;)
>          at org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>          at
> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>          at
> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>          at
> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>          at
> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>          at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>          at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>          at
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>          at
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>          at
> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
> )
>          at
> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
> xt.java:69)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
> va:269)
>          - locked<0x2e5d7e70>  (a java.lang.Object)
>          at
> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
> text.java:247)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
> tionContextExecutor.java:222)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
> ionContextExecutor.java:175)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
> java:175)
>          at
> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>          at java.lang.Thread.run(Thread.java:619)
>
> "SpringOsgiExtenderThread-27" prio=6 tid=0x312cc400 nid=0xa74 in
> Object.wait() [0x328af000..0x328afd14]
>     java.lang.Thread.State: BLOCKED (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.lang.Object.wait(Object.java:485)
>          at
> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>          - locked<0x0fea0670>  (a [Ljava.lang.Object;)
>          at org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>          at
> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>          at
> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>          at
> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>          at
> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>          at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>          at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>          at
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>          at
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>          at
> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
> )
>          at
> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
> xt.java:69)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
> va:269)
>          - locked<0x2e5dbbf0>  (a java.lang.Object)
>          at
> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
> text.java:247)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
> tionContextExecutor.java:222)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
> ionContextExecutor.java:175)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
> java:175)
>          at
> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>          at java.lang.Thread.run(Thread.java:619)
>
> "SpringOsgiExtenderThread-26" prio=6 tid=0x3256a400 nid=0x6b8 in
> Object.wait() [0x32a3f000..0x32a3fd94]
>     java.lang.Thread.State: BLOCKED (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.lang.Object.wait(Object.java:485)
>          at
> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
>          - locked<0x0fea0670>  (a [Ljava.lang.Object;)
>          at org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
>          at
> org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
>          at
> org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
>          at
> org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
>          at
> org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
>          at
> org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
>          at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
>          at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
>          at
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
>          at
> org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
>          at
> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
> )
>          at
> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
> xt.java:69)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
> va:269)
>          - locked<0x2e5e0520>  (a java.lang.Object)
>          at
> org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
> text.java:247)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
> tionContextExecutor.java:222)
>          at
> org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
> ionContextExecutor.java:175)
>          at
> org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
> java:175)
>          at
> org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
>          at java.lang.Thread.run(Thread.java:619)
>
> Regards,
>
> Charles Moulliard
> Senior Enterprise Architect
> Apache Camel Committer
>
> *****************************
> blog : http://cmoulliard.blogspot.com
>
>
> On Tue, Jun 30, 2009 at 1:42 PM, Guillaume Nodet<gn...@gmail.com>  wrote:
>
>    
>> Could you grab a thread dump of the vm when the shutdown is waiting ?
>> There must be a deadlock somewhere but without any thread dump, it's
>> kinda impossible to see what happen.
>>
>> On Tue, Jun 30, 2009 at 13:25, Charles Moulliard<cm...@gmail.com>
>> wrote:
>>      
>>> Hi,
>>>
>>> When Camel tries to stop one of its context, the following message is
>>> displayed in the log:
>>>
>>> 13:10:59,213 | INFO  | Timer-2          | DefaultCamelContext
>>>        
>>   |
>>      
>>> e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
>>> (CamelContext:InToOut) is stopping
>>> 13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext
>>>        
>>   |
>>      
>>> e.camel.impl.DefaultCamelContext  975 | Apache Camel 2.0-SNAPSHOT
>>> (CamelContext:InToOut) stopped
>>> 13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext
>>>        
>>   |
>>      
>>> e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
>>> (CamelContext:OutToIn) is stopping
>>> *13:11:09,181 | ERROR | Thread-47        | RunnableTimedExecution
>>> | oncurrent.RunnableTimedExecution  109 | Closing runnable for context
>>> OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-core,
>>> config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
>>> consider taking a snapshot and then shutdown the VM in case the thread
>>>        
>> still
>>      
>>> hangs
>>> 13:11:19,181 | ERROR | Thread-47        | RunnableTimedExecution
>>>        
>>    |
>>      
>>> oncurrent.RunnableTimedExecution  109 | Closing runnable for context
>>> OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-service,
>>> config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
>>> consider taking a snapshot and then shutdown the VM in case the thread
>>>        
>> still
>>      
>>> hangs*
>>>
>>> It seems to be related to the following Spring DM error :
>>> http://jira.springframework.org/browse/OSGI-611 but I use Spring OSGI
>>>        
>> 1.2
>>      
>>> and the error is still there
>>>
>>> Does anybody knows how to solve this ?
>>>
>>> Regards,
>>>
>>> Charles Moulliard
>>> Senior Enterprise Architect
>>> Apache Camel Committer
>>>
>>> *****************************
>>> blog : http://cmoulliard.blogspot.com
>>>
>>>        
>>
>> --
>> Cheers,
>> Guillaume Nodet
>> ------------------------
>> Blog: http://gnodet.blogspot.com/
>> ------------------------
>> Open Source SOA
>> http://fusesource.com
>>
>>      
>
>    

Re: Apache Karaf & Spring DM : consider taking a snapshot and then shutdown the VM in case the thread still hangs

Posted by Charles Moulliard <cm...@gmail.com>.
I have been able to reproduce the problem. This error is generated when I
launch consecutively two refresh commands of the same bundle without waiting
that the first refresh has finished. The bundle refreshed contains spring
XML config files + services which are exposed as OSGI service

Here is a trace of the thread dump :

2009-07-01 16:12:52
Full thread dump Java HotSpot(TM) Server VM (11.2-b01 mixed mode):

"SpringOsgiExtenderThread-31" prio=6 tid=0x31e42400 nid=0xa18 in
Object.wait() [0x33d4f000..0x33d4fd14]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at
org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
        - locked <0x0fea0670> (a [Ljava.lang.Object;)
        at org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
        at
org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
        at
org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
        at
org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
        at
org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
        at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
        at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
        at
org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
        at
org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
        at
org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
)
        at
org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
xt.java:69)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
va:269)
        - locked <0x2c998110> (a java.lang.Object)
        at
org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
text.java:247)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
tionContextExecutor.java:222)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
ionContextExecutor.java:175)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
java:175)
        at
org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
        at java.lang.Thread.run(Thread.java:619)

"SpringOsgiExtenderThread-30" prio=6 tid=0x32196c00 nid=0x988 in
Object.wait() [0x3357f000..0x3357fd94]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at
org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
        - locked <0x0fea0670> (a [Ljava.lang.Object;)
        at org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
        at
org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
        at
org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
        at
org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
        at
org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
        at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
        at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
        at
org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
        at
org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
        at
org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
)
        at
org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
xt.java:69)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
va:269)
        - locked <0x2c97e368> (a java.lang.Object)
        at
org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
text.java:247)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
tionContextExecutor.java:222)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
ionContextExecutor.java:175)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
java:175)
        at
org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
        at java.lang.Thread.run(Thread.java:619)

"RMI TCP Connection(7)-172.16.10.128" daemon prio=6 tid=0x3198c400 nid=0xe98
runnable [0x3294f000..0x3294fa14]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        - locked <0x2c6ad740> (a java.io.BufferedInputStream)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
        at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
        at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

"InactivityMonitor Async Task:
java.util.concurrent.ThreadPoolExecutor$Worker@57c459" daemon prio=6
tid=0x324cd400 nid=0xbf0 waiting on condition [0x3
34df000..0x334dfb14]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x10cba430> (a
java.util.concurrent.SynchronousQueue$TransferStack)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
        at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
        at
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"SpringOsgiExtenderThread-29" prio=6 tid=0x324e6c00 nid=0x8e4 in
Object.wait() [0x332af000..0x332afb94]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at
org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
        - locked <0x0fea0670> (a [Ljava.lang.Object;)
        at org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
        at
org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
        at
org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
        at
org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
        at
org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
        at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
        at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
        at
org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
        at
org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
        at
org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
)
        at
org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
xt.java:69)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
va:269)
        - locked <0x2e5d6ba8> (a java.lang.Object)
        at
org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
text.java:247)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
tionContextExecutor.java:222)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
ionContextExecutor.java:175)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
java:175)
        at
org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
        at java.lang.Thread.run(Thread.java:619)

"SpringOsgiExtenderThread-28" prio=6 tid=0x3207ac00 nid=0xfe4 in
Object.wait() [0x333ef000..0x333efc14]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at
org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
        - locked <0x0fea0670> (a [Ljava.lang.Object;)
        at org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
        at
org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
        at
org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
        at
org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
        at
org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
        at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
        at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
        at
org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
        at
org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
        at
org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
)
        at
org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
xt.java:69)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
va:269)
        - locked <0x2e5d7e70> (a java.lang.Object)
        at
org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
text.java:247)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
tionContextExecutor.java:222)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
ionContextExecutor.java:175)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
java:175)
        at
org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
        at java.lang.Thread.run(Thread.java:619)

"SpringOsgiExtenderThread-27" prio=6 tid=0x312cc400 nid=0xa74 in
Object.wait() [0x328af000..0x328afd14]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at
org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
        - locked <0x0fea0670> (a [Ljava.lang.Object;)
        at org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
        at
org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
        at
org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
        at
org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
        at
org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
        at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
        at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
        at
org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
        at
org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
        at
org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
)
        at
org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
xt.java:69)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
va:269)
        - locked <0x2e5dbbf0> (a java.lang.Object)
        at
org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
text.java:247)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
tionContextExecutor.java:222)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
ionContextExecutor.java:175)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
java:175)
        at
org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
        at java.lang.Thread.run(Thread.java:619)

"SpringOsgiExtenderThread-26" prio=6 tid=0x3256a400 nid=0x6b8 in
Object.wait() [0x32a3f000..0x32a3fd94]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at
org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4343)
        - locked <0x0fea0670> (a [Ljava.lang.Object;)
        at org.apache.felix.framework.Felix.resolveBundles(Felix.java:3012)
        at
org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1268)
        at
org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:225)
        at
org.springframework.osgi.io.OsgiBundleResource.getAllUrlsFromBundleSpace(OsgiBundleResource.java:485)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:182)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findPathMatchingResources(OsgiBundleResourcePatternResolver.java:562)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.findResources(OsgiBundleResourcePatternResolver.java:167)
        at
org.springframework.osgi.io.OsgiBundleResourcePatternResolver.getResources(OsgiBundleResourcePatternResolver.java:197)
        at
org.springframework.osgi.context.support.AbstractOsgiBundleApplicationContext.getResources(AbstractOsgiBundleApplicationContext.java:425)
        at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:177)
        at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:149)
        at
org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:176)
        at
org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:142)
        at
org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:123
)
        at
org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:422)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationConte
xt.java:69)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.ja
va:269)
        - locked <0x2e5e0520> (a java.lang.Object)
        at
org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationCon
text.java:247)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplica
tionContextExecutor.java:222)
        at
org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicat
ionContextExecutor.java:175)
        at
org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.
java:175)
        at
org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:718)
        at java.lang.Thread.run(Thread.java:619)

Regards,

Charles Moulliard
Senior Enterprise Architect
Apache Camel Committer

*****************************
blog : http://cmoulliard.blogspot.com


On Tue, Jun 30, 2009 at 1:42 PM, Guillaume Nodet <gn...@gmail.com> wrote:

> Could you grab a thread dump of the vm when the shutdown is waiting ?
> There must be a deadlock somewhere but without any thread dump, it's
> kinda impossible to see what happen.
>
> On Tue, Jun 30, 2009 at 13:25, Charles Moulliard<cm...@gmail.com>
> wrote:
> > Hi,
> >
> > When Camel tries to stop one of its context, the following message is
> > displayed in the log:
> >
> > 13:10:59,213 | INFO  | Timer-2          | DefaultCamelContext
>  |
> > e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
> > (CamelContext:InToOut) is stopping
> > 13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext
>  |
> > e.camel.impl.DefaultCamelContext  975 | Apache Camel 2.0-SNAPSHOT
> > (CamelContext:InToOut) stopped
> > 13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext
>  |
> > e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
> > (CamelContext:OutToIn) is stopping
> > *13:11:09,181 | ERROR | Thread-47        | RunnableTimedExecution
> > | oncurrent.RunnableTimedExecution  109 | Closing runnable for context
> > OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-core,
> > config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
> > consider taking a snapshot and then shutdown the VM in case the thread
> still
> > hangs
> > 13:11:19,181 | ERROR | Thread-47        | RunnableTimedExecution
>   |
> > oncurrent.RunnableTimedExecution  109 | Closing runnable for context
> > OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-service,
> > config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
> > consider taking a snapshot and then shutdown the VM in case the thread
> still
> > hangs*
> >
> > It seems to be related to the following Spring DM error :
> > http://jira.springframework.org/browse/OSGI-611 but I use Spring OSGI
> 1.2
> > and the error is still there
> >
> > Does anybody knows how to solve this ?
> >
> > Regards,
> >
> > Charles Moulliard
> > Senior Enterprise Architect
> > Apache Camel Committer
> >
> > *****************************
> > blog : http://cmoulliard.blogspot.com
> >
>
>
>
> --
> Cheers,
> Guillaume Nodet
> ------------------------
> Blog: http://gnodet.blogspot.com/
> ------------------------
> Open Source SOA
> http://fusesource.com
>

Re: Apache Karaf & Spring DM : consider taking a snapshot and then shutdown the VM in case the thread still hangs

Posted by Guillaume Nodet <gn...@gmail.com>.
Could you grab a thread dump of the vm when the shutdown is waiting ?
There must be a deadlock somewhere but without any thread dump, it's
kinda impossible to see what happen.

On Tue, Jun 30, 2009 at 13:25, Charles Moulliard<cm...@gmail.com> wrote:
> Hi,
>
> When Camel tries to stop one of its context, the following message is
> displayed in the log:
>
> 13:10:59,213 | INFO  | Timer-2          | DefaultCamelContext              |
> e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
> (CamelContext:InToOut) is stopping
> 13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext              |
> e.camel.impl.DefaultCamelContext  975 | Apache Camel 2.0-SNAPSHOT
> (CamelContext:InToOut) stopped
> 13:11:00,244 | INFO  | Timer-2          | DefaultCamelContext              |
> e.camel.impl.DefaultCamelContext  965 | Apache Camel 2.0-SNAPSHOT
> (CamelContext:OutToIn) is stopping
> *13:11:09,181 | ERROR | Thread-47        | RunnableTimedExecution
> | oncurrent.RunnableTimedExecution  109 | Closing runnable for context
> OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-core,
> config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
> consider taking a snapshot and then shutdown the VM in case the thread still
> hangs
> 13:11:19,181 | ERROR | Thread-47        | RunnableTimedExecution           |
> oncurrent.RunnableTimedExecution  109 | Closing runnable for context
> OsgiBundleXmlApplicationContext(bundle=com.xpectis.x3s.x3s-service,
> config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms;
> consider taking a snapshot and then shutdown the VM in case the thread still
> hangs*
>
> It seems to be related to the following Spring DM error :
> http://jira.springframework.org/browse/OSGI-611 but I use Spring OSGI 1.2
> and the error is still there
>
> Does anybody knows how to solve this ?
>
> Regards,
>
> Charles Moulliard
> Senior Enterprise Architect
> Apache Camel Committer
>
> *****************************
> blog : http://cmoulliard.blogspot.com
>



-- 
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
Open Source SOA
http://fusesource.com