You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@felix.apache.org by Jad Naous <ja...@nerati.com> on 2013/01/06 18:32:46 UTC
iPOJO should not grab locks during synchronous callbacks
This is a fork of the earlier thread from before on deadlocks during
refresh.
iPOJO grabs the InstanceCreator lock (using synchronized method calls) when
stopping a bundle, while another iPOJO thread is attempting to register
services, which requires resolving a class, which requires the framework
lock already grabbed by the other thread, creating a deadlock.
In general, iPOJO should not be grabbing locks during synchronous callbacks
because the framework might already have some locks grabbed. Resolving this
through the framework seems impossible because the framework spec afaik
requires preventing anything else from happening in the framework while
changing bundle states.
The resolution seems to be that applications should not grab locks in
synchronous callbacks that could also be grabbed by other application
threads.
I've been stuck on this for weeks, and it would be really painful to
extract ipojo now and replace it with something else. Any help in resolving
this is really appreciated. I was thinking of going to study the ipojo code
to take out the locks in InstanceCreator, but that seems like a major
undertaking for me, and we have a release deadline soon...
Here are the stack traces of the deadlocked threads:
Name: Thread-2
State: WAITING on [Ljava.lang.Object;@4a018e1b
Total blocked: 38,871,649 Total waited: 38,871,650
Stack trace:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:5033)
org.apache.felix.framework.StatefulResolver.resolve(StatefulResolver.java:451)
org.apache.felix.framework.BundleWiringImpl.searchDynamicImports(BundleWiringImpl.java:1578)
org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1478)
org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:75)
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1882)
java.lang.ClassLoader.loadClass(ClassLoader.java:247)
org.apache.felix.framework.BundleWiringImpl.getClassByDelegation(BundleWiringImpl.java:1356)
org.apache.felix.framework.ServiceRegistrationImpl$ServiceReferenceImpl.isAssignableTo(ServiceRegistrationImpl.java:548)
org.apache.felix.framework.util.Util.isServiceAssignable(Util.java:280)
org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:916)
org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:793)
org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:543)
org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4346)
org.apache.felix.framework.Felix.registerService(Felix.java:3356)
org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346)
org.apache.felix.ipojo.IPojoFactory.start(IPojoFactory.java:613)
- locked org.apache.felix.ipojo.ComponentFactory@468034b6
org.apache.felix.ipojo.Extender.createAbstractFactory(Extender.java:520)
org.apache.felix.ipojo.Extender.parse(Extender.java:301)
org.apache.felix.ipojo.Extender.startManagementFor(Extender.java:237)
org.apache.felix.ipojo.Extender.access$600(Extender.java:52)
org.apache.felix.ipojo.Extender$CreatorThread.run(Extender.java:769)
java.lang.Thread.run(Thread.java:662)
Name: FelixFrameworkWiring
State: BLOCKED on org.apache.felix.ipojo.ComponentFactory@468034b6 owned
by: Thread-2
Total blocked: 7 Total waited: 1
Stack trace:
org.apache.felix.ipojo
.IPojoFactory.removeFactoryStateListener(IPojoFactory.java:511)
org.apache.felix.ipojo
.InstanceCreator.removeFactory(InstanceCreator.java:199)
org.apache.felix.ipojo.Extender.closeManagementFor(Extender.java:180)
org.apache.felix.ipojo.Extender.bundleChanged(Extender.java:153)
org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:868)
org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:789)
org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:514)
org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4330)
org.apache.felix.framework.Felix.stopBundle(Felix.java:2451)
org.apache.felix.framework.Felix$RefreshHelper.stop(Felix.java:4715)
org.apache.felix.framework.Felix.refreshPackages(Felix.java:4037)
org.apache.felix.framework.FrameworkWiringImpl.run(FrameworkWiringImpl.java:178)
java.lang.Thread.run(Thread.java:662)
Re: iPOJO should not grab locks during synchronous callbacks
Posted by Jad Naous <ja...@nerati.com>.
On Mon, Jan 7, 2013 at 12:29 AM, clement escoffier <
clement.escoffier@gmail.com> wrote:
> Hi,
>
> Sorry just back (needed some time to recover).
Hope you're better.
> I will have a look today.
> Could you create a jira issue ?
>
Created FELIX-3839 <https://issues.apache.org/jira/browse/FELIX-3839>
> We have ongoing work on the instance creator and factories to simplify the
> design and synchronization protocol. I need to discuss with Guillaume
> Sauthier (leading this effort) to see how your issue is handled.
>
> Anyway, I will have a look and try to provide a fix ASAP (hopefully today).
>
>
Much appreciated!
Thanks,
Jad.
> Regards,
>
> Clement
>
>
> 2013/1/6 Jad Naous <ja...@nerati.com>
>
> > This is a fork of the earlier thread from before on deadlocks during
> > refresh.
> >
> > iPOJO grabs the InstanceCreator lock (using synchronized method calls)
> when
> > stopping a bundle, while another iPOJO thread is attempting to register
> > services, which requires resolving a class, which requires the framework
> > lock already grabbed by the other thread, creating a deadlock.
> >
> > In general, iPOJO should not be grabbing locks during synchronous
> callbacks
> > because the framework might already have some locks grabbed. Resolving
> this
> > through the framework seems impossible because the framework spec afaik
> > requires preventing anything else from happening in the framework while
> > changing bundle states.
> >
> > The resolution seems to be that applications should not grab locks in
> > synchronous callbacks that could also be grabbed by other application
> > threads.
> >
> > I've been stuck on this for weeks, and it would be really painful to
> > extract ipojo now and replace it with something else. Any help in
> resolving
> > this is really appreciated. I was thinking of going to study the ipojo
> code
> > to take out the locks in InstanceCreator, but that seems like a major
> > undertaking for me, and we have a release deadline soon...
> >
> > Here are the stack traces of the deadlocked threads:
> >
> > Name: Thread-2
> > State: WAITING on [Ljava.lang.Object;@4a018e1b
> > Total blocked: 38,871,649 Total waited: 38,871,650
> >
> > Stack trace:
> > java.lang.Object.wait(Native Method)
> > java.lang.Object.wait(Object.java:485)
> > org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:5033)
> >
> >
> org.apache.felix.framework.StatefulResolver.resolve(StatefulResolver.java:451)
> >
> >
> org.apache.felix.framework.BundleWiringImpl.searchDynamicImports(BundleWiringImpl.java:1578)
> >
> >
> org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1478)
> >
> >
> org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:75)
> >
> >
> org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1882)
> > java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >
> >
> org.apache.felix.framework.BundleWiringImpl.getClassByDelegation(BundleWiringImpl.java:1356)
> >
> >
> org.apache.felix.framework.ServiceRegistrationImpl$ServiceReferenceImpl.isAssignableTo(ServiceRegistrationImpl.java:548)
> > org.apache.felix.framework.util.Util.isServiceAssignable(Util.java:280)
> >
> >
> org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:916)
> >
> >
> org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:793)
> >
> >
> org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:543)
> > org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4346)
> > org.apache.felix.framework.Felix.registerService(Felix.java:3356)
> >
> >
> org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346)
> > org.apache.felix.ipojo.IPojoFactory.start(IPojoFactory.java:613)
> > - locked org.apache.felix.ipojo.ComponentFactory@468034b6
> > org.apache.felix.ipojo.Extender.createAbstractFactory(Extender.java:520)
> > org.apache.felix.ipojo.Extender.parse(Extender.java:301)
> > org.apache.felix.ipojo.Extender.startManagementFor(Extender.java:237)
> > org.apache.felix.ipojo.Extender.access$600(Extender.java:52)
> > org.apache.felix.ipojo.Extender$CreatorThread.run(Extender.java:769)
> > java.lang.Thread.run(Thread.java:662)
> >
> > Name: FelixFrameworkWiring
> > State: BLOCKED on org.apache.felix.ipojo.ComponentFactory@468034b6 owned
> > by: Thread-2
> > Total blocked: 7 Total waited: 1
> >
> > Stack trace:
> > org.apache.felix.ipojo
> > .IPojoFactory.removeFactoryStateListener(IPojoFactory.java:511)
> > org.apache.felix.ipojo
> > .InstanceCreator.removeFactory(InstanceCreator.java:199)
> > org.apache.felix.ipojo.Extender.closeManagementFor(Extender.java:180)
> > org.apache.felix.ipojo.Extender.bundleChanged(Extender.java:153)
> >
> >
> org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:868)
> >
> >
> org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:789)
> >
> >
> org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:514)
> > org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4330)
> > org.apache.felix.framework.Felix.stopBundle(Felix.java:2451)
> > org.apache.felix.framework.Felix$RefreshHelper.stop(Felix.java:4715)
> > org.apache.felix.framework.Felix.refreshPackages(Felix.java:4037)
> >
> >
> org.apache.felix.framework.FrameworkWiringImpl.run(FrameworkWiringImpl.java:178)
> > java.lang.Thread.run(Thread.java:662)
> >
>
Re: iPOJO should not grab locks during synchronous callbacks
Posted by clement escoffier <cl...@gmail.com>.
Hi,
Sorry just back (needed some time to recover). I will have a look today.
Could you create a jira issue ?
We have ongoing work on the instance creator and factories to simplify the
design and synchronization protocol. I need to discuss with Guillaume
Sauthier (leading this effort) to see how your issue is handled.
Anyway, I will have a look and try to provide a fix ASAP (hopefully today).
Regards,
Clement
2013/1/6 Jad Naous <ja...@nerati.com>
> This is a fork of the earlier thread from before on deadlocks during
> refresh.
>
> iPOJO grabs the InstanceCreator lock (using synchronized method calls) when
> stopping a bundle, while another iPOJO thread is attempting to register
> services, which requires resolving a class, which requires the framework
> lock already grabbed by the other thread, creating a deadlock.
>
> In general, iPOJO should not be grabbing locks during synchronous callbacks
> because the framework might already have some locks grabbed. Resolving this
> through the framework seems impossible because the framework spec afaik
> requires preventing anything else from happening in the framework while
> changing bundle states.
>
> The resolution seems to be that applications should not grab locks in
> synchronous callbacks that could also be grabbed by other application
> threads.
>
> I've been stuck on this for weeks, and it would be really painful to
> extract ipojo now and replace it with something else. Any help in resolving
> this is really appreciated. I was thinking of going to study the ipojo code
> to take out the locks in InstanceCreator, but that seems like a major
> undertaking for me, and we have a release deadline soon...
>
> Here are the stack traces of the deadlocked threads:
>
> Name: Thread-2
> State: WAITING on [Ljava.lang.Object;@4a018e1b
> Total blocked: 38,871,649 Total waited: 38,871,650
>
> Stack trace:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:5033)
>
> org.apache.felix.framework.StatefulResolver.resolve(StatefulResolver.java:451)
>
> org.apache.felix.framework.BundleWiringImpl.searchDynamicImports(BundleWiringImpl.java:1578)
>
> org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1478)
>
> org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:75)
>
> org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1882)
> java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>
> org.apache.felix.framework.BundleWiringImpl.getClassByDelegation(BundleWiringImpl.java:1356)
>
> org.apache.felix.framework.ServiceRegistrationImpl$ServiceReferenceImpl.isAssignableTo(ServiceRegistrationImpl.java:548)
> org.apache.felix.framework.util.Util.isServiceAssignable(Util.java:280)
>
> org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:916)
>
> org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:793)
>
> org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:543)
> org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4346)
> org.apache.felix.framework.Felix.registerService(Felix.java:3356)
>
> org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346)
> org.apache.felix.ipojo.IPojoFactory.start(IPojoFactory.java:613)
> - locked org.apache.felix.ipojo.ComponentFactory@468034b6
> org.apache.felix.ipojo.Extender.createAbstractFactory(Extender.java:520)
> org.apache.felix.ipojo.Extender.parse(Extender.java:301)
> org.apache.felix.ipojo.Extender.startManagementFor(Extender.java:237)
> org.apache.felix.ipojo.Extender.access$600(Extender.java:52)
> org.apache.felix.ipojo.Extender$CreatorThread.run(Extender.java:769)
> java.lang.Thread.run(Thread.java:662)
>
> Name: FelixFrameworkWiring
> State: BLOCKED on org.apache.felix.ipojo.ComponentFactory@468034b6 owned
> by: Thread-2
> Total blocked: 7 Total waited: 1
>
> Stack trace:
> org.apache.felix.ipojo
> .IPojoFactory.removeFactoryStateListener(IPojoFactory.java:511)
> org.apache.felix.ipojo
> .InstanceCreator.removeFactory(InstanceCreator.java:199)
> org.apache.felix.ipojo.Extender.closeManagementFor(Extender.java:180)
> org.apache.felix.ipojo.Extender.bundleChanged(Extender.java:153)
>
> org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:868)
>
> org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:789)
>
> org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:514)
> org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4330)
> org.apache.felix.framework.Felix.stopBundle(Felix.java:2451)
> org.apache.felix.framework.Felix$RefreshHelper.stop(Felix.java:4715)
> org.apache.felix.framework.Felix.refreshPackages(Felix.java:4037)
>
> org.apache.felix.framework.FrameworkWiringImpl.run(FrameworkWiringImpl.java:178)
> java.lang.Thread.run(Thread.java:662)
>
Re: iPOJO should not grab locks during synchronous callbacks
Posted by Jad Naous <ja...@nerati.com>.
Sorry, please replace InstanceCreator with IPojoFactory (though
InstanceCreator also has locks that are obtained in callbacks).
On Sun, Jan 6, 2013 at 9:32 AM, Jad Naous <ja...@nerati.com> wrote:
> This is a fork of the earlier thread from before on deadlocks during
> refresh.
>
> iPOJO grabs the InstanceCreator lock (using synchronized method calls)
> when stopping a bundle, while another iPOJO thread is attempting to
> register services, which requires resolving a class, which requires the
> framework lock already grabbed by the other thread, creating a deadlock.
>
> In general, iPOJO should not be grabbing locks during synchronous
> callbacks because the framework might already have some locks grabbed.
> Resolving this through the framework seems impossible because the
> framework spec afaik requires preventing anything else from happening in
> the framework while changing bundle states.
>
> The resolution seems to be that applications should not grab locks in
> synchronous callbacks that could also be grabbed by other application
> threads.
>
> I've been stuck on this for weeks, and it would be really painful to
> extract ipojo now and replace it with something else. Any help in resolving
> this is really appreciated. I was thinking of going to study the ipojo code
> to take out the locks in InstanceCreator, but that seems like a major
> undertaking for me, and we have a release deadline soon...
>
> Here are the stack traces of the deadlocked threads:
>
> Name: Thread-2
> State: WAITING on [Ljava.lang.Object;@4a018e1b
> Total blocked: 38,871,649 Total waited: 38,871,650
>
> Stack trace:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:5033)
>
> org.apache.felix.framework.StatefulResolver.resolve(StatefulResolver.java:451)
>
> org.apache.felix.framework.BundleWiringImpl.searchDynamicImports(BundleWiringImpl.java:1578)
>
> org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1478)
>
> org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:75)
>
> org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1882)
> java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>
> org.apache.felix.framework.BundleWiringImpl.getClassByDelegation(BundleWiringImpl.java:1356)
>
> org.apache.felix.framework.ServiceRegistrationImpl$ServiceReferenceImpl.isAssignableTo(ServiceRegistrationImpl.java:548)
> org.apache.felix.framework.util.Util.isServiceAssignable(Util.java:280)
>
> org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:916)
>
> org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:793)
>
> org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:543)
> org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4346)
> org.apache.felix.framework.Felix.registerService(Felix.java:3356)
>
> org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346)
> org.apache.felix.ipojo.IPojoFactory.start(IPojoFactory.java:613)
> - locked org.apache.felix.ipojo.ComponentFactory@468034b6
> org.apache.felix.ipojo.Extender.createAbstractFactory(Extender.java:520)
> org.apache.felix.ipojo.Extender.parse(Extender.java:301)
> org.apache.felix.ipojo.Extender.startManagementFor(Extender.java:237)
> org.apache.felix.ipojo.Extender.access$600(Extender.java:52)
> org.apache.felix.ipojo.Extender$CreatorThread.run(Extender.java:769)
> java.lang.Thread.run(Thread.java:662)
>
> Name: FelixFrameworkWiring
> State: BLOCKED on org.apache.felix.ipojo.ComponentFactory@468034b6 owned
> by: Thread-2
> Total blocked: 7 Total waited: 1
>
> Stack trace:
> org.apache.felix.ipojo
> .IPojoFactory.removeFactoryStateListener(IPojoFactory.java:511)
> org.apache.felix.ipojo
> .InstanceCreator.removeFactory(InstanceCreator.java:199)
> org.apache.felix.ipojo.Extender.closeManagementFor(Extender.java:180)
> org.apache.felix.ipojo.Extender.bundleChanged(Extender.java:153)
>
> org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:868)
>
> org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:789)
>
> org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:514)
> org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4330)
> org.apache.felix.framework.Felix.stopBundle(Felix.java:2451)
> org.apache.felix.framework.Felix$RefreshHelper.stop(Felix.java:4715)
> org.apache.felix.framework.Felix.refreshPackages(Felix.java:4037)
>
> org.apache.felix.framework.FrameworkWiringImpl.run(FrameworkWiringImpl.java:178)
> java.lang.Thread.run(Thread.java:662)
>
>