You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tapestry.apache.org by "Douglas Hauge (Updated) (JIRA)" <ji...@apache.org> on 2011/10/04 02:29:33 UTC
[jira] [Updated] (TAP5-1650) On a cold start with a large number of
incoming requests, Tapestry can deadlock inside
PlasticClassLoader/PlasticClassPool
[ https://issues.apache.org/jira/browse/TAP5-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Douglas Hauge updated TAP5-1650:
--------------------------------
Attachment: tapestry-deadlock-fix-1.patch
Attached is a potential fix for the deadlock on startup (at least after applying the patch, we are no longer able to reproduce the deadlock). The problem was that there were multiple orders in which the 3 Tapestry classloader locks could be acquired.
# standard: (prop expr conduit CL) -> (component CL) -> (service CL)
# problematic: (componet CL) -> (prop expr conduit CL)
The second happens when *ComponentInstantiatorSourceImpl.getInstantiator* is called, but while assembling the component classes a service class needs to be loaded. The rough path for this is
# ComponentInstantiatorSourceimpl.getInstantiator
# -> ComponentInstantiatorSourceImpl.createInstantiatorForClass
# -> PlasticManager.getClassInstantiator (component classloader manager, not that in 'proxyFactory')
# -> PlasticClassPool.getClassInstantiator
# -> PlasticClassLoader.loadClass (acquires component class loader lock)
# -> ComponentInstantiatorSourceImpl.transform -> RegistryImpl.run -> ... -> DefaultInjectionProvider.provideInjection
# -> ComponentClassCacheImpl.forName
# -> ComponentClassCacheImpl.lookupClassForType (gets the property expression conduit class loader from 'PlasticProxyFactory')
# -> PlasticInternalUtils.toClass -> Class.forName (acquires the lock on the passed in class loader, the property expression conduit one).
The exact stacks of the deadlocked threads vary a little, but look similar to:
Bad lock order:
{code}
"TP-Processor44":
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.tapestry5.internal.plastic.PlasticInternalUtils.toClass(PlasticInternalUtils.java:306)
at org.apache.tapestry5.internal.services.ComponentClassCacheImpl.lookupClassForType(ComponentClassCacheImpl.java:78)
at org.apache.tapestry5.internal.services.ComponentClassCacheImpl.forName(ComponentClassCacheImpl.java:65)
at $ComponentClassCache_1089cca7fe04a.forName(Unknown Source)
at org.apache.tapestry5.internal.services.DefaultInjectionProvider.provideInjection(DefaultInjectionProvider.java:65)
at $InjectionProvider2_1089cca7fe169.provideInjection(Unknown Source)
at $InjectionProvider2_1089cca7fe15d.provideInjection(Unknown Source)
at org.apache.tapestry5.internal.transform.InjectWorker$2.run(InjectWorker.java:73)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl$1.invoke(OperationTrackerImpl.java:51)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl$1.invoke(OperationTrackerImpl.java:48)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl.invoke(OperationTrackerImpl.java:74)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl.run(OperationTrackerImpl.java:47)
at org.apache.tapestry5.ioc.internal.PerThreadOperationTracker.run(PerThreadOperationTracker.java:76)
at org.apache.tapestry5.ioc.internal.RegistryImpl.run(RegistryImpl.java:1005)
at org.apache.tapestry5.internal.transform.InjectWorker.transform(InjectWorker.java:67)
at $ComponentClassTransformWorker2_1089cca7fe165.transform(Unknown Source)
at $ComponentClassTransformWorker2_1089cca7fe157.transform(Unknown Source)
at org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl$3.run(ComponentInstantiatorSourceImpl.java:320)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl$1.invoke(OperationTrackerImpl.java:51)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl$1.invoke(OperationTrackerImpl.java:48)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl.invoke(OperationTrackerImpl.java:74)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl.run(OperationTrackerImpl.java:47)
at org.apache.tapestry5.ioc.internal.PerThreadOperationTracker.run(PerThreadOperationTracker.java:76)
at org.apache.tapestry5.ioc.internal.RegistryImpl.run(RegistryImpl.java:1005)
at org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl.transform(ComponentInstantiatorSourceImpl.java:273)
at org.apache.tapestry5.internal.plastic.PlasticClassPool.loadAndTransformClass(PlasticClassPool.java:313)
at org.apache.tapestry5.internal.plastic.PlasticClassLoader.loadClass(PlasticClassLoader.java:38)
- locked <0x00000003027edc98> (a org.apache.tapestry5.internal.plastic.PlasticClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at org.apache.tapestry5.internal.plastic.PlasticClassPool.getClassInstantiator(PlasticClassPool.java:416)
- locked <0x00000003027edc98> (a org.apache.tapestry5.internal.plastic.PlasticClassLoader)
at org.apache.tapestry5.plastic.PlasticManager.getClassInstantiator(PlasticManager.java:189)
at org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl$2.invoke(ComponentInstantiatorSourceImpl.java:228)
at org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl$2.invoke(ComponentInstantiatorSourceImpl.java:222)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl.invoke(OperationTrackerImpl.java:74)
at org.apache.tapestry5.ioc.internal.PerThreadOperationTracker.invoke(PerThreadOperationTracker.java:87)
at org.apache.tapestry5.ioc.internal.RegistryImpl.invoke(RegistryImpl.java:1010)
at org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl.createInstantiatorForClass(ComponentInstantiatorSourceImpl.java:220)
at org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl.getInstantiator(ComponentInstantiatorSourceImpl.java:210)
- locked <0x0000000304c07bc8> (a org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl)
at $ComponentInstantiatorSource_1089cca7fe152.getInstantiator(Unknown Source)
{code}
Good lock order:
{code}
"TP-Processor100":
at org.apache.tapestry5.internal.plastic.PlasticClassLoader.loadClass(PlasticClassLoader.java:31)
- waiting to lock <0x00000003027edc98> (a org.apache.tapestry5.internal.plastic.PlasticClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:295)
- locked <0x000000039f889ab8> (a org.apache.tapestry5.internal.plastic.PlasticClassLoader)
at org.apache.tapestry5.internal.plastic.PlasticClassLoader.loadClass(PlasticClassLoader.java:46)
- locked <0x000000039f889ab8> (a org.apache.tapestry5.internal.plastic.PlasticClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at $InternalPropertyConduit_1089cca7ffb3f.getRoot(Unknown Source)
at $InternalPropertyConduit_1089cca7ffb3f.get(Unknown Source)
at org.apache.tapestry5.internal.bindings.PropBinding.get(PropBinding.java:59)
at org.apache.tapestry5.internal.transform.ParameterWorker$3$1.readFromBinding(ParameterWorker.java:254)
at org.apache.tapestry5.internal.transform.ParameterWorker$3$1.get(ParameterWorker.java:372)
at lithium.tapestry.mixins.IsVisible.getfieldvalue_isVisible(IsVisible.java)
at lithium.tapestry.mixins.IsVisible.setupRender(IsVisible.java:32)
at lithium.tapestry.mixins.IsVisible.setupRender(IsVisible.java)
at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$SetupRenderPhase.invokeComponent(ComponentPageElementImpl.java:174)
at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$AbstractPhase.invoke(ComponentPageElementImpl.java:143)
at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$SetupRenderPhase.render(ComponentPageElementImpl.java:181)
at org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72)
at org.apache.tapestry5.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:124)
{code}
> On a cold start with a large number of incoming requests, Tapestry can deadlock inside PlasticClassLoader/PlasticClassPool
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: TAP5-1650
> URL: https://issues.apache.org/jira/browse/TAP5-1650
> Project: Tapestry 5
> Issue Type: Bug
> Components: plastic
> Affects Versions: 5.3
> Reporter: Howard M. Lewis Ship
> Assignee: Howard M. Lewis Ship
> Priority: Critical
> Labels: deadlock, performance
> Attachments: tapestry-deadlock-fix-1.patch
>
>
> Discovered during load testing; the scenario is a cold start for the application then hitting it with large numbers of concurrent requests with no ramp up. The deadlocks involve two methods:
> PlasticClassLoader.loadClass()
> PlasticClassPool.getClassInstantiator()
> A proper solution would to make these methods share a single explicit Lock, rather than be synchronized.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira