You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by "Aled Sage (JIRA)" <ji...@apache.org> on 2017/04/27 15:56:04 UTC
[jira] [Resolved] (BROOKLYN-424) Web-server hangs after several
days
[ https://issues.apache.org/jira/browse/BROOKLYN-424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aled Sage resolved BROOKLYN-424.
--------------------------------
Resolution: Fixed
Fix Version/s: 0.11.0
> Web-server hangs after several days
> -----------------------------------
>
> Key: BROOKLYN-424
> URL: https://issues.apache.org/jira/browse/BROOKLYN-424
> Project: Brooklyn
> Issue Type: Bug
> Reporter: Aled Sage
> Fix For: 0.11.0
>
>
> Using Brooklyn 0.10.0-SNAPSHOT from (approximately) 13-09-16...
> The web-server becomes unresponsive. Refreshing the page doesn't help.
> There are (probably) approx 100 catalog items in the persisted state, but no apps deployed.
> Looking at the jstack output, there are 16 threads (all with names matching "brooklyn-jetty-server-8081-qtp97076093-.*") that are parked on the same groovy LockableObject (0x00000007030f2d50).
> Most of the stacktraces looks something like:
> {noformat}
> "brooklyn-jetty-server-8081-qtp97076093-17830" prio=10 tid=0x00007f430c5db800 nid=0x1d4d waiting on condition [0x00007f42c5cd8000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000007030f2d50> (a org.codehaus.groovy.util.LockableObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
> at org.codehaus.groovy.util.LockableObject.lock(LockableObject.java:34)
> at org.codehaus.groovy.reflection.ClassInfo.lock(ClassInfo.java:327)
> at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:239)
> at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:255)
> at groovy.lang.MetaClassImpl.pickStaticMethod(MetaClassImpl.java:1511)
> at groovy.lang.MetaClassImpl.retrieveStaticMethod(MetaClassImpl.java:1401)
> at groovy.lang.MetaClassImpl.invokeStaticMethod(MetaClassImpl.java:1435)
> at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:875)
> at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:166)
> at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.isCase(ScriptBytecodeAdapter.java:657)
> at org.apache.brooklyn.util.groovy.GroovyJavaMethods.fix(GroovyJavaMethods.java:176)
> at org.apache.brooklyn.util.groovy.GroovyJavaMethods.elvis(GroovyJavaMethods.java:155)
> at org.apache.brooklyn.util.core.flags.FlagUtils.getFlagConfigKeyRecord(FlagUtils.java:277)
> at org.apache.brooklyn.util.core.flags.FlagUtils.findAllFlagsAndConfigKeys(FlagUtils.java:250)
> at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynComponentTemplateResolver.findAllFlagsAndConfigKeyValues(BrooklynComponentTemplateResolver.java:378)
> at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynComponentTemplateResolver.configureEntityConfig(BrooklynComponentTemplateResolver.java:275)
> at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynComponentTemplateResolver.decorateSpec(BrooklynComponentTemplateResolver.java:255)
> at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynComponentTemplateResolver.populateSpec(BrooklynComponentTemplateResolver.java:246)
> at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynComponentTemplateResolver.resolveSpec(BrooklynComponentTemplateResolver.java:185)
> at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynAssemblyTemplateInstantiator.buildTemplateServicesAsSpecs(BrooklynAssemblyTemplateInstantiator.java:114)
> at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynAssemblyTemplateInstantiator.createServiceSpecs(BrooklynAssemblyTemplateInstantiator.java:73)
> at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynAssemblyTemplateInstantiator.createApplicationSpec(BrooklynAssemblyTemplateInstantiator.java:90)
> at org.apache.brooklyn.camp.brooklyn.spi.creation.CampResolver.createEntitySpecFromServicesBlock(CampResolver.java:145)
> at org.apache.brooklyn.camp.brooklyn.spi.creation.CampResolver.createSpecFromFull(CampResolver.java:109)
> at org.apache.brooklyn.camp.brooklyn.spi.creation.CampToSpecTransformer.createCatalogSpec(CampToSpecTransformer.java:102)
> at org.apache.brooklyn.core.catalog.internal.BasicBrooklynCatalog$1.apply(BasicBrooklynCatalog.java:331)
> at org.apache.brooklyn.core.catalog.internal.BasicBrooklynCatalog$1.apply(BasicBrooklynCatalog.java:328)
> at org.apache.brooklyn.core.plan.PlanToSpecFactory.attemptWithLoaders(PlanToSpecFactory.java:126)
> at org.apache.brooklyn.core.plan.PlanToSpecFactory.attemptWithLoaders(PlanToSpecFactory.java:118)
> at org.apache.brooklyn.core.catalog.internal.BasicBrooklynCatalog.internalCreateSpecLegacy(BasicBrooklynCatalog.java:328)
> at org.apache.brooklyn.core.catalog.internal.BasicBrooklynCatalog.createSpec(BasicBrooklynCatalog.java:310)
> at org.apache.brooklyn.rest.transform.CatalogTransformer.catalogEntitySummary(CatalogTransformer.java:81)
> at org.apache.brooklyn.rest.transform.CatalogTransformer.catalogItemSummary(CatalogTransformer.java:117)
> at org.apache.brooklyn.rest.resources.CatalogResource$1.apply(CatalogResource.java:91)
> at org.apache.brooklyn.rest.resources.CatalogResource$1.apply(CatalogResource.java:88)
> at com.google.common.collect.Lists$TransformingRandomAccessList$1.transform(Lists.java:582)
> at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContentsUsing(CollectionSerializer.java:139)
> at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContents(CollectionSerializer.java:87)
> at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContents(CollectionSerializer.java:23)
> at com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.serialize(AsArraySerializerBase.java:183)
> at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:250)
> at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:604)
> at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:648)
> at org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1378)
> at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.serializeMessage(JAXRSOutInterceptor.java:235)
> at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.processResponse(JAXRSOutInterceptor.java:118)
> at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleMessage(JAXRSOutInterceptor.java:81)
> at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
> - locked <0x00000007dec2d290> (a org.apache.cxf.phase.PhaseInterceptorChain)
> at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:83)
> at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
> - locked <0x00000007dec17dc0> (a org.apache.cxf.phase.PhaseInterceptorChain)
> at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
> at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:251)
> at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
> at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
> at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
> at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:180)
> at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:293)
> at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:217)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:268)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
> at org.apache.brooklyn.rest.filter.LoggingFilter.doFilter(LoggingFilter.java:91)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at org.apache.brooklyn.rest.filter.RequestTaggingFilter.doFilter(RequestTaggingFilter.java:51)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:542)
> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> And one looks like:
> {noformat}
> "brooklyn-jetty-server-8081-qtp97076093-16662" prio=10 tid=0x00007f430c0da000 nid=0x34ee runnable [0x00007f431df67000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2367)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
> at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
> at java.lang.StringBuilder.append(StringBuilder.java:132)
> at java.lang.reflect.Method.toGenericString(Method.java:495)
> at java.beans.MethodRef.find(MethodRef.java:79)
> at java.beans.MethodRef.get(MethodRef.java:62)
> at java.beans.PropertyDescriptor.getReadMethod(PropertyDescriptor.java:207)
> - locked <0x0000000703139d30> (a java.beans.PropertyDescriptor)
> at groovy.lang.MetaClassImpl.applyPropertyDescriptors(MetaClassImpl.java:2470)
> at groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:2250)
> at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:3255)
> at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:3226)
> - locked <0x00000007defcff68> (a groovy.lang.MetaClassImpl)
> at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:210)
> at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:241)
> at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:255)
> at groovy.lang.MetaClassImpl.pickStaticMethod(MetaClassImpl.java:1511)
> at groovy.lang.MetaClassImpl.retrieveStaticMethod(MetaClassImpl.java:1401)
> at groovy.lang.MetaClassImpl.invokeStaticMethod(MetaClassImpl.java:1435)
> at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:875)
> at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:166)
> at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.isCase(ScriptBytecodeAdapter.java:657)
> at org.apache.brooklyn.util.groovy.GroovyJavaMethods.fix(GroovyJavaMethods.java:176)
> at org.apache.brooklyn.util.groovy.GroovyJavaMethods.elvis(GroovyJavaMethods.java:155)
> at org.apache.brooklyn.core.effector.MethodEffector.<init>(MethodEffector.java:142)
> at org.apache.brooklyn.core.effector.MethodEffector.<init>(MethodEffector.java:133)
> at org.apache.brooklyn.core.effector.MethodEffector.create(MethodEffector.java:55)
> at org.apache.brooklyn.core.entity.EntityDynamicType.findEffectors(EntityDynamicType.java:317)
> at org.apache.brooklyn.core.entity.EntityDynamicType.<init>(EntityDynamicType.java:82)
> at org.apache.brooklyn.core.entity.EntityDynamicType.<init>(EntityDynamicType.java:76)
> at org.apache.brooklyn.core.objs.BrooklynTypes$ImmutableEntityType.<init>(BrooklynTypes.java:45)
> at org.apache.brooklyn.core.objs.BrooklynTypes$DefinedBrooklynTypeLoader.call(BrooklynTypes.java:88)
> at org.apache.brooklyn.core.objs.BrooklynTypes$DefinedBrooklynTypeLoader.call(BrooklynTypes.java:77)
> at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4739)
> at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
> at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
> at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
> - locked <0x00000007def8ed90> (a com.google.common.cache.LocalCache$WeakEntry)
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
> at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
> at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4736)
> at org.apache.brooklyn.core.objs.BrooklynTypes.getDefinedBrooklynType(BrooklynTypes.java:113)
> at org.apache.brooklyn.core.objs.BasicSpecParameter$ParseClassParameters.parseParameters(BasicSpecParameter.java:358)
> at org.apache.brooklyn.core.objs.BasicSpecParameter.fromSpec(BasicSpecParameter.java:168)
> at org.apache.brooklyn.core.objs.BasicSpecParameter.addParameters(BasicSpecParameter.java:400)
> at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynEntityDecorationResolver$SpecParameterResolver.decorate(BrooklynEntityDecorationResolver.java:213)
> {noformat}
> There is a discussion at http://groovy.329449.n5.nabble.com/Deadlock-on-concurrent-metaClass-access-tp3275702p3276031.html (from 2010) about what could cause this.
> For us, we should *not* be calling groovy code here! I think that would get rid of the problem.
> The results of {{top}} showed that the server was heavily loaded:
> {noformat}
> top - 11:57:51 up 25 days, 6:05, 2 users, load average: 5.51, 5.02, 4.74
> Tasks: 150 total, 1 running, 149 sleeping, 0 stopped, 0 zombie
> Cpu(s): 13.3%us, 1.3%sy, 0.0%ni, 85.2%id, 0.1%wa, 0.0%hi, 0.1%si, 0.0%st
> Mem: 8061144k total, 5431376k used, 2629768k free, 195796k buffers
> Swap: 2097148k total, 0k used, 2097148k free, 626448k cached
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 23196 brooklyn 20 0 7673m 3.9g 25m S 243.4 50.3 17960:32 java
> 12577 brooklyn 20 0 15028 1124 816 R 3.6 0.0 0:00.40 top
> ...
> {noformat}
> Looking at the 11 "RUNNABLE" threads in jstack (a pooor man's profiling!), only one was in groovy code (shown below):
> {noformat}
> "brooklyn-jetty-server-8081-qtp97076093-16808" prio=10 tid=0x00007f430c0ff000 nid=0x7ca9 runnable [0x00007f4351cc2000]
> java.lang.Thread.State: RUNNABLE
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2625)
> at java.lang.Class.privateGetPublicMethods(Class.java:2743)
> at java.lang.Class.getMethods(Class.java:1480)
> at java.beans.MethodRef.find(MethodRef.java:77)
> at java.beans.MethodRef.get(MethodRef.java:62)
> at java.beans.PropertyDescriptor.getReadMethod(PropertyDescriptor.java:207)
> - locked <0x000000070312ebb8> (a java.beans.PropertyDescriptor)
> at groovy.lang.MetaClassImpl.applyPropertyDescriptors(MetaClassImpl.java:2470)
> at groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:2250)
> at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:3255)
> at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:3226)
> - locked <0x00000007df006650> (a groovy.lang.MetaClassImpl)
> at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:210)
> at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:241)
> at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:251)
> at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:259)
> at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:855)
> at org.codehaus.groovy.runtime.InvokerHelper.invokePojoMethod(InvokerHelper.java:888)
> at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:880)
> at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.castToBoolean(DefaultTypeTransformation.java:180)
> at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.booleanUnbox(DefaultTypeTransformation.java:69)
> at org.apache.brooklyn.util.groovy.GroovyJavaMethods.truth(GroovyJavaMethods.java:149)
> at org.apache.brooklyn.util.groovy.GroovyJavaMethods.elvis(GroovyJavaMethods.java:155)
> at org.apache.brooklyn.util.core.flags.FlagUtils.getFlagConfigKeyRecord(FlagUtils.java:277)
> at org.apache.brooklyn.util.core.flags.FlagUtils.findAllFlagsAndConfigKeys(FlagUtils.java:250)
> at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynComponentTemplateResolver.findAllFlagsAndConfigKeyValues(BrooklynComponentTemplateResolver.java:376)
> ...
> {noformat}
> Memory usage was high (according to {{jmap -histo:live}}), but not unreasonable given our use of soft references:
> {noformat}
> num #instances #bytes class name
> ----------------------------------------------
> 1: 14196525 1728415160 [C
> ...
> Total 73158949 3812605624
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)