You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/01/13 09:00:34 UTC
[GitHub] [pulsar] fantapsody opened a new issue #9201: Saw "Unknown HK2 failure detected" Exception in the log when broker starts with function worker enabled
fantapsody opened a new issue #9201:
URL: https://github.com/apache/pulsar/issues/9201
**Describe the bug**
I started the broker of version `2.7.0.3` with `functionWorkerEnabled=true` in a K8s cluster and saw error logs like follows during broker start:
> 04:54:54.810 [pulsar-web-45-3] WARN org.glassfish.jersey.internal.Errors - The following warnings have been detected: WARNING: Unknown HK2 failure detected:
> MultiException stack 1 of 2
> javax.ws.rs.WebApplicationException: HTTP 404 Not Found
> at org.apache.pulsar.common.configuration.VipStatus.checkStatus(VipStatus.java:58)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1268)
> at org.jvnet.hk2.internal.ClazzCreator.methodMe(ClazzCreator.java:295)
> at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:348)
> at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
> at org.glassfish.jersey.inject.hk2.RequestContext.findOrCreate(RequestContext.java:59)
> at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
> at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:758)
> at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:721)
> at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:691)
> at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:160)
> at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:30)
> at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:105)
> at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:260)
> at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:51)
> at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:86)
> at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
> at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
> at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
> at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:69)
> at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:38)
> at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)
> at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:245)
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
> at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
> at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
> at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
> at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
> at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
> at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
> at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
> at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
> at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
> at org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:58)
> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
> at org.apache.pulsar.broker.web.PreInterceptFilter.doFilter(PreInterceptFilter.java:65)
> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:516)
> at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
> at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> MultiException stack 2 of 2
> java.lang.IllegalStateException: Unable to perform operation: method inject on org.apache.pulsar.common.configuration.VipStatus
> at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:369)
> at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
> at org.glassfish.jersey.inject.hk2.RequestContext.findOrCreate(RequestContext.java:59)
> at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
> at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:758)
> at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:721)
> at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:691)
> at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:160)
> at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:30)
> at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:105)
> at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:260)
> at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:51)
> at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:86)
> at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
> at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
> at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
> at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:69)
> at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:38)
> at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)
> at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:245)
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
> at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
> at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
> at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
> at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
> at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
> at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
> at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
> at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
> at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
> at org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:58)
> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
> at org.apache.pulsar.broker.web.PreInterceptFilter.doFilter(PreInterceptFilter.java:65)
> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:516)
> at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
> at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> 04:54:54.829 [pulsar-web-45-3] INFO org.eclipse.jetty.server.RequestLog - 10.88.128.7 - - [13/Jan/2021:04:54:54 +0000] "GET /status.html HTTP/1.1" 404 502 "-" "kube-probe/1.16+" 23
It seems the exceptions were triggered by the liveness/readiness probe of the container during the start. Sometimes the pod failed to become ready and got killed after consecutive failures of liveness probe. Even the pod finally became running, it's annoying and confusing to see error logs like this.
These logs didn't show up if the broker was started without `functionWorkerEnabled=true` or if the version of the broker is 2.6.0.
**To Reproduce**
Start the broker with `functionWorkerEnabled=true` during start and access the `/status.html` for status.
**Expected behavior**
No error logs like described.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie closed issue #9201: Saw "Unknown HK2 failure detected" Exception in the log when broker starts with function worker enabled
Posted by GitBox <gi...@apache.org>.
sijie closed issue #9201:
URL: https://github.com/apache/pulsar/issues/9201
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #9201: Saw "Unknown HK2 failure detected" Exception in the log when broker starts with function worker enabled
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #9201:
URL: https://github.com/apache/pulsar/issues/9201#issuecomment-759944162
This is a false alert. I have worked with @fantapsody on this offline. The exception was a bit misleading. It was thrown when readiness probe kicks in before the service is ready. We have resolved the problem by tuning the readiness probe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org