[jira] [Created] (ZOOKEEPER-4540) Zookeeper 3.7.0: /metrics java.lang.IllegalArgumentException: Invalid thread ID parameter: 0

["Guido Iaquinti (Jira)" <ji...@apache.org>]

Guido Iaquinti created ZOOKEEPER-4540:
-----------------------------------------

             Summary: Zookeeper 3.7.0: /metrics java.lang.IllegalArgumentException: Invalid thread ID parameter: 0
                 Key: ZOOKEEPER-4540
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-4540
             Project: ZooKeeper
          Issue Type: Bug
            Reporter: Guido Iaquinti


Zookeeper node running version 3.7.0 part of a cluster with 2 other nodes. I'm getting the following log while accessing the /metrics (Prometheus) endpoint:
{code:java}
2022-05-10 16:11:01,122 [myid:0] - WARN  [qtp850551034-16:HttpChannel@677] - /metrics
java.lang.IllegalArgumentException: Invalid thread ID parameter: 0
        at java.management/sun.management.ThreadImpl.verifyThreadId(ThreadImpl.java:165)
        at java.management/sun.management.ThreadImpl.verifyThreadIds(ThreadImpl.java:174)
        at java.management/sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:180)
        at io.prometheus.client.hotspot.ThreadExports.getThreadStateCountMap(ThreadExports.java:98)
        at io.prometheus.client.hotspot.ThreadExports.addThreadMetrics(ThreadExports.java:86)
        at io.prometheus.client.hotspot.ThreadExports.collect(ThreadExports.java:123)
        at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:190)
        at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:223)
        at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:144)
        at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)
        at io.prometheus.client.exporter.MetricsServlet.doGet(MetricsServlet.java:49)
        at org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider$MetricsServletImpl.doGet(PrometheusMetricsProvider.java:406)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:279)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
        at java.base/java.lang.Thread.run(Thread.java:829) {code}
the response of the endpoint is also partial:
{code:java}
ssm-user@dev-iad-zk-0:/var/snap/amazon-ssm-agent/5165$ curl http://localhost:7000/metrics
# HELP learner_proposal_received_count learner_proposal_received_count
# TYPE learner_proposal_received_count counter
learner_proposal_received_count 0.0{code}

Few additional things:
 # the other 2 nodes don't experience the same issue (same config)
 # the node is otherwise fine (the service is working)
 # the overall cluster health is good 

Not a critical issue but it's creating us issues from a service visibility perspective. Thank you!



--
This message was sent by Atlassian Jira
(v8.20.7#820007)