You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/01/24 14:25:04 UTC
[GitHub] [pulsar] gfexensa opened a new issue #6134: broker prometheus
metrics export fails when bookie is launched by the broker
gfexensa opened a new issue #6134: broker prometheus metrics export fails when bookie is launched by the broker
URL: https://github.com/apache/pulsar/issues/6134
#### Expected behavior
in our setting, the bookeeper was started at the same time as the broker with:
% bin/pulsar broker --run-bookie --run-bookie-autorecovery
and I was able to get bookkeeper metrics on
'http://pulsar-node-1.mydomain.net:8000/metrics'
and I should had been able to get metrics from the broker too here :
http://pulsar-node-1.mydomain.net:8080/metrics
#### Actual behavior
The following exception is raised at every request of the broker metrics.
Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: 06:25:02.139 [pulsar-web-59-8] INFO org.eclipse.jetty.server.RequestLog - 62.210.75.103 - - [24/Jan/2020:06:25:02 +0100] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.1.0+ds" 0
Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: 06:25:02.139 [prometheus-stats-68-1] ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught
Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: java.lang.NoClassDefFoundError: Could not initialize class org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator
Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsServlet.lambda$doGet$0(PrometheusMetricsServlet.java:70) ~[org.apache.pulsar-pulsar-broker-2.4.2.jar:2.4.2]
Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) ~[org.apache.pulsar-managed-ledger-original-2.4.2.jar:2.4.2]
Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232]
Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232]
Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_232]
Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_232]
Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: 06:25:02.141 [pulsar-web-59-7] INFO org.eclipse.jetty.server.RequestLog - 62.210.75.103 - - [24/Jan/2020:06:24:32 +0100] "GET /metrics/ HTTP/1.1" 500 331 "http://pulsar-node-1.mydomain.net:8080/metrics" "Prometheus/2.1.0+ds" 30000
Note that it's bookkeeper which is complaining that it can't found 'org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator', which is a class from the broker.
The 8080 port is the default port for the bookkeeper httpServerPort, so it's probably hiding the prometheus export. Unfortunatly, we can't change the prometheus port in the broker configuration.
If both the bookie and the broker are launched separately, then both metrics are correctly exposed.
#### Steps to reproduce
Start the bookie along the broker and request the broker metric.
#### System configuration
**Pulsar version**: 2.4.2
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] gfexensa commented on issue #6134: broker prometheus
metrics export fails when bookie is launched by the broker
Posted by GitBox <gi...@apache.org>.
gfexensa commented on issue #6134: broker prometheus metrics export fails when bookie is launched by the broker
URL: https://github.com/apache/pulsar/issues/6134#issuecomment-583260886
> @gfexensa right. This is because bookie/broker using the same port of 8000 for prometheus.
The broker use a different port 8000 to expose its metrics than bookkeeper (8080). Its the bookkeeper httpServerPort which share the same port as the broker prometheus metrics.
it should work if the httpServerPort is modified, but it's not exactly a desirable solution. IMHO, both broker prometheus port and bookkeeper http server port should be different, and eventually allows to specify a different broker port for prometheus.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] jiazhai commented on issue #6134: broker prometheus
metrics export fails when bookie is launched by the broker
Posted by GitBox <gi...@apache.org>.
jiazhai commented on issue #6134: broker prometheus metrics export fails when bookie is launched by the broker
URL: https://github.com/apache/pulsar/issues/6134#issuecomment-583215246
@gfexensa right. This is because bookie/broker using the same port of 8000 for prometheus.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services