You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2018/12/03 14:46:20 UTC
[GitHub] yifan opened a new issue #3112: broker error for prometheus metrics
yifan opened a new issue #3112: broker error for prometheus metrics
URL: https://github.com/apache/pulsar/issues/3112
#### Expected behavior
I deployed pulsar with helm and minikube configuration in kubernetes. I tried multiple times, I having problem with one of the brokers, sometimes both brokers. The problem has been consistent for three deployments so far.
#### Steps to reproduce
deploy pulsar with helm
#### System configuration
**Pulsar version**: x.y
image: apachepulsar/pulsar-all
#### Error in Broker.log
06:40:34.280 [pulsar-web-30-30] INFO org.eclipse.jetty.server.RequestLog - 10.244.6.29 - - [03/Dec/2018:06:40:34 +0000] "GET //xx.xx.xx.xx:8080/metrics HTTP/1.1" 302 0 "-" "Prometheus/1.6.3" 1
06:40:34.287 [prometheus-stats-31-1] ERROR org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsServlet - Failed to generate prometheus stats
org.eclipse.jetty.io.EofException: null
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:197) ~[org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:420) ~[org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:375) ~[org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.SelectChannelEndPoint$3.run(SelectChannelEndPoint.java:107) ~[org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:193) ~[org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:283) ~[org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181) ~[org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) ~[org.eclipse.jetty-jetty-util-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) ~[org.eclipse.jetty-jetty-util-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) ~[org.eclipse.jetty-jetty-util-9.3.11.v20160721.jar:9.3.11.v20160721]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [org.apache.pulsar-pulsar-functions-metrics-2.2.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:1.8.0_171]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:1.8.0_171]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:1.8.0_171]
at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[?:1.8.0_171]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[?:1.8.0_171]
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:175) ~[org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
... 13 more
#### Error in Prometheus
text format parsing error in line 182: second TYPE line for metric name "pulsar_subscription_back_log", or TYPE reported after samples
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services