You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/12/17 09:40:06 UTC
[GitHub] [pulsar] eolivelli opened a new issue #13379: Metrics: noise stacktrace if Prometheus closes the connection due to a timeout
eolivelli opened a new issue #13379:
URL: https://github.com/apache/pulsar/issues/13379
**Describe the bug**
if Prometheus closes the connection due to a timeout or some other reason you will see these errors in the logs and this is very noisy.
We can simply report a better message in the logs.
10:35:18.642 [prometheus-stats-58-1] ERROR org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsServlet - Failed to generate prometheus stats
org.eclipse.jetty.io.EofException: null
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279) ~[jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422) ~[jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277) ~[jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381) ~[jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:826) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241) ~[jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223) ~[jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:550) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:915) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:987) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:285) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:269) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:897) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator.generate(PrometheusMetricsGenerator.java:127) .... 4.1.60.Final.jar:4.1.60.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.writev0(Native Method) ~[?:?]
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51) ~[?:?]
at sun.nio.ch.IOUtil.write(IOUtil.java:182) ~[?:?]
at sun.nio.ch.IOUtil.write(IOUtil.java:130) ~[?:?]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:493) ~[?:?]
at java.nio.channels.SocketChannel.write(SocketChannel.java:507) ~[?:?]
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:273) ~[jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629]
... 23 more
**Expected behavior**
The logs should not be full of this spammy log
**Additional context**
The error may occour if you have many functions in the broker and collecting the metrics is too slow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #13379: Metrics: noise stacktrace if Prometheus closes the connection due to a timeout
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #13379:
URL: https://github.com/apache/pulsar/issues/13379#issuecomment-996645287
This is another common error that happens when the timeout is elapsed on the server side
java.lang.IllegalStateException: AsyncContext completed and/or Request lifecycle recycled
at org.eclipse.jetty.server.AsyncContextState.state(AsyncContextState.java:52) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.AsyncContextState.complete(AsyncContextState.java:72) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsServlet.lambda$doGet$0(PrometheusMetricsServlet.java:85) ~[pulsar-broker.jar:2.7.2.1.1.22-SNAPSHOT]
at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) ~[managed-ledger.jar:2.7.2.1.1.22-SNAPSHOT]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [bookkeeper-common-4.14.3.jar:4.14.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.60.Final.jar:4.1.60.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli closed issue #13379: Metrics: noise stacktrace if Prometheus closes the connection due to a timeout
Posted by GitBox <gi...@apache.org>.
eolivelli closed issue #13379:
URL: https://github.com/apache/pulsar/issues/13379
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org