You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/04/20 09:00:13 UTC
[GitHub] [druid] raisalpa commented on issue #11033: Metrics Emitter timeout after 12-16ms
raisalpa commented on issue #11033:
URL: https://github.com/apache/druid/issues/11033#issuecomment-1103656068
@tanisdlj How did you resolve it? I am facing the same error.
My config is:
druid.emitter=http
druid.emitter.http.recipientBaseUrl=http://druid-production-exporter:8000/druid
druid.emitter.http.minHttpTimeoutMillis=10
druid.emitter.http.flushMillis=90000
druid.emitter.http.batchQueueSizeLimit=250
druid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor"]
error:
```
2022-04-19T18:14:05,946 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Failed to send events to url[http://druid-production-exporter:8000/druid]
java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-production-exporter/xxxxxxx:8000 after 42 ms
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emit(HttpPostEmitter.java:574) [druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emitBatches(HttpPostEmitter.java:550) [druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:503) [druid-core-0.22.1.jar:0.22.1]
Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-production-exporter/xxxxxxx:8000 after 42 ms
at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:669) ~[netty-common-4.1.68.Final.jar:4.1.68.Final]
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:744) ~[netty-common-4.1.68.Final.jar:4.1.68.Final]
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:469) ~[netty-common-4.1.68.Final.jar:4.1.68.Final]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
2022-04-19T18:14:06,146 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [21] ms, timeout [42] ms
2022-04-19T18:14:07,545 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [21] ms, timeout [42] ms
2022-04-19T18:14:10,146 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [21] ms, timeout [42] ms
2022-04-19T18:14:11,446 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [21] ms, timeout [42] ms
2022-04-19T18:14:12,546 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [21] ms, timeout [42] ms
2022-04-19T18:15:04,746 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [18] ms, timeout [36] ms
2022-04-19T18:15:04,746 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Failed to send events to url[http://druid-production-exporter:8000/druid]
java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-production-exporter/xxxxxxx:8000 after 36 ms
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emit(HttpPostEmitter.java:574) [druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emitBatches(HttpPostEmitter.java:550) [druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:503) [druid-core-0.22.1.jar:0.22.1]
Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-production-exporter/xxxxxxx:8000 after 36 ms
at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:669) ~[netty-common-4.1.68.Final.jar:4.1.68.Final]
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:744) ~[netty-common-4.1.68.Final.jar:4.1.68.Final]
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:469) ~[netty-common-4.1.68.Final.jar:4.1.68.Final]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
```
@FrankChen021 Any suggessions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org