You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/03/25 12:02:42 UTC
[GitHub] [druid] tanisdlj opened a new issue #11033: Metrics Emitter timeout after 12-16ms
tanisdlj opened a new issue #11033:
URL: https://github.com/apache/druid/issues/11033
Happening in the Broker, but I am fairly sure it happened in Overlord and Coordinator too.
### Affected Version
Druid 0.20.1
### Description
Cluster:
- 2 x Router
- 2 x Overlord
- 2 x Coordinator
- 2 x Brokers
- 32 x Middlemanager
- 32 x Historicals
From time to time we stop receiving metrics on our metrics exporter. Druid logs starts throwing exceptions like:
```
Mar 25 11:40:35 druid-queryserver-1 java[20898]: 2021-03-25T11:40:35,758 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [8] ms, timeout [16] ms
Mar 25 11:40:35 druid-queryserver-1 java[20898]: 2021-03-25T11:40:35,758 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Failed to send events to url[http://druid-exporter-1:8080/druid]
Mar 25 11:40:35 druid-queryserver-1 java[20898]: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:XXXX after 16 ms
Mar 25 11:40:35 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
Mar 25 11:40:35 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
Mar 25 11:40:35 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:40:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:40:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:40:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:40:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:40:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:40:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:40:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:40:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:40:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emit(HttpPostEmitter.java:574) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:40:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emitBatches(HttpPostEmitter.java:550) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:40:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:503) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:40:35 druid-queryserver-1 java[20898]: Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:XXXX after 16 ms
Mar 25 11:40:35 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:40:35 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:40:35 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:40:35 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:40:35 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:40:35 druid-queryserver-1 java[20898]: at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
```
I understand that this can timeout if the receiving end is slow, but timeout at 12-16ms? That's a bit excessive.
I was not able to find a config to change this value. Could you point it to me if I missed it??
Thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #11033: Metrics Emitter timeout after 12-16ms
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #11033:
URL: https://github.com/apache/druid/issues/11033#issuecomment-807877142
You could take a look at how timeout is calculated here.
https://github.com/apache/druid/blob/bf20f9e9798417c9293a195690b6adcb48f44d3f/core/src/main/java/org/apache/druid/java/util/emitter/core/HttpPostEmitter.java#L805
And according to the log, I think you might also need to configure following property.
|Property|Description|Default|
|--------|-----------|-------|
|`druid.emitter.http.minHttpTimeoutMillis`|If the speed of filling batches imposes timeout smaller than that, not even trying to send batch to endpoint, because it will likely fail, not being able to send the data that fast. Configure this depending based on emitter/successfulSending/minTimeMs metric. Reasonable values are 10ms..100ms.|0|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] tanisdlj commented on issue #11033: Metrics Emitter timeout after 12-16ms
Posted by GitBox <gi...@apache.org>.
tanisdlj commented on issue #11033:
URL: https://github.com/apache/druid/issues/11033#issuecomment-806614146
In case of "I want more logs", this is copy paste "as is", just removing IPs:
```
Mar 25 11:36:25 druid-queryserver-1 java[20898]: 2021-03-25T11:36:25,457 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [140] ms, timeout [126] ms
Mar 25 11:36:25 druid-queryserver-1 java[20898]: 2021-03-25T11:36:25,457 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Failed to send events to url[http://druid-exporter-1:8080/druid]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 126 ms
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emit(HttpPostEmitter.java:574) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emitBatches(HttpPostEmitter.java:550) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:503) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 126 ms
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: 2021-03-25T11:36:25,758 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [300] ms, timeout [270] ms
Mar 25 11:36:25 druid-queryserver-1 java[20898]: 2021-03-25T11:36:25,758 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Failed to send events to url[http://druid-exporter-1:8080/druid]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 270 ms
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emit(HttpPostEmitter.java:574) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emitBatches(HttpPostEmitter.java:550) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:503) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 270 ms
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: 2021-03-25T11:36:25,957 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [217] ms, timeout [195] ms
Mar 25 11:36:25 druid-queryserver-1 java[20898]: 2021-03-25T11:36:25,957 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Failed to send events to url[http://druid-exporter-1:8080/druid]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 195 ms
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emit(HttpPostEmitter.java:574) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emitBatches(HttpPostEmitter.java:550) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:503) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 195 ms
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:25 druid-queryserver-1 java[20898]: at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: 2021-03-25T11:36:26,157 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [217] ms, timeout [195] ms
Mar 25 11:36:26 druid-queryserver-1 java[20898]: 2021-03-25T11:36:26,158 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Failed to send events to url[http://druid-exporter-1:8080/druid]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 195 ms
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emit(HttpPostEmitter.java:574) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emitBatches(HttpPostEmitter.java:550) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:503) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 195 ms
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: 2021-03-25T11:36:26,358 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [217] ms, timeout [195] ms
Mar 25 11:36:26 druid-queryserver-1 java[20898]: 2021-03-25T11:36:26,358 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Failed to send events to url[http://druid-exporter-1:8080/druid]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 195 ms
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emit(HttpPostEmitter.java:574) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emitBatches(HttpPostEmitter.java:550) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:503) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 195 ms
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:26 druid-queryserver-1 java[20898]: at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: 2021-03-25T11:36:27,657 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [388] ms, timeout [349] ms
Mar 25 11:36:27 druid-queryserver-1 java[20898]: 2021-03-25T11:36:27,658 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Failed to send events to url[http://druid-exporter-1:8080/druid]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 349 ms
Mar 25 11:36:27 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emit(HttpPostEmitter.java:574) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emitBatches(HttpPostEmitter.java:550) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:503) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 349 ms
Mar 25 11:36:27 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:27 druid-queryserver-1 java[20898]: at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
Mar 25 11:36:33 druid-queryserver-1 java[20898]: 2021-03-25T11:36:33,758 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [303] ms, timeout [606] ms
Mar 25 11:36:33 druid-queryserver-1 java[20898]: 2021-03-25T11:36:33,758 WARN [HttpPostEmitter-1] org.apache.druid.java.util.common.RetryUtils - Retrying (1 of 2) in 992ms.
Mar 25 11:36:33 druid-queryserver-1 java[20898]: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 606 ms
Mar 25 11:36:33 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
Mar 25 11:36:33 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
Mar 25 11:36:33 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.tryEmitOneFailedBuffer(HttpPostEmitter.java:639) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:504) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:33 druid-queryserver-1 java[20898]: Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 606 ms
Mar 25 11:36:33 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:33 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:33 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:33 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:33 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:33 druid-queryserver-1 java[20898]: at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
Mar 25 11:36:35 druid-queryserver-1 java[20898]: 2021-03-25T11:36:35,358 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [303] ms, timeout [606] ms
Mar 25 11:36:35 druid-queryserver-1 java[20898]: 2021-03-25T11:36:35,358 WARN [HttpPostEmitter-1] org.apache.druid.java.util.common.RetryUtils - Retrying (2 of 2) in 2,697ms.
Mar 25 11:36:35 druid-queryserver-1 java[20898]: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 606 ms
Mar 25 11:36:35 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
Mar 25 11:36:35 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
Mar 25 11:36:35 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.tryEmitOneFailedBuffer(HttpPostEmitter.java:639) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:35 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:504) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:35 druid-queryserver-1 java[20898]: Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 606 ms
Mar 25 11:36:35 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:35 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:35 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:35 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:35 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:35 druid-queryserver-1 java[20898]: at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: 2021-03-25T11:36:38,158 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [20] ms, timeout [40] ms
Mar 25 11:36:38 druid-queryserver-1 java[20898]: 2021-03-25T11:36:38,158 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Failed to send events to url[http://druid-exporter-1:8080/druid]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 40 ms
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.tryEmitOneFailedBuffer(HttpPostEmitter.java:639) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:504) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 40 ms
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: 2021-03-25T11:36:38,257 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [20] ms, timeout [40] ms
Mar 25 11:36:38 druid-queryserver-1 java[20898]: 2021-03-25T11:36:38,257 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Failed to send events to url[http://druid-exporter-1:8080/druid]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 40 ms
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emit(HttpPostEmitter.java:574) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emitBatches(HttpPostEmitter.java:550) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:503) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 40 ms
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: 2021-03-25T11:36:38,357 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [20] ms, timeout [40] ms
Mar 25 11:36:38 druid-queryserver-1 java[20898]: 2021-03-25T11:36:38,357 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Failed to send events to url[http://druid-exporter-1:8080/druid]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 40 ms
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emit(HttpPostEmitter.java:574) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emitBatches(HttpPostEmitter.java:550) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:503) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 40 ms
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:38 druid-queryserver-1 java[20898]: at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: 2021-03-25T11:36:52,557 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [392] ms, timeout [784] ms
Mar 25 11:36:52 druid-queryserver-1 java[20898]: 2021-03-25T11:36:52,557 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Failed to send events to url[http://druid-exporter-1:8080/druid]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 784 ms
Mar 25 11:36:52 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emit(HttpPostEmitter.java:574) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emitBatches(HttpPostEmitter.java:550) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:503) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 784 ms
Mar 25 11:36:52 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:52 druid-queryserver-1 java[20898]: at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
Mar 25 11:36:53 druid-queryserver-1 java[20898]: 2021-03-25T11:36:53,358 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [392] ms, timeout [784] ms
Mar 25 11:36:53 druid-queryserver-1 java[20898]: 2021-03-25T11:36:53,359 WARN [HttpPostEmitter-1] org.apache.druid.java.util.common.RetryUtils - Retrying (1 of 2) in 993ms.
Mar 25 11:36:53 druid-queryserver-1 java[20898]: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 784 ms
Mar 25 11:36:53 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
Mar 25 11:36:53 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
Mar 25 11:36:53 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:53 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:53 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:53 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:53 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:53 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:53 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:53 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:53 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:53 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.tryEmitOneFailedBuffer(HttpPostEmitter.java:639) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:53 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:504) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:36:53 druid-queryserver-1 java[20898]: Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 784 ms
Mar 25 11:36:53 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:53 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:36:53 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:53 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:53 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:36:53 druid-queryserver-1 java[20898]: at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
Mar 25 11:37:04 druid-queryserver-1 java[20898]: 2021-03-25T11:37:04,557 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [231] ms, timeout [462] ms
Mar 25 11:37:04 druid-queryserver-1 java[20898]: 2021-03-25T11:37:04,557 WARN [HttpPostEmitter-1] org.apache.druid.java.util.common.RetryUtils - Retrying (1 of 2) in 872ms.
Mar 25 11:37:04 druid-queryserver-1 java[20898]: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 462 ms
Mar 25 11:37:04 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
Mar 25 11:37:04 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
Mar 25 11:37:04 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:37:04 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:04 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:04 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:04 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:04 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:04 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:04 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:04 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:04 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.tryEmitOneFailedBuffer(HttpPostEmitter.java:639) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:04 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:504) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:04 druid-queryserver-1 java[20898]: Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 462 ms
Mar 25 11:37:04 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:37:04 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:37:04 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:37:04 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:37:04 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:37:04 druid-queryserver-1 java[20898]: at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: 2021-03-25T11:37:09,957 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [7] ms, timeout [14] ms
Mar 25 11:37:09 druid-queryserver-1 java[20898]: 2021-03-25T11:37:09,958 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Failed to send events to url[http://druid-exporter-1:8080/druid]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 14 ms
Mar 25 11:37:09 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emit(HttpPostEmitter.java:574) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emitBatches(HttpPostEmitter.java:550) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:503) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 14 ms
Mar 25 11:37:09 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:37:09 druid-queryserver-1 java[20898]: at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: 2021-03-25T11:37:10,426 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [159] ms, timeout [318] ms
Mar 25 11:37:10 druid-queryserver-1 java[20898]: 2021-03-25T11:37:10,426 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Failed to send events to url[http://druid-exporter-1:8080/druid]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 318 ms
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emit(HttpPostEmitter.java:574) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emitBatches(HttpPostEmitter.java:550) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:503) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 318 ms
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: 2021-03-25T11:37:10,957 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [254] ms, timeout [508] ms
Mar 25 11:37:10 druid-queryserver-1 java[20898]: 2021-03-25T11:37:10,957 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Failed to send events to url[http://druid-exporter-1:8080/druid]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 508 ms
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emit(HttpPostEmitter.java:574) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emitBatches(HttpPostEmitter.java:550) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:503) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 508 ms
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:37:10 druid-queryserver-1 java[20898]: at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: 2021-03-25T11:37:33,158 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Timing out emitter batch send, last batch fill time [11] ms, timeout [22] ms
Mar 25 11:37:33 druid-queryserver-1 java[20898]: 2021-03-25T11:37:33,158 ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Failed to send events to url[http://druid-exporter-1:8080/druid]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 22 ms
Mar 25 11:37:33 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_275]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_275]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:759) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:672) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:668) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:666) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emit(HttpPostEmitter.java:574) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emitBatches(HttpPostEmitter.java:550) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:503) [druid-core-0.20.1.jar:0.20.1]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: Caused by: java.util.concurrent.TimeoutException: Request timeout to druid-exporter-1/XX.XX.XX.XX:8080 after 22 ms
Mar 25 11:37:33 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50) ~[async-http-client-2.5.3.jar:?]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
Mar 25 11:37:33 druid-queryserver-1 java[20898]: at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org