You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/05/19 10:05:32 UTC
[GitHub] [druid] Jonarzz commented on issue #7099: broker metadata refresh failing
Jonarzz commented on issue #7099:
URL: https://github.com/apache/druid/issues/7099#issuecomment-630721470
We're experiancing a similar issue on Druid 0.17.0:
```
2020-05-13T16:38:59,797 INFO [ServerInventoryView-0] org.apache.druid.client.BatchServerInventoryView - Server Disappeared[DruidServerMetadata{name='hostname:8101', hostAndPort='hostname:8101', hostAndTlsPort='null', maxSize=0, tier='_default_tier', type=indexer-executor, priority=0}]
2020-05-13T16:38:59,797 INFO [ServerInventoryView-0] org.apache.druid.client.BatchServerInventoryView - Server Disappeared[DruidServerMetadata{name='hostname:8101', hostAndPort='hostname:8101', hostAndTlsPort='null', maxSize=0, tier='_default_tier', type=indexer-executor, priority=0}]
2020-05-13T16:43:03,020 WARN [HttpClient-Netty-Worker-70] org.apache.druid.java.util.http.client.NettyHttpClient - [POST http://hostname2:8101/druid/v2/] Channel disconnected before response complete
2020-05-13T16:43:03,020 WARN [HttpClient-Netty-Worker-71] org.apache.druid.java.util.http.client.NettyHttpClient - [POST http://hostname2:8101/druid/v2/] Channel disconnected before response complete
2020-05-13T16:43:03,020 WARN [HttpClient-Netty-Worker-69] org.apache.druid.java.util.http.client.NettyHttpClient - [POST http://hostname2:8101/druid/v2/] Channel disconnected before response complete
2020-05-13T16:43:03,020 WARN [HttpClient-Netty-Worker-72] org.apache.druid.java.util.http.client.NettyHttpClient - [POST http://hostname2:8101/druid/v2/] Channel disconnected before response complete
2020-05-13T16:43:03,029 WARN [qtp1952473604-259[groupBy_[eff_test]_30788e21-c073-41c0-8ad4-843f1b8f2f77]] org.apache.druid.server.QueryLifecycle - Exception while processing queryId [30788e21-c073-41c0-8ad4-843f1b8f2f77] (QueryInterruptedException{msg=org.jboss.netty.channel.ChannelException: Channel disconnected, code=Unknown exception, class=java.util.concurrent.ExecutionException, host=hostname2:8101})
2020-05-13T16:43:03,029 WARN [qtp1952473604-296[groupBy_[eff_test]_224dba66-3d1d-41cd-b83a-7d8b345f44ad]] org.apache.druid.server.QueryLifecycle - Exception while processing queryId [224dba66-3d1d-41cd-b83a-7d8b345f44ad] (QueryInterruptedException{msg=org.jboss.netty.channel.ChannelException: Channel disconnected, code=Unknown exception, class=java.util.concurrent.ExecutionException, host=hostname2:8101})
2020-05-13T16:43:03,029 WARN [DruidSchema-Cache-0] org.apache.druid.server.QueryLifecycle - Exception while processing queryId [361a9aa2-7ed7-4363-a64c-65b1813b66a5] (QueryInterruptedException{msg=org.jboss.netty.channel.ChannelException: Channel disconnected, code=Unknown exception, class=java.util.concurrent.ExecutionException, host=hostname2:8101})
...
2020-05-13T16:43:03,029 WARN [DruidSchema-Cache-0] org.apache.druid.sql.calcite.schema.DruidSchema - Metadata refresh failed, trying again soon.
org.apache.druid.query.QueryInterruptedException: org.jboss.netty.channel.ChannelException: Channel disconnected
at org.apache.druid.client.JsonParserIterator.interruptQuery(JsonParserIterator.java:189) ~[druid-server-0.17.0.jar:0.17.0]
at org.apache.druid.client.JsonParserIterator.init(JsonParserIterator.java:173) ~[druid-server-0.17.0.jar:0.17.0]
at org.apache.druid.client.JsonParserIterator.hasNext(JsonParserIterator.java:95) ~[druid-server-0.17.0.jar:0.17.0]
at org.apache.druid.java.util.common.guava.BaseSequence.makeYielder(BaseSequence.java:89) ~[druid-core-0.17.0.jar:0.17.0]
at org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:69) ~[druid-core-0.17.0.jar:0.17.0]
at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[druid-core-0.17.0.jar:0.17.0]
at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$ResultBatch.fromSequence(ParallelMergeCombiningSequence.java:847) ~[druid-core-0.17.0.jar:0.17.0]
at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$SequenceBatcher.block(ParallelMergeCombiningSequence.java:897) ~[druid-core-0.17.0.jar:0.17.0]
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3313) ~[?:1.8.0_242]
at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$SequenceBatcher.getBatchYielder(ParallelMergeCombiningSequence.java:886) ~[druid-core-0.17.0.jar:0.17.0]
at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$YielderBatchedResultsCursor.initialize(ParallelMergeCombiningSequence.java:993) ~[druid-core-0.17.0.jar:0.17.0]
at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$PrepareMergeCombineInputsAction.compute(ParallelMergeCombiningSequence.java:702) ~[druid-core-0.17.0.jar:0.17.0]
at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189) ~[?:1.8.0_242]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_242]
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) ~[?:1.8.0_242]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) ~[?:1.8.0_242]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) ~[?:1.8.0_242]
Caused by: java.util.concurrent.ExecutionException: org.jboss.netty.channel.ChannelException: Channel disconnected
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:272) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96) ~[guava-16.0.1.jar:?]
at org.apache.druid.client.JsonParserIterator.init(JsonParserIterator.java:138) ~[druid-server-0.17.0.jar:0.17.0]
... 15 more
Caused by: org.jboss.netty.channel.ChannelException: Channel disconnected
at org.apache.druid.java.util.http.client.NettyHttpClient$1.channelDisconnected(NettyHttpClient.java:351) ~[druid-core-0.17.0.jar:0.17.0]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:208) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:208) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.cleanup(ReplayingDecoder.java:570) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:365) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[netty-3.10.6.Final.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
2020-05-13T16:43:10,235 WARN [ServerInventoryView-0] org.apache.druid.client.BatchServerInventoryView - Not adding or running callbacks for existing segment[eff_test_2020-05-13T00:00:00.000Z_2020-05-14T00:00:00.000Z_2020-05-12T22:38:57.174Z_3] on server[hostname2:8101]
```
Nothing special going on the aforementioned machines - at least not in logs:
```
2020-05-13T16:37:48,563 INFO [forking-task-runner-0] org.apache.druid.indexing.overlord.ForkingTaskRunner - Logging task index_kafka_eff_test_269717372a47df9_hippkgkp output to: var/druid/task/index_kafka_eff_test_269717372a47df9_hippkgkp/log
2020-05-13T16:44:56,880 INFO [forking-task-runner-0-[index_kafka_eff_test_269717372a47df9_hippkgkp]] org.apache.druid.indexing.overlord.ForkingTaskRunner - Process exited with status[3] for task: index_kafka_eff_test_269717372a47df9_hippkgkp
```
Is there a way to find the cause of this issue? Seems to occur randomly (happened twice in 20 hours span).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org