You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/11/26 03:09:44 UTC
[GitHub] [druid] silverflow opened a new issue #10607: When I ingestion, It'll reset if it's over 10 million rows.
silverflow opened a new issue #10607:
URL: https://github.com/apache/druid/issues/10607
Please provide a detailed title (e.g. "Broker crashes when using TopN query with Bound filter" instead of just "Broker crashes").
### Affected Version
**apache druid 0.20.0**
### Description
I try to load the data using kafka. but Task was failed. It was reset to task after almost 10,442,979 row.
There is an task error log.
```
2020-11-26T02:01:04,048 INFO [parent-monitor-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Triggering JVM shutdown.
2020-11-26T02:01:04,051 INFO [Thread-25] org.apache.druid.cli.CliPeon - Running shutdown hook
2020-11-26T02:01:04,051 INFO [Thread-25] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [ANNOUNCEMENTS]
2020-11-26T02:01:04,054 INFO [Thread-25] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/announcements/localhost:8101]
2020-11-26T02:01:04,079 INFO [Thread-25] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:06:50.068Z_8e88ca0552d247ae96b6fbbec1328d635]
2020-11-26T02:01:04,090 INFO [Thread-25] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:28:12.297Z_36d5d67f7e9d466b9b38c1a7e29716af8]
2020-11-26T02:01:04,092 INFO [Thread-25] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:38:20.561Z_1fa942f9e8d8412ca44d227d08565ab61]
2020-11-26T02:01:04,093 INFO [Thread-25] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:14:09.148Z_0bcc28e53a85461d81d683f5ace4f4ed6]
2020-11-26T02:01:04,097 INFO [Thread-25] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:45:10.147Z_42363f4e23754b69ad78fc9f1750e30b2]
2020-11-26T02:01:04,098 INFO [Thread-25] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:21:34.428Z_fad4b4fb98374ea4a7b4f8ef565687ed7]
2020-11-26T02:01:04,101 INFO [Thread-25] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:30:46.379Z_5b6fd3bc23a14a6096c14a38097a50200]
2020-11-26T02:01:04,102 INFO [Thread-25] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:00:01.340Z_7b3f1a77464849f9850116421dbe7bca4]
2020-11-26T02:01:04,104 INFO [Thread-25] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:52:49.096Z_d2bbb9966c5d42249fd6484d8ff4170b3]
2020-11-26T02:01:04,112 INFO [Thread-25] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/internal-discovery/PEON/localhost:8101]
2020-11-26T02:01:04,114 INFO [Thread-25] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER]
2020-11-26T02:01:04,125 INFO [Thread-25] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@5315266{HTTP/1.1, (http/1.1)}{0.0.0.0:8101}
2020-11-26T02:01:04,126 INFO [Thread-25] org.eclipse.jetty.server.session - node0 Stopped scavenging
2020-11-26T02:01:04,135 INFO [Thread-25] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@1a0ffb9b{/,null,UNAVAILABLE}
2020-11-26T02:01:04,150 INFO [Thread-25] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL]
2020-11-26T02:01:04,150 INFO [Thread-25] org.apache.druid.server.coordination.ZkCoordinator - Stopping ZkCoordinator for [DruidServerMetadata{name='localhost:8101', hostAndPort='localhost:8101', hostAndTlsPort='null', maxSize=0, tier='_default_tier', type=indexer-executor, priority=0}]
2020-11-26T02:01:04,151 INFO [Thread-25] org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopping...
2020-11-26T02:01:04,151 INFO [Thread-25] org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopped.
2020-11-26T02:01:04,152 INFO [Thread-25] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Starting graceful shutdown of task[index_kafka_news_29e33becd2f8188_gpkgficc].
2020-11-26T02:01:04,152 INFO [Thread-25] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Stopping forcefully (status: [PUBLISHING])
2020-11-26T02:01:04,162 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Error while publishing segments for sequenceNumber[SequenceMetadata{sequenceId=0, sequenceName='index_kafka_news_29e33becd2f8188_0', assignments=[], startOffsets={0=0}, exclusiveStartPartitions=[], endOffsets={0=10451319}, sentinel=false, checkpointed=true}]
java.util.concurrent.CancellationException: Task was cancelled.
at com.google.common.util.concurrent.AbstractFuture.cancellationExceptionWithCause(AbstractFuture.java:392) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:306) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1170) [guava-16.0.1.jar:?]
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) [guava-16.0.1.jar:?]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) [guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture.cancel(AbstractFuture.java:134) [guava-16.0.1.jar:?]
at com.google.common.util.concurrent.Futures$ChainingListenableFuture.cancel(Futures.java:826) [guava-16.0.1.jar:?]
at com.google.common.util.concurrent.Futures$CombinedFuture$1.run(Futures.java:1505) [guava-16.0.1.jar:?]
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) [guava-16.0.1.jar:?]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) [guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture.cancel(AbstractFuture.java:134) [guava-16.0.1.jar:?]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:841) [druid-indexing-service-0.20.0.jar:0.20.0]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:267) [druid-indexing-service-0.20.0.jar:0.20.0]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:145) [druid-indexing-service-0.20.0.jar:0.20.0]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:451) [druid-indexing-service-0.20.0.jar:0.20.0]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:423) [druid-indexing-service-0.20.0.jar:0.20.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
Caused by: java.util.concurrent.CancellationException: Future.cancel() was called.
at com.google.common.util.concurrent.AbstractFuture$Sync.complete(AbstractFuture.java:378) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture$Sync.cancel(AbstractFuture.java:355) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture.cancel(AbstractFuture.java:131) ~[guava-16.0.1.jar:?]
... 15 more
2020-11-26T02:01:04,189 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[news_2019-10-08T00:00:00.000Z_2019-10-09T00:00:00.000Z_2020-11-25T07:03:21.633Z_2] at path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:38:20.561Z_1fa942f9e8d8412ca44d227d08565ab61]
2020-11-26T02:01:04,191 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[news_2019-02-16T00:00:00.000Z_2019-02-17T00:00:00.000Z_2020-11-25T07:06:08.110Z_2] at path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:38:20.561Z_1fa942f9e8d8412ca44d227d08565ab61]
2020-11-26T02:01:04,193 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[news_2020-03-11T00:00:00.000Z_2020-03-12T00:00:00.000Z_2020-11-25T07:02:31.447Z_2] at path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:38:20.561Z_1fa942f9e8d8412ca44d227d08565ab61]
2020-11-26T02:01:04,194 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[news_2020-04-27T00:00:00.000Z_2020-04-28T00:00:00.000Z_2020-11-25T07:51:45.769Z_2] at path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:28:12.297Z_36d5d67f7e9d466b9b38c1a7e29716af8]
2020-11-26T02:01:04,195 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[news_2020-01-02T00:00:00.000Z_2020-01-03T00:00:00.000Z_2020-11-25T07:48:09.421Z_2] at path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:21:34.428Z_fad4b4fb98374ea4a7b4f8ef565687ed7]
2020-11-26T02:01:04,195 INFO [LookupExtractorFactoryContainerProvider-MainThread] org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited. Lookup notices are not handled anymore.
2020-11-26T02:01:04,197 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[news_2019-01-31T00:00:00.000Z_2019-02-01T00:00:00.000Z_2020-11-25T07:11:34.639Z_2] at path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:45:10.147Z_42363f4e23754b69ad78fc9f1750e30b2]
2020-11-26T02:01:04,199 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[news_2019-08-27T00:00:00.000Z_2019-08-28T00:00:00.000Z_2020-11-25T06:57:55.971Z_2] at path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:30:46.379Z_5b6fd3bc23a14a6096c14a38097a50200]
2020-11-26T02:01:04,200 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[news_2019-07-02T00:00:00.000Z_2019-07-03T00:00:00.000Z_2020-11-25T07:17:01.648Z_2] at path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:52:49.096Z_d2bbb9966c5d42249fd6484d8ff4170b3]
2020-11-26T02:01:04,203 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[news_2020-02-12T00:00:00.000Z_2020-02-13T00:00:00.000Z_2020-11-25T07:22:34.133Z_2] at path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:52:49.096Z_d2bbb9966c5d42249fd6484d8ff4170b3]
2020-11-26T02:01:04,205 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[news_2019-01-24T00:00:00.000Z_2019-01-25T00:00:00.000Z_2020-11-25T07:26:56.883Z_2] at path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:00:01.340Z_7b3f1a77464849f9850116421dbe7bca4]
2020-11-26T02:01:04,208 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[news_2019-01-16T00:00:00.000Z_2019-01-17T00:00:00.000Z_2020-11-25T07:53:19.373Z_2] at path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:28:12.297Z_36d5d67f7e9d466b9b38c1a7e29716af8]
2020-11-26T02:01:04,210 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[news_2019-07-20T00:00:00.000Z_2019-07-21T00:00:00.000Z_2020-11-25T07:25:05.005Z_2] at path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:00:01.340Z_7b3f1a77464849f9850116421dbe7bca4]
2020-11-26T02:01:04,218 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[news_2019-07-24T00:00:00.000Z_2019-07-25T00:00:00.000Z_2020-11-25T07:29:21.789Z_2] at path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:00:01.340Z_7b3f1a77464849f9850116421dbe7bca4]
2020-11-26T02:01:04,220 INFO [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
2020-11-26T02:01:04,222 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[news_2020-05-01T00:00:00.000Z_2020-05-02T00:00:00.000Z_2020-11-25T07:35:31.105Z_2] at path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:06:50.068Z_8e88ca0552d247ae96b6fbbec1328d635]
2020-11-26T02:01:04,224 INFO [Thread-25] org.apache.zookeeper.ZooKeeper - Session: 0x100037867f60030 closed
2020-11-26T02:01:04,224 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x100037867f60030
2020-11-26T02:01:04,224 INFO [Thread-25] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT]
2020-11-26T02:01:04,223 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[news_2019-03-11T00:00:00.000Z_2019-03-12T00:00:00.000Z_2020-11-25T07:47:58.558Z_2] at path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:21:34.428Z_fad4b4fb98374ea4a7b4f8ef565687ed7]
```
I want to know why this happens.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] SrinathRayala edited a comment on issue #10607: When I ingest, It'll reset if it's over 10 million rows.
Posted by GitBox <gi...@apache.org>.
SrinathRayala edited a comment on issue #10607:
URL: https://github.com/apache/druid/issues/10607#issuecomment-751320120
Hi,Did you resolve the above issue?Im also getting same issue for druid matviews. Can please let me know how you resolved?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] silverflow edited a comment on issue #10607: When I ingest, It'll reset if it's over 10 million rows.
Posted by GitBox <gi...@apache.org>.
silverflow edited a comment on issue #10607:
URL: https://github.com/apache/druid/issues/10607#issuecomment-774808585
Unfortunately, it hasn't been resolved yet. One would guess it's just a Kafka problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] SrinathRayala commented on issue #10607: When I ingest, It'll reset if it's over 10 million rows.
Posted by GitBox <gi...@apache.org>.
SrinathRayala commented on issue #10607:
URL: https://github.com/apache/druid/issues/10607#issuecomment-751320120
Hello- Did you resolve the above issue?Im also getting same issue for druid matviews. Can please me know how you resolved?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] silverflow commented on issue #10607: When I ingest, It'll reset if it's over 10 million rows.
Posted by GitBox <gi...@apache.org>.
silverflow commented on issue #10607:
URL: https://github.com/apache/druid/issues/10607#issuecomment-788698528
@liubo-it Can you tell me how it was possible? I want to do know that
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] silverflow commented on issue #10607: When I ingest, It'll reset if it's over 10 million rows.
Posted by GitBox <gi...@apache.org>.
silverflow commented on issue #10607:
URL: https://github.com/apache/druid/issues/10607#issuecomment-774808585
Unfortunately, it hasn't been resolved yet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] silverflow commented on issue #10607: When I ingest, It'll reset if it's over 10 million rows.
Posted by GitBox <gi...@apache.org>.
silverflow commented on issue #10607:
URL: https://github.com/apache/druid/issues/10607#issuecomment-788698968
is it just waiting?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] liubo-it commented on issue #10607: When I ingest, It'll reset if it's over 10 million rows.
Posted by GitBox <gi...@apache.org>.
liubo-it commented on issue #10607:
URL: https://github.com/apache/druid/issues/10607#issuecomment-781777964
@silverflow Hi, apache druid 0.20.0 , I have the same problem,But the final state of my task is success。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org