You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2019/12/23 16:44:30 UTC
[GitHub] [druid] terry19850829 opened a new issue #9091: Compaction task
failed after hdfs namenode from single namenode to HA
terry19850829 opened a new issue #9091: Compaction task failed after hdfs namenode from single namenode to HA
URL: https://github.com/apache/druid/issues/9091
### Affected Version
0.16.0
### Description
when I test the 0.16 version druid, the kafka indexing task works well for single namenode.
But after the namenode mode change to HA, the datasource's compaction task throw unknowhostException.
I checked the peon's log, the segments to be merged records the single namenode file path.
```
....
2019-12-23T16:11:27,222 INFO [NodeTypeWatcher[OVERLORD]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeTypeWatcher - Received INITIALIZED in node watcher.
2019-12-23T16:11:27,225 INFO [NodeTypeWatcher[COORDINATOR]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeTypeWatcher - Node[localhost:8081:DiscoveryDruidNode{druidNode=DruidNode{serviceName='druid/coordinator', host='localhost', bindOnHost=false, port=-1, plaintextPort=8081, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}, nodeType='COORDINATOR', services={}}] appeared.
2019-12-23T16:11:27,225 INFO [NodeTypeWatcher[COORDINATOR]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeTypeWatcher - Received INITIALIZED in node watcher.
2019-12-23T16:11:27,308 INFO [main] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Running with task: {
"type" : "compact",
"id" : "compact_trade_ds_2019-12-23T16:11:21.750Z",
"resource" : {
"availabilityGroup" : "compact_trade_ds_2019-12-23T16:11:21.750Z",
"requiredCapacity" : 1
},
"dataSource" : "trade_ds",
"interval" : null,
"segments" : [ {
"dataSource" : "trade_ds",
"interval" : "2019-12-21T20:00:00.000Z/2019-12-21T21:00:00.000Z",
"version" : "2019-12-21T20:01:07.206Z",
"loadSpec" : {
"type" : "hdfs",
"path" : "hdfs://hb3-test-data-hdfs-001:9000/druid_one_for_all/segments/trade_ds/20191221T200000.000Z_20191221T210000.000Z/2019-12-21T20_01_07.206Z/0_99c160f7-026e-48fe-a616-130f69eb57eb_index.zip"
},
"dimensions" : "merchant_id,payway",
"metrics" : "cnt,amount,ef_amount,ef_cnt,success_cnt,success_amount",
"shardSpec" : {
"type" : "numbered",
"partitionNum" : 0,
"partitions" : 0
},
"binaryVersion" : 9,
"size" : 5994,
"identifier" : "trade_ds_2019-12-21T20:00:00.000Z_2019-12-21T21:00:00.000Z_2019-12-21T20:01:07.206Z"
}, {
"dataSource" : "trade_ds",
"interval" : "2019-12-21T20:00:00.000Z/2019-12-21T21:00:00.000Z",
"version" : "2019-12-21T20:01:07.206Z",
"loadSpec" : {
"type" : "hdfs",
"path" : "hdfs://hb3-test-data-hdfs-001:9000/druid_one_for_all/segments/trade_ds/20191221T200000.000Z_20191221T210000.000Z/2019-12-21T20_01_07.206Z/1_6ddb004b-edfe-4ed2-8e67-47706469cada_index.zip"
},
....
2019-12-23T16:11:27,635 WARN [task-runner-0-priority-0] org.apache.hadoop.ipc.Client - Failed to connect to server: hb3-test-data-hdfs-001/192.168.100.48:9000: try once and fail.
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_222]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:1.8.0_222]
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) ~[hadoop-common-2.8.3.jar:?]
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) ~[hadoop-common-2.8.3.jar:?]
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:685) [hadoop-common-2.8.3.jar:?]
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:788) [hadoop-common-2.8.3.jar:?]
at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:410) [hadoop-common-2.8.3.jar:?]
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1550) [hadoop-common-2.8.3.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1381) [hadoop-common-2.8.3.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1345) [hadoop-common-2.8.3.jar:?]
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) [hadoop-common-2.8.3.jar:?]
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) [hadoop-common-2.8.3.jar:?]
at com.sun.proxy.$Proxy67.getFileInfo(Unknown Source) [?:?]
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:796) [hadoop-hdfs-client-2.8.3.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_222]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_222]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409) [hadoop-common-2.8.3.jar:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) [hadoop-common-2.8.3.jar:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155) [hadoop-common-2.8.3.jar:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) [hadoop-common-2.8.3.jar:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346) [hadoop-common-2.8.3.jar:?]
at com.sun.proxy.$Proxy68.getFileInfo(Unknown Source) [?:?]
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1717) [hadoop-hdfs-client-2.8.3.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1437) [hadoop-hdfs-client-2.8.3.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1434) [hadoop-hdfs-client-2.8.3.jar:?]
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) [hadoop-common-2.8.3.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1434) [hadoop-hdfs-client-2.8.3.jar:?]
at org.apache.hadoop.fs.FileSystem.isDirectory(FileSystem.java:1450) [hadoop-common-2.8.3.jar:?]
at org.apache.druid.storage.hdfs.HdfsDataSegmentPuller.getSegmentFiles(HdfsDataSegmentPuller.java:177) [druid-hdfs-storage-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.storage.hdfs.HdfsLoadSpec.loadSegment(HdfsLoadSpec.java:62) [druid-hdfs-storage-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:236) [druid-server-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:224) [druid-server-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:185) [druid-server-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:164) [druid-server-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.indexing.common.TaskToolbox.fetchSegments(TaskToolbox.java:247) [druid-indexing-service-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.indexing.common.task.CompactionTask.prepareSegments(CompactionTask.java:541) [druid-indexing-service-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.indexing.common.task.CompactionTask.createIngestionSchema(CompactionTask.java:415) [druid-indexing-service-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.indexing.common.task.CompactionTask.runTask(CompactionTask.java:315) [druid-indexing-service-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.indexing.common.task.AbstractBatchIndexTask.run(AbstractBatchIndexTask.java:137) [druid-indexing-service-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:419) [druid-indexing-service-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:391) [druid-indexing-service-0.16.0-incubating.jar:0.16.0-incubating]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
2019-12-23T16:11:27,653 ERROR [task-runner-0-priority-0] org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager - Failed to load segment in current location [/app/apache-druid-0.16.0-incubating/var/druid/task/compact_trade_ds_2019-12-23T16:11:21.750Z/work], try next location if any: {class=org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager, exceptionType=class org.apache.druid.segment.loading.SegmentLoadingException, exceptionMessage=Error loading [hdfs://hb3-test-data-hdfs-001:9000/druid_one_for_all/segments/trade_ds/20191221T200000.000Z_20191221T210000.000Z/2019-12-21T20_01_07.206Z/0_99c160f7-026e-48fe-a616-130f69eb57eb_index.zip], location=/app/apache-druid-0.16.0-incubating/var/druid/task/compact_trade_ds_2019-12-23T16:11:21.750Z/work}
org.apache.druid.segment.loading.SegmentLoadingException: Error loading [hdfs://hb3-test-data-hdfs-001:9000/druid_one_for_all/segments/trade_ds/20191221T200000.000Z_20191221T210000.000Z/2019-12-21T20_01_07.206Z/0_99c160f7-026e-48fe-a616-130f69eb57eb_index.zip]
at org.apache.druid.storage.hdfs.HdfsDataSegmentPuller.getSegmentFiles(HdfsDataSegmentPuller.java:272) ~[?:?]
at org.apache.druid.storage.hdfs.HdfsLoadSpec.loadSegment(HdfsLoadSpec.java:62) ~[?:?]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:236) ~[druid-server-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:224) ~[druid-server-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:185) [druid-server-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:164) [druid-server-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.indexing.common.TaskToolbox.fetchSegments(TaskToolbox.java:247) [druid-indexing-service-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.indexing.common.task.CompactionTask.prepareSegments(CompactionTask.java:541) [druid-indexing-service-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.indexing.common.task.CompactionTask.createIngestionSchema(CompactionTask.java:415) [druid-indexing-service-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.indexing.common.task.CompactionTask.runTask(CompactionTask.java:315) [druid-indexing-service-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.indexing.common.task.AbstractBatchIndexTask.run(AbstractBatchIndexTask.java:137) [druid-indexing-service-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:419) [druid-indexing-service-0.16.0-incubating.jar:0.16.0-incubating]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:391) [druid-indexing-service-0.16.0-incubating.jar:0.16.0-incubating]
.......
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org