You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Tim Armstrong (JIRA)" <ji...@apache.org> on 2018/06/05 19:48:00 UTC
[jira] [Updated] (IMPALA-7122) Data load failure: Failed to replace
a bad datanode on the existing pipeline due to no more good datanodes being
available to try
[ https://issues.apache.org/jira/browse/IMPALA-7122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong updated IMPALA-7122:
----------------------------------
Attachment: impalad.ec2-m2-4xlarge-centos-6-4-0570.vpc.cloudera.com.jenkins.log.INFO.20180604-205755.5587
> Data load failure: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-7122
> URL: https://issues.apache.org/jira/browse/IMPALA-7122
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 3.1.0
> Reporter: Tim Armstrong
> Assignee: Joe McDonnell
> Priority: Critical
> Labels: flaky
> Attachments: impalad.ec2-m2-4xlarge-centos-6-4-0570.vpc.cloudera.com.jenkins.log.INFO.20180604-205755.5587, load-functional-query.log
>
>
> {noformat}
> 20:58:29 Started Loading functional-query data in background; pid 6813.
> 20:58:29 Loading functional-query data (logging to /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/load-functional-query.log)...
> 20:58:29 Started Loading TPC-H data in background; pid 6814.
> 20:58:29 Loading TPC-H data (logging to /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/load-tpch.log)...
> 20:58:29 Started Loading TPC-DS data in background; pid 6815.
> 20:58:29 Loading TPC-DS data (logging to /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/load-tpcds.log)...
> 21:35:26 FAILED (Took: 36 min 57 sec)
> 21:35:26 'load-data functional-query exhaustive' failed. Tail of log:
> 21:35:26 at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
> 21:35:26 at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)
> 21:35:26 18/06/04 21:20:29 WARN hdfs.DataStreamer: Error Recovery for BP-1407206351-127.0.0.1-1528170335185:blk_1073743620_2799 in pipeline [DatanodeInfoWithStorage[127.0.0.1:31000,DS-37cfc57c-ab39-443c-80c9-e440cb18b63d,DISK], DatanodeInfoWithStorage[127.0.0.1:31001,DS-2bc41558-4f2c-460f-ae87-5d1a6acbf42f,DISK], DatanodeInfoWithStorage[127.0.0.1:31002,DS-4ba4d3a0-af31-4eaf-b43d-89b408231481,DISK]]: datanode 0(DatanodeInfoWithStorage[127.0.0.1:31000,DS-37cfc57c-ab39-443c-80c9-e440cb18b63d,DISK]) is bad.
> 21:35:26 18/06/04 21:21:29 INFO hdfs.DataStreamer: Exception in createBlockOutputStream blk_1073743620_2799
> 21:35:26 java.io.IOException: Got error, status=ERROR, status message , ack with firstBadLink as 127.0.0.1:31002
> 21:35:26 at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:134)
> 21:35:26 at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:110)
> 21:35:26 at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1778)
> 21:35:26 at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1507)
> 21:35:26 at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481)
> 21:35:26 at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256)
> 21:35:26 at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667)
> 21:35:26 18/06/04 21:21:29 WARN hdfs.DataStreamer: Error Recovery for BP-1407206351-127.0.0.1-1528170335185:blk_1073743620_2799 in pipeline [DatanodeInfoWithStorage[127.0.0.1:31001,DS-2bc41558-4f2c-460f-ae87-5d1a6acbf42f,DISK], DatanodeInfoWithStorage[127.0.0.1:31002,DS-4ba4d3a0-af31-4eaf-b43d-89b408231481,DISK]]: datanode 1(DatanodeInfoWithStorage[127.0.0.1:31002,DS-4ba4d3a0-af31-4eaf-b43d-89b408231481,DISK]) is bad.
> 21:35:26 18/06/04 21:21:29 WARN hdfs.DataStreamer: DataStreamer Exception
> 21:35:26 java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[127.0.0.1:31001,DS-2bc41558-4f2c-460f-ae87-5d1a6acbf42f,DISK]], original=[DatanodeInfoWithStorage[127.0.0.1:31001,DS-2bc41558-4f2c-460f-ae87-5d1a6acbf42f,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
> 21:35:26 at org.apache.hadoop.hdfs.DataStreamer.findNewDatanode(DataStreamer.java:1304)
> 21:35:26 at org.apache.hadoop.hdfs.DataStreamer.addDatanode2ExistingPipeline(DataStreamer.java:1372)
> 21:35:26 at org.apache.hadoop.hdfs.DataStreamer.handleDatanodeReplacement(DataStreamer.java:1598)
> 21:35:26 at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1499)
> 21:35:26 at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481)
> 21:35:26 at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256)
> 21:35:26 at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667)
> 21:35:26 put: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[127.0.0.1:31001,DS-2bc41558-4f2c-460f-ae87-5d1a6acbf42f,DISK]], original=[DatanodeInfoWithStorage[127.0.0.1:31001,DS-2bc41558-4f2c-460f-ae87-5d1a6acbf42f,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
> 21:35:26 18/06/04 21:24:25 INFO hdfs.DFSClient: Could not complete /test-warehouse/testescape_17_crlf/126._COPYING_ retrying...
> 21:35:26 be loaded.
> 21:35:26 Empty base table load for chars_tiny. Skipping load generation
> 21:35:26 HDFS path: /test-warehouse/widetable_250_cols does not exists or is empty. Data will be loaded.
> 21:35:26 HDFS path: /test-warehouse/widetable_500_cols does not exists or is empty. Data will be loaded.
> 21:35:26 HDFS path: /test-warehouse/widetable_1000_cols does not exists or is empty. Data will be loaded.
> 21:35:26 Skipping 'functional.avro_decimal_tbl' due to include constraint match.
> 21:35:26 Skipping 'functional.no_avro_schema' due to include constraint match.
> 21:35:26 HDFS path: /test-warehouse/table_no_newline does not exists or is empty. Data will be loaded.
> 21:35:26 Empty base table load for table_no_newline. Skipping load generation
> 21:35:26 HDFS path: /test-warehouse/table_no_newline_part does not exists or is empty. Data will be loaded.
> 21:35:26 Empty base table load for table_no_newline_part. Skipping load generation
> 21:35:26 HDFS path: /test-warehouse/testescape_16_lf does not exists or is empty. Data will be loaded.
> 21:35:26 Empty base table load for testescape_16_lf. Skipping load generation
> 21:35:26 HDFS path: /test-warehouse/testescape_16_crlf does not exists or is empty. Data will be loaded.
> 21:35:26 Empty base table load for testescape_16_crlf. Skipping load generation
> 21:35:26 HDFS path: /test-warehouse/testescape_17_lf does not exists or is empty. Data will be loaded.
> 21:35:26 Empty base table load for testescape_17_lf. Skipping load generation
> 21:35:26 Traceback (most recent call last):
> 21:35:26 File "/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/testdata/bin/generate-schema-statements.py", line 836, in <module>
> 21:35:26 test_vectors, sections, include_constraints, exclude_constraints, only_constraints)
> 21:35:26 File "/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/testdata/bin/generate-schema-statements.py", line 595, in generate_statements
> 21:35:26 load = eval_section(section['LOAD'])
> 21:35:26 File "/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/testdata/bin/generate-schema-statements.py", line 533, in eval_section
> 21:35:26 assert p.returncode == 0
> 21:35:26 AssertionError
> 21:35:26 21:35:26 Error generating schema statements for workload: functional-query
> 21:35:26 Background task Loading functional-query data (pid 6813) failed.
> 21:48:12 FAILED (Took: 49 min 43 sec)
> 21:48:12 'load-data tpch core' failed. Tail of log:
> 21:48:13 20:59:36 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-text-none-none.sql
> 21:48:13 20:59:36 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-text-gzip-block.sql
> 21:48:13 20:59:36 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-seq-snap-block.sql
> 21:48:13 20:59:36 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-avro-none-none.sql
> 21:48:13 20:59:36 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-seq-gzip-block.sql
> 21:48:13 20:59:36 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-avro-snap-block.sql
> 21:48:13 20:59:36 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-parquet-none-none.sql
> 21:48:13 20:59:36 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-rc-none-none.sql
> 21:48:13 20:59:53 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-parquet-none-none.sql
> 21:48:13 20:59:53 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-orc-def-block.sql
> 21:48:13 20:59:53 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-text-none-none.sql
> 21:48:13 20:59:53 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-kudu-none-none.sql
> 21:48:13 20:59:53 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-text-gzip-block.sql
> 21:48:13 20:59:53 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-seq-gzip-block.sql
> 21:48:13 20:59:53 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-avro-snap-block.sql
> 21:48:13 20:59:54 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-avro-none-none.sql
> 21:48:13 20:59:59 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-seq-snap-block.sql
> 21:48:13 20:59:59 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-rc-none-none.sql
> 21:48:13 20:59:59 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-orc-def-block.sql
> 21:48:13 21:00:21 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-kudu-none-none.sql
> 21:48:13 21:00:21 Beginning execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-text-none-none.sql
> 21:48:13 21:01:21 Finished execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-text-none-none.sql
> 21:48:13 21:01:21 Beginning execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-text-gzip-block.sql
> 21:48:13 21:01:21 Beginning execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-seq-gzip-block.sql
> 21:48:13 21:01:21 Beginning execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-rc-none-none.sql
> 21:48:13 21:01:21 Beginning execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-avro-none-none.sql
> 21:48:13 21:01:21 Beginning execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-seq-snap-block.sql
> 21:48:13 21:01:21 Beginning execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-avro-snap-block.sql
> 21:48:13 21:01:21 Beginning execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-orc-def-block.sql
> 21:48:13 21:21:55 Finished execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-seq-snap-block.sql
> 21:48:13 21:22:22 Finished execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-orc-def-block.sql
> 21:48:13 21:26:17 Finished execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-text-gzip-block.sql
> 21:48:13 21:28:15 Finished execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-rc-none-none.sql
> 21:48:13 21:29:13 Finished execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-seq-gzip-block.sql
> 21:48:13 21:29:43 Finished execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-avro-snap-block.sql
> 21:48:13 21:37:08 Finished execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-avro-none-none.sql
> 21:48:13 21:37:08 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/invalidate-tpch-core-impala-generated.sql
> 21:48:13 21:37:31 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/invalidate-tpch-core-impala-generated.sql
> 21:48:13 21:37:31 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-impala-generated-kudu-none-none.sql
> 21:48:13 21:37:31 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-impala-generated-parquet-none-none.sql
> 21:48:13 21:48:12 Error executing impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-impala-generated-parquet-none-none.sql See: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-impala-generated-parquet-none-none.sql.log
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org