You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Tianyi Wang (JIRA)" <ji...@apache.org> on 2018/05/09 23:03:00 UTC
[jira] [Comment Edited] (IMPALA-6394) GVO failed in "Waiting for
HDFS replication"
[ https://issues.apache.org/jira/browse/IMPALA-6394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16469647#comment-16469647 ]
Tianyi Wang edited comment on IMPALA-6394 at 5/9/18 11:02 PM:
--------------------------------------------------------------
I think the root cause of this issue is that the HDFS property dfs.namenode.redundancy.considerLoad defaults to true. Namenode would exclude a data node from being considered for replication if the number of connections on that data node is more than 2x greater than the average among all data nodes.
I found it while debugging IMPALA-7004.
was (Author: tianyiwang):
I think the root cause of this issue is that the HDFS property dfs.namenode.redundancy.considerLoad defaults to true. Namenode would exclude a data node from being considered for replication if the number of connections on that data node is more than 2x greater than the average among all data nodes.
I found it while debugging IMPALA-7003.
> GVO failed in "Waiting for HDFS replication"
> --------------------------------------------
>
> Key: IMPALA-6394
> URL: https://issues.apache.org/jira/browse/IMPALA-6394
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 2.12.0
> Reporter: Tim Armstrong
> Assignee: Tianyi Wang
> Priority: Critical
> Labels: broken-build, flaky
> Fix For: Impala 2.12.0
>
> Attachments: wait-hdfs-replication.log
>
>
> A GVO failed at this step in data loading: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/942/console . I attached the log tail is:
> {noformat}
> /test-warehouse/zipcode_incomes_seq_gzip/000000_0: Under replicated BP-611512709-127.0.0.1-1515787500736:blk_1073751054_10230. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
> ..
> /test-warehouse/zipcode_incomes_seq_record_def/000000_0: Under replicated BP-611512709-127.0.0.1-1515787500736:blk_1073750916_10092. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
> ..
> /test-warehouse/zipcode_incomes_seq_record_snap/000000_0: Under replicated BP-611512709-127.0.0.1-1515787500736:blk_1073751714_10890. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
> ...
> /test-warehouse/zipcode_incomes_text_gzip/000000_0.gz: Under replicated BP-611512709-127.0.0.1-1515787500736:blk_1073748020_7196. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
> ..
> /test-warehouse/zipcode_incomes_text_lzo/000000_0.lzo.index: Under replicated BP-611512709-127.0.0.1-1515787500736:blk_1073752971_12147. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
> .Status: HEALTHY
> Total size: 8422045399 B
> Total dirs: 8441
> Total files: 10228
> Total symlinks: 0
> Total blocks (validated): 10249 (avg. block size 821743 B)
> Minimally replicated blocks: 10249 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 1523 (14.859986 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 3
> Average block replication: 2.8514001
> Corrupt blocks: 0
> Missing replicas: 1523 (4.9533286 %)
> Number of data-nodes: 3
> Number of racks: 1
> FSCK ended at Fri Jan 12 20:57:37 UTC 2018 in 140 milliseconds
> The filesystem under path '/test-warehouse' is HEALTHY
> Some HDFS blocks are still under replicated after 30s.
> Some tests cannot pass without fully replicated blocks (IMPALA-3887).
> Failing the data loading.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org