You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Alex Behm (Code Review)" <ge...@cloudera.org> on 2017/10/31 04:34:02 UTC
[Impala-ASF-CR] IMPALA-3887: Use dfs.namenode.replication.min=3
Alex Behm has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8426
Change subject: IMPALA-3887: Use dfs.namenode.replication.min=3
......................................................................
IMPALA-3887: Use dfs.namenode.replication.min=3
Changes the HDFS and HBase configuration of our
mini-cluster to use:
dfs.namenode.replication.min=3
Several of our tests including planner tests
expect HDFS blocks to have exactly three replicas.
With the default HDFS configuration of
dfs.namenode.replication.min=1, HDFS acks writes
as soon as the min replication is satisfied. This
can lead to test failures because HDFS replication
is racing to replicate blocks while our tests
are executing and expecting to see all replicas.
A more detailed description is in the JIRA.
Testing:
- A core/hdfs ran passed
- A core/hdfs run with data loading passed
Change-Id: Ie09efdd9512cc4f0e338919af42bcdcede2cfb93
---
M fe/src/test/resources/hbase-site.xml.template
M testdata/cluster/node_templates/common/etc/hadoop/conf/hdfs-site.xml.tmpl
2 files changed, 26 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/8426/1
--
To view, visit http://gerrit.cloudera.org:8080/8426
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie09efdd9512cc4f0e338919af42bcdcede2cfb93
Gerrit-Change-Number: 8426
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Behm <al...@cloudera.com>
[Impala-ASF-CR] IMPALA-3887: Use dfs.namenode.replication.min=3
Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/8426 )
Change subject: IMPALA-3887: Use dfs.namenode.replication.min=3
......................................................................
Patch Set 3:
I'm still testing this change. With this change we seem to be more prone to hitting the Hive/mapred local executor race (MAPREDUCE-6441 MAPREDUCE-6992). Will sync with PhilZ to see how to proceed.
--
To view, visit http://gerrit.cloudera.org:8080/8426
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie09efdd9512cc4f0e338919af42bcdcede2cfb93
Gerrit-Change-Number: 8426
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Comment-Date: Tue, 31 Oct 2017 18:29:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3887: Use dfs.namenode.replication.min=3
Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/8426 )
Change subject: IMPALA-3887: Use dfs.namenode.replication.min=3
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/8426/1/fe/src/test/resources/hbase-site.xml.template
File fe/src/test/resources/hbase-site.xml.template:
http://gerrit.cloudera.org:8080/#/c/8426/1/fe/src/test/resources/hbase-site.xml.template@38
PS1, Line 38: <!-- Our tests expect to have 3 replicas. Force completing all replicas before acknowledging
: writes to avoid test failures due to temporarily underreplicated blocks. -->
: <property>
: <name>dfs.namenode.replication.min</name>
: <value>3</value>
: </property>
> Good question, let me give it a try.
I tried without the min replication here and it works fine. Removed it.
--
To view, visit http://gerrit.cloudera.org:8080/8426
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie09efdd9512cc4f0e338919af42bcdcede2cfb93
Gerrit-Change-Number: 8426
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Comment-Date: Tue, 31 Oct 2017 15:57:16 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3887: Use dfs.namenode.replication.min=3
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/8426 )
Change subject: IMPALA-3887: Use dfs.namenode.replication.min=3
......................................................................
Patch Set 1: Code-Review+2
(1 comment)
Just curious, how does this change affect the total runtime of the core/exhaustive test-suite? +2'ing since the change is simple.
http://gerrit.cloudera.org:8080/#/c/8426/1/fe/src/test/resources/hbase-site.xml.template
File fe/src/test/resources/hbase-site.xml.template:
http://gerrit.cloudera.org:8080/#/c/8426/1/fe/src/test/resources/hbase-site.xml.template@38
PS1, Line 38: <!-- Our tests expect to have 3 replicas. Force completing all replicas before acknowledging
: writes to avoid test failures due to temporarily underreplicated blocks. -->
: <property>
: <name>dfs.namenode.replication.min</name>
: <value>3</value>
: </property>
Isn't this redundant? Per my understanding, this only goes into HDFS service side settings and clients (HBase in this case) only control dfs.replication.
--
To view, visit http://gerrit.cloudera.org:8080/8426
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie09efdd9512cc4f0e338919af42bcdcede2cfb93
Gerrit-Change-Number: 8426
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Comment-Date: Tue, 31 Oct 2017 05:11:41 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3887: Use dfs.namenode.replication.min=3
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/8426 )
Change subject: IMPALA-3887: Use dfs.namenode.replication.min=3
......................................................................
Patch Set 2: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/8426/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/8426/2//COMMIT_MSG@9
PS2, Line 9: and HBase configuration of our
Update?
--
To view, visit http://gerrit.cloudera.org:8080/8426
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie09efdd9512cc4f0e338919af42bcdcede2cfb93
Gerrit-Change-Number: 8426
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Comment-Date: Tue, 31 Oct 2017 16:52:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3887: Use dfs.namenode.replication.min=3
Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/8426 )
Change subject: IMPALA-3887: Use dfs.namenode.replication.min=3
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/8426/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/8426/2//COMMIT_MSG@9
PS2, Line 9: and HBase configuration of our
> Update?
Done
--
To view, visit http://gerrit.cloudera.org:8080/8426
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie09efdd9512cc4f0e338919af42bcdcede2cfb93
Gerrit-Change-Number: 8426
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Comment-Date: Tue, 31 Oct 2017 18:26:38 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3887: Use dfs.namenode.replication.min=3
Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/8426
to look at the new patch set (#3).
Change subject: IMPALA-3887: Use dfs.namenode.replication.min=3
......................................................................
IMPALA-3887: Use dfs.namenode.replication.min=3
Changes the HDFS config of our mini-cluster to use:
dfs.namenode.replication.min=3
Several of our tests including planner tests
expect HDFS blocks to have exactly three replicas.
With the default HDFS configuration of
dfs.namenode.replication.min=1, HDFS acks writes
as soon as the min replication is satisfied. This
can lead to test failures because HDFS replication
is racing to replicate blocks while our tests
are executing and expecting to see all replicas.
A more detailed description is in the JIRA.
Testing:
- A core/hdfs ran passed
- A core/hdfs run with data loading passed
Change-Id: Ie09efdd9512cc4f0e338919af42bcdcede2cfb93
---
M fe/src/test/resources/hbase-site.xml.template
M testdata/cluster/node_templates/common/etc/hadoop/conf/hdfs-site.xml.tmpl
2 files changed, 21 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/8426/3
--
To view, visit http://gerrit.cloudera.org:8080/8426
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie09efdd9512cc4f0e338919af42bcdcede2cfb93
Gerrit-Change-Number: 8426
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
[Impala-ASF-CR] IMPALA-3887: Use dfs.namenode.replication.min=3
Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/8426 )
Change subject: IMPALA-3887: Use dfs.namenode.replication.min=3
......................................................................
Patch Set 3:
> Patch Set 3:
>
> I'm still testing this change. With this change we seem to be more prone to hitting the Hive/mapred local executor race (MAPREDUCE-6441 MAPREDUCE-6992). Will sync with PhilZ to see how to proceed.
https://gerrit.cloudera.org/c/8405/ is ready to commit. We were holding off because Lars said to hold off until the build stabilized.
I don't know where you're running into MAPREDUCE-6441. My change only addresses it for data load, since it's an option you have to pass to beeline.
c/8405 is splittable into two parts (beeline change for the option; parallelization), but I don't see much benefit into splitting it.
--
To view, visit http://gerrit.cloudera.org:8080/8426
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie09efdd9512cc4f0e338919af42bcdcede2cfb93
Gerrit-Change-Number: 8426
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Tue, 31 Oct 2017 20:38:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3887: Use dfs.namenode.replication.min=3
Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/8426
to look at the new patch set (#2).
Change subject: IMPALA-3887: Use dfs.namenode.replication.min=3
......................................................................
IMPALA-3887: Use dfs.namenode.replication.min=3
Changes the HDFS and HBase configuration of our
mini-cluster to use:
dfs.namenode.replication.min=3
Several of our tests including planner tests
expect HDFS blocks to have exactly three replicas.
With the default HDFS configuration of
dfs.namenode.replication.min=1, HDFS acks writes
as soon as the min replication is satisfied. This
can lead to test failures because HDFS replication
is racing to replicate blocks while our tests
are executing and expecting to see all replicas.
A more detailed description is in the JIRA.
Testing:
- A core/hdfs ran passed
- A core/hdfs run with data loading passed
Change-Id: Ie09efdd9512cc4f0e338919af42bcdcede2cfb93
---
M fe/src/test/resources/hbase-site.xml.template
M testdata/cluster/node_templates/common/etc/hadoop/conf/hdfs-site.xml.tmpl
2 files changed, 21 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/8426/2
--
To view, visit http://gerrit.cloudera.org:8080/8426
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie09efdd9512cc4f0e338919af42bcdcede2cfb93
Gerrit-Change-Number: 8426
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
[Impala-ASF-CR] IMPALA-3887: Use dfs.namenode.replication.min=3
Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/8426 )
Change subject: IMPALA-3887: Use dfs.namenode.replication.min=3
......................................................................
Patch Set 3:
I am running into MAPREDUCE-6441 with this patch precisely during data load.
--
To view, visit http://gerrit.cloudera.org:8080/8426
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie09efdd9512cc4f0e338919af42bcdcede2cfb93
Gerrit-Change-Number: 8426
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Tue, 31 Oct 2017 21:15:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3887: Use dfs.namenode.replication.min=3
Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has abandoned this change. ( http://gerrit.cloudera.org:8080/8426 )
Change subject: IMPALA-3887: Use dfs.namenode.replication.min=3
......................................................................
Abandoned
It's sad but I couldn't make this change work reliably. Data loading frequently fails in Hive due to underreplicated HDFS blocks.
Abandoning to reflect reality.
--
To view, visit http://gerrit.cloudera.org:8080/8426
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: Ie09efdd9512cc4f0e338919af42bcdcede2cfb93
Gerrit-Change-Number: 8426
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
[Impala-ASF-CR] IMPALA-3887: Use dfs.namenode.replication.min=3
Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/8426 )
Change subject: IMPALA-3887: Use dfs.namenode.replication.min=3
......................................................................
Patch Set 1:
(1 comment)
Don't have many data points regarding the effects on build+test runtimes.
The master-core tests have pretty high variance.
A quick examination showed times ranging roughly from 4:50 to 5:20.
The same job with this patch ran in 5:12 so at least it seems within the same ballpark.
I'll do an exhaustive test run and report numbers on that.
http://gerrit.cloudera.org:8080/#/c/8426/1/fe/src/test/resources/hbase-site.xml.template
File fe/src/test/resources/hbase-site.xml.template:
http://gerrit.cloudera.org:8080/#/c/8426/1/fe/src/test/resources/hbase-site.xml.template@38
PS1, Line 38: <!-- Our tests expect to have 3 replicas. Force completing all replicas before acknowledging
: writes to avoid test failures due to temporarily underreplicated blocks. -->
: <property>
: <name>dfs.namenode.replication.min</name>
: <value>3</value>
: </property>
> Isn't this redundant? Per my understanding, this only goes into HDFS servic
Good question, let me give it a try.
I originally tried without modifying hbase-site at all and that didn't work, maybe setting dfs.replication is enough. I agree that setting the min replication here seems weird/unnecessary.
--
To view, visit http://gerrit.cloudera.org:8080/8426
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie09efdd9512cc4f0e338919af42bcdcede2cfb93
Gerrit-Change-Number: 8426
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Comment-Date: Tue, 31 Oct 2017 05:27:03 +0000
Gerrit-HasComments: Yes