You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Tianyi Wang (Code Review)" <ge...@cloudera.org> on 2018/06/16 01:19:46 UTC
[Impala-ASF-CR] IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10742
Change subject: IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
......................................................................
IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
HDFS trash checkpointing renames files in the trash folder and breaks
impala tests. Impala set the trash checkpointing interval to 1440 to try
to postpone it for 24 hours. Unfortunately that told HDFS to do it when
the UNIX time is a multiple of 1440 * 60 and it broke trash-related
tests run around midnight in GMT. This patch sets the interval to
541728000 so that HDFS won't do the checkpointing until Jan 1st 3000,
and HDFS will checkpoint every 1030 years after that.
Change-Id: I9452f7e44c7679f86a947cd20115c078757223d8
---
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl
1 file changed, 2 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/10742/1
--
To view, visit http://gerrit.cloudera.org:8080/10742
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9452f7e44c7679f86a947cd20115c078757223d8
Gerrit-Change-Number: 10742
Gerrit-PatchSet: 1
Gerrit-Owner: Tianyi Wang <tw...@cloudera.com>
[Impala-ASF-CR] IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10742 )
Change subject: IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
......................................................................
Patch Set 1: Code-Review+2
Fascinating find!
--
To view, visit http://gerrit.cloudera.org:8080/10742
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9452f7e44c7679f86a947cd20115c078757223d8
Gerrit-Change-Number: 10742
Gerrit-PatchSet: 1
Gerrit-Owner: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Sun, 17 Jun 2018 18:03:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10742 )
Change subject: IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2691/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/10742
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9452f7e44c7679f86a947cd20115c078757223d8
Gerrit-Change-Number: 10742
Gerrit-PatchSet: 2
Gerrit-Owner: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Jun 2018 05:14:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10742 )
Change subject: IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2711/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/10742
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9452f7e44c7679f86a947cd20115c078757223d8
Gerrit-Change-Number: 10742
Gerrit-PatchSet: 3
Gerrit-Owner: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 Jun 2018 22:05:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
Posted by "Tianyi Wang (Code Review)" <ge...@cloudera.org>.
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10742 )
Change subject: IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
......................................................................
Patch Set 2:
> Patch Set 2:
>
> We can remerge this, right? Since it's a flaky test fix.
I think so. (The fix has been merged)
--
To view, visit http://gerrit.cloudera.org:8080/10742
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9452f7e44c7679f86a947cd20115c078757223d8
Gerrit-Change-Number: 10742
Gerrit-PatchSet: 2
Gerrit-Owner: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 Jun 2018 22:05:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
Posted by "Tianyi Wang (Code Review)" <ge...@cloudera.org>.
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10742 )
Change subject: IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
......................................................................
Patch Set 2:
> fyi, the build failure is related to https://issues.apache.org/jira/browse/IMPALA-7130
>
> Vincent is working on the test to deflake it.
That JIRA is marked fixed yesterday. Can we GVO now?
--
To view, visit http://gerrit.cloudera.org:8080/10742
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9452f7e44c7679f86a947cd20115c078757223d8
Gerrit-Change-Number: 10742
Gerrit-PatchSet: 2
Gerrit-Owner: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Jun 2018 19:03:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10742 )
Change subject: IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
......................................................................
IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
HDFS trash checkpointing renames files in the trash folder and breaks
impala tests. Impala set the trash checkpointing interval to 1440 to try
to postpone it for 24 hours. Unfortunately that told HDFS to do it when
the UNIX time is a multiple of 1440 * 60 and it broke trash-related
tests run around midnight in GMT. This patch sets the interval to
541728000 so that HDFS won't do the checkpointing until Jan 1st 3000,
and HDFS will checkpoint every 1030 years after that.
Change-Id: I9452f7e44c7679f86a947cd20115c078757223d8
Reviewed-on: http://gerrit.cloudera.org:8080/10742
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl
1 file changed, 2 insertions(+), 2 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/10742
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I9452f7e44c7679f86a947cd20115c078757223d8
Gerrit-Change-Number: 10742
Gerrit-PatchSet: 4
Gerrit-Owner: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/10742 )
Change subject: IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
......................................................................
Patch Set 2: Code-Review+2
The flakiness related to IMPALA-7130 is not fixed yet ( it is on review: https://gerrit.cloudera.org/#/c/10747/ ).
I just mentioned it to ensure you that it is not related to your change, so a new jenkins verify job may be successful.
--
To view, visit http://gerrit.cloudera.org:8080/10742
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9452f7e44c7679f86a947cd20115c078757223d8
Gerrit-Change-Number: 10742
Gerrit-PatchSet: 2
Gerrit-Owner: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Jun 2018 20:02:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10742 )
Change subject: IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
......................................................................
Patch Set 2:
We can remerge this, right? Since it's a flaky test fix.
--
To view, visit http://gerrit.cloudera.org:8080/10742
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9452f7e44c7679f86a947cd20115c078757223d8
Gerrit-Change-Number: 10742
Gerrit-PatchSet: 2
Gerrit-Owner: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 Jun 2018 21:58:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/10742 )
Change subject: IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
......................................................................
Patch Set 2:
fyi, the build failure is related to https://issues.apache.org/jira/browse/IMPALA-7130
Vincent is working on the test to deflake it.
--
To view, visit http://gerrit.cloudera.org:8080/10742
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9452f7e44c7679f86a947cd20115c078757223d8
Gerrit-Change-Number: 10742
Gerrit-PatchSet: 2
Gerrit-Owner: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Jun 2018 15:16:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10742 )
Change subject: IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
......................................................................
Patch Set 2: Code-Review+1
I'm a little late to this party, but thought I would +1.
--
To view, visit http://gerrit.cloudera.org:8080/10742
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9452f7e44c7679f86a947cd20115c078757223d8
Gerrit-Change-Number: 10742
Gerrit-PatchSet: 2
Gerrit-Owner: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Jun 2018 23:19:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10742 )
Change subject: IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/10742
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9452f7e44c7679f86a947cd20115c078757223d8
Gerrit-Change-Number: 10742
Gerrit-PatchSet: 2
Gerrit-Owner: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Jun 2018 05:14:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10742 )
Change subject: IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/10742
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9452f7e44c7679f86a947cd20115c078757223d8
Gerrit-Change-Number: 10742
Gerrit-PatchSet: 3
Gerrit-Owner: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 Jun 2018 22:05:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10742 )
Change subject: IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/10742
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9452f7e44c7679f86a947cd20115c078757223d8
Gerrit-Change-Number: 10742
Gerrit-PatchSet: 3
Gerrit-Owner: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Jun 2018 01:32:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10742 )
Change subject: IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
......................................................................
Patch Set 2: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2691/
--
To view, visit http://gerrit.cloudera.org:8080/10742
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9452f7e44c7679f86a947cd20115c078757223d8
Gerrit-Change-Number: 10742
Gerrit-PatchSet: 2
Gerrit-Owner: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Jun 2018 08:40:25 +0000
Gerrit-HasComments: No