You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Thomas Marshall (Code Review)" <ge...@cloudera.org> on 2018/07/24 22:24:35 UTC
[Impala-ASF-CR] IMPALA-7170: Update data generator.py for Hadoop 3
Thomas Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11041
Change subject: IMPALA-7170: Update data_generator.py for Hadoop 3
......................................................................
IMPALA-7170: Update data_generator.py for Hadoop 3
After the move to Hadoop 3, data_generator.py was broken. The issue
seems to be that we rely on additional jars not in the classpath. The
solution is to pass the location of these jars into the 'hadoop'
command using the '-libjars' parameter.
This patch also updates tests/comparison/README to add instructions
for dealing with Yarn, since during the move to Hadoop 3 we switched
to no longer running Yarn as part of the minicluster by default.
Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a
---
M tests/comparison/README
M tests/comparison/cluster.py
M tests/comparison/data_generator.py
3 files changed, 9 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/11041/1
--
To view, visit http://gerrit.cloudera.org:8080/11041
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a
Gerrit-Change-Number: 11041
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall <th...@cmu.edu>
[Impala-ASF-CR] IMPALA-7170: Update data generator.py for Hadoop 3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11041 )
Change subject: IMPALA-7170: Update data_generator.py for Hadoop 3
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2872/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/11041
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a
Gerrit-Change-Number: 11041
Gerrit-PatchSet: 3
Gerrit-Owner: Thomas Marshall <th...@cmu.edu>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Jul 2018 21:06:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7170: Update data generator.py for Hadoop 3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11041 )
Change subject: IMPALA-7170: Update data_generator.py for Hadoop 3
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/40/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11041
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a
Gerrit-Change-Number: 11041
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Marshall <th...@cmu.edu>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 24 Jul 2018 23:07:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7170: Update data generator.py for Hadoop 3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11041 )
Change subject: IMPALA-7170: Update data_generator.py for Hadoop 3
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/11041
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a
Gerrit-Change-Number: 11041
Gerrit-PatchSet: 4
Gerrit-Owner: Thomas Marshall <th...@cmu.edu>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Marshall <th...@cmu.edu>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sun, 29 Jul 2018 02:25:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7170: Update data generator.py for Hadoop 3
Posted by "Thomas Marshall (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11041
to look at the new patch set (#2).
Change subject: IMPALA-7170: Update data_generator.py for Hadoop 3
......................................................................
IMPALA-7170: Update data_generator.py for Hadoop 3
After the move to Hadoop 3, data_generator.py was broken. The issue
seems to be that we rely on additional jars not in the classpath. The
solution is to pass the location of these jars into the 'hadoop'
command using the '-libjars' parameter.
This patch also updates tests/comparison/README to add instructions
for dealing with Yarn, since during the move to Hadoop 3 we switched
to no longer running Yarn as part of the minicluster by default.
Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a
---
M tests/comparison/README
M tests/comparison/data_generator.py
2 files changed, 8 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/11041/2
--
To view, visit http://gerrit.cloudera.org:8080/11041
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a
Gerrit-Change-Number: 11041
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Marshall <th...@cmu.edu>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-7170: Update data generator.py for Hadoop 3
Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11041 )
Change subject: IMPALA-7170: Update data_generator.py for Hadoop 3
......................................................................
Patch Set 5: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/11041/5/tests/comparison/data_generator.py
File tests/comparison/data_generator.py:
http://gerrit.cloudera.org:8080/#/c/11041/5/tests/comparison/data_generator.py@237
PS5, Line 237: -D mapred.reduce.tasks=%s \
If your instructions work, then we could add "-D mapreduce.framework.name=local" hereabouts, and then remove the instructions? It seems like the more bound this is to the defaults, the more likely it is someone will be able to work it?
--
To view, visit http://gerrit.cloudera.org:8080/11041
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a
Gerrit-Change-Number: 11041
Gerrit-PatchSet: 5
Gerrit-Owner: Thomas Marshall <th...@cmu.edu>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Thomas Marshall <th...@cmu.edu>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 30 Jul 2018 16:47:58 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7170: Update data generator.py for Hadoop 3
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11041 )
Change subject: IMPALA-7170: Update data_generator.py for Hadoop 3
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/11041
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a
Gerrit-Change-Number: 11041
Gerrit-PatchSet: 3
Gerrit-Owner: Thomas Marshall <th...@cmu.edu>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Jul 2018 20:45:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7170: Update data generator.py for Hadoop 3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11041 )
Change subject: IMPALA-7170: Update data_generator.py for Hadoop 3
......................................................................
Patch Set 3: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2872/
--
To view, visit http://gerrit.cloudera.org:8080/11041
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a
Gerrit-Change-Number: 11041
Gerrit-PatchSet: 3
Gerrit-Owner: Thomas Marshall <th...@cmu.edu>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Jul 2018 00:18:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7170: Update data generator.py for Hadoop 3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11041 )
Change subject: IMPALA-7170: Update data_generator.py for Hadoop 3
......................................................................
Patch Set 2:
Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/40/
Running initial code review checks. This is experimental - please report any issues to tarmstrong@cloudera.com or on this JIRA: IMPALA-7317
--
To view, visit http://gerrit.cloudera.org:8080/11041
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a
Gerrit-Change-Number: 11041
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Marshall <th...@cmu.edu>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 24 Jul 2018 22:28:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7170: Update data generator.py for Hadoop 3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11041 )
Change subject: IMPALA-7170: Update data_generator.py for Hadoop 3
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2874/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/11041
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a
Gerrit-Change-Number: 11041
Gerrit-PatchSet: 4
Gerrit-Owner: Thomas Marshall <th...@cmu.edu>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Marshall <th...@cmu.edu>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Jul 2018 23:08:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7170: Update data generator.py for Hadoop 3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11041 )
Change subject: IMPALA-7170: Update data_generator.py for Hadoop 3
......................................................................
IMPALA-7170: Update data_generator.py for Hadoop 3
After the move to Hadoop 3, data_generator.py was broken. The issue
seems to be that we rely on additional jars not in the classpath. The
solution is to pass the location of these jars into the 'hadoop'
command using the '-libjars' parameter.
This patch also updates tests/comparison/README to add instructions
for dealing with Yarn, since during the move to Hadoop 3 we switched
to no longer running Yarn as part of the minicluster by default.
Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a
Reviewed-on: http://gerrit.cloudera.org:8080/11041
Reviewed-by: Thomas Marshall <th...@cmu.edu>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M tests/comparison/README
M tests/comparison/data_generator.py
2 files changed, 8 insertions(+), 2 deletions(-)
Approvals:
Thomas Marshall: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/11041
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a
Gerrit-Change-Number: 11041
Gerrit-PatchSet: 5
Gerrit-Owner: Thomas Marshall <th...@cmu.edu>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Marshall <th...@cmu.edu>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-7170: Update data generator.py for Hadoop 3
Posted by "Thomas Marshall (Code Review)" <ge...@cloudera.org>.
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/11041 )
Change subject: IMPALA-7170: Update data_generator.py for Hadoop 3
......................................................................
Patch Set 4: Code-Review+2
gvo failed because it was rebased on the wrong thing
--
To view, visit http://gerrit.cloudera.org:8080/11041
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a
Gerrit-Change-Number: 11041
Gerrit-PatchSet: 4
Gerrit-Owner: Thomas Marshall <th...@cmu.edu>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Marshall <th...@cmu.edu>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Jul 2018 23:07:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7170: Update data generator.py for Hadoop 3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11041 )
Change subject: IMPALA-7170: Update data_generator.py for Hadoop 3
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/39/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11041
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a
Gerrit-Change-Number: 11041
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall <th...@cmu.edu>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 24 Jul 2018 23:00:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7170: Update data generator.py for Hadoop 3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11041 )
Change subject: IMPALA-7170: Update data_generator.py for Hadoop 3
......................................................................
Patch Set 1:
Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/39/
Running initial code review checks. This is experimental - please report any issues to tarmstrong@cloudera.com or on this JIRA: IMPALA-7317
--
To view, visit http://gerrit.cloudera.org:8080/11041
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a
Gerrit-Change-Number: 11041
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall <th...@cmu.edu>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 24 Jul 2018 22:24:47 +0000
Gerrit-HasComments: No