You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org> on 2019/08/16 14:28:07 UTC
[Impala-ASF-CR] IMPALA-8841: Try to fix Tez related dataload flakiness
Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14081
Change subject: IMPALA-8841: Try to fix Tez related dataload flakiness
......................................................................
IMPALA-8841: Try to fix Tez related dataload flakiness
The flakiness may be related to startinf Hive queries in parallel which
triggers initializing Tez resources in parallel (only needed at the
first statement that uses Tez). Doing a non-parallel statement at first
may solve the issue.
Change-Id: Id21d57483fe7a4f72f450fb71f8f53b3c1ef6327
---
M testdata/bin/create-load-data.sh
1 file changed, 16 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/14081/1
--
To view, visit http://gerrit.cloudera.org:8080/14081
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id21d57483fe7a4f72f450fb71f8f53b3c1ef6327
Gerrit-Change-Number: 14081
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
[Impala-ASF-CR] IMPALA-8841: Try to fix Tez related dataload flakiness
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14081 )
Change subject: IMPALA-8841: Try to fix Tez related dataload flakiness
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/14081/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14081/2//COMMIT_MSG@10
PS2, Line 10: initializing Tez resources in parallel
my observation is that the tez resources are allocated per session and since every hive query which is run during dataload happens in a separate beeline session, it may still need to initialize the resources. Right?
http://gerrit.cloudera.org:8080/#/c/14081/2/testdata/bin/create-load-data.sh
File testdata/bin/create-load-data.sh:
http://gerrit.cloudera.org:8080/#/c/14081/2/testdata/bin/create-load-data.sh@620
PS2, Line 620: before before
typo
--
To view, visit http://gerrit.cloudera.org:8080/14081
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id21d57483fe7a4f72f450fb71f8f53b3c1ef6327
Gerrit-Change-Number: 14081
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (521)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 16 Aug 2019 16:35:19 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8841: Try to fix Tez related dataload flakiness
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14081 )
Change subject: IMPALA-8841: Try to fix Tez related dataload flakiness
......................................................................
Patch Set 3: Verified+1 Code-Review+2
Change from 2->3 was only a comment, I'll just carry the verification.
--
To view, visit http://gerrit.cloudera.org:8080/14081
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id21d57483fe7a4f72f450fb71f8f53b3c1ef6327
Gerrit-Change-Number: 14081
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (521)
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 16 Aug 2019 22:59:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8841: Try to fix Tez related dataload flakiness
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14081 )
Change subject: IMPALA-8841: Try to fix Tez related dataload flakiness
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/4279/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/14081
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id21d57483fe7a4f72f450fb71f8f53b3c1ef6327
Gerrit-Change-Number: 14081
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 16 Aug 2019 15:08:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8841: Try to fix Tez related dataload flakiness
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Anonymous Coward (521), Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14081
to look at the new patch set (#2).
Change subject: IMPALA-8841: Try to fix Tez related dataload flakiness
......................................................................
IMPALA-8841: Try to fix Tez related dataload flakiness
The flakiness may be related to starting Hive queries in parallel which
triggers initializing Tez resources in parallel (only needed at the
first statement that uses Tez). Doing a non-parallel statement at first
may solve the issue.
Also includes a fix for a recent issue in 'build-and-copy-hive-udfs'
introduced by the version bump
in https://gerrit.cloudera.org/#/c/14043/
Change-Id: Id21d57483fe7a4f72f450fb71f8f53b3c1ef6327
---
M testdata/bin/create-load-data.sh
M tests/test-hive-udfs/pom.xml
2 files changed, 20 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/14081/2
--
To view, visit http://gerrit.cloudera.org:8080/14081
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id21d57483fe7a4f72f450fb71f8f53b3c1ef6327
Gerrit-Change-Number: 14081
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (521)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-8841: Try to fix Tez related dataload flakiness
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14081 )
Change subject: IMPALA-8841: Try to fix Tez related dataload flakiness
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/14081
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id21d57483fe7a4f72f450fb71f8f53b3c1ef6327
Gerrit-Change-Number: 14081
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (521)
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 16 Aug 2019 20:30:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8841: Try to fix Tez related dataload flakiness
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Vihang Karajgaonkar, Anonymous Coward (521), Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14081
to look at the new patch set (#3).
Change subject: IMPALA-8841: Try to fix Tez related dataload flakiness
......................................................................
IMPALA-8841: Try to fix Tez related dataload flakiness
The flakiness may be related to starting Hive queries in parallel which
triggers initializing Tez resources in parallel (only needed at the
first statement that uses Tez). Doing a non-parallel statement at first
may solve the issue.
Also includes a fix for a recent issue in 'build-and-copy-hive-udfs'
introduced by the version bump
in https://gerrit.cloudera.org/#/c/14043/
Change-Id: Id21d57483fe7a4f72f450fb71f8f53b3c1ef6327
---
M testdata/bin/create-load-data.sh
M tests/test-hive-udfs/pom.xml
2 files changed, 19 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/14081/3
--
To view, visit http://gerrit.cloudera.org:8080/14081
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id21d57483fe7a4f72f450fb71f8f53b3c1ef6327
Gerrit-Change-Number: 14081
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (521)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
[Impala-ASF-CR] IMPALA-8841: Try to fix Tez related dataload flakiness
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/14081 )
Change subject: IMPALA-8841: Try to fix Tez related dataload flakiness
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/14081/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14081/2//COMMIT_MSG@10
PS2, Line 10: initializing Tez resources in parallel
> my observation is that the tez resources are allocated per session and sinc
My experience is that the .jar in question is not overwritten by new sessions if the file already exists, so I have some hopes about helping with this change.
http://gerrit.cloudera.org:8080/#/c/14081/2/testdata/bin/create-load-data.sh
File testdata/bin/create-load-data.sh:
http://gerrit.cloudera.org:8080/#/c/14081/2/testdata/bin/create-load-data.sh@620
PS2, Line 620: before before
> typo
Done
--
To view, visit http://gerrit.cloudera.org:8080/14081
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id21d57483fe7a4f72f450fb71f8f53b3c1ef6327
Gerrit-Change-Number: 14081
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (521)
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 16 Aug 2019 18:00:45 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8841: Try to fix Tez related dataload flakiness
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14081 )
Change subject: IMPALA-8841: Try to fix Tez related dataload flakiness
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/4281/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/14081
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id21d57483fe7a4f72f450fb71f8f53b3c1ef6327
Gerrit-Change-Number: 14081
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (521)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 16 Aug 2019 16:54:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8841: Try to fix Tez related dataload flakiness
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14081 )
Change subject: IMPALA-8841: Try to fix Tez related dataload flakiness
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4805/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/14081
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id21d57483fe7a4f72f450fb71f8f53b3c1ef6327
Gerrit-Change-Number: 14081
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (521)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 16 Aug 2019 16:24:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8841: Try to fix Tez related dataload flakiness
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14081 )
Change subject: IMPALA-8841: Try to fix Tez related dataload flakiness
......................................................................
IMPALA-8841: Try to fix Tez related dataload flakiness
The flakiness may be related to starting Hive queries in parallel which
triggers initializing Tez resources in parallel (only needed at the
first statement that uses Tez). Doing a non-parallel statement at first
may solve the issue.
Also includes a fix for a recent issue in 'build-and-copy-hive-udfs'
introduced by the version bump
in https://gerrit.cloudera.org/#/c/14043/
Change-Id: Id21d57483fe7a4f72f450fb71f8f53b3c1ef6327
Reviewed-on: http://gerrit.cloudera.org:8080/14081
Reviewed-by: Vihang Karajgaonkar <vi...@cloudera.com>
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Tim Armstrong <ta...@cloudera.com>
---
M testdata/bin/create-load-data.sh
M tests/test-hive-udfs/pom.xml
2 files changed, 19 insertions(+), 0 deletions(-)
Approvals:
Vihang Karajgaonkar: Looks good to me, approved
Tim Armstrong: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/14081
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Id21d57483fe7a4f72f450fb71f8f53b3c1ef6327
Gerrit-Change-Number: 14081
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (521)
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
[Impala-ASF-CR] IMPALA-8841: Try to fix Tez related dataload flakiness
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Anonymous Coward (521) has posted comments on this change. ( http://gerrit.cloudera.org:8080/14081 )
Change subject: IMPALA-8841: Try to fix Tez related dataload flakiness
......................................................................
Patch Set 1: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/14081
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id21d57483fe7a4f72f450fb71f8f53b3c1ef6327
Gerrit-Change-Number: 14081
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (521)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 16 Aug 2019 15:33:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8841: Try to fix Tez related dataload flakiness
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14081 )
Change subject: IMPALA-8841: Try to fix Tez related dataload flakiness
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/14081
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id21d57483fe7a4f72f450fb71f8f53b3c1ef6327
Gerrit-Change-Number: 14081
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (521)
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 16 Aug 2019 18:45:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8841: Try to fix Tez related dataload flakiness
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14081 )
Change subject: IMPALA-8841: Try to fix Tez related dataload flakiness
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/4286/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/14081
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id21d57483fe7a4f72f450fb71f8f53b3c1ef6327
Gerrit-Change-Number: 14081
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (521)
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 16 Aug 2019 19:37:03 +0000
Gerrit-HasComments: No