You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2022/08/22 01:20:37 UTC
[Impala-ASF-CR](branch-4.1.1) IMPALA-10057: Fix log spew by using jars in the classpath
Hello Michael Smith, Impala Public Jenkins,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/18879
to review the following change.
Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................
IMPALA-10057: Fix log spew by using jars in the classpath
Some tests saw log spew that causes the INFO log files to
be filled with output like this:
E0903 02:25:39.453887 12060 TransactionKeepalive.java:137] Unexpected exception thrown
Java exception follows:
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/impala/common/TransactionKeepalive$HeartbeatContext
at org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114)
at java.lang.Thread.run(Thread.java:748)
...
It turns out that the catalogd/impalad use a CLASSPATH in
tests that refers to fe/target/classes. The maven command
that runs frontend tests recompiles these classes and
causes the files in fe/target/classes to be deleted and
recreated. There are race conditions where this causes
the symptoms above.
This changes the CLASSPATH to use the frontend jars, which
are not impacted by the machinations on fe/target/classes.
To find the appropriate jar, set-classpath.sh needs to
know the Impala version. This adds IMPALA_VERSION in
bin/impala-config.sh to provide an easy to use
environment variable.
To make the versioning more uniform, this modifies
bin/save-version.sh to use this environment variable.
It also adds a check to make sure that the Java pom.xml
files use the same version as the environment variable.
It fails the build if the Java pom.xml files do not
match.
Testing:
- Ran core jobs
- Checked the log file sizes on jobs
- Changed a Java pom.xml's version and verified that
bin/validate-java-pom-versions.sh fails
Merge conflicts:
- Change version string "4.1.0" to "4.1.1".
Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Reviewed-on: http://gerrit.cloudera.org:8080/18415
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Michael Smith <mi...@cloudera.com>
Reviewed-by: Quanlong Huang <hu...@gmail.com>
---
M bin/impala-config.sh
M bin/jenkins/build-all-flag-combinations.sh
M bin/save-version.sh
M bin/set-classpath.sh
A bin/validate-java-pom-versions.sh
M fe/pom.xml
M java/CMakeLists.txt
M java/TableFlattener/pom.xml
M java/datagenerator/pom.xml
M java/executor-deps/pom.xml
M java/ext-data-source/api/pom.xml
M java/ext-data-source/pom.xml
M java/ext-data-source/sample/pom.xml
M java/ext-data-source/test/pom.xml
M java/pom.xml
M java/query-event-hook-api/pom.xml
M java/shaded-deps/hive-exec/pom.xml
M java/shaded-deps/s3a-aws-sdk/pom.xml
M java/test-hive-udfs/pom.xml
M java/yarn-extras/pom.xml
20 files changed, 108 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/18879/1
--
To view, visit http://gerrit.cloudera.org:8080/18879
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18879
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR](branch-4.1.1) IMPALA-10057: Fix log spew by using jars in the classpath
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has removed a vote on this change.
Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................
Removed Verified-1 by Impala Public Jenkins <im...@cloudera.com>
--
To view, visit http://gerrit.cloudera.org:8080/18879
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18879
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR](branch-4.1.1) IMPALA-10057: Fix log spew by using jars in the classpath
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18879 )
Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................
Patch Set 1: Verified+1
> Patch Set 1: Verified-1
>
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8481/
The failure is due to IMPALA-11295 (https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/6195) which is unrelated to this patch. Merging it. Thanks for Joe's review!
--
To view, visit http://gerrit.cloudera.org:8080/18879
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18879
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 23 Aug 2022 03:16:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-10057: Fix log spew by using jars in the classpath
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18879 )
Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................
Patch Set 1: Code-Review+2
This looks good to me
--
To view, visit http://gerrit.cloudera.org:8080/18879
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18879
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Aug 2022 22:59:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-10057: Fix log spew by using jars in the classpath
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18879 )
Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................
Patch Set 1: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8481/
--
To view, visit http://gerrit.cloudera.org:8080/18879
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18879
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Aug 2022 03:10:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-10057: Fix log spew by using jars in the classpath
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18879 )
Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8481/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18879
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18879
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Aug 2022 22:36:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-10057: Fix log spew by using jars in the classpath
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18879 )
Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................
IMPALA-10057: Fix log spew by using jars in the classpath
Some tests saw log spew that causes the INFO log files to
be filled with output like this:
E0903 02:25:39.453887 12060 TransactionKeepalive.java:137] Unexpected exception thrown
Java exception follows:
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/impala/common/TransactionKeepalive$HeartbeatContext
at org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114)
at java.lang.Thread.run(Thread.java:748)
...
It turns out that the catalogd/impalad use a CLASSPATH in
tests that refers to fe/target/classes. The maven command
that runs frontend tests recompiles these classes and
causes the files in fe/target/classes to be deleted and
recreated. There are race conditions where this causes
the symptoms above.
This changes the CLASSPATH to use the frontend jars, which
are not impacted by the machinations on fe/target/classes.
To find the appropriate jar, set-classpath.sh needs to
know the Impala version. This adds IMPALA_VERSION in
bin/impala-config.sh to provide an easy to use
environment variable.
To make the versioning more uniform, this modifies
bin/save-version.sh to use this environment variable.
It also adds a check to make sure that the Java pom.xml
files use the same version as the environment variable.
It fails the build if the Java pom.xml files do not
match.
Testing:
- Ran core jobs
- Checked the log file sizes on jobs
- Changed a Java pom.xml's version and verified that
bin/validate-java-pom-versions.sh fails
Merge conflicts:
- Change version string "4.1.0" to "4.1.1".
Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Reviewed-on: http://gerrit.cloudera.org:8080/18415
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Michael Smith <mi...@cloudera.com>
Reviewed-by: Quanlong Huang <hu...@gmail.com>
Reviewed-on: http://gerrit.cloudera.org:8080/18879
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Quanlong Huang <hu...@gmail.com>
---
M bin/impala-config.sh
M bin/jenkins/build-all-flag-combinations.sh
M bin/save-version.sh
M bin/set-classpath.sh
A bin/validate-java-pom-versions.sh
M fe/pom.xml
M java/CMakeLists.txt
M java/TableFlattener/pom.xml
M java/datagenerator/pom.xml
M java/executor-deps/pom.xml
M java/ext-data-source/api/pom.xml
M java/ext-data-source/pom.xml
M java/ext-data-source/sample/pom.xml
M java/ext-data-source/test/pom.xml
M java/pom.xml
M java/query-event-hook-api/pom.xml
M java/shaded-deps/hive-exec/pom.xml
M java/shaded-deps/s3a-aws-sdk/pom.xml
M java/test-hive-udfs/pom.xml
M java/yarn-extras/pom.xml
20 files changed, 108 insertions(+), 18 deletions(-)
Approvals:
Joe McDonnell: Looks good to me, approved
Quanlong Huang: Verified
--
To view, visit http://gerrit.cloudera.org:8080/18879
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: merged
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18879
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>