You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Joe McDonnell (Code Review)" <ge...@cloudera.org> on 2022/04/13 22:53:25 UTC

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18415


Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................

IMPALA-10057: Fix log spew by using jars in the classpath

Some tests saw log spew that causes the INFO log files to
be filled with output like this:
E0903 02:25:39.453887 12060 TransactionKeepalive.java:137] Unexpected exception thrown
Java exception follows:
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/impala/common/TransactionKeepalive$HeartbeatContext
	at org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114)
	at java.lang.Thread.run(Thread.java:748)
...

It turns out that the catalogd/impalad use a CLASSPATH in
tests that refers to fe/target/classes. The maven command
that runs frontend tests recompiles these classes and
causes the files in fe/target/classes to be deleted and
recreated. There are race conditions where this causes
the symptoms above.

This changes the CLASSPATH to use the frontend jars, which
are not impacted by the machinations on fe/target/classes.
To find the appropriate jar, set-classpath.sh needs to
know the Impala version. This adds IMPALA_VERSION in
bin/impala-config.sh to provide an easy to use
environment variable.

To make the versioning more uniform, this modifies
bin/save-version.sh to use this environment variable.
It also adds a check to make sure that the Java pom.xml
files use the same version as the environment variable.
It fails the build if the Java pom.xml files do not
match.

Testing:
 - Ran core jobs
 - Checked the log file sizes on jobs
 - Changed a Java pom.xml's version and verified that
   bin/validate-java-pom-versions.sh fails

Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
---
M bin/impala-config.sh
M bin/jenkins/build-all-flag-combinations.sh
M bin/save-version.sh
M bin/set-classpath.sh
A bin/validate-java-pom-versions.sh
M java/CMakeLists.txt
M java/pom.xml
7 files changed, 101 insertions(+), 4 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/18415/1
-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18415/5/bin/validate-java-pom-versions.sh
File bin/validate-java-pom-versions.sh:

http://gerrit.cloudera.org:8080/#/c/18415/5/bin/validate-java-pom-versions.sh@34
PS5, Line 34: declare -a POM_XML_FILES=(
Could we use `git ls-files '**/pom.xml'` to enumerate all pom.xml in the repo instead of hard-coding it?



-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 06 May 2022 22:46:28 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 6: Code-Review+1

Looks clear to me, though I haven't actually run it myself.


-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 6
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 09 May 2022 16:23:31 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 6: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 6
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 09 May 2022 00:21:42 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8083/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 6
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sun, 08 May 2022 19:57:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 3: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8076/


-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 May 2022 09:45:56 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8080/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 06 May 2022 21:37:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18415

to look at the new patch set (#5).

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................

IMPALA-10057: Fix log spew by using jars in the classpath

Some tests saw log spew that causes the INFO log files to
be filled with output like this:
E0903 02:25:39.453887 12060 TransactionKeepalive.java:137] Unexpected exception thrown
Java exception follows:
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/impala/common/TransactionKeepalive$HeartbeatContext
	at org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114)
	at java.lang.Thread.run(Thread.java:748)
...

It turns out that the catalogd/impalad use a CLASSPATH in
tests that refers to fe/target/classes. The maven command
that runs frontend tests recompiles these classes and
causes the files in fe/target/classes to be deleted and
recreated. There are race conditions where this causes
the symptoms above.

This changes the CLASSPATH to use the frontend jars, which
are not impacted by the machinations on fe/target/classes.
To find the appropriate jar, set-classpath.sh needs to
know the Impala version. This adds IMPALA_VERSION in
bin/impala-config.sh to provide an easy to use
environment variable.

To make the versioning more uniform, this modifies
bin/save-version.sh to use this environment variable.
It also adds a check to make sure that the Java pom.xml
files use the same version as the environment variable.
It fails the build if the Java pom.xml files do not
match.

Testing:
 - Ran core jobs
 - Checked the log file sizes on jobs
 - Changed a Java pom.xml's version and verified that
   bin/validate-java-pom-versions.sh fails

Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
---
M bin/impala-config.sh
M bin/jenkins/build-all-flag-combinations.sh
M bin/save-version.sh
M bin/set-classpath.sh
A bin/validate-java-pom-versions.sh
M java/CMakeLists.txt
M java/pom.xml
7 files changed, 102 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/18415/5
-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10540/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 06 May 2022 21:47:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 6: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 6
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 09 May 2022 01:22:55 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 2: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8061/


-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 23 Apr 2022 06:08:50 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 1: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8045/


-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Apr 2022 03:43:21 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 5: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8080/


-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 07 May 2022 02:09:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10483/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 23 Apr 2022 01:47:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8076/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 May 2022 05:20:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Michael Smith, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18415

to look at the new patch set (#6).

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................

IMPALA-10057: Fix log spew by using jars in the classpath

Some tests saw log spew that causes the INFO log files to
be filled with output like this:
E0903 02:25:39.453887 12060 TransactionKeepalive.java:137] Unexpected exception thrown
Java exception follows:
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/impala/common/TransactionKeepalive$HeartbeatContext
	at org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114)
	at java.lang.Thread.run(Thread.java:748)
...

It turns out that the catalogd/impalad use a CLASSPATH in
tests that refers to fe/target/classes. The maven command
that runs frontend tests recompiles these classes and
causes the files in fe/target/classes to be deleted and
recreated. There are race conditions where this causes
the symptoms above.

This changes the CLASSPATH to use the frontend jars, which
are not impacted by the machinations on fe/target/classes.
To find the appropriate jar, set-classpath.sh needs to
know the Impala version. This adds IMPALA_VERSION in
bin/impala-config.sh to provide an easy to use
environment variable.

To make the versioning more uniform, this modifies
bin/save-version.sh to use this environment variable.
It also adds a check to make sure that the Java pom.xml
files use the same version as the environment variable.
It fails the build if the Java pom.xml files do not
match.

Testing:
 - Ran core jobs
 - Checked the log file sizes on jobs
 - Changed a Java pom.xml's version and verified that
   bin/validate-java-pom-versions.sh fails

Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
---
M bin/impala-config.sh
M bin/jenkins/build-all-flag-combinations.sh
M bin/save-version.sh
M bin/set-classpath.sh
A bin/validate-java-pom-versions.sh
M java/CMakeLists.txt
6 files changed, 94 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/18415/6
-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 6
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18415/3/bin/validate-java-pom-versions.sh
File bin/validate-java-pom-versions.sh:

http://gerrit.cloudera.org:8080/#/c/18415/3/bin/validate-java-pom-versions.sh@54
PS3, Line 54:     
> nit: 2 spaces indention
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 06 May 2022 21:37:05 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8061/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 23 Apr 2022 01:40:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10445/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Apr 2022 23:16:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 2:

Build Successful 

https://ec2-35-162-169-52.us-west-2.compute.amazonaws.com/job/gerrit-code-review-checks/10205/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 23 Apr 2022 01:47:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 3: Code-Review+1

(1 comment)

Thanks for fixing this!

http://gerrit.cloudera.org:8080/#/c/18415/3/bin/validate-java-pom-versions.sh
File bin/validate-java-pom-versions.sh:

http://gerrit.cloudera.org:8080/#/c/18415/3/bin/validate-java-pom-versions.sh@54
PS3, Line 54:     
nit: 2 spaces indention



-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 05 May 2022 11:31:36 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 6: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 6
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 10 May 2022 00:17:45 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18415

to look at the new patch set (#2).

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................

IMPALA-10057: Fix log spew by using jars in the classpath

Some tests saw log spew that causes the INFO log files to
be filled with output like this:
E0903 02:25:39.453887 12060 TransactionKeepalive.java:137] Unexpected exception thrown
Java exception follows:
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/impala/common/TransactionKeepalive$HeartbeatContext
	at org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114)
	at java.lang.Thread.run(Thread.java:748)
...

It turns out that the catalogd/impalad use a CLASSPATH in
tests that refers to fe/target/classes. The maven command
that runs frontend tests recompiles these classes and
causes the files in fe/target/classes to be deleted and
recreated. There are race conditions where this causes
the symptoms above.

This changes the CLASSPATH to use the frontend jars, which
are not impacted by the machinations on fe/target/classes.
To find the appropriate jar, set-classpath.sh needs to
know the Impala version. This adds IMPALA_VERSION in
bin/impala-config.sh to provide an easy to use
environment variable.

To make the versioning more uniform, this modifies
bin/save-version.sh to use this environment variable.
It also adds a check to make sure that the Java pom.xml
files use the same version as the environment variable.
It fails the build if the Java pom.xml files do not
match.

Testing:
 - Ran core jobs
 - Checked the log file sizes on jobs
 - Changed a Java pom.xml's version and verified that
   bin/validate-java-pom-versions.sh fails

Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
---
M bin/impala-config.sh
M bin/jenkins/build-all-flag-combinations.sh
M bin/save-version.sh
M bin/set-classpath.sh
A bin/validate-java-pom-versions.sh
M java/CMakeLists.txt
M java/pom.xml
7 files changed, 102 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/18415/2
-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8045/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Apr 2022 23:16:45 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18415/5/bin/validate-java-pom-versions.sh
File bin/validate-java-pom-versions.sh:

http://gerrit.cloudera.org:8080/#/c/18415/5/bin/validate-java-pom-versions.sh@34
PS5, Line 34: declare -a POM_XML_FILES=(
> Could we use `git ls-files '**/pom.xml'` to enumerate all pom.xml in the re
Good point, it's better to autodetect this.

For Apache Impala releases, we produce a source tarball without git information. We do want the build to work for that source tarball.

I changed this to autodetect pom.xml files via "find ${IMPALA_HOME} -name pom.xml". If this is a git checkout, it ignores pom.xml files that are not known to git.



-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 07 May 2022 00:53:18 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10541/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 6
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 07 May 2022 01:14:31 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10057: Fix log spew by using jars in the classpath

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18415 )

Change subject: IMPALA-10057: Fix log spew by using jars in the classpath
......................................................................

IMPALA-10057: Fix log spew by using jars in the classpath

Some tests saw log spew that causes the INFO log files to
be filled with output like this:
E0903 02:25:39.453887 12060 TransactionKeepalive.java:137] Unexpected exception thrown
Java exception follows:
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/impala/common/TransactionKeepalive$HeartbeatContext
	at org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114)
	at java.lang.Thread.run(Thread.java:748)
...

It turns out that the catalogd/impalad use a CLASSPATH in
tests that refers to fe/target/classes. The maven command
that runs frontend tests recompiles these classes and
causes the files in fe/target/classes to be deleted and
recreated. There are race conditions where this causes
the symptoms above.

This changes the CLASSPATH to use the frontend jars, which
are not impacted by the machinations on fe/target/classes.
To find the appropriate jar, set-classpath.sh needs to
know the Impala version. This adds IMPALA_VERSION in
bin/impala-config.sh to provide an easy to use
environment variable.

To make the versioning more uniform, this modifies
bin/save-version.sh to use this environment variable.
It also adds a check to make sure that the Java pom.xml
files use the same version as the environment variable.
It fails the build if the Java pom.xml files do not
match.

Testing:
 - Ran core jobs
 - Checked the log file sizes on jobs
 - Changed a Java pom.xml's version and verified that
   bin/validate-java-pom-versions.sh fails

Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Reviewed-on: http://gerrit.cloudera.org:8080/18415
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Michael Smith <mi...@cloudera.com>
Reviewed-by: Quanlong Huang <hu...@gmail.com>
---
M bin/impala-config.sh
M bin/jenkins/build-all-flag-combinations.sh
M bin/save-version.sh
M bin/set-classpath.sh
A bin/validate-java-pom-versions.sh
M java/CMakeLists.txt
6 files changed, 94 insertions(+), 4 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Michael Smith: Looks good to me, but someone else must approve
  Quanlong Huang: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/18415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Gerrit-Change-Number: 18415
Gerrit-PatchSet: 7
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>