You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by jo...@apache.org on 2022/08/23 03:23:11 UTC
[impala] branch branch-4.1.1 updated: IMPALA-10057: Fix log spew by using jars in the classpath
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a commit to branch branch-4.1.1
in repository https://gitbox.apache.org/repos/asf/impala.git
The following commit(s) were added to refs/heads/branch-4.1.1 by this push:
new 81a21bac0 IMPALA-10057: Fix log spew by using jars in the classpath
81a21bac0 is described below
commit 81a21bac0f6e9f3c22f93bb0919b2d465906215b
Author: Joe McDonnell <jo...@cloudera.com>
AuthorDate: Wed Apr 13 15:18:17 2022 -0700
IMPALA-10057: Fix log spew by using jars in the classpath
Some tests saw log spew that causes the INFO log files to
be filled with output like this:
E0903 02:25:39.453887 12060 TransactionKeepalive.java:137] Unexpected exception thrown
Java exception follows:
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/impala/common/TransactionKeepalive$HeartbeatContext
at org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114)
at java.lang.Thread.run(Thread.java:748)
...
It turns out that the catalogd/impalad use a CLASSPATH in
tests that refers to fe/target/classes. The maven command
that runs frontend tests recompiles these classes and
causes the files in fe/target/classes to be deleted and
recreated. There are race conditions where this causes
the symptoms above.
This changes the CLASSPATH to use the frontend jars, which
are not impacted by the machinations on fe/target/classes.
To find the appropriate jar, set-classpath.sh needs to
know the Impala version. This adds IMPALA_VERSION in
bin/impala-config.sh to provide an easy to use
environment variable.
To make the versioning more uniform, this modifies
bin/save-version.sh to use this environment variable.
It also adds a check to make sure that the Java pom.xml
files use the same version as the environment variable.
It fails the build if the Java pom.xml files do not
match.
Testing:
- Ran core jobs
- Checked the log file sizes on jobs
- Changed a Java pom.xml's version and verified that
bin/validate-java-pom-versions.sh fails
Merge conflicts:
- Change version string "4.1.0" to "4.1.1".
Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Reviewed-on: http://gerrit.cloudera.org:8080/18415
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Michael Smith <mi...@cloudera.com>
Reviewed-by: Quanlong Huang <hu...@gmail.com>
Reviewed-on: http://gerrit.cloudera.org:8080/18879
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Quanlong Huang <hu...@gmail.com>
---
bin/impala-config.sh | 10 +++++
bin/jenkins/build-all-flag-combinations.sh | 4 ++
bin/save-version.sh | 2 +-
bin/set-classpath.sh | 8 +++-
bin/validate-java-pom-versions.sh | 68 ++++++++++++++++++++++++++++++
fe/pom.xml | 2 +-
java/CMakeLists.txt | 6 ++-
java/TableFlattener/pom.xml | 2 +-
java/datagenerator/pom.xml | 2 +-
java/executor-deps/pom.xml | 2 +-
java/ext-data-source/api/pom.xml | 2 +-
java/ext-data-source/pom.xml | 2 +-
java/ext-data-source/sample/pom.xml | 2 +-
java/ext-data-source/test/pom.xml | 2 +-
java/pom.xml | 2 +-
java/query-event-hook-api/pom.xml | 2 +-
java/shaded-deps/hive-exec/pom.xml | 2 +-
java/shaded-deps/s3a-aws-sdk/pom.xml | 2 +-
java/test-hive-udfs/pom.xml | 2 +-
java/yarn-extras/pom.xml | 2 +-
20 files changed, 108 insertions(+), 18 deletions(-)
diff --git a/bin/impala-config.sh b/bin/impala-config.sh
index 51e20b762..6cc6ff2b6 100755
--- a/bin/impala-config.sh
+++ b/bin/impala-config.sh
@@ -63,6 +63,15 @@ fi
# variable from a previously-sourced impala-config.sh overrides the new value. #
#######################################################################################
+# The current Impala version that will be embedded in the Impala binary. This is
+# also used to find the Impala frontend jar files, so the version must match
+# the version in our Maven pom.xml files. This is validated via
+# bin/validate-java-pom-version.sh during the build.
+# WARNING: If changing this value, also run these commands:
+# cd ${IMPALA_HOME}/java
+# mvn versions:set -DnewVersion=YOUR_NEW_VERSION
+export IMPALA_VERSION=4.1.1-SNAPSHOT
+
# The unique build id of the toolchain to use if bootstrapping. This is generated by the
# native-toolchain build when publishing its build artifacts. This should be changed when
# moving to a different build of the toolchain, e.g. when a version is bumped or a
@@ -778,6 +787,7 @@ CLASSPATH="$IMPALA_FE_DIR/src/test/resources:$CLASSPATH"
# A marker in the environment to prove that we really did source this file
export IMPALA_CONFIG_SOURCED=1
+echo "IMPALA_VERSION = $IMPALA_VERSION"
echo "IMPALA_HOME = $IMPALA_HOME"
echo "HADOOP_HOME = $HADOOP_HOME"
echo "HADOOP_CONF_DIR = $HADOOP_CONF_DIR"
diff --git a/bin/jenkins/build-all-flag-combinations.sh b/bin/jenkins/build-all-flag-combinations.sh
index a6c0aba9b..94acd94fc 100755
--- a/bin/jenkins/build-all-flag-combinations.sh
+++ b/bin/jenkins/build-all-flag-combinations.sh
@@ -107,6 +107,10 @@ if [[ $# == 1 && $1 == "--dryrun" ]]; then
echo $DESCRIPTION
else
# Note: this command modifies files in the git checkout
+ pushd bin
+ sed "s#export IMPALA_VERSION=.*#export IMPALA_VERSION=${NEW_MAVEN_VERSION}#g"\
+ -i impala-config.sh
+ popd
pushd java
mvn versions:set -DnewVersion=${NEW_MAVEN_VERSION}
popd
diff --git a/bin/save-version.sh b/bin/save-version.sh
index 017a237e0..85e5f899e 100755
--- a/bin/save-version.sh
+++ b/bin/save-version.sh
@@ -21,7 +21,7 @@
# Note: for internal (aka pre-release) versions, the version should have
# "-INTERNAL" appended. Parts of the code will look for this to distinguish
# between released and internal versions.
-VERSION=4.1.1-SNAPSHOT
+VERSION=${IMPALA_VERSION}
GIT_HASH=$(git rev-parse HEAD 2> /dev/null)
if [ -z $GIT_HASH ]
then
diff --git a/bin/set-classpath.sh b/bin/set-classpath.sh
index 8eac82fba..d15bc00aa 100644
--- a/bin/set-classpath.sh
+++ b/bin/set-classpath.sh
@@ -28,11 +28,15 @@ if [ "$0" = "$BASH_SOURCE" ]; then
exit 1
fi
+# IMPALA-10057: The fe/target/classes and fe/target/test-classes
+# directory can get cleaned out / rebuilt as part of running the
+# frontend tests. This uses jars to avoid issues related to that
+# inconsistency.
CLASSPATH=\
"$IMPALA_HOME"/fe/src/test/resources:\
-"$IMPALA_HOME"/fe/target/classes:\
+"$IMPALA_HOME"/fe/target/impala-frontend-${IMPALA_VERSION}.jar:\
"$IMPALA_HOME"/fe/target/dependency:\
-"$IMPALA_HOME"/fe/target/test-classes:
+"$IMPALA_HOME"/fe/target/impala-frontend-${IMPALA_VERSION}-tests.jar:
FE_CP_FILE="$IMPALA_HOME/fe/target/build-classpath.txt"
diff --git a/bin/validate-java-pom-versions.sh b/bin/validate-java-pom-versions.sh
new file mode 100755
index 000000000..a996eb666
--- /dev/null
+++ b/bin/validate-java-pom-versions.sh
@@ -0,0 +1,68 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# This script verifies that the Java pom.xml files use the
+# IMPALA_VERSION. This can fail if the IMPALA_VERSION changes
+# without the version in the pom.xml changing. To update the
+# pom.xml files, run:
+# cd $IMPALA_HOME/java
+# mvn versions:set -DnewVersion=NEW_MAVEN_VERSION
+
+set -euo pipefail
+
+MAVEN_VERSION_STRING="<version>${IMPALA_VERSION}</version>"
+
+# Detect if IMPALA_HOME is a git repository
+pushd ${IMPALA_HOME} > /dev/null 2>&1
+IS_GIT_CHECKOUT=false
+if git ls-files --error-unmatch > /dev/null 2>&1 ; then
+ IS_GIT_CHECKOUT=true
+fi;
+popd > /dev/null 2>&1
+
+RETVAL=0
+NO_MATCH_FILES=()
+for pom_file in $(find ${IMPALA_HOME} -name pom.xml); do
+ # If this is a git checkout, then only do the check for pom.xml
+ # files known to git. If this is not a git checkout, then it should
+ # be building from a tarball, and there should not be extra
+ # pom.xml files.
+ if ${IS_GIT_CHECKOUT} &&
+ ! git ls-files --error-unmatch ${pom_file} > /dev/null 2>&1 ; then
+ # This pom.xml file is not known to git.
+ continue;
+ fi
+ if ! grep $MAVEN_VERSION_STRING "${pom_file}" > /dev/null; then
+ NO_MATCH_FILES+=(${pom_file})
+ RETVAL=1
+ fi
+done
+
+if [[ $RETVAL != 0 ]]; then
+ echo "Check for Java pom.xml versions FAILED"
+ echo "Expected ${MAVEN_VERSION_STRING}"
+ echo "Not found in:"
+ for pom_file in ${NO_MATCH_FILES[@]}; do
+ echo " ${pom_file}"
+ done
+ echo "The pom.xml files can be updated automatically via:"
+ echo 'cd ${IMPALA_HOME}/java; mvn versions:set -DnewVersion=YOUR_NEW_VERSION'
+fi
+
+exit $RETVAL
diff --git a/fe/pom.xml b/fe/pom.xml
index e3a28cfc8..e2ee4897a 100644
--- a/fe/pom.xml
+++ b/fe/pom.xml
@@ -23,7 +23,7 @@ under the License.
<parent>
<groupId>org.apache.impala</groupId>
<artifactId>impala-parent</artifactId>
- <version>4.1.0-SNAPSHOT</version>
+ <version>4.1.1-SNAPSHOT</version>
<relativePath>../java/pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
diff --git a/java/CMakeLists.txt b/java/CMakeLists.txt
index 08feb30ea..7fcef3312 100644
--- a/java/CMakeLists.txt
+++ b/java/CMakeLists.txt
@@ -15,6 +15,10 @@
# specific language governing permissions and limitations
# under the License.
-add_custom_target(java ALL DEPENDS gen-deps function-registry
+add_custom_target(validate_java_pom_versions ALL
+ COMMAND $ENV{IMPALA_HOME}/bin/validate-java-pom-versions.sh
+)
+
+add_custom_target(java ALL DEPENDS gen-deps function-registry validate_java_pom_versions
COMMAND $ENV{IMPALA_HOME}/bin/mvn-quiet.sh -B install -DskipTests
)
diff --git a/java/TableFlattener/pom.xml b/java/TableFlattener/pom.xml
index 789fd09c8..079719d5f 100644
--- a/java/TableFlattener/pom.xml
+++ b/java/TableFlattener/pom.xml
@@ -22,7 +22,7 @@
<parent>
<groupId>org.apache.impala</groupId>
<artifactId>impala-parent</artifactId>
- <version>4.1.0-SNAPSHOT</version>
+ <version>4.1.1-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>nested-table-flattener</artifactId>
diff --git a/java/datagenerator/pom.xml b/java/datagenerator/pom.xml
index 89e64ca1b..eec068a67 100644
--- a/java/datagenerator/pom.xml
+++ b/java/datagenerator/pom.xml
@@ -23,7 +23,7 @@ under the License.
<parent>
<groupId>org.apache.impala</groupId>
<artifactId>impala-parent</artifactId>
- <version>4.1.0-SNAPSHOT</version>
+ <version>4.1.1-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
diff --git a/java/executor-deps/pom.xml b/java/executor-deps/pom.xml
index 05d4aeb1c..436e15436 100644
--- a/java/executor-deps/pom.xml
+++ b/java/executor-deps/pom.xml
@@ -34,7 +34,7 @@ under the License.
<parent>
<groupId>org.apache.impala</groupId>
<artifactId>impala-parent</artifactId>
- <version>4.1.0-SNAPSHOT</version>
+ <version>4.1.1-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<groupId>org.apache.impala</groupId>
diff --git a/java/ext-data-source/api/pom.xml b/java/ext-data-source/api/pom.xml
index 11ef1ba71..738f6eb3f 100644
--- a/java/ext-data-source/api/pom.xml
+++ b/java/ext-data-source/api/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.impala</groupId>
<artifactId>impala-data-source</artifactId>
- <version>4.1.0-SNAPSHOT</version>
+ <version>4.1.1-SNAPSHOT</version>
</parent>
<artifactId>impala-data-source-api</artifactId>
<name>Apache Impala External Data Source API</name>
diff --git a/java/ext-data-source/pom.xml b/java/ext-data-source/pom.xml
index 89a514e29..ccd2788c5 100644
--- a/java/ext-data-source/pom.xml
+++ b/java/ext-data-source/pom.xml
@@ -22,7 +22,7 @@
<parent>
<groupId>org.apache.impala</groupId>
<artifactId>impala-parent</artifactId>
- <version>4.1.0-SNAPSHOT</version>
+ <version>4.1.1-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>impala-data-source</artifactId>
diff --git a/java/ext-data-source/sample/pom.xml b/java/ext-data-source/sample/pom.xml
index aa12f0d1f..c64024407 100644
--- a/java/ext-data-source/sample/pom.xml
+++ b/java/ext-data-source/sample/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.impala</groupId>
<artifactId>impala-data-source</artifactId>
- <version>4.1.0-SNAPSHOT</version>
+ <version>4.1.1-SNAPSHOT</version>
</parent>
<artifactId>impala-data-source-sample</artifactId>
<name>Apache Impala External Data Source Sample</name>
diff --git a/java/ext-data-source/test/pom.xml b/java/ext-data-source/test/pom.xml
index fd16e7892..282f00936 100644
--- a/java/ext-data-source/test/pom.xml
+++ b/java/ext-data-source/test/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.impala</groupId>
<artifactId>impala-data-source</artifactId>
- <version>4.1.0-SNAPSHOT</version>
+ <version>4.1.1-SNAPSHOT</version>
</parent>
<artifactId>impala-data-source-test</artifactId>
<name>Apache Impala External Data Source Test Library</name>
diff --git a/java/pom.xml b/java/pom.xml
index dbb931325..9ec75aa51 100644
--- a/java/pom.xml
+++ b/java/pom.xml
@@ -21,7 +21,7 @@ under the License.
<modelVersion>4.0.0</modelVersion>
<groupId>org.apache.impala</groupId>
<artifactId>impala-parent</artifactId>
- <version>4.1.0-SNAPSHOT</version>
+ <version>4.1.1-SNAPSHOT</version>
<packaging>pom</packaging>
<name>Apache Impala Parent POM</name>
diff --git a/java/query-event-hook-api/pom.xml b/java/query-event-hook-api/pom.xml
index eeae99595..4d49c5e77 100644
--- a/java/query-event-hook-api/pom.xml
+++ b/java/query-event-hook-api/pom.xml
@@ -22,7 +22,7 @@
<parent>
<groupId>org.apache.impala</groupId>
<artifactId>impala-parent</artifactId>
- <version>4.1.0-SNAPSHOT</version>
+ <version>4.1.1-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>query-event-hook-api</artifactId>
diff --git a/java/shaded-deps/hive-exec/pom.xml b/java/shaded-deps/hive-exec/pom.xml
index 642ba3e6e..ef4e4a484 100644
--- a/java/shaded-deps/hive-exec/pom.xml
+++ b/java/shaded-deps/hive-exec/pom.xml
@@ -27,7 +27,7 @@ the same dependencies
<parent>
<groupId>org.apache.impala</groupId>
<artifactId>impala-parent</artifactId>
- <version>4.1.0-SNAPSHOT</version>
+ <version>4.1.1-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
diff --git a/java/shaded-deps/s3a-aws-sdk/pom.xml b/java/shaded-deps/s3a-aws-sdk/pom.xml
index 9040a4635..1a782ab41 100644
--- a/java/shaded-deps/s3a-aws-sdk/pom.xml
+++ b/java/shaded-deps/s3a-aws-sdk/pom.xml
@@ -25,7 +25,7 @@ though some of them might not be necessary. The exclusions are sorted alphabetic
<parent>
<groupId>org.apache.impala</groupId>
<artifactId>impala-parent</artifactId>
- <version>4.1.0-SNAPSHOT</version>
+ <version>4.1.1-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
diff --git a/java/test-hive-udfs/pom.xml b/java/test-hive-udfs/pom.xml
index cd241bdae..24bc9fe2c 100644
--- a/java/test-hive-udfs/pom.xml
+++ b/java/test-hive-udfs/pom.xml
@@ -22,7 +22,7 @@ under the License.
<parent>
<groupId>org.apache.impala</groupId>
<artifactId>impala-parent</artifactId>
- <version>4.1.0-SNAPSHOT</version>
+ <version>4.1.1-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
diff --git a/java/yarn-extras/pom.xml b/java/yarn-extras/pom.xml
index d45fcd3e6..c2db16a63 100644
--- a/java/yarn-extras/pom.xml
+++ b/java/yarn-extras/pom.xml
@@ -22,7 +22,7 @@
<parent>
<groupId>org.apache.impala</groupId>
<artifactId>impala-parent</artifactId>
- <version>4.1.0-SNAPSHOT</version>
+ <version>4.1.1-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>yarn-extras</artifactId>