You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by jo...@apache.org on 2022/05/10 00:19:47 UTC
[impala] branch master updated (1f20b7aa5 -> 4118522b9)
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
from 1f20b7aa5 IMPALA-11234: Bump CDP_BUILD_NUMBER to 26065553
new 51b537a3c IMPALA-11244: Run the minicluster for docker-based BE tests
new 4118522b9 IMPALA-10057: Fix log spew by using jars in the classpath
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
bin/impala-config.sh | 10 +++++
bin/jenkins/build-all-flag-combinations.sh | 4 ++
bin/save-version.sh | 2 +-
bin/set-classpath.sh | 8 +++-
bin/validate-java-pom-versions.sh | 68 ++++++++++++++++++++++++++++++
docker/entrypoint.sh | 9 +---
java/CMakeLists.txt | 6 ++-
7 files changed, 95 insertions(+), 12 deletions(-)
create mode 100755 bin/validate-java-pom-versions.sh
[impala] 02/02: IMPALA-10057: Fix log spew by using jars in the classpath
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 4118522b9cd4d27dbabde6ead3d5647d72f8369d
Author: Joe McDonnell <jo...@cloudera.com>
AuthorDate: Wed Apr 13 15:18:17 2022 -0700
IMPALA-10057: Fix log spew by using jars in the classpath
Some tests saw log spew that causes the INFO log files to
be filled with output like this:
E0903 02:25:39.453887 12060 TransactionKeepalive.java:137] Unexpected exception thrown
Java exception follows:
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/impala/common/TransactionKeepalive$HeartbeatContext
at org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114)
at java.lang.Thread.run(Thread.java:748)
...
It turns out that the catalogd/impalad use a CLASSPATH in
tests that refers to fe/target/classes. The maven command
that runs frontend tests recompiles these classes and
causes the files in fe/target/classes to be deleted and
recreated. There are race conditions where this causes
the symptoms above.
This changes the CLASSPATH to use the frontend jars, which
are not impacted by the machinations on fe/target/classes.
To find the appropriate jar, set-classpath.sh needs to
know the Impala version. This adds IMPALA_VERSION in
bin/impala-config.sh to provide an easy to use
environment variable.
To make the versioning more uniform, this modifies
bin/save-version.sh to use this environment variable.
It also adds a check to make sure that the Java pom.xml
files use the same version as the environment variable.
It fails the build if the Java pom.xml files do not
match.
Testing:
- Ran core jobs
- Checked the log file sizes on jobs
- Changed a Java pom.xml's version and verified that
bin/validate-java-pom-versions.sh fails
Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Reviewed-on: http://gerrit.cloudera.org:8080/18415
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Michael Smith <mi...@cloudera.com>
Reviewed-by: Quanlong Huang <hu...@gmail.com>
---
bin/impala-config.sh | 10 +++++
bin/jenkins/build-all-flag-combinations.sh | 4 ++
bin/save-version.sh | 2 +-
bin/set-classpath.sh | 8 +++-
bin/validate-java-pom-versions.sh | 68 ++++++++++++++++++++++++++++++
java/CMakeLists.txt | 6 ++-
6 files changed, 94 insertions(+), 4 deletions(-)
diff --git a/bin/impala-config.sh b/bin/impala-config.sh
index 49eb3f20a..e5c1c4527 100755
--- a/bin/impala-config.sh
+++ b/bin/impala-config.sh
@@ -63,6 +63,15 @@ fi
# variable from a previously-sourced impala-config.sh overrides the new value. #
#######################################################################################
+# The current Impala version that will be embedded in the Impala binary. This is
+# also used to find the Impala frontend jar files, so the version must match
+# the version in our Maven pom.xml files. This is validated via
+# bin/validate-java-pom-version.sh during the build.
+# WARNING: If changing this value, also run these commands:
+# cd ${IMPALA_HOME}/java
+# mvn versions:set -DnewVersion=YOUR_NEW_VERSION
+export IMPALA_VERSION=4.1.0-SNAPSHOT
+
# The unique build id of the toolchain to use if bootstrapping. This is generated by the
# native-toolchain build when publishing its build artifacts. This should be changed when
# moving to a different build of the toolchain, e.g. when a version is bumped or a
@@ -778,6 +787,7 @@ CLASSPATH="$IMPALA_FE_DIR/src/test/resources:$CLASSPATH"
# A marker in the environment to prove that we really did source this file
export IMPALA_CONFIG_SOURCED=1
+echo "IMPALA_VERSION = $IMPALA_VERSION"
echo "IMPALA_HOME = $IMPALA_HOME"
echo "HADOOP_HOME = $HADOOP_HOME"
echo "HADOOP_CONF_DIR = $HADOOP_CONF_DIR"
diff --git a/bin/jenkins/build-all-flag-combinations.sh b/bin/jenkins/build-all-flag-combinations.sh
index a6c0aba9b..94acd94fc 100755
--- a/bin/jenkins/build-all-flag-combinations.sh
+++ b/bin/jenkins/build-all-flag-combinations.sh
@@ -107,6 +107,10 @@ if [[ $# == 1 && $1 == "--dryrun" ]]; then
echo $DESCRIPTION
else
# Note: this command modifies files in the git checkout
+ pushd bin
+ sed "s#export IMPALA_VERSION=.*#export IMPALA_VERSION=${NEW_MAVEN_VERSION}#g"\
+ -i impala-config.sh
+ popd
pushd java
mvn versions:set -DnewVersion=${NEW_MAVEN_VERSION}
popd
diff --git a/bin/save-version.sh b/bin/save-version.sh
index 84843ec76..85e5f899e 100755
--- a/bin/save-version.sh
+++ b/bin/save-version.sh
@@ -21,7 +21,7 @@
# Note: for internal (aka pre-release) versions, the version should have
# "-INTERNAL" appended. Parts of the code will look for this to distinguish
# between released and internal versions.
-VERSION=4.1.0-SNAPSHOT
+VERSION=${IMPALA_VERSION}
GIT_HASH=$(git rev-parse HEAD 2> /dev/null)
if [ -z $GIT_HASH ]
then
diff --git a/bin/set-classpath.sh b/bin/set-classpath.sh
index 8eac82fba..d15bc00aa 100644
--- a/bin/set-classpath.sh
+++ b/bin/set-classpath.sh
@@ -28,11 +28,15 @@ if [ "$0" = "$BASH_SOURCE" ]; then
exit 1
fi
+# IMPALA-10057: The fe/target/classes and fe/target/test-classes
+# directory can get cleaned out / rebuilt as part of running the
+# frontend tests. This uses jars to avoid issues related to that
+# inconsistency.
CLASSPATH=\
"$IMPALA_HOME"/fe/src/test/resources:\
-"$IMPALA_HOME"/fe/target/classes:\
+"$IMPALA_HOME"/fe/target/impala-frontend-${IMPALA_VERSION}.jar:\
"$IMPALA_HOME"/fe/target/dependency:\
-"$IMPALA_HOME"/fe/target/test-classes:
+"$IMPALA_HOME"/fe/target/impala-frontend-${IMPALA_VERSION}-tests.jar:
FE_CP_FILE="$IMPALA_HOME/fe/target/build-classpath.txt"
diff --git a/bin/validate-java-pom-versions.sh b/bin/validate-java-pom-versions.sh
new file mode 100755
index 000000000..a996eb666
--- /dev/null
+++ b/bin/validate-java-pom-versions.sh
@@ -0,0 +1,68 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# This script verifies that the Java pom.xml files use the
+# IMPALA_VERSION. This can fail if the IMPALA_VERSION changes
+# without the version in the pom.xml changing. To update the
+# pom.xml files, run:
+# cd $IMPALA_HOME/java
+# mvn versions:set -DnewVersion=NEW_MAVEN_VERSION
+
+set -euo pipefail
+
+MAVEN_VERSION_STRING="<version>${IMPALA_VERSION}</version>"
+
+# Detect if IMPALA_HOME is a git repository
+pushd ${IMPALA_HOME} > /dev/null 2>&1
+IS_GIT_CHECKOUT=false
+if git ls-files --error-unmatch > /dev/null 2>&1 ; then
+ IS_GIT_CHECKOUT=true
+fi;
+popd > /dev/null 2>&1
+
+RETVAL=0
+NO_MATCH_FILES=()
+for pom_file in $(find ${IMPALA_HOME} -name pom.xml); do
+ # If this is a git checkout, then only do the check for pom.xml
+ # files known to git. If this is not a git checkout, then it should
+ # be building from a tarball, and there should not be extra
+ # pom.xml files.
+ if ${IS_GIT_CHECKOUT} &&
+ ! git ls-files --error-unmatch ${pom_file} > /dev/null 2>&1 ; then
+ # This pom.xml file is not known to git.
+ continue;
+ fi
+ if ! grep $MAVEN_VERSION_STRING "${pom_file}" > /dev/null; then
+ NO_MATCH_FILES+=(${pom_file})
+ RETVAL=1
+ fi
+done
+
+if [[ $RETVAL != 0 ]]; then
+ echo "Check for Java pom.xml versions FAILED"
+ echo "Expected ${MAVEN_VERSION_STRING}"
+ echo "Not found in:"
+ for pom_file in ${NO_MATCH_FILES[@]}; do
+ echo " ${pom_file}"
+ done
+ echo "The pom.xml files can be updated automatically via:"
+ echo 'cd ${IMPALA_HOME}/java; mvn versions:set -DnewVersion=YOUR_NEW_VERSION'
+fi
+
+exit $RETVAL
diff --git a/java/CMakeLists.txt b/java/CMakeLists.txt
index 08feb30ea..7fcef3312 100644
--- a/java/CMakeLists.txt
+++ b/java/CMakeLists.txt
@@ -15,6 +15,10 @@
# specific language governing permissions and limitations
# under the License.
-add_custom_target(java ALL DEPENDS gen-deps function-registry
+add_custom_target(validate_java_pom_versions ALL
+ COMMAND $ENV{IMPALA_HOME}/bin/validate-java-pom-versions.sh
+)
+
+add_custom_target(java ALL DEPENDS gen-deps function-registry validate_java_pom_versions
COMMAND $ENV{IMPALA_HOME}/bin/mvn-quiet.sh -B install -DskipTests
)
[impala] 01/02: IMPALA-11244: Run the minicluster for docker-based BE tests
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 51b537a3cd4e7d5f3448f0bb8ffce7d69711f512
Author: Joe McDonnell <jo...@cloudera.com>
AuthorDate: Wed Apr 13 14:02:51 2022 -0700
IMPALA-11244: Run the minicluster for docker-based BE tests
As an optimization, the docker-based tests didn't run
the minicluster for BE tests. Some BE tests now require
the minicluster (DiskIoMgrTest.WriteToRemote*), so this
cannot work with the optimization.
This changes the docker-based tests to start the minicluster
for the BE tests.
Testing:
- Ran a docker-based test job
Change-Id: I784a63a02886852e10ccca7c118c22ff7d38b8a3
Reviewed-on: http://gerrit.cloudera.org:8080/18414
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Joe McDonnell <jo...@cloudera.com>
---
docker/entrypoint.sh | 9 +--------
1 file changed, 1 insertion(+), 8 deletions(-)
diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh
index 3c3cfbb1f..a26585a1a 100755
--- a/docker/entrypoint.sh
+++ b/docker/entrypoint.sh
@@ -407,16 +407,9 @@ function test_suite() {
SKIP_TOOLCHAIN_BOOTSTRAP=true ./buildall.sh -noclean -notests -asan
fi
- # BE tests don't require the minicluster, so we can run them directly.
+ # Build the BE test binaries if needed.
if [[ $1 = BE_TEST* ]]; then
make -j$(nproc) --load-average=$(nproc) be-test be-benchmarks
- if ! bin/run-backend-tests.sh; then
- echo "Tests $1 failed!"
- return 1
- else
- echo "Tests $1 succeeded!"
- return 0
- fi
fi
if [[ $1 == RAT_CHECK ]]; then