You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by mi...@apache.org on 2023/07/25 17:22:21 UTC
[impala] branch master updated (ee069687f -> 7fb6a9a1d)
This is an automated email from the ASF dual-hosted git repository.
michaelsmith pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
from ee069687f IMPALA-12212: Bump Maven to 3.9.2, pull dependencies in parallel
new 0d3fc33bb IMPALA-12300: (addendum) Remove HDFS specific assertion
new b7dd3c267 IMPALA-12296: Fix filenames in query profile download page
new 7fb6a9a1d IMPALA-11941: (Addendum) Use released jamm 0.4.0
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/common/init.cc | 4 +-
bin/run-all-tests.sh | 2 +-
fe/pom.xml | 12 ++----
java/.gitignore | 1 -
java/CMakeLists.txt | 20 +--------
java/pom.xml | 1 -
java/shaded-deps/jamm/.gitignore | 1 -
java/shaded-deps/jamm/pom.xml | 64 ----------------------------
java/toolchains.xml.tmpl | 35 ---------------
tests/custom_cluster/test_executor_groups.py | 1 -
tests/verifiers/test_banned_log_messages.py | 3 --
www/query_profile.tmpl | 9 ++--
12 files changed, 13 insertions(+), 140 deletions(-)
delete mode 100644 java/.gitignore
delete mode 100644 java/shaded-deps/jamm/.gitignore
delete mode 100644 java/shaded-deps/jamm/pom.xml
delete mode 100644 java/toolchains.xml.tmpl
[impala] 02/03: IMPALA-12296: Fix filenames in query profile download page
Posted by mi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
michaelsmith pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit b7dd3c267a6e5ec092297bde853681f77c0da50c
Author: Surya Hebbar <sh...@cloudera.com>
AuthorDate: Tue Jul 18 20:49:55 2023 +0530
IMPALA-12296: Fix filenames in query profile download page
The downloads of query profiles from the profile page contain ':'
in the filename, this is not supported by some filesystems
such as NTFS.
Most browsers replace it with a white-space during the download.
Hence, characters except letters and white spaces in downloaded
profile's filename should be replaced with '_'.
Tests in 'test_web_pages.py' do not need modifications as
the endpoint for profile downloads has not changed.
Change-Id: Ia170e0ae1adc6f3347b57ca8572d65a05ebf9544
Reviewed-on: http://gerrit.cloudera.org:8080/20215
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
www/query_profile.tmpl | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/www/query_profile.tmpl b/www/query_profile.tmpl
index 55c79fb94..2f1f194c6 100644
--- a/www/query_profile.tmpl
+++ b/www/query_profile.tmpl
@@ -27,13 +27,13 @@ under the License.
<br/>
<div>
<h4>Download Profile (Available Formats):
- <a style="font-size:16px;" class="btn btn-primary"
+ <a style="font-size:16px;" class="btn btn-primary profile-download"
href="{{ __common__.host-url }}/query_profile_encoded?query_id={{query_id}}"
download="thrift_profile_{{query_id}}">Thrift</a>
- <a style="font-size:16px;" class="btn btn-primary"
+ <a style="font-size:16px;" class="btn btn-primary profile-download"
href="{{ __common__.host-url }}/query_profile_json?query_id={{query_id}}"
download="json_profile_{{query_id}}">Json</a>
- <a style="font-size:16px;" class="btn btn-primary"
+ <a style="font-size:16px;" class="btn btn-primary profile-download"
href="{{ __common__.host-url }}/query_profile_plain_text?query_id={{query_id}}"
download="profile_{{query_id}}">Text</a>
</h4>
@@ -43,6 +43,9 @@ under the License.
<script>
$("#profile-tab").addClass("active");
+document.querySelectorAll('.profile-download').forEach(function (profile_link) {
+ profile_link.download = profile_link.download.replace(/\W/g,'_');
+});
</script>
{{> www/common-footer.tmpl }}
[impala] 01/03: IMPALA-12300: (addendum) Remove HDFS specific assertion
Posted by mi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
michaelsmith pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 0d3fc33bbbe8313788547d30b507a0b0effc4951
Author: Riza Suminto <ri...@cloudera.com>
AuthorDate: Mon Jul 24 08:02:13 2023 -0700
IMPALA-12300: (addendum) Remove HDFS specific assertion
test_75_percent_availability fail against Ozone and S3 test environment
because test expects the string "SCAN HDFS" to be found in the profile.
Instead of it there's "SCAN OZONE" and "SCAN S3" for Ozone and S3 test
environment respectively. This patch fix the test by removing that
assertion from test_75_percent_availability. The remaining assertion is
enough to verify that FE planner and BE scheduler can see cluster
membership change.
Change-Id: Id14934d2fce0f6cf03242c36c0142bc697b4180e
Reviewed-on: http://gerrit.cloudera.org:8080/20259
Reviewed-by: Wenzhe Zhou <wz...@cloudera.com>
Reviewed-by: Michael Smith <mi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
tests/custom_cluster/test_executor_groups.py | 1 -
1 file changed, 1 deletion(-)
diff --git a/tests/custom_cluster/test_executor_groups.py b/tests/custom_cluster/test_executor_groups.py
index 42c900b47..bd6ffca81 100644
--- a/tests/custom_cluster/test_executor_groups.py
+++ b/tests/custom_cluster/test_executor_groups.py
@@ -1437,5 +1437,4 @@ class TestExecutorGroups(CustomClusterTestSuite):
assert ("Scheduler Warning: Cluster membership might changed between planning and "
"scheduling, F00 scheduled instance count (16) is higher than its effective "
"count (12)") in profile, profile
- assert "00:SCAN HDFS 8 16" in profile, profile
self.client.close_query(handle)
[impala] 03/03: IMPALA-11941: (Addendum) Use released jamm 0.4.0
Posted by mi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
michaelsmith pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 7fb6a9a1d2e5f524498848502da372a8a5965268
Author: Michael Smith <mi...@cloudera.com>
AuthorDate: Thu Jul 20 15:45:14 2023 -0700
IMPALA-11941: (Addendum) Use released jamm 0.4.0
Switches to the 0.4.0 release of jamm, as building a shaded JAR from
source was a temporary measure.
Change-Id: I5b88b479580f7d0baff502ad9551d2764971babf
Reviewed-on: http://gerrit.cloudera.org:8080/20237
Reviewed-by: Laszlo Gaal <la...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
be/src/common/init.cc | 4 +-
bin/run-all-tests.sh | 2 +-
fe/pom.xml | 12 ++----
java/.gitignore | 1 -
java/CMakeLists.txt | 20 +--------
java/pom.xml | 1 -
java/shaded-deps/jamm/.gitignore | 1 -
java/shaded-deps/jamm/pom.xml | 64 -----------------------------
java/toolchains.xml.tmpl | 35 ----------------
tests/verifiers/test_banned_log_messages.py | 3 --
10 files changed, 7 insertions(+), 136 deletions(-)
diff --git a/be/src/common/init.cc b/be/src/common/init.cc
index 8229f0a08..3409dde52 100644
--- a/be/src/common/init.cc
+++ b/be/src/common/init.cc
@@ -351,11 +351,11 @@ static Status JavaAddJammAgent() {
istringstream classpath {getenv("CLASSPATH")};
string jamm_path, test_path;
while (getline(classpath, test_path, ':')) {
- jamm_path = FileSystemUtil::FindFileInPath(test_path, "impala-jamm-.*.jar");
+ jamm_path = FileSystemUtil::FindFileInPath(test_path, "jamm-.*.jar");
if (!jamm_path.empty()) break;
}
if (jamm_path.empty()) {
- return Status("Could not find impala-jamm-*.jar in Java CLASSPATH");
+ return Status("Could not find jamm-*.jar in Java CLASSPATH");
}
val_out << "-javaagent:" << jamm_path;
diff --git a/bin/run-all-tests.sh b/bin/run-all-tests.sh
index 81be57ae0..8f419c777 100755
--- a/bin/run-all-tests.sh
+++ b/bin/run-all-tests.sh
@@ -245,7 +245,7 @@ do
pushd "${IMPALA_FE_DIR}"
# Add Jamm as javaagent for CatalogdMetaProviderTest.testWeights
- JAMM_JAR=$(compgen -G ${IMPALA_HOME}/fe/target/dependency/impala-jamm-*.jar)
+ JAMM_JAR=$(compgen -G ${IMPALA_HOME}/fe/target/dependency/jamm-*.jar)
export JAVA_TOOL_OPTIONS="${JAVA_TOOL_OPTIONS-} -javaagent:${JAMM_JAR}"
if $JAVA -version 2>&1 | grep -q -E ' version "(9|[1-9][0-9])\.'; then
diff --git a/fe/pom.xml b/fe/pom.xml
index 3b9fd5fab..b817f3fb9 100644
--- a/fe/pom.xml
+++ b/fe/pom.xml
@@ -364,15 +364,9 @@ under the License.
</dependency>
<dependency>
- <groupId>org.apache.impala</groupId>
- <artifactId>impala-jamm</artifactId>
- <version>${project.version}</version>
- <exclusions>
- <exclusion>
- <groupId>*</groupId>
- <artifactId>*</artifactId>
- </exclusion>
- </exclusions>
+ <groupId>com.github.jbellis</groupId>
+ <artifactId>jamm</artifactId>
+ <version>0.4.0</version>
</dependency>
<dependency>
diff --git a/java/.gitignore b/java/.gitignore
deleted file mode 100644
index 173495683..000000000
--- a/java/.gitignore
+++ /dev/null
@@ -1 +0,0 @@
-jamm-prefix/
diff --git a/java/CMakeLists.txt b/java/CMakeLists.txt
index c39cee3b6..d26c4614e 100644
--- a/java/CMakeLists.txt
+++ b/java/CMakeLists.txt
@@ -19,24 +19,6 @@ add_custom_target(validate_java_pom_versions ALL
COMMAND $ENV{IMPALA_HOME}/bin/validate-java-pom-versions.sh
)
-SET(JAMM_MAVEN_FLAGS --show-version --batch-mode)
-# Discover and download dependencies in parallel, using 10 threads
-SET(JAMM_MAVEN_FLAGS ${JAMM_MAVEN_FLAGS} -Daether.dependencyCollector.impl=bf)
-SET(JAMM_MAVEN_FLAGS ${JAMM_MAVEN_FLAGS} -Daether.connector.basic.threads=10)
-SET(JAMM_MAVEN_FLAGS ${JAMM_MAVEN_FLAGS} -Daether.dependencyCollector.bf.threads=10)
-
-include(ExternalProject)
-ExternalProject_Add(jamm
- GIT_REPOSITORY https://github.com/jbellis/jamm
- # 0.4.0 RC. A release is expected soon: https://github.com/jbellis/jamm/issues/44
- GIT_TAG 49c3e691c43a39ddaccaf19839ef316807009377
- PATCH_COMMAND sed -i.bak s/0.4.0-SNAPSHOT/0.4.0-IMPALA/ pom.xml
- CONFIGURE_COMMAND sed s:JAVA_HOME:$ENV{JAVA_HOME}:g ${CMAKE_CURRENT_SOURCE_DIR}/toolchains.xml.tmpl > toolchains.xml
- BUILD_IN_SOURCE true
- BUILD_COMMAND mvn install ${JAMM_MAVEN_FLAGS} --toolchains toolchains.xml -Dmaven.test.skip=true -DskipTests
- INSTALL_COMMAND ""
-)
-
-add_custom_target(java ALL DEPENDS gen-deps function-registry geospatial-udf-wrappers validate_java_pom_versions jamm
+add_custom_target(java ALL DEPENDS gen-deps function-registry geospatial-udf-wrappers validate_java_pom_versions
COMMAND $ENV{IMPALA_HOME}/bin/mvn-quiet.sh -B install -DskipTests
)
diff --git a/java/pom.xml b/java/pom.xml
index 29a8a7c3b..76364f06d 100644
--- a/java/pom.xml
+++ b/java/pom.xml
@@ -370,7 +370,6 @@ under the License.
<module>../fe</module>
<module>query-event-hook-api</module>
<module>shaded-deps/hive-exec</module>
- <module>shaded-deps/jamm</module>
<module>shaded-deps/s3a-aws-sdk</module>
<module>TableFlattener</module>
<module>test-hive-udfs</module>
diff --git a/java/shaded-deps/jamm/.gitignore b/java/shaded-deps/jamm/.gitignore
deleted file mode 100644
index 916e17c09..000000000
--- a/java/shaded-deps/jamm/.gitignore
+++ /dev/null
@@ -1 +0,0 @@
-dependency-reduced-pom.xml
diff --git a/java/shaded-deps/jamm/pom.xml b/java/shaded-deps/jamm/pom.xml
deleted file mode 100644
index 345b762a4..000000000
--- a/java/shaded-deps/jamm/pom.xml
+++ /dev/null
@@ -1,64 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-<project xmlns="http://maven.apache.org/POM/4.0.0"
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
- http://maven.apache.org/xsd/maven-4.0.0.xsd">
-
- <!-- This pom shades an unreleased version of jamm, as others need the JAR available to
- use Impala jars. -->
- <parent>
- <groupId>org.apache.impala</groupId>
- <artifactId>impala-parent</artifactId>
- <version>4.3.0-SNAPSHOT</version>
- <relativePath>../../pom.xml</relativePath>
- </parent>
- <modelVersion>4.0.0</modelVersion>
- <artifactId>impala-jamm</artifactId>
- <packaging>jar</packaging>
-
- <dependencies>
- <dependency>
- <groupId>com.github.jbellis</groupId>
- <artifactId>jamm</artifactId>
- <version>0.4.0-IMPALA</version>
- </dependency>
- </dependencies>
- <build>
- <plugins>
- <plugin>
- <artifactId>maven-shade-plugin</artifactId>
- <version>3.4.1</version>
- <executions>
- <execution>
- <phase>package</phase>
- <goals>
- <goal>shade</goal>
- </goals>
- <configuration>
- <transformers>
- <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
- <manifestEntries>
- <Premain-Class>org.github.jamm.MemoryMeter</Premain-Class>
- </manifestEntries>
- </transformer>
- </transformers>
- </configuration>
- </execution>
- </executions>
- </plugin>
- </plugins>
- </build>
-</project>
diff --git a/java/toolchains.xml.tmpl b/java/toolchains.xml.tmpl
deleted file mode 100644
index c5e0215ef..000000000
--- a/java/toolchains.xml.tmpl
+++ /dev/null
@@ -1,35 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<toolchains>
- <!-- Provide toolchain entries for all versions Jamm expects. We don't actually use
- 11 or 17 to build or run tests however, so what they point to doesn't matter. -->
- <toolchain>
- <type>jdk</type>
- <provides>
- <version>1.8</version>
- <vendor>Oracle Corporation</vendor>
- </provides>
- <configuration>
- <jdkHome>JAVA_HOME</jdkHome>
- </configuration>
- </toolchain>
- <toolchain>
- <type>jdk</type>
- <provides>
- <version>11</version>
- <vendor>Oracle Corporation</vendor>
- </provides>
- <configuration>
- <jdkHome>JAVA_HOME</jdkHome>
- </configuration>
- </toolchain>
- <toolchain>
- <type>jdk</type>
- <provides>
- <version>17</version>
- <vendor>Oracle Corporation</vendor>
- </provides>
- <configuration>
- <jdkHome>JAVA_HOME</jdkHome>
- </configuration>
- </toolchain>
-</toolchains>
diff --git a/tests/verifiers/test_banned_log_messages.py b/tests/verifiers/test_banned_log_messages.py
index 627434139..1f4d52162 100644
--- a/tests/verifiers/test_banned_log_messages.py
+++ b/tests/verifiers/test_banned_log_messages.py
@@ -33,9 +33,6 @@ class TestBannedLogMessages(ImpalaTestSuite):
def assert_message_absent(self, message, log_dir=os.environ["IMPALA_LOGS_DIR"]):
for root, _, files in os.walk(log_dir):
for file in files:
- if file == 'mvn.log':
- # Skip mvn.log as some builds warn about extra shaded classes
- continue
log_file_path = os.path.join(root, file)
returncode = subprocess.call(['grep', message, log_file_path])
assert returncode == 1, "%s contains '%s'" % (log_file_path, message)