You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by ac...@apache.org on 2024/09/17 17:00:25 UTC
(kudu) branch branch-1.17.x updated (f8399fa7b -> bf74530ff)
This is an automated email from the ASF dual-hosted git repository.
achennaka pushed a change to branch branch-1.17.x
in repository https://gitbox.apache.org/repos/asf/kudu.git
from f8399fa7b [java] Fix build failures
new f1839b76c [cmake] work around of Gradle-related issues
new bf74530ff [cmake] work around of Gradle-related issues (take 2)
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CMakeLists.txt | 24 +++++++++++++++++++++++-
src/kudu/hms/CMakeLists.txt | 4 +++-
2 files changed, 26 insertions(+), 2 deletions(-)
(kudu) 02/02: [cmake] work around of Gradle-related issues (take 2)
Posted by ac...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
achennaka pushed a commit to branch branch-1.17.x
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit bf74530fff9d5c7888071395af31ba0670bd24bc
Author: Alexey Serbin <al...@apache.org>
AuthorDate: Tue Aug 13 20:26:50 2024 -0700
[cmake] work around of Gradle-related issues (take 2)
This is a follow-up to a1cc227df5d5dc7586d3971135643320682b2709.
Change-Id: I5875d7fff1304be1da93c9f4f9b6fb28bda3bdbb
Reviewed-on: http://gerrit.cloudera.org:8080/21669
Tested-by: Kudu Jenkins
Reviewed-by: Zoltan Chovan <zc...@cloudera.com>
Reviewed-by: Abhishek Chennaka <ac...@cloudera.com>
(cherry picked from commit ff69c2e312e5385037db214f50605979fd26a9f5)
Reviewed-on: http://gerrit.cloudera.org:8080/21810
Reviewed-by: Alexey Serbin <al...@apache.org>
Tested-by: Abhishek Chennaka <ac...@cloudera.com>
---
CMakeLists.txt | 36 +++++++++++++++++++++---------------
src/kudu/hms/CMakeLists.txt | 4 +++-
2 files changed, 24 insertions(+), 16 deletions(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index f8c3ac328..ba80831aa 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -112,21 +112,7 @@ separate_arguments(GRADLE_FLAGS)
# When invoking Gradle:
# * Log errors only, reducing the verbosity of the Gradle's output
# * Make the output better fit for plain consoles, avoiding garbled logs
-# * Avoid building projects in parallel: when the kudu-subprocess.jar and
-# the hms-plugin.jar targets are built in parallel, it often ends up with
-# java.lang.ArrayIndexOutOfBoundsException while building shadowJar
-# for corresponding Java dependencies, and while it's not yet clear whether
-# the issue is in shadowJar implementation or elsewhere, it's quite simple
-# to address the issue by not building multiple Gradle targets concurrently
-# * Disable the build cache: sometimes, once a build process is interrupted,
-# follow-up Gradle invocations end up with an error like below, and while
-# it's not yet clear whether the issue is in not properly invoking Gradle
-# or it's a bug in signal handling elsewhere, it's quite simple to address
-# the issue by not using the Gradle's build cache
-#
-# Cannot lock Java compile cache (...) as it has already been locked by this process.
-#
-list(APPEND GRADLE_FLAGS --quiet --console=plain --no-parallel --no-build-cache)
+list(APPEND GRADLE_FLAGS --quiet --console=plain)
list(REMOVE_DUPLICATES GRADLE_FLAGS)
# Initialize Gradle to ensure the wrapper is downloaded.
@@ -150,6 +136,26 @@ add_custom_command(OUTPUT ${PROTO_JAR}
DEPENDS init_gradle)
add_custom_target(proto_jar DEPENDS ${PROTO_JAR})
+# An artificial target for sequencing of building the subprocess_jar and the
+# hms_plugin_jar targets. The sequencing is to work around the following issues
+# when running multiple gradle instances in parallel, concurrently building
+# the targets:
+#
+# * Sometimes it ends up with java.lang.ArrayIndexOutOfBoundsException
+# while building shadowJar for corresponding Java dependencies,
+# and while it's not yet clear whether the issue is in shadowJar
+# implementation or elsewhere, it's quite simple to address the issue
+# by not running multiple Gradle instances concurrently
+#
+# * Sometimes the following error is reported, as seen elsewhere [1][2]:
+#
+# Timeout waiting to lock Java compile cache (...). It is currently in use by another Gradle instance.
+#
+# [1] https://github.com/gradle/gradle/issues/8375
+# [2] https://github.com/gradle/gradle/issues/11861
+#
+add_custom_target(subprocess_jar_seq DEPENDS subprocess_jar)
+
# Allow "make install" to not depend on all targets.
#
# Must be declared in the top-level CMakeLists.txt.
diff --git a/src/kudu/hms/CMakeLists.txt b/src/kudu/hms/CMakeLists.txt
index f777e007d..531710b73 100644
--- a/src/kudu/hms/CMakeLists.txt
+++ b/src/kudu/hms/CMakeLists.txt
@@ -72,7 +72,9 @@ add_custom_command(OUTPUT ${HMS_PLUGIN_JAR}
"${HMS_PLUGIN_JAR}"
WORKING_DIRECTORY "${JAVA_DIR}"
DEPENDS proto_jar)
-add_custom_target(hms_plugin_jar DEPENDS ${HMS_PLUGIN_JAR})
+# NOTE: 'subprocess_jar_seq' is an artificial dependency for 'hms_plugin_jar';
+# see the comment in the project's top-level CMakeLists.txt.
+add_custom_target(hms_plugin_jar DEPENDS ${HMS_PLUGIN_JAR} subprocess_jar_seq)
# The mini_hms is used only for tests.
if (KUDU_CLI_TEST_TOOL_ENABLED)
(kudu) 01/02: [cmake] work around of Gradle-related issues
Posted by ac...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
achennaka pushed a commit to branch branch-1.17.x
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit f1839b76cc76ad0e17dab2f8a763a8a5ebfd3370
Author: Alexey Serbin <al...@apache.org>
AuthorDate: Wed Jul 31 11:58:55 2024 -0700
[cmake] work around of Gradle-related issues
Once Gradle started being invoked to build kudu-subprocess and kudu-hms
JARs as dependency for the C++ server-side of the project, it's become
an issue to get 100% success rate when building it from source. With
the recent Gradle upgrade the build success rate has reduced further,
it seems.
I decided to address that by adding a few extra flags to the Gradle's
invocations under cmake when building corresponding targets. Probably,
there is a better way of addressing that (say, finding the root cause
of the issues, reporting, and fixing them), but this simple solution
works well for me. Also, this solution doesn't affect the build times
of the project (even when using ccache) since the targets built with
Gradle constitute a small part of the rest of the targets that are built
concurrently in a very robust manner.
Prior to this patch, Gradle would error out on follow-up invocations
after the build process has been interrupted, not being able to access
its build cache with an error like below:
--------------------------------------------------
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':kudu-hive:compileJava'.
> Could not create service of type DefaultGeneralCompileCaches using GradleScopeCompileServices.createGeneralCompileCaches().
> Cannot lock Java compile cache (...) as it has already been locked by this process.
--------------------------------------------------
Prior to this patch, building the project would fail sometimes when
invoking 'cmake -jN' (where N > 1) with errors like below while Gradle
was generating kudu-subprocess.jar and hms-plugin.jar concurrently:
--------------------------------------------------
...
[ 4%] Generating ../../../bin/kudu-subprocess.jar
...
[ 6%] Generating ../../../bin/hms-plugin.jar
...
java.lang.ArrayIndexOutOfBoundsException: 6
at shadow.org.objectweb.asm.ClassReader.readShort(ClassReader.java:3573)
...
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
at java.lang.Thread.run(Thread.java:750)
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':kudu-client:shadowJar'.
> 6
...
BUILD FAILED in 1m 0s
make[2]: *** [bin/hms-plugin.jar] Error 1
make[1]: *** [src/kudu/hms/CMakeFiles/hms_plugin_jar.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
java.lang.ArrayIndexOutOfBoundsException: 6
at shadow.org.objectweb.asm.ClassReader.readShort(ClassReader.java:3573)
...
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:624)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnabl
e.run(ThreadFactoryImpl.java:56)
at java.lang.Thread.run(Thread.java:750)
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':kudu-subprocess:shadowJar'.
> 6
...
BUILD FAILED in 1m 3s
make[2]: *** [bin/kudu-subprocess.jar] Error 1
make[1]: *** [src/kudu/subprocess/CMakeFiles/subprocess_jar.dir/all] Error 2
make: *** [all] Error 2
--------------------------------------------------
Change-Id: Ib7d1637854233592ede5388bfc2287df96ec9865
Reviewed-on: http://gerrit.cloudera.org:8080/21626
Reviewed-by: Abhishek Chennaka <ac...@cloudera.com>
Tested-by: Alexey Serbin <al...@apache.org>
(cherry picked from commit a1cc227df5d5dc7586d3971135643320682b2709)
Reviewed-on: http://gerrit.cloudera.org:8080/21808
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <al...@apache.org>
---
CMakeLists.txt | 20 ++++++++++++++++++--
1 file changed, 18 insertions(+), 2 deletions(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index d4c9f4fc7..f8c3ac328 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -109,8 +109,24 @@ set(JAVA_DIR ${CMAKE_CURRENT_SOURCE_DIR}/java)
# This solves the problem of escaped spaces in the EXTRA_GRADLE_FLAGS environment variable.
list(APPEND GRADLE_FLAGS $ENV{EXTRA_GRADLE_FLAGS})
separate_arguments(GRADLE_FLAGS)
-# We always want Gradle to use the plain console and quiet flag when called from cmake.
-list(APPEND GRADLE_FLAGS --quiet --console=plain)
+# When invoking Gradle:
+# * Log errors only, reducing the verbosity of the Gradle's output
+# * Make the output better fit for plain consoles, avoiding garbled logs
+# * Avoid building projects in parallel: when the kudu-subprocess.jar and
+# the hms-plugin.jar targets are built in parallel, it often ends up with
+# java.lang.ArrayIndexOutOfBoundsException while building shadowJar
+# for corresponding Java dependencies, and while it's not yet clear whether
+# the issue is in shadowJar implementation or elsewhere, it's quite simple
+# to address the issue by not building multiple Gradle targets concurrently
+# * Disable the build cache: sometimes, once a build process is interrupted,
+# follow-up Gradle invocations end up with an error like below, and while
+# it's not yet clear whether the issue is in not properly invoking Gradle
+# or it's a bug in signal handling elsewhere, it's quite simple to address
+# the issue by not using the Gradle's build cache
+#
+# Cannot lock Java compile cache (...) as it has already been locked by this process.
+#
+list(APPEND GRADLE_FLAGS --quiet --console=plain --no-parallel --no-build-cache)
list(REMOVE_DUPLICATES GRADLE_FLAGS)
# Initialize Gradle to ensure the wrapper is downloaded.