You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by gr...@apache.org on 2020/09/15 19:16:25 UTC
[kudu] branch master updated (0f9ff5f -> 09463d3)
This is an automated email from the ASF dual-hosted git repository.
granthenke pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git.
from 0f9ff5f KUDU-2612 p6 (b): add {Abort,BeginCommit}Transaction() to TxnSystemClient
new dc28392 [transactions] Adjust Java package for transactions.proto
new c29359d [benchmarks] fix Y-axis labels for dense_node-itest
new e9d62d3 [java] Fix shading of Micrometer dependency
new 6953f53 KUDU-3157: Ensure we don’t shade slf4j classes
new 09463d3 [build] Fail the LINT build if verify_jars.pl fails
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
build-support/jenkins/build-and-test.sh | 14 ++++++++------
build-support/verify_jars.pl | 2 ++
java/gradle/shadow.gradle | 20 ++++++++++++++++++++
java/kudu-subprocess/build.gradle | 3 +++
src/kudu/scripts/benchmarks.sh | 8 ++++----
src/kudu/transactions/transactions.proto | 2 ++
6 files changed, 39 insertions(+), 10 deletions(-)
[kudu] 01/05: [transactions] Adjust Java package for
transactions.proto
Posted by gr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
granthenke pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit dc28392314605880583ba9b8926e2b86d15768f6
Author: Grant Henke <gr...@apache.org>
AuthorDate: Tue Sep 15 09:43:15 2020 -0500
[transactions] Adjust Java package for transactions.proto
This patch moves the Java package for transactions.proto
from `kudu.transactions` to `org.apache.kudu.transactions`
to match the location of the other generated Java classes.
Change-Id: I69866e0688a9ef9378b7615ecbb5c3089c3d85fd
Reviewed-on: http://gerrit.cloudera.org:8080/16451
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Reviewed-by: Andrew Wong <aw...@cloudera.com>
---
src/kudu/transactions/transactions.proto | 2 ++
1 file changed, 2 insertions(+)
diff --git a/src/kudu/transactions/transactions.proto b/src/kudu/transactions/transactions.proto
index 0d669e5..7013935 100644
--- a/src/kudu/transactions/transactions.proto
+++ b/src/kudu/transactions/transactions.proto
@@ -17,6 +17,8 @@
syntax = "proto2";
package kudu.transactions;
+option java_package = "org.apache.kudu.transactions";
+
enum TxnStatePB {
UNKNOWN = 0;
OPEN = 1;
[kudu] 02/05: [benchmarks] fix Y-axis labels for dense_node-itest
Posted by gr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
granthenke pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit c29359d3730828764677d76ef33819dd75e7a0ad
Author: Alexey Serbin <al...@apache.org>
AuthorDate: Mon Sep 14 10:30:10 2020 -0700
[benchmarks] fix Y-axis labels for dense_node-itest
Change-Id: I7381b117308e75479f4dcca188cd7befefc32b3e
Reviewed-on: http://gerrit.cloudera.org:8080/16448
Tested-by: Kudu Jenkins
Reviewed-by: Bankim Bhavsar <ba...@cloudera.com>
Reviewed-by: Grant Henke <gr...@apache.org>
---
src/kudu/scripts/benchmarks.sh | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/src/kudu/scripts/benchmarks.sh b/src/kudu/scripts/benchmarks.sh
index 5a16d3a..74d0a49 100755
--- a/src/kudu/scripts/benchmarks.sh
+++ b/src/kudu/scripts/benchmarks.sh
@@ -783,10 +783,10 @@ load_stats_and_generate_plots() {
load_and_generate_plot "${FS_SCANINSERT_DISK}%_scan%" fs-withdisk-scan
load_and_generate_plot "${DENSE_NODE_ITEST}_time%" dense-node-bench-times
- load_and_generate_plot "${DENSE_NODE_ITEST}_num%containers%" dense-node-bench-containers
- load_and_generate_plot "${DENSE_NODE_ITEST}_num_blocks%" dense-node-bench-blocks
- load_and_generate_plot "${DENSE_NODE_ITEST}_num_threads%" dense-node-bench-threads
- load_and_generate_plot "${DENSE_NODE_ITEST}_num_bytes%" dense-node-bench-bytes
+ load_and_generate_plot "${DENSE_NODE_ITEST}_num%containers%" dense-node-bench-containers "number of log block containers"
+ load_and_generate_plot "${DENSE_NODE_ITEST}_num_blocks%" dense-node-bench-blocks "number of blocks"
+ load_and_generate_plot "${DENSE_NODE_ITEST}_num_threads%" dense-node-bench-threads "threads running"
+ load_and_generate_plot "${DENSE_NODE_ITEST}_num_bytes%" dense-node-bench-bytes "bytes under management"
load_and_generate_plot "${BLOCKING_QUEUE_SYMMETRIC_TEST}%" blocking-queue-symmetric "function call rate (req/sec)"
load_and_generate_plot "${BLOCKING_QUEUE_NON_SYMMETRIC_TEST}%" blocking-queue-non-symmetric "function call rate (req/sec)"
[kudu] 05/05: [build] Fail the LINT build if verify_jars.pl fails
Posted by gr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
granthenke pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 09463d31f4c3a26b534e0eb51621d3341b03466d
Author: Grant Henke <gr...@apache.org>
AuthorDate: Tue Sep 15 11:24:14 2020 -0500
[build] Fail the LINT build if verify_jars.pl fails
In order to ensure we don’t introduce regressions in our
Java packaging, this change adds `verify_jars.pl` to the
LINT build.
Change-Id: Iefbb89950c332e9549bc0eb9867933dcb2810af7
Reviewed-on: http://gerrit.cloudera.org:8080/16454
Reviewed-by: Andrew Wong <aw...@cloudera.com>
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <as...@cloudera.com>
---
build-support/jenkins/build-and-test.sh | 14 ++++++++------
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git a/build-support/jenkins/build-and-test.sh b/build-support/jenkins/build-and-test.sh
index 21eaa49..fe4c3ed 100755
--- a/build-support/jenkins/build-and-test.sh
+++ b/build-support/jenkins/build-and-test.sh
@@ -410,11 +410,19 @@ if [ "$BUILD_TYPE" = "LINT" ]; then
LINT_RESULT=0
if [ "$BUILD_JAVA" == "1" ]; then
+ # Run Java static code analysis via Gradle.
pushd $SOURCE_ROOT/java
if ! ./gradlew $EXTRA_GRADLE_FLAGS clean check -x test $EXTRA_GRADLE_TEST_FLAGS; then
LINT_RESULT=1
LINT_FAILURES="$LINT_FAILURES"$'Java Gradle check failed\n'
fi
+
+ # Verify the contents of the JARs to ensure the shading and
+ # packaging is correct.
+ if ! $SOURCE_ROOT/build-support/verify_jars.pl .; then
+ LINT_RESULT=1
+ LINT_FAILURES="$LINT_FAILURES"$'Java verify Jars check failed\n'
+ fi
popd
fi
@@ -535,17 +543,11 @@ if [ "$BUILD_JAVA" == "1" ]; then
FAILURES="$FAILURES"$'Could not submit Java distributed test job\n'
fi
else
- # TODO: Run `gradle check` in BUILD_TYPE DEBUG when static code analysis is fixed
if ! ./gradlew $EXTRA_GRADLE_FLAGS clean test $EXTRA_GRADLE_TEST_FLAGS; then
TESTS_FAILED=1
FAILURES="$FAILURES"$'Java Gradle build/test failed\n'
fi
fi
-
- # Run a script to verify the contents of the JARs to ensure the shading and
- # packaging is correct.
- $SOURCE_ROOT/build-support/verify_jars.pl .
-
set +x
popd
fi
[kudu] 04/05: KUDU-3157: Ensure we don’t shade slf4j classes
Posted by gr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
granthenke pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 6953f53fb1f0ef617f596ec89690ab0e1adfc4e4
Author: Grant Henke <gr...@apache.org>
AuthorDate: Tue Sep 15 11:09:09 2020 -0500
KUDU-3157: Ensure we don’t shade slf4j classes
We should almost never shade slf4j classes but due to a behavior
issue in the Shadow Gradle plugin, exclusions from parent
modules are not respected in modules that use them. This meant
that we were shading slf4j classes into the kudu-spark module
jars and any modules that depend on kudu-spark.
This patch fixes the issue by universally excluding slf4j from
being shaded unless `shadowIncludeSlf4j` is explicitly specified.
Currently we only do this in the kudu-subrocess module.
Change-Id: I72605594bdc70670f82eff4154247c5c0252de7f
Reviewed-on: http://gerrit.cloudera.org:8080/16453
Tested-by: Grant Henke <gr...@apache.org>
Reviewed-by: Grant Henke <gr...@apache.org>
---
java/gradle/shadow.gradle | 17 +++++++++++++++++
java/kudu-subprocess/build.gradle | 3 +++
2 files changed, 20 insertions(+)
diff --git a/java/gradle/shadow.gradle b/java/gradle/shadow.gradle
index 5d9eb18..9eb6cfd 100644
--- a/java/gradle/shadow.gradle
+++ b/java/gradle/shadow.gradle
@@ -26,6 +26,11 @@ knows.enabled = false // Disable the "easter egg" knows task.
knows.group = "" // Hide the "easter egg" knows task.
shadowJar.group = "" // Hide shadowJar task since it's used by the default build.
+// Add a property to explicitly allow slf4j shading.
+ext {
+ shadowIncludeSlf4j = false
+}
+
// Configure a shaded jar to replace the default jar
shadowJar.classifier = null // Configure shadow jar to have the default classifier.
jar.finalizedBy(shadowJar) // Generate the shaded jar anytime the jar task is run.
@@ -94,6 +99,18 @@ afterEvaluate {
}
}
}
+
+ // Ensure we never shade SLF4J unless we explicitly specify it.
+ // This is a workaround because in the shadow plugin exclusions from
+ // parent modules are not respected in modules that use them.
+ if (!shadowIncludeSlf4j) {
+ shadowJar {
+ dependencies {
+ exclude(dependency("org.slf4j:slf4j-api:.*"))
+ }
+ }
+ }
+
// Ensure compileUnshaded dependencies are not compiled into shadowJar.
project.configurations.compileUnshaded.dependencies.each { dep ->
def depStr = "${dep.group}:${dep.name}:${dep.version}"
diff --git a/java/kudu-subprocess/build.gradle b/java/kudu-subprocess/build.gradle
index 1a9cdca..4463cc2 100644
--- a/java/kudu-subprocess/build.gradle
+++ b/java/kudu-subprocess/build.gradle
@@ -18,6 +18,9 @@
apply from: "$rootDir/gradle/protobuf.gradle"
apply from: "$rootDir/gradle/shadow.gradle"
+// Explicitly allow slf4j to be included in this jar.
+shadowIncludeSlf4j = true
+
dependencies {
compile(libs.hadoopCommon) {
// hadoopCommon and rangerPlugin use different versions of jersey.
[kudu] 03/05: [java] Fix shading of Micrometer dependency
Posted by gr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
granthenke pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit e9d62d39f657a08d9ee5846e7b6f00fdc1c75c1c
Author: Grant Henke <gr...@apache.org>
AuthorDate: Tue Sep 15 10:25:25 2020 -0500
[java] Fix shading of Micrometer dependency
This patch adjusts the Java build to ensure Micrometer classes
and its dependencies are relocated properly. It also adjusts
verify_jars.pl to account for the change.
Change-Id: Id2547b5703e5d300de531aa51d55f37718a27b61
Reviewed-on: http://gerrit.cloudera.org:8080/16452
Reviewed-by: Andrew Wong <aw...@cloudera.com>
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Tested-by: Grant Henke <gr...@apache.org>
---
build-support/verify_jars.pl | 2 ++
java/gradle/shadow.gradle | 3 +++
2 files changed, 5 insertions(+)
diff --git a/build-support/verify_jars.pl b/build-support/verify_jars.pl
index a1afd73..4bcef20 100755
--- a/build-support/verify_jars.pl
+++ b/build-support/verify_jars.pl
@@ -42,6 +42,8 @@ my $pat_allow_kudu_shaded =
org/apache/(?:commons|http)|
org/checkerframework|
org/HdrHistogram|
+ org/LatencyUtils|
+ io/micrometer|
io/netty|
scopt)
}x;
diff --git a/java/gradle/shadow.gradle b/java/gradle/shadow.gradle
index c09b06f..5d9eb18 100644
--- a/java/gradle/shadow.gradle
+++ b/java/gradle/shadow.gradle
@@ -58,6 +58,9 @@ shadowJar {
relocate "org.checkerframework", "org.apache.kudu.shaded.org.checkerframework"
relocate "org.hamcrest", "org.apache.kudu.shaded.org.hamcrest"
relocate "org.HdrHistogram", "org.apache.kudu.shaded.org.HdrHistogram"
+ // Pulled in via Micrometer.
+ relocate "org.LatencyUtils", "org.apache.kudu.shaded.org.LatencyUtils"
+ relocate "io.micrometer", "org.apache.kudu.shaded.io.micrometer"
relocate "io.netty", "org.apache.kudu.shaded.io.netty"
relocate "scopt", "org.apache.kudu.shaded.scopt"
}