You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kafka.apache.org by ij...@apache.org on 2020/06/03 20:43:50 UTC
[kafka] branch 2.6 updated (ac4c589 -> c13ce7a)
This is an automated email from the ASF dual-hosted git repository.
ijuma pushed a change to branch 2.6
in repository https://gitbox.apache.org/repos/asf/kafka.git.
from ac4c589 KAFKA-10089 The stale ssl engine factory is not closed after reconfigure (#8792)
new 91f9c26 MINOR: Upgrade spotbugs and spotbugsPlugin (#8790)
new c13ce7a MINOR: Update to Gradle 6.5 and tweak build jvm config (#8751)
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
build.gradle | 23 ++++++++++++++---------
gradle.properties | 2 +-
gradle/dependencies.gradle | 6 +++---
gradle/wrapper/gradle-wrapper.properties | 2 +-
gradlew | 2 +-
5 files changed, 20 insertions(+), 15 deletions(-)
[kafka] 02/02: MINOR: Update to Gradle 6.5 and tweak build jvm
config (#8751)
Posted by ij...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ijuma pushed a commit to branch 2.6
in repository https://gitbox.apache.org/repos/asf/kafka.git
commit c13ce7a0bd688b80a607f0cf985878c72ac0c3b4
Author: Ismael Juma <is...@juma.me.uk>
AuthorDate: Wed Jun 3 13:18:50 2020 -0700
MINOR: Update to Gradle 6.5 and tweak build jvm config (#8751)
Gradle 6.5 includes a fix for https://github.com/gradle/gradle/pull/12866, which
affects the performance of Scala compilation.
I profiled the scalac build with async profiler and 54% of the time was on GC
even after the Gradle upgrade (it was more than 60% before), so I switched to
the throughput GC (GC latency is less important for batch builds) and it
was reduced to 38%.
I also centralized the jvm configuration in `build.gradle` and simplified it a bit
by removing the minHeapSize configuration from the test tasks.
On my desktop, the time to execute clean builds with no cached Gradle daemon
was reduced from 127 seconds to 97 seconds. With a cached daemon, it was
reduced from 120 seconds to 88 seconds. The performance regression when
we upgraded to Gradle 6.x was 27 seconds with a cached daemon
(https://github.com/apache/kafka/pull/7677#issuecomment-616271179), so it
should be fixed now.
Gradle 6.4 with no cached daemon:
```
BUILD SUCCESSFUL in 2m 7s
115 actionable tasks: 112 executed, 3 up-to-date
./gradlew clean compileScala compileJava compileTestScala compileTestJava 1.15s user 0.12s system 0% cpu 2:08.06 total
```
Gradle 6.4 with cached daemon:
```
BUILD SUCCESSFUL in 2m 0s
115 actionable tasks: 111 executed, 4 up-to-date
./gradlew clean compileScala compileJava compileTestScala compileTestJava 0.95s user 0.10s system 0% cpu 2:01.42 total
```
Gradle 6.5 with no cached daemon:
```
BUILD SUCCESSFUL in 1m 46s
115 actionable tasks: 111 executed, 4 up-to-date
./gradlew clean compileScala compileJava compileTestScala compileTestJava 1.27s user 0.12s system 1% cpu 1:47.71 total
```
Gradle 6.5 with cached daemon:
```
BUILD SUCCESSFUL in 1m 37s
115 actionable tasks: 111 executed, 4 up-to-date
./gradlew clean compileScala compileJava compileTestScala compileTestJava 1.02s user 0.10s system 1% cpu 1:38.31 total
```
This PR with no cached Gradle daemon:
```
BUILD SUCCESSFUL in 1m 37s
115 actionable tasks: 81 executed, 34 up-to-date
./gradlew clean compileScala compileJava compileTestScala compileTestJava 1.27s user 0.10s system 1% cpu 1:38.70 total
```
This PR with cached Gradle daemon:
```
BUILD SUCCESSFUL in 1m 28s
115 actionable tasks: 111 executed, 4 up-to-date
./gradlew clean compileScala compileJava compileTestScala compileTestJava 1.02s user 0.10s system 1% cpu 1:29.35 total
```
Reviewers: Manikumar Reddy <ma...@gmail.com>, Chia-Ping Tsai <ch...@gmail.com>
---
build.gradle | 23 ++++++++++++++---------
gradle.properties | 2 +-
gradle/dependencies.gradle | 2 +-
gradle/wrapper/gradle-wrapper.properties | 2 +-
gradlew | 2 +-
5 files changed, 18 insertions(+), 13 deletions(-)
diff --git a/build.gradle b/build.gradle
index 696e1c4..65f18e5 100644
--- a/build.gradle
+++ b/build.gradle
@@ -96,6 +96,9 @@ ext {
minJavaVersion = "8"
buildVersionFileName = "kafka-version.properties"
+ defaultMaxHeapSize = "2g"
+ defaultJvmArgs = ["-Xss4m", "-XX:+UseParallelGC"]
+
userMaxForks = project.hasProperty('maxParallelForks') ? maxParallelForks.toInteger() : null
userMaxTestRetries = project.hasProperty('maxTestRetries') ? maxTestRetries.toInteger() : 0
@@ -293,8 +296,8 @@ subprojects {
test {
maxParallelForks = userMaxForks ?: Runtime.runtime.availableProcessors()
- minHeapSize = "256m"
- maxHeapSize = "2048m"
+ maxHeapSize = defaultMaxHeapSize
+ jvmArgs = defaultJvmArgs
testLogging {
events = userTestLoggingEvents ?: testLoggingEvents
@@ -316,8 +319,9 @@ subprojects {
task integrationTest(type: Test, dependsOn: compileJava) {
maxParallelForks = userMaxForks ?: Runtime.runtime.availableProcessors()
- minHeapSize = "256m"
- maxHeapSize = "2048m"
+ maxHeapSize = defaultMaxHeapSize
+ jvmArgs = defaultJvmArgs
+
testLogging {
events = userTestLoggingEvents ?: testLoggingEvents
@@ -343,8 +347,8 @@ subprojects {
task unitTest(type: Test, dependsOn: compileJava) {
maxParallelForks = userMaxForks ?: Runtime.runtime.availableProcessors()
- minHeapSize = "256m"
- maxHeapSize = "2048m"
+ maxHeapSize = defaultMaxHeapSize
+ jvmArgs = defaultJvmArgs
testLogging {
events = userTestLoggingEvents ?: testLoggingEvents
@@ -506,8 +510,8 @@ subprojects {
scalaCompileOptions.additionalParameters += ["-release", minJavaVersion]
configure(scalaCompileOptions.forkOptions) {
- memoryMaximumSize = '1g'
- jvmArgs = ['-Xss4m']
+ memoryMaximumSize = defaultMaxHeapSize
+ jvmArgs = defaultJvmArgs
}
}
@@ -542,7 +546,8 @@ subprojects {
xml.enabled(project.hasProperty('xmlSpotBugsReport') || project.hasProperty('xmlFindBugsReport'))
html.enabled(!project.hasProperty('xmlSpotBugsReport') && !project.hasProperty('xmlFindBugsReport'))
}
- maxHeapSize = "2g"
+ maxHeapSize = defaultMaxHeapSize
+ jvmArgs = defaultJvmArgs
}
// Ignore core since its a scala project
diff --git a/gradle.properties b/gradle.properties
index a34495d..e742057 100644
--- a/gradle.properties
+++ b/gradle.properties
@@ -23,4 +23,4 @@ group=org.apache.kafka
version=2.6.0-SNAPSHOT
scalaVersion=2.13.2
task=build
-org.gradle.jvmargs=-Xmx2g -Xss2m
+org.gradle.jvmargs=-Xmx2g -Xss4m -XX:+UseParallelGC
diff --git a/gradle/dependencies.gradle b/gradle/dependencies.gradle
index d803f1c..630312d 100644
--- a/gradle/dependencies.gradle
+++ b/gradle/dependencies.gradle
@@ -61,7 +61,7 @@ versions += [
bcpkix: "1.64",
checkstyle: "8.20",
commonsCli: "1.4",
- gradle: "6.4.1",
+ gradle: "6.5",
gradleVersionsPlugin: "0.28.0",
grgit: "4.0.1",
httpclient: "4.5.11",
diff --git a/gradle/wrapper/gradle-wrapper.properties b/gradle/wrapper/gradle-wrapper.properties
index 21e622d..186b715 100644
--- a/gradle/wrapper/gradle-wrapper.properties
+++ b/gradle/wrapper/gradle-wrapper.properties
@@ -1,5 +1,5 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-6.4.1-all.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-6.5-all.zip
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
diff --git a/gradlew b/gradlew
index 609813f..f0890d7 100755
--- a/gradlew
+++ b/gradlew
@@ -84,7 +84,7 @@ esac
# Loop in case we encounter an error.
for attempt in 1 2 3; do
if [ ! -e $APP_HOME/gradle/wrapper/gradle-wrapper.jar ]; then
- if ! curl -s -S --retry 3 -L -o "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" "https://raw.githubusercontent.com/gradle/gradle/v6.4.1/gradle/wrapper/gradle-wrapper.jar"; then
+ if ! curl -s -S --retry 3 -L -o "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" "https://raw.githubusercontent.com/gradle/gradle/v6.5.0/gradle/wrapper/gradle-wrapper.jar"; then
rm -f "$APP_HOME/gradle/wrapper/gradle-wrapper.jar"
# Pause for a bit before looping in case the server throttled us.
sleep 5
[kafka] 01/02: MINOR: Upgrade spotbugs and spotbugsPlugin (#8790)
Posted by ij...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ijuma pushed a commit to branch 2.6
in repository https://gitbox.apache.org/repos/asf/kafka.git
commit 91f9c2652f5833817f826690b70fc597de9bc223
Author: showuon <43...@users.noreply.github.com>
AuthorDate: Thu Jun 4 04:17:31 2020 +0800
MINOR: Upgrade spotbugs and spotbugsPlugin (#8790)
Upgrade spotbugsPlugin to have clear output indicating where the error is.
When investigating KAFKA-10081, I found the error output of spotbugs is very poor. It doesn't even tell you where the error is and how many errors found, it will take a lot of time for the developers to find out where the error is, and then fix it.
https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk8/detail/kafka-trunk-jdk8/4596/pipeline/
Then, I found out there's a bug in spotbugsPlugin in V4.0.x, and got fixed in V4.2.x
https://github.com/spotbugs/spotbugs-gradle-plugin/issues/210
So, after upgrading to V4.2.x (I followed to the latest version V4.2.4), the output is like this:
So you know there's 1 error and you can also open the report file to find out the error.
Upgraded to the latest bug fix release of spotbugs (4.0.3) while at it.
Reviewers: Ismael Juma <is...@juma.me.uk>
---
gradle/dependencies.gradle | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/gradle/dependencies.gradle b/gradle/dependencies.gradle
index 294d0c4..d803f1c 100644
--- a/gradle/dependencies.gradle
+++ b/gradle/dependencies.gradle
@@ -111,8 +111,8 @@ versions += [
shadowPlugin: "5.2.0",
slf4j: "1.7.30",
snappy: "1.1.7.3",
- spotbugs: "4.0.2",
- spotbugsPlugin: "4.0.5",
+ spotbugs: "4.0.3",
+ spotbugsPlugin: "4.2.4",
spotlessPlugin: "3.28.1",
testRetryPlugin: "1.1.5",
zinc: "1.3.5",