You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "dejan2609 (via GitHub)" <gi...@apache.org> on 2025/11/12 13:31:30 UTC
[PR] KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.2.2 [kafka]
dejan2609 opened a new pull request, #20866:
URL: https://github.com/apache/kafka/pull/20866
Release notes for Shadow plugin:
- https://gradleup.com/shadow/changes/#922-2025-09-26
I did some testing, and results look just fine :white_check_mark::
- gradle builds `./gradlew :jmh-benchmarks:shadowJar` and `./gradlew releaseTarGz` are successful
- generated distributions (`kafka_2.13-4.2.0-SNAPSHOT-site-docs.tgz and` and `kafka_2.13-4.2.0-SNAPSHOT.tgz`) have identical file numbers (for trunk and this branch)
- Kafka cluster starts and messages can be produced and consumed: https://kafka.apache.org/quickstart
:warning: disclaimer notes:
- solution is generated via Gemini CLI
- Gemini prompt is based on Jira ticket (previously created by me): KAFKA-19761
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.2.2 [kafka]
Posted by "dejan2609 (via GitHub)" <gi...@apache.org>.
dejan2609 commented on PR #20866:
URL: https://github.com/apache/kafka/pull/20866#issuecomment-3558929395
I'll rebase this onto trunk (to include a new, updated Gradle version into this branch/PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.3.1 [kafka]
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on PR #20866:
URL: https://github.com/apache/kafka/pull/20866#issuecomment-3729179462
Thanks for the change and apologies for the delay!
It may be because my expertise in Gradle is not my strong suit, but I would encourage you to give a brief overview of why all the changes in the PR are required!
The real problem is that if we just bump Gradle Shadow from 8 to 9 we run into
```
* What went wrong:
Execution failed for task ':jmh-benchmarks:shadowJar'.
> Cannot expand ZIP '/Users/REDACTED/Documents/kafka/core/build/classes/java/test' as it does not exist.
```
which appears to be because :core only has build/classes/scala while all other dependencies have build/classes/java and Gradle Shadow 9 appears to be more strict in requiring that directory to exist.
I am a bit on the fence for this change. On one hand, I don't like depending on .sourceSets.test.output in some places in the build.gradle file and on configurations/artifacts in others. On the other hand, I also don't like generating an empty folder (i.e. build/classes/java) just to satisfy a stricter expectation.
What are the opinions of others in the community?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.3.1 [kafka]
Posted by "dejan2609 (via GitHub)" <gi...@apache.org>.
dejan2609 commented on PR #20866:
URL: https://github.com/apache/kafka/pull/20866#issuecomment-3734991125
> _Instead of working around it, I think it's better to refactor the JMH module and remove the dependency on core test output._
I concur with that @chia7712.
Moving this PR into draft state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.3.0 [kafka]
Posted by "dejan2609 (via GitHub)" <gi...@apache.org>.
dejan2609 commented on PR #20866:
URL: https://github.com/apache/kafka/pull/20866#issuecomment-3649532046
:bulb: I will try to execute entire test suite once more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.2.2 [kafka]
Posted by "dejan2609 (via GitHub)" <gi...@apache.org>.
dejan2609 commented on code in PR #20866:
URL: https://github.com/apache/kafka/pull/20866#discussion_r2519747075
##########
build.gradle:
##########
@@ -713,6 +713,17 @@ subprojects {
exclude 'junit-platform.properties'
}
+ configurations {
+ tests {
Review Comment:
:bulb: Mental note to myself: configuration name `tests` should be changed to something more appropriate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.3.0 [kafka]
Posted by "dejan2609 (via GitHub)" <gi...@apache.org>.
dejan2609 commented on PR #20866:
URL: https://github.com/apache/kafka/pull/20866#issuecomment-3620543198
Hi @apoorvmittal10,
Can you please allocate some time to provide a review ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.3.1 [kafka]
Posted by "dejan2609 (via GitHub)" <gi...@apache.org>.
dejan2609 commented on PR #20866:
URL: https://github.com/apache/kafka/pull/20866#issuecomment-3730024846
>_It may be because my expertise in Gradle is not my strong suit, but I would encourage you to give a brief overview of why all the changes in the PR are required!_
**_Rationale_** section is added into description above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.2.2 [kafka]
Posted by "dejan2609 (via GitHub)" <gi...@apache.org>.
dejan2609 commented on code in PR #20866:
URL: https://github.com/apache/kafka/pull/20866#discussion_r2538151375
##########
build.gradle:
##########
@@ -713,6 +713,17 @@ subprojects {
exclude 'junit-platform.properties'
}
+ configurations {
+ tests {
Review Comment:
Configuration name changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.3.1 [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #20866:
URL: https://github.com/apache/kafka/pull/20866#issuecomment-3733281925
> What are the opinions of others in the community?
The issue stems from `implementation project(':core').sourceSets.test.output` including non-existent paths due to joint compilation. I actually reported this Gradle before (see https://github.com/gradle/gradle/issues/35293).
Instead of working around it, I think it's better to refactor the JMH module and remove the dependency on core test output. This saves us the trouble of dealing with missing paths. I have opened https://issues.apache.org/jira/browse/KAFKA-20060 for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.2.2 [kafka]
Posted by "dejan2609 (via GitHub)" <gi...@apache.org>.
dejan2609 commented on PR #20866:
URL: https://github.com/apache/kafka/pull/20866#issuecomment-3523763650
@chia7712 Please review when you have some time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.3.1 [kafka]
Posted by "dejan2609 (via GitHub)" <gi...@apache.org>.
dejan2609 commented on PR #20866:
URL: https://github.com/apache/kafka/pull/20866#issuecomment-3729462271
Hi @clolov and thanks for a review/comment ! You are rasing some valid questions here.
But first: can you please share your environment details (thing is that Gradle build task `:jmh-benchmarks:shadowJar` works just fine on my end):
```
dejan@box:~/kafka$ date
Fri 9 Jan 16:24:51 CET 2026
dejan@box:~/kafka$ uname -r
6.8.0-90-generic
dejan@box:~/kafka$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04.3 LTS
Release: 24.04
Codename: noble
dejan@box:~/kafka$ java -version
openjdk version "25.0.1" 2025-10-21 LTS
OpenJDK Runtime Environment Temurin-25.0.1+8 (build 25.0.1+8-LTS)
OpenJDK 64-Bit Server VM Temurin-25.0.1+8 (build 25.0.1+8-LTS, mixed mode, sharing)
dejan@box:~/kafka$ git log -2 --oneline
70dac31e9f (HEAD -> KAFKA-19761, origin/KAFKA-19761) KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.3.1
2f90af70ee (upstream/trunk, upstream/HEAD, origin/trunk, origin/HEAD, trunk) MINOR: Remove onPartitionsDeleted from GroupCoordinator interface (#21263)
dejan@box:~/kafka$ ./gradlew clean :jmh-benchmarks:shadowJar -q
Starting build with version 4.3.0-SNAPSHOT (commit id 70dac31e) using Gradle 9.2.1, Java 25 and Scala 2.13.18
Build properties: ignoreFailures=false, maxParallelForks=8, maxScalacThreads=8, maxTestRetries=0
MessageGenerator: processed 1 Kafka message JSON file(s).
MessageGenerator: processed 1 Kafka message JSON file(s).
MessageGenerator: processed 5 Kafka message JSON file(s).
MessageGenerator: processed 4 Kafka message JSON file(s).
MessageGenerator: processed 2 Kafka message JSON file(s).
MessageGenerator: processed 26 Kafka message JSON file(s).
MessageGenerator: processed 44 Kafka message JSON file(s).
MessageGenerator: processed 197 Kafka message JSON file(s).
MessageGenerator: processed 5 Kafka message JSON file(s).
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::arrayBaseOffset has been called by net.openhft.hashing.UnsafeAccess (file:/home/dejan/.gradle/caches/modules-2/files-2.1/net.openhft/zero-allocation-hashing/0.16/ca252f328160ed5d027f100a4fe525d6d21daaf/zero-allocation-hashing-0.16.jar)
WARNING: Please consider reporting this to the maintainers of class net.openhft.hashing.UnsafeAccess
WARNING: sun.misc.Unsafe::arrayBaseOffset will be removed in a future release
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /home/dejan/kafka/coordinator-common/src/test/java/org/apache/kafka/coordinator/common/runtime/CoordinatorRuntimeTest.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Note: /home/dejan/kafka/group-coordinator/src/test/java/org/apache/kafka/coordinator/group/streams/StreamsGroupTest.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
dejan@box:~/kafka$ sdk use java 17.0.17-tem
Using java version 17.0.17-tem in this shell.
dejan@box:~/kafka$ java -version
openjdk version "17.0.17" 2025-10-21
OpenJDK Runtime Environment Temurin-17.0.17+10 (build 17.0.17+10)
OpenJDK 64-Bit Server VM Temurin-17.0.17+10 (build 17.0.17+10, mixed mode, sharing)
dejan@box:~/kafka$ ./gradlew clean :jmh-benchmarks:shadowJar -q
Starting build with version 4.3.0-SNAPSHOT (commit id 70dac31e) using Gradle 9.2.1, Java 17 and Scala 2.13.18
Build properties: ignoreFailures=false, maxParallelForks=8, maxScalacThreads=8, maxTestRetries=0
MessageGenerator: processed 4 Kafka message JSON file(s).
MessageGenerator: processed 1 Kafka message JSON file(s).
MessageGenerator: processed 5 Kafka message JSON file(s).
MessageGenerator: processed 2 Kafka message JSON file(s).
MessageGenerator: processed 1 Kafka message JSON file(s).
MessageGenerator: processed 26 Kafka message JSON file(s).
MessageGenerator: processed 44 Kafka message JSON file(s).
MessageGenerator: processed 197 Kafka message JSON file(s).
MessageGenerator: processed 5 Kafka message JSON file(s).
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /home/dejan/kafka/coordinator-common/src/test/java/org/apache/kafka/coordinator/common/runtime/CoordinatorRuntimeTest.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Note: /home/dejan/kafka/group-coordinator/src/test/java/org/apache/kafka/coordinator/group/streams/StreamsGroupTest.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
dejan@box:~/kafka$ date
Fri 9 Jan 16:38:25 CET 2026
dejan@box:~/kafka$
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.3.1 [kafka]
Posted by "dejan2609 (via GitHub)" <gi...@apache.org>.
dejan2609 commented on PR #20866:
URL: https://github.com/apache/kafka/pull/20866#issuecomment-3729570051
> _I am a bit on the fence for this change. On one hand, I don't like depending on .sourceSets.test.output in some places in the build.gradle file and on configurations/artifacts in others. On the other hand, I also don't like generating an empty folder (i.e. build/classes/java) just to satisfy a stricter expectation._
I can relate with all of this @clolov. Thing is that Kafka Gradle build is complicated due to a fact that people want Kafka to perform various complicated things.
And on top of that: dependencies versions just have to move forward: sooner or later this Shadow gradle plugin version will have to be bumped, one way or another.
All-in-all, we can either choose to:
1. put some duct tape and/or WD-40 and call it a day (like here, in this PR :slightly_smiling_face:)
2. re-factor Gradle build (and remove some technical debt all along) - :warning: pitfall - it can be tricky
3. something else :nerd_face:
Open invitation to entire community: what is your take on this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.3.1 [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #20866:
URL: https://github.com/apache/kafka/pull/20866#issuecomment-3784568816
@dejan2609 #21310 is already merged, so please rebase code. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.3.0 [kafka]
Posted by "dejan2609 (via GitHub)" <gi...@apache.org>.
dejan2609 commented on PR #20866:
URL: https://github.com/apache/kafka/pull/20866#issuecomment-3650680784
All tests are fine now :white_check_mark:
@chia7712 can you allocate some time to review this one ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.3.0 [kafka]
Posted by "dejan2609 (via GitHub)" <gi...@apache.org>.
dejan2609 commented on PR #20866:
URL: https://github.com/apache/kafka/pull/20866#issuecomment-3649531057
Few tests are flaky/failing:
<img width="2083" height="985" alt="image" src="https://github.com/user-attachments/assets/e1f06c7e-1bcd-4298-ba12-c0a962c85134" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.3.1 [kafka]
Posted by "dejan2609 (via GitHub)" <gi...@apache.org>.
dejan2609 commented on PR #20866:
URL: https://github.com/apache/kafka/pull/20866#issuecomment-3786328401
Done, PR is rebased. I did some smoke testing (https://kafka.apache.org/quickstart) and everything seems fine (messages can be produced and consumed).
Notes/remarks:
- given the fact that this is now a simple one-line change we could opt to also add Gradle minor version update here (from 9.2.1 to 9.3.0)
- :warning: pitfall: Gradle 9.3.0 has some known issues (https://docs.gradle.org/9.3.0/release-notes.html#known-issues) so maybe we should stay on the safe side and wait for Gradle 9.3.1 or 9.4.0
@chia7712 let me know what you think about this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org