You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "uros-db (via GitHub)" <gi...@apache.org> on 2025/07/15 17:31:48 UTC
[PR] [SPARK-52804] Revert faulty annotation processing policy [spark]
uros-db opened a new pull request, #51500:
URL: https://github.com/apache/spark/pull/51500
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
Revert the code changes from: https://github.com/apache/spark/commit/ac12869a09e2739f7c86c5eb41bc0d73a3933a81.
### Why are the changes needed?
Re-enable Maven build at apache/spark master tip.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing CI tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52804] Revert faulty annotation processing policy [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #51500:
URL: https://github.com/apache/spark/pull/51500#issuecomment-3074711983
I conducted the following tests locally:
1. Cloned a clean repository using `git clone git@github.com:apache/spark.git spark-test`. The commit I checked out was:
```
commit d15ddd676a07c1336a12cbd7b4d9b718f7fd7117 (HEAD -> master, origin/master, origin/HEAD)
Author: Kent Yao <ya...@apache.org>
Date: Tue Jul 15 09:28:37 2025 -0700
[SPARK-52788][SQL] Fix error of converting binary value in BinaryType to XML
```
2. Renamed the Maven local cache with `mv ~/.m2 ~/.m2.bak` to start with a fresh cache.
3. Ran `./build/mvn -DskipTests clean package`.
The Java version I used was:
```
java -version
openjdk version "17.0.15" 2025-04-15 LTS
OpenJDK Runtime Environment Zulu17.58+21-CA (build 17.0.15+6-LTS)
OpenJDK 64-Bit Server VM Zulu17.58+21-CA (build 17.0.15+6-LTS, mixed mode, sharing)
```
The compilation succeeded in my local test:
```
[INFO] --- cyclonedx:2.9.1:makeBom (default) @ spark-streaming-kafka-0-10-assembly_2.13 ---
[INFO] CycloneDX: Resolving Dependencies
[INFO] CycloneDX: Creating BOM version 1.6 with 156 component(s)
[INFO] CycloneDX: Writing and validating BOM (XML): /Users/yangjie01/SourceCode/git/spark-test/connector/kafka-0-10-assembly/target/bom.xml
[INFO] attaching as spark-streaming-kafka-0-10-assembly_2.13-4.1.0-SNAPSHOT-cyclonedx.xml
[INFO] CycloneDX: Writing and validating BOM (JSON): /Users/yangjie01/SourceCode/git/spark-test/connector/kafka-0-10-assembly/target/bom.json
[INFO] attaching as spark-streaming-kafka-0-10-assembly_2.13-4.1.0-SNAPSHOT-cyclonedx.json
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 4.1.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [06:00 min]
[INFO] Spark Project Tags ................................. SUCCESS [ 20.466 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 4.271 s]
[INFO] Spark Project Common Utils ......................... SUCCESS [ 40.843 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 27.671 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 33.253 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 6.365 s]
[INFO] Spark Project Variant .............................. SUCCESS [ 2.372 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 17.410 s]
[INFO] Spark Project Connect Shims ........................ SUCCESS [ 2.170 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 14.076 s]
[INFO] Spark Project Core ................................. SUCCESS [04:01 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 28.151 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 16.006 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 23.697 s]
[INFO] Spark Project SQL API .............................. SUCCESS [ 40.092 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [01:09 min]
[INFO] Spark Project SQL .................................. SUCCESS [02:30 min]
[INFO] Spark Project ML Library ........................... SUCCESS [01:13 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 6.405 s]
[INFO] Spark Project Declarative Pipelines Library ........ SUCCESS [ 17.524 s]
[INFO] Spark Project Hive ................................. SUCCESS [01:33 min]
[INFO] Spark Project Connect Common ....................... SUCCESS [01:42 min]
[INFO] Spark Avro ......................................... SUCCESS [ 16.355 s]
[INFO] Spark Protobuf ..................................... SUCCESS [ 19.714 s]
[INFO] Spark Project REPL ................................. SUCCESS [ 11.344 s]
[INFO] Spark Project Connect Server ....................... SUCCESS [ 30.343 s]
[INFO] Spark Project Connect Client ....................... SUCCESS [01:09 min]
[INFO] Spark Project Assembly ............................. SUCCESS [ 7.095 s]
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 12.916 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 52.302 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [ 22.664 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 18.481 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 7.018 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 31:14 min
[INFO] Finished at: 2025-07-16T01:47:29+08:00
[INFO] ------------------------------------------------------------------------
```
The full compilation log is attached here:
[compile-20250716.log](https://github.com/user-attachments/files/21240161/compile-20250716.log)
Therefore, I also vote -1 on this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-52804] Revert faulty annotation processing policy [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #51500:
URL: https://github.com/apache/spark/pull/51500#issuecomment-3075395316
To @uros-db , I already mentioned that on the original PR here. The whole Java community recommended to update Java versions (including Java 17).
https://github.com/apache/spark/pull/51144#issuecomment-2964844099
Let me re-iterated here for you. It was `April 2024` which means over 1 and quarter ago. Please deliver the security patches first to the production if you didn't do that for 15 months.
- https://inside.java/2024/06/18/quality-heads-up/
> As of the April 2024 JDK security updates, support for -proc:full has been backported to 17u (17.0.11) and 11u (11.0.23) for both Oracle JDK and OpenJDK distributions. Additionally, Oracle’s 8u release (8u411) also supports -proc:full.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52804] Revert faulty annotation processing policy [spark]
Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on PR #51500:
URL: https://github.com/apache/spark/pull/51500#issuecomment-3075028321
In conclusion: the Java version needs to be upgraded in order to support
Although the build actually passes on latest versions of Java, this still completely breaks developers on Java 17.0.10 for example.
We should maybe think about improving the build requirements / documentation / dev guides, or error messages to reflect this better.
cc @cloud-fan @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52804] Revert faulty annotation processing policy [spark]
Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on PR #51500:
URL: https://github.com/apache/spark/pull/51500#issuecomment-3074686501
@LuciferYang & @dongjoon-hyun Sounds good, let's test it out and pinpoint the exact issue that causes builds to fail, given that there is some additional factor or dependency here that I might be missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52804] Revert faulty annotation processing policy [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #51500:
URL: https://github.com/apache/spark/pull/51500#issuecomment-3074606538
As I've mentioned [here](https://github.com/apache/spark/pull/51144#issuecomment-3074439496), please confirm or provide the steps to reproduce the issue. I am currently attempting to reproduce the problem by following the steps I described. I have compiled up to the sql module so far, and the issue you mentioned hasn't occurred yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52804] Revert faulty annotation processing policy [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #51500:
URL: https://github.com/apache/spark/pull/51500#issuecomment-3074716319
Thank you for double-checking, @LuciferYang .
Also, Java 17 clean build CI passed and merged.
- https://github.com/apache/spark/pull/51499
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52804] Revert faulty annotation processing policy [spark]
Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on PR #51500:
URL: https://github.com/apache/spark/pull/51500#issuecomment-3074587030
Also cc @cloud-fan and @MaxGekk, who have context about issues I've noticed with the Maven build in Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52804] Revert faulty annotation processing policy [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #51500:
URL: https://github.com/apache/spark/pull/51500#issuecomment-3074721934
Feel free to reopen this with the reproducible example, @uros-db . I'll close this to prevent any accidental merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52804] Revert faulty annotation processing policy [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #51500: [SPARK-52804] Revert faulty annotation processing policy
URL: https://github.com/apache/spark/pull/51500
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52804] Revert faulty annotation processing policy [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #51500:
URL: https://github.com/apache/spark/pull/51500#issuecomment-3074649239
@uros-db I'd like to confirm another issue. Which version of Java are you using? I'm conducting tests with version 17.0.15.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org