You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2026/06/26 06:07:43 UTC
[PR] [SPARK-57712][SS][KAFKA] Work around JDK-8298446 C2 miscompilation causing IllegalMonitorStateException in Kafka tests [spark]
HyukjinKwon opened a new pull request, #56803:
URL: https://github.com/apache/spark/pull/56803
### What changes were proposed in this pull request?
Add a single JVM workaround flag to the shared `extraJavaTestArgs` in the root `pom.xml`:
```
-XX:CompileCommand=exclude,org.apache.kafka.clients.producer.KafkaProducer::waitOnMetadata
```
This excludes only `KafkaProducer.waitOnMetadata` from C2 JIT compilation in the test JVM, so it runs in the interpreter. It is preceded in `extraJavaTestArgs` by the pre-existing `-XX:+IgnoreUnrecognizedVMOptions`, so it is a no-op on JVMs that do not recognize or do not need it.
This is the `master` version so the fix can be backported down to `branch-4.2` (where the failure is currently observed). Companion backport PR: #56801.
### Why are the changes needed?
The Maven Kafka test job intermittently fails nearly the whole Kafka suite (~227 tests) with:
```
java.lang.IllegalMonitorStateException: current thread is not owner
at java.base/java.lang.Object.wait(Native Method)
at org.apache.kafka.common.utils.SystemTime.waitObject(SystemTime.java:60)
at org.apache.kafka.clients.producer.internals.ProducerMetadata.awaitUpdate(ProducerMetadata.java:123)
at org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata(KafkaProducer.java:1180)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:1027)
...
```
`SystemTime.waitObject` in kafka-clients 3.9.2 calls `obj.wait()` correctly inside `synchronized (obj)` (verified from the jar bytecode: `monitorenter` -> `Object.wait` (line 60) -> `monitorexit`), so a correct JVM cannot throw `IllegalMonitorStateException` here. This is the HotSpot **C2 JIT bug [JDK-8298446](https://bugs.openjdk.org/browse/JDK-8298446)**: once C2 compiles `KafkaProducer.waitOnMetadata`, the monitor on the synchronized `obj.wait()` is dropped and every producer metadata wait throws "current thread is not owner". It is triggered nondeterministically by whether C2 compiles that hot method during the run, which is why the **identical** JDK fails on some runs and passes on others.
CI evidence:
- Before (failing): https://github.com/apache/spark/actions/runs/28182892330 — scheduled `Build / Maven (branch-4.2, Scala 2.13, JDK 17)`, Kafka job mass-fails with the stack above.
- After (this change, fork CI): https://github.com/HyukjinKwon/spark/actions/runs/28220508364
### Does this PR introduce _any_ user-facing change?
No. This is a test-only JVM argument; it does not affect any Spark runtime behavior or shipped artifact.
### How was this patch tested?
Existing Kafka test suites. The flag only forces one `kafka-clients` producer method to run interpreted in the test JVM. Because the underlying C2 miscompilation is nondeterministic, the primary justification is the JDK-8298446 analysis plus the `kafka-clients` bytecode (the `wait()` is correctly guarded by `synchronized`); the linked fork CI run exercises the change.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57712][SS][KAFKA] Work around JDK-8298446 C2 miscompilation causing IllegalMonitorStateException in Kafka tests [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #56803: [SPARK-57712][SS][KAFKA] Work around JDK-8298446 C2 miscompilation causing IllegalMonitorStateException in Kafka tests
URL: https://github.com/apache/spark/pull/56803
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57712][SS][KAFKA] Work around JDK-8298446 C2 miscompilation causing IllegalMonitorStateException in Kafka tests [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #56803:
URL: https://github.com/apache/spark/pull/56803#discussion_r3481092582
##########
pom.xml:
##########
@@ -344,6 +344,7 @@
--sun-misc-unsafe-memory-access=allow
--enable-native-access=ALL-UNNAMED
-XX:+EnableDynamicAgentLoading
+ -XX:CompileCommand=exclude,org.apache.kafka.clients.producer.KafkaProducer::waitOnMetadata
Review Comment:
Also, nit: there's no inline pom comment referencing SPARK-57712/JDK-8298446 next to the new line (the nearby block comments SPARK-36796); future readers won't know why the method is excluded or when to drop it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57712][SS][KAFKA] Work around JDK-8298446 C2 miscompilation causing IllegalMonitorStateException in Kafka tests [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #56803:
URL: https://github.com/apache/spark/pull/56803#issuecomment-4814681093
Let me close this for now, and take a fresh look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57712][SS][KAFKA] Work around JDK-8298446 C2 miscompilation causing IllegalMonitorStateException in Kafka tests [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #56803:
URL: https://github.com/apache/spark/pull/56803#discussion_r3481088816
##########
pom.xml:
##########
@@ -344,6 +344,7 @@
--sun-misc-unsafe-memory-access=allow
--enable-native-access=ALL-UNNAMED
-XX:+EnableDynamicAgentLoading
+ -XX:CompileCommand=exclude,org.apache.kafka.clients.producer.KafkaProducer::waitOnMetadata
Review Comment:
The workaround is unconditional/permanent: it doesn't gate on the buggy JDK range (JDK-8298446 is fixed in 17.0.14 / 21.0.3), so once CI moves to a fixed JDK the exclusion silently lingers, permanently forcing waitOnMetadata interpreted and reducing Kafka-test JIT fidelity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org