You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/05/14 18:22:12 UTC
[I] Frequent CI failures for Spark 4.0.2 / JDK 21 [datafusion-comet]
andygrove opened a new issue, #4327:
URL: https://github.com/apache/datafusion-comet/issues/4327
### Describe the bug
Example: https://github.com/apache/datafusion-comet/actions/runs/25837642339/job/75918339673?pr=4308
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Frequent CI failures for Spark 4.0.2 / JDK 21 [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #4327:
URL: https://github.com/apache/datafusion-comet/issues/4327#issuecomment-4479558592
## Root cause
The failing assertion is always the same in every recent failing run of `spark-sql-auto-sql_core-1/spark-4.0.2-jdk21`:
```
[info] ParquetFileFormatV{1,2}Suite (or OrcSourceV{1,2}Suite) *** ABORTED ***
[info] "There are 1 possibly leaked file streams." (SharedSparkSession.scala:189)
```
This is Spark's `DebugFilesystem.assertNoOpenStreams()` in `SharedSparkSession.afterEach`, retried via `eventually` for ~10s.
### Why only Spark 4.0.2 / JDK 21
These V1/V2 Parquet and ORC source suites are flaky on JDK 21 even in apache/spark's own CI. Spark's workaround is `DEDICATED_JVM_SBT_TESTS`, which forks a separate JVM per listed suite. Comet's workflow (`.github/workflows/spark_sql_test.yml:199`) sets this var, but only for `spark-short == '4.0'`.
### Why the workaround silently fails
The workflow also unconditionally sets `SERIAL_SBT_TESTS: \"1\"` at line 191 (added in #4285 to fix OOM on standard runners). In Spark 4.0.2's `project/SparkBuild.scala`:
```scala
if (!sys.env.contains(\"SERIAL_SBT_TESTS\")) {
allProjects.foreach(enable(SparkParallelTestGrouping.settings))
}
```
When `SERIAL_SBT_TESTS` is set, `SparkParallelTestGrouping.settings` is never installed, and that block is the only consumer of `DEDICATED_JVM_SBT_TESTS`. The env var is read into a set that is never used; SBT falls back to a single forked JVM and runs every suite there.
Evidence from a failing log (run [26004020697](https://github.com/apache/datafusion-comet/actions/runs/26004020697/job/76441382377)):
- thread-leak warning mentions `readingParquetFooters-ForkJoinPool-12260-worker-1` (high pool counter = many prior suites in this JVM)
- unrelated suites (`AnalysisConfOverrideSuite`, `TPCDSModifiedPlanStabilityWithStatsSuite`, state-store suites) appear in the same JVM right before these Parquet/ORC suites
- no SBT fork-restart markers between unrelated suites
So all ~11,600 tests share one JVM. By the time these flaky suites run, accumulated state is enough to make the leak-detection eventually-loop time out.
### Options
The two env vars are mutually exclusive by Spark's design. Either:
1. Drop `SERIAL_SBT_TESTS=1` for the 4.0.2/JDK21 row so the dedicated-JVM workaround actually fires (likely requires raising the heap to avoid OOM regression).
2. Move the four flaky suites into their own CI step/job for the 4.0.2/JDK21 matrix entry, achieving isolation outside SBT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Frequent CI failures for Spark 4.0.2 / JDK 21 [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich closed issue #4327: Frequent CI failures for Spark 4.0.2 / JDK 21
URL: https://github.com/apache/datafusion-comet/issues/4327
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Frequent CI failures for Spark 4.0.2 / JDK 21 [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #4327:
URL: https://github.com/apache/datafusion-comet/issues/4327#issuecomment-4478711265
Another one https://github.com/apache/datafusion-comet/actions/runs/25951502106/job/76290385532?pr=4328
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org