You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "dmefs (via GitHub)" <gi...@apache.org> on 2026/04/28 06:24:17 UTC
[PR] ci: pin JDK per Spark version in Iceberg workflow matrix [datafusion-comet]
dmefs opened a new pull request, #4120:
URL: https://github.com/apache/datafusion-comet/pull/4120
## Which issue does this PR close?
Closes #4101 .
## Rationale for this change
- Iceberg's own CI does not exhaustively cross JDK with every Spark version; we are overcovering relative to upstream.
- JDK 11 vs 17 differences that affect Comet are caught elsewhere: pr_build_linux.yml and spark_sql_test.yml both run JDK 11 + 17 against multiple Spark versions.
- The Iceberg suites are about Iceberg/Comet integration, not JVM-level differences; the same JDK delta is unlikely to surface there but not in the broader matrix.
- Frees CI capacity to add Spark 4.0.1 and 4.1.1 to the matrix without a net increase in PR runtime.
## What changes are included in this PR?
The java-version: [11, 17] axis is removed from the base matrix of all three test jobs (iceberg-spark, iceberg-spark-extensions, iceberg-spark-runtime) and replaced with an `include:` block that injects the appropriate JDK per Spark version:
- Spark 3.4.3 → JDK 11
- Spark 3.5.8 → JDK 17
This reduces test jobs from 36 to 18 per PR with no changes to step logic or variable references.
## How are these changes tested?
CI on this PR shows 18 jobs instead of 36.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: pin JDK per Spark version in Iceberg workflow matrix [datafusion-comet]
Posted by "dmefs (via GitHub)" <gi...@apache.org>.
dmefs commented on PR #4120:
URL: https://github.com/apache/datafusion-comet/pull/4120#issuecomment-4333667593
Confirmed that there are 6 jobs per matrix and 3 matrices, for a total of 18 jobs per PR.
<img width="643" height="488" alt="Screenshot 2026-04-28 at 4 32 06 PM" src="https://github.com/user-attachments/assets/5a2c6508-527f-42e2-abb6-9508425f4a74" />
https://github.com/dmefs/datafusion-comet/actions/runs/25040787129
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: pin JDK per Spark version in Iceberg workflow matrix [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #4120:
URL: https://github.com/apache/datafusion-comet/pull/4120#issuecomment-4332950073
Thank you for the PR @dmefs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: pin JDK per Spark version in Iceberg workflow matrix [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich merged PR #4120:
URL: https://github.com/apache/datafusion-comet/pull/4120
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: pin JDK per Spark version in Iceberg workflow matrix [datafusion-comet]
Posted by "dmefs (via GitHub)" <gi...@apache.org>.
dmefs commented on PR #4120:
URL: https://github.com/apache/datafusion-comet/pull/4120#issuecomment-4333414550
Hi @coderfender, Thanks for the suggestion!
I'll verify the matrix build using my fork first. I'll convert this PR to draft while checking the action results.
Here is the run:
https://github.com/dmefs/datafusion-comet/actions/runs/25040787129.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: pin JDK per Spark version in Iceberg workflow matrix [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #4120:
URL: https://github.com/apache/datafusion-comet/pull/4120#issuecomment-4381192918
Thanks @dmefs. LGTM but I'd like @mbutrovich to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org