You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2026/05/18 19:13:11 UTC
[I] spark-ci walltime improvements [iceberg]
kevinjqliu opened a new issue, #16397:
URL: https://github.com/apache/iceberg/issues/16397
### Context
`spark-ci` dominates the overall time CI takes to run. Based on [most recent stats](https://github.com/apache/iceberg/actions/metrics/performance), it averages 1h5m while the next slowest ci job (`flink-ci`) averages around only 21m.
This means PRs must wait at least 1h for `spark-ci` to complete before merging.
A couple of things we can do to improve the overall time it takes to run `spark-ci`
1. fix the gradle cache. reduce time to download dependencies. fixed in https://github.com/apache/iceberg/pull/16356
2. parallelize execution. enabling `-DtestParallelism=auto` for all tests, following `flink-ci`. And split `spark-ci` into 2 concurrent jobs. Fixed in https://github.com/apache/iceberg/pull/16357. This reduced Spark CI walltime from ~84 min to ~58 min (−31%)
### Ask
One more potential fix is to reduce the runtime of the tests inside `spark-ci`. I've pulled data on how long each test class takes to run. See https://gist.github.com/kevinjqliu/86827bedf9d02adad36881f476484208#top-25--core-spark for more details.
Some of the tests are inherently slow, but exacerbated by parameterized matrix that dramatically multiples the number of runs.
Is it worth to dig into these tests and see if there are any optimizations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] spark-ci walltime improvements [iceberg]
Posted by "MaxNevermind (via GitHub)" <gi...@apache.org>.
MaxNevermind commented on issue #16397:
URL: https://github.com/apache/iceberg/issues/16397#issuecomment-4538669307
Hey @kevinjqliu,
Have you considered reusing a single Spark Session for multiple tests classes? I believe Spark's Session setup is quite resource consuming and atm as I understand each class create individual Spark Session inside of `TestBase.startMetastoreAndSpark()`. It is probably not an easy/quick fix though even if applicable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] spark-ci walltime improvements [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #16397:
URL: https://github.com/apache/iceberg/issues/16397#issuecomment-4481176641
another potential optimization is to minimize the number of triggers for `spark-ci`. ie
- flink changes dont need to trigger spark-ci
- spark 3.5 changes dont need to trigger spark 4.x
- etc
we should be careful here and only apply optimizations when absolutely sure
this will reduce the number of jobs, but the walltime is still dominated by a single matrix run (which after #16357 should be around 58 mins)
removing spark 3.4 is the same flavor of change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org