You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "jbonofre (via GitHub)" <gi...@apache.org> on 2026/03/08 07:07:19 UTC
[PR] Build: Define timeout on Flink CI and cap the number of workers [iceberg]
jbonofre opened a new pull request, #15541:
URL: https://github.com/apache/iceberg/pull/15541
I have observed intermittent timeout on CI for the Flink jobs.
Usually, the CI Flink jobs take around 30mn, but sometime, they are running forever, and timing out after 6 hours.
This change is to timeout after 60 minutes (no need to wait 6 hours), and cap the number of workers for parallelism test to reduce the memory pressure on CI runners.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Build: Cap the number of workers in Flink CI [iceberg]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #15541:
URL: https://github.com/apache/iceberg/pull/15541#issuecomment-4054948312
@pvary my idea is to set maxParallelForks at 4 when testParallelism=auto.
Currently, the formula was ceil(availableProcessors / 2) with no upper bound.
Our CI runner has 16 vCPUs, that meant 8 parallel test forks -- each configured with 1500m heap (according to `build.gradle`, at line 246).
It means a total of ~12 GB just for test JVMs (without counting Gradle itself, etc).
The GitHub ubuntu-24.04 runner has 4 CPUs, so caping here is useless.
Let me close this PR and investigate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Build: Define timeout on Flink CI and cap the number of workers [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #15541:
URL: https://github.com/apache/iceberg/pull/15541#issuecomment-4025077507
@zheliu2: The Flink test suite typically runs for around 30 minutes. There are tests where timeouts are not configured correctly, and if a PR introduces a bug, those tests can end up running indefinitely. We’re gradually adding proper timeouts to the individual tests, but if a new case slips through, introducing a global timeout could be a pragmatic way to free up resources in the meantime. This wouldn’t replace adding timeouts to the tests themselves; it would simply help limit the impact when something goes wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Build: Define timeout on Flink CI and cap the number of workers [iceberg]
Posted by "zheliu2 (via GitHub)" <gi...@apache.org>.
zheliu2 commented on PR #15541:
URL: https://github.com/apache/iceberg/pull/15541#issuecomment-4025037162
A few quesitons
- Why 60 minutes specifically? Are there known cases where legitimate Flink CI runs exceed 60 minutes
(e.g., under heavy GH Actions load)? Want to make sure we don't fail valid jobs unexpectedly.
- Do we understand the root cause of the hangs? If not, the timeout is a good band-aid but we might want to track down why jobs run forever in the first place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Build: Define timeout on Flink CI and cap the number of workers [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #15541:
URL: https://github.com/apache/iceberg/pull/15541#discussion_r2902342884
##########
build.gradle:
##########
@@ -165,7 +165,8 @@ subprojects {
if (testParallelism != null) {
def numTestWorkers
if (testParallelism.equalsIgnoreCase('auto')) {
- numTestWorkers = Math.ceil(Runtime.getRuntime().availableProcessors() / 2.0) as int
+ // Cap at 4 to avoid memory pressure on CI runners (each fork uses up to 1500m heap)
+ numTestWorkers = Math.min(Math.ceil(Runtime.getRuntime().availableProcessors() / 2.0) as int, 4)
Review Comment:
[Question] Do we want to fix Flink only, or is this intentionally targeting all modules?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Build: Cap the number of workers in Flink CI [iceberg]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #15541:
URL: https://github.com/apache/iceberg/pull/15541#issuecomment-4053683179
So, we now have a PR for the Flink timeout specifially: #15617
This PR is now focusing on the Flink CI "flakiness" and avoid memory pressure on all Iceberg runners on CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Build: Define timeout on Flink CI and cap the number of workers [iceberg]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #15541:
URL: https://github.com/apache/iceberg/pull/15541#issuecomment-4053586274
Let me split this PR in two PRs:
1. One with just the timeout: I think it's good enough to avoid to wait for hours (typically the flink CI passes in 30 minutes).
2. One to relax the memory "pressure" on the worker and investigate a fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Build: Cap the number of workers in Flink CI [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #15541:
URL: https://github.com/apache/iceberg/pull/15541#issuecomment-4066509693
Thank you for working on this @jbonofre!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Build: Define timeout on Flink CI and cap the number of workers [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15541:
URL: https://github.com/apache/iceberg/pull/15541#discussion_r2904204257
##########
build.gradle:
##########
@@ -165,7 +165,8 @@ subprojects {
if (testParallelism != null) {
def numTestWorkers
if (testParallelism.equalsIgnoreCase('auto')) {
- numTestWorkers = Math.ceil(Runtime.getRuntime().availableProcessors() / 2.0) as int
+ // Cap at 4 to avoid memory pressure on CI runners (each fork uses up to 1500m heap)
+ numTestWorkers = Math.min(Math.ceil(Runtime.getRuntime().availableProcessors() / 2.0) as int, 4)
Review Comment:
Could we just separate the 2 changes?
The Flink timeout is non-questionable. I would like to merge it ASAP
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Build: Cap the number of workers in Flink CI [iceberg]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #15541:
URL: https://github.com/apache/iceberg/pull/15541#discussion_r2929911331
##########
build.gradle:
##########
@@ -165,7 +165,8 @@ subprojects {
if (testParallelism != null) {
def numTestWorkers
if (testParallelism.equalsIgnoreCase('auto')) {
- numTestWorkers = Math.ceil(Runtime.getRuntime().availableProcessors() / 2.0) as int
+ // Cap at 4 to avoid memory pressure on CI runners (each fork uses up to 1500m heap)
+ numTestWorkers = Math.min(Math.ceil(Runtime.getRuntime().availableProcessors() / 2.0) as int, 4)
Review Comment:
Done, I created a PR for the timeout (#15617). The intent of this PR is to avoid memory pressure on CI runners only to investigate why Flink sometime runs for more than 1 hour (flaky).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Build: Cap the number of workers in Flink CI [iceberg]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #15541:
URL: https://github.com/apache/iceberg/pull/15541#issuecomment-4056732643
@pvary @mxm by the way, I took a look on the Flink CI which timed out on different jobs: they are all stuck on:
```
[TaskExecutorFileMergingManager shutdown hook] INFO org.apache.flink.runtime.state.TaskExecutorFileMergingManager - Shutting down TaskExecutorFileMergingManager.
```
So, I went on the Flink community, and this is a known Flink issue.
The TaskExecutorFileMergingManager shutdown hook can deadlock when the JVM is shutting down concurrently with the mini-cluster.
I'm investigating if I can find a better fix that just the timeout I already did.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Build: Define timeout on Flink CI and cap the number of workers [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #15541:
URL: https://github.com/apache/iceberg/pull/15541#issuecomment-4035988807
Latest failure due to exceeding maximum running time.
https://github.com/apache/iceberg/actions/runs/22915304106/job/66498887695
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Build: Cap the number of workers in Flink CI [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #15541:
URL: https://github.com/apache/iceberg/pull/15541#issuecomment-4054877287
@jbonofre: What is the result of decreasing the number of workers? What did you see that prompted the change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Build: Cap the number of workers in Flink CI [iceberg]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre closed pull request #15541: Build: Cap the number of workers in Flink CI
URL: https://github.com/apache/iceberg/pull/15541
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org