You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/13 20:07:15 UTC
[I] Job hangs indefinitely instead of failing when all executors are lost [datafusion-ballista]
andygrove opened a new issue, #2029:
URL: https://github.com/apache/datafusion-ballista/issues/2029
## Describe the bug
If every executor dies while a query is running, the job **never terminates**. The scheduler waits indefinitely rather than failing the query, so the client hangs forever.
## To Reproduce
Reproduced end to end on a real multi-process cluster by the chaos harness in #PR_NUM (scenario `killing_every_executor_terminates_the_job`, which fails under **both** AQE on and AQE off):
1. Start a scheduler and 2 executors.
2. Submit a query (the harness slows a stage with a delay UDF so the kill lands while tasks are genuinely running).
3. Wait until stage 1 has running tasks.
4. `SIGKILL` **both** executors.
5. Wait.
The job does not terminate within 120 seconds. The harness asserts only that the job *ends* — it makes no claim about success — and even that does not hold.
## Expected behavior
The job should fail with a clear error once the scheduler determines its tasks cannot be scheduled — there are no executors left to run them, and no prospect of recovery without a new executor registering.
Some judgement is needed on the exact policy, and reasonable people could disagree:
- Failing fast once all executors are lost is the simplest behavior and is what a client most likely wants.
- Alternatively, if the intent is to *wait* for executors to come back (a plausible design for a long-lived cluster), then that wait should be **bounded** by a configurable timeout, after which the job fails — and ideally the job should be visibly reported as blocked-on-resources in the meantime rather than looking indistinguishable from a job that is simply slow.
Either way, an unbounded silent hang is not a good outcome: the caller has no signal and no way to distinguish it from progress.
## Additional context
Found while building an HA chaos-testing harness (#PR_NUM) that drives the scheduler's fault-tolerance paths on a real multi-process cluster. This scenario is deliberately a **hang detector** rather than a correctness test.
Lower severity than the two sibling issues (the `FetchFailed` and `Shared`-wrapping misclassifications), since total executor loss is a more extreme condition than a single executor dying — but the failure mode (silent unbounded hang) is worse than a clean error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Job hangs indefinitely instead of failing when all executors are lost [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2029:
URL: https://github.com/apache/datafusion-ballista/issues/2029#issuecomment-5950423448
Fixed on `main` by #2212, with the remaining task-launch failure path addressed by #2240 and the default grace-period behavior corrected by #2279. Kubernetes regression coverage for both AQE modes was added in #2425.
The issue appears to have remained open because #2212 used malformed auto-close syntax (`Closes #. ...`). Closing as completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Job hangs indefinitely instead of failing when all executors are lost [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #2029: Job hangs indefinitely instead of failing when all executors are lost
URL: https://github.com/apache/datafusion-ballista/issues/2029
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org