You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/09 18:14:21 UTC
[PR] feat: reclaim intermediate shuffle data on job success [datafusion-ballista]
andygrove opened a new pull request, #1982:
URL: https://github.com/apache/datafusion-ballista/pull/1982
# Which issue does this PR close?
Closes #1981.
Related to #1980 (denser shuffle compression codec is a complementary, separate lever for the same disk pressure).
# Rationale for this change
When a query job succeeds, Ballista keeps the entire job directory on the executors — intermediate inter-stage shuffle data *and* the final-stage output — and only deletes it after a delay of `finished_job_data_clean_up_interval_seconds` (default 300s). That delay exists for a good reason: on success the scheduler returns the final stage's output partition locations to the client, and in the default (pull) model the client fetches those results from the executor's job directory *after* the job is marked successful, so the final output must survive for the client to read it.
The cost of that delay is disk. The intermediate shuffle data — which is the bulk of a job's on-disk footprint — is dead the moment the job succeeds (every stage's output has already been consumed by its downstream stage), yet it lingers for the full interval. In back-to-back runs (for example the TPC-H SF10 CI job, which runs all 22 queries with AQE off and on), intermediate shuffle from many just-completed queries accumulates at once and intermittently exhausts the runner's disk (`No space left on device`, os error 28).
This change reclaims the intermediate shuffle data immediately on job success while retaining the final-stage output for the existing delayed cleanup, so the client can still fetch results.
# What changes are included in this PR?
- **Protobuf:** a new `repeated uint32 remove_stage_ids` field on `RemoveJobDataParams` (push path) and `CleanJobDataParams` (pull/poll path). Empty means "remove the whole job directory" — identical to today's behavior, so the change is backward compatible on the wire.
- **Executor:** the whole-job `remove_job_dir` becomes stage-aware `remove_job_data(work_dir, job_id, remove_stage_ids)` — empty removes the whole job dir; non-empty removes only the given `work_dir/{job_id}/{stage_id}` subdirectories (missing ones skipped), keeping the existing `is_subdirectory` safety guard per path.
- **Scheduler:** the cleanup plumbing carries `remove_stage_ids` through both the push (direct RPC) and pull (`pending_cleanup_jobs` → poll response) paths. On job success the scheduler computes the intermediate stage set (`all stages − final stages`, where a final stage has empty `output_links`) and fires an immediate reclaim via a new `clean_up_intermediate_job_data`, in addition to the existing delayed whole-job cleanup that now just removes the retained final-stage output.
Safety invariant: the immediate reclaim only ever deletes stages in `all − final`; a final-stage output directory can never be reached by it, and any unresolvable/failed job path degrades to an empty set (no reclaim, existing delayed cleanup only). Failed-job cleanup is unchanged.
# Are there any user-facing changes?
Behavioral, not API. By default, intermediate shuffle files for a completed query are removed from executors immediately on job success rather than after `finished_job_data_clean_up_interval_seconds`. Final query results are unaffected — they are retained for the existing delayed-cleanup window exactly as before. No public API or configuration changes; no new configuration is required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: delete intermediate shuffle files on job success [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #1982:
URL: https://github.com/apache/datafusion-ballista/pull/1982#discussion_r3564386734
##########
ballista/scheduler/src/state/execution_graph.rs:
##########
@@ -240,6 +240,16 @@ pub trait ExecutionGraph: Debug {
/// Exposes executions stages and stage id's
fn stages(&self) -> &HashMap<usize, ExecutionStage>;
+ /// Stage ids of all non-final (intermediate) stages — those whose
+ /// `output_links` is non-empty. The final stage(s) are excluded.
+ fn intermediate_stage_ids(&self) -> Vec<u32> {
Review Comment:
You are correct. The current PR does no harm under AQE (and we run TPC-H in CI with AQE enabled). I filed a follow-on issue to implement this for AQE.
https://github.com/apache/datafusion-ballista/issues/1996
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: delete intermediate shuffle files on job success [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1982:
URL: https://github.com/apache/datafusion-ballista/pull/1982#issuecomment-4946974308
Merged. Thanks @milenkovicm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: delete intermediate shuffle files on job success [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #1982:
URL: https://github.com/apache/datafusion-ballista/pull/1982
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: delete intermediate shuffle files on job success [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #1982:
URL: https://github.com/apache/datafusion-ballista/pull/1982#discussion_r3564284209
##########
ballista/scheduler/src/state/execution_graph.rs:
##########
@@ -240,6 +240,16 @@ pub trait ExecutionGraph: Debug {
/// Exposes executions stages and stage id's
fn stages(&self) -> &HashMap<usize, ExecutionStage>;
+ /// Stage ids of all non-final (intermediate) stages — those whose
+ /// `output_links` is non-empty. The final stage(s) are excluded.
+ fn intermediate_stage_ids(&self) -> Vec<u32> {
Review Comment:
this might not work with AQE as output links are not populated, have to check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org