You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/23 17:41:58 UTC
[PR] feat(executor): default to a bounded, auto-sized memory pool [datafusion-ballista]
andygrove opened a new pull request, #2160:
URL: https://github.com/apache/datafusion-ballista/pull/2160
# Which issue does this PR close?
<!-- No tracked issue yet; can be linked if one is filed. -->
# Rationale for this change
By default (`--memory-pool-size` unset), an executor runs with DataFusion's unbounded memory pool, so spillable operators (sort, spillable aggregate/join, the sort-shuffle writer) never spill. On heavy plans — for example TPC-H q9 stage 1, with several concurrent tasks per executor each scanning `lineitem` and writing shuffle output — memory grows until the OS (or the container cgroup) OOM-kills the process. The executor is then restarted, the scheduler re-dispatches the same tasks, and it OOMs again: a reset → re-dispatch → reset livelock that surfaces to the user as executors "heartbeat timing out" and the query never completing.
Installing a bounded `FairSpillPool` avoids this by letting DataFusion spill to disk instead of exhausting memory. Making that the default means a fresh cluster does not OOM on heavy plans without manual tuning.
# What changes are included in this PR?
Executor-side only (`ballista/executor/src/executor_process.rs`, `ballista/executor/src/config.rs`), plus a doc update.
- `--memory-pool-size` now has three-way semantics:
- **unset (default):** a bounded `FairSpillPool` sized at a fraction (default 70%) of the detected host/cgroup memory limit, split evenly across vcores.
- **`0`:** unbounded (DataFusion's default, no spilling) — the explicit opt-out that preserves the previous behavior.
- **`N`:** exactly `N` bytes (unchanged).
- New `--memory-pool-fraction` flag (range `(0.0, 1.0]`, default `0.70`) tunes the auto path; ignored when `--memory-pool-size` is set explicitly.
- Cgroup-aware detection: reads cgroup v2 `memory.max` / v1 `memory.limit_in_bytes` and uses `min(host, cgroup)`, so container memory limits are respected. Falls back to host memory (via `sysinfo`) when no cgroup limit applies, and to unbounded-with-warning if no limit can be detected.
- A startup log line reporting the resolved pool size and its source, e.g. `Executor memory pool: 22.4 GiB (auto: 70% of host memory), split across 8 vcores (2.8 GiB/vcore)`.
- Updated `docs/source/user-guide/tuning-guide.md` to describe the new default, the escape hatch, and the new flag.
The pure resolution logic (classification, `min`/fraction, cgroup/host detection) is unit-tested across the full matrix, including cgroup-v2 `max`, the cgroup-v1 "unlimited" sentinel, host-only, cgroup-only, and undetected cases.
# Are there any user-facing changes?
Yes — this changes default executor behavior:
- **Default is now a bounded, spilling memory pool.** Executors with `--memory-pool-size` unset will now spill to disk under memory pressure instead of running unbounded. Ensure the executor's `--work-dir` has adequate scratch space.
- **`--memory-pool-size 0` now means "unbounded."** Previously an explicit `0` would have failed at startup (per-vcore share of zero); it is now the supported way to opt back into the old unbounded pool.
- **New `--memory-pool-fraction` flag** (default `0.70`).
- **`ExecutorProcessConfig::default()` changed:** library embedders and standalone/example executors constructed via `..Default::default()` now get the auto-sized bounded pool instead of the unbounded one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(executor): default to a bounded, auto-sized memory pool [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #2160:
URL: https://github.com/apache/datafusion-ballista/pull/2160#issuecomment-5061740213
cc @phillipleblanc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(executor): default to a bounded, auto-sized memory pool [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #2160:
URL: https://github.com/apache/datafusion-ballista/pull/2160
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org