You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "junknown (via GitHub)" <gi...@apache.org> on 2026/06/09 12:37:31 UTC
[I] Add a configurable cap on spill-file merge fan-in (max open files during external merge) [datafusion]
junknown opened a new issue, #22848:
URL: https://github.com/apache/datafusion/issues/22848
### Is your feature request related to a problem or challenge?
When a spilling operator (hash aggregate, sort) merges its spill files, the number of
spill files opened simultaneously in a single merge pass is bounded only by available
memory, never by a file-count ceiling.
In `MultiLevelMergeBuilder::get_sorted_spill_files_to_merge`
(datafusion-physical-plan/src/sorts/multi_level_merge.rs), the loop keeps adding spill
files to the current pass as long as `reservation.try_grow(...)` succeeds — i.e. as long
as the (often large, shared) memory pool has room. There is no upper bound on how many
files are opened at once; the only floor is "at least 2 streams". `DiskManager` exposes
`max_temp_directory_size` (total bytes on disk) but nothing for the number of open
files / merge fan-in.
This becomes a process-level problem on high-core machines. The aggregation runs
`target_partitions` streams in parallel (= num CPUs by default), and each partition runs
its own multi-level merge. With a generous memory pool, each merge opens many spill files
at once, so peak open file descriptors ≈ `target_partitions × fan_in_per_pass`. On a
many-core node a heavy `GROUP BY` / `COUNT(DISTINCT ...)` over many spilled files fails
with:
IO error: Too many open files (os error 24)
on a spill temp file. The error is a `Result` and is handled gracefully, but EMFILE is a
*process-wide* condition: at the peak it can transiently break unrelated operations in the
same process (opening other files, network connections, logging, other concurrent queries).
Today the only real control is the OS `ulimit -n`. Lowering `target_partitions` works but
penalizes *every* query on the node — including the vast majority that never spill — which
is the wrong trade-off, since non-spilling queries never enter the merge path at all.
### Describe the solution you'd like
A configuration knob that caps the merge fan-in — the maximum number of spill files
opened simultaneously in a single merge pass — independently of available memory. When
the cap is hit, the merge simply performs more passes (slower, but bounded file
descriptors), which is exactly the desired trade-off for "make a very heavy aggregation
feasible, even if slower".
Proposed shape:
- Add `max_merge_fanin: usize` (0 = unbounded, the current behavior and the default) to
`DiskManagerBuilder` / `DiskManager` in datafusion-execution, alongside the existing
`max_temp_directory_size`. It is a natural home: it is the type that already carries the
spill/disk caps, and `SpillManager` already reaches it via `env.disk_manager`.
- In `MultiLevelMergeBuilder::get_sorted_spill_files_to_merge`, after the existing
memory-driven loop, clamp the count:
let cap = self.spill_manager.env().disk_manager.max_merge_fanin();
if cap != 0 {
number_of_spills_to_read_for_current_phase =
number_of_spills_to_read_for_current_phase.min(cap.max(2));
}
The `.max(2)` preserves the existing "at least 2 streams per merge" invariant, and the
surrounding algorithm already handles leftover spill files by doing additional passes.
Crucially, this is zero-cost for queries that do not spill: `MultiLevelMergeBuilder` is
only constructed when there are spill files to merge, so non-spilling queries never
execute this branch. `target_partitions` can stay high (full parallelism for everyone),
and only the heavy spilling operators pay the price — more merge passes in exchange for a
peak FD bound of `target_partitions × max_merge_fanin`.
Optionally surface it as an execution config option (e.g.
`datafusion.execution.max_merge_fanin`) so it can be set per-session.
### Describe alternatives you've considered
- Lowering `target_partitions`: reduces the FD multiplier, but it is a per-node global
that slows down *all* queries, including the majority that never spill. Wrong altitude:
it taxes the common case to fix the rare heavy-aggregation case.
- Lowering the memory pool size: indirectly shrinks the per-pass fan-in (toward the floor
of 2), but the pool is shared across all concurrent queries, the relationship to FD count
is indirect, and a smaller pool makes *more* queries spill.
- Raising `ulimit -n` / `LimitNOFILE`: a necessary operational mitigation, but it only
moves the ceiling; it does not give the engine a way to bound its own FD usage, and the
bound still scales with cores × data.
- `max_temp_directory_size`: caps total bytes on disk, not the number of files open at once,
so it does not address EMFILE.
None of these give a direct, memory-independent bound on simultaneously-open spill files
that is also free for non-spilling queries — which the proposed fan-in cap does.
### Additional context
Observed on DataFusion 53.1.0; the same code path is present on `main`.
Repro shape: a `SELECT COUNT(DISTINCT col_b), col_a FROM t GROUP BY col_a` over a large
dataset that forces the hash aggregate to spill many files, on a many-core machine with a
default-ish `ulimit -n` (e.g. 1024). `COUNT(DISTINCT ...)` is expanded into two stacked
aggregation layers, both of which can spill, compounding the FD pressure.
Relevant code:
- datafusion-physical-plan/src/sorts/multi_level_merge.rs —
`MultiLevelMergeBuilder::get_sorted_spill_files_to_merge` (the unbounded, memory-driven
fan-in loop).
- datafusion-physical-plan/src/aggregates/row_hash.rs — the hash aggregate hands all
accumulated spill files to `StreamingMergeBuilder::with_sorted_spill_files`.
- datafusion-execution/src/disk_manager.rs — `DiskManagerBuilder` / `DiskManager`, where
`max_temp_directory_size` lives and where `max_merge_fanin` would be added.
Related: #7858 touches spilling improvements in the hash aggregate and notes the
"too many open files" concern in passing, but does not propose a fan-in cap.
Happy to open a PR implementing the above if the approach sounds reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add a configurable cap on spill-file merge fan-in (max open files during external merge) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #22848: Add a configurable cap on spill-file merge fan-in (max open files during external merge)
URL: https://github.com/apache/datafusion/issues/22848
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org