You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "JanKaul (via GitHub)" <gi...@apache.org> on 2026/05/06 15:32:38 UTC

[PR] Cooperative memory reclaim via async `MemoryReclaimer` [datafusion]

JanKaul opened a new pull request, #22043:
URL: https://github.com/apache/datafusion/pull/22043

   Adds an async hook that lets a `MemoryPool` ask other consumers to free
   memory before failing an allocation.
   
   This PR is **complementary to [#21425](https://github.com/apache/datafusion/pull/21425)**
   — it is not a replacement. It exists to broaden the design discussion
   there with a concrete alternative, not to supersede that work.
   
   ## Design
   
   - **`trait MemoryReclaimer`** (async) attached to a `MemoryConsumer` via
     `with_reclaimer`. Implements: `reclaim(target)`, optional
     `reclaimable_bytes`, optional `priority`.
   - **`MemoryPool::try_grow_async`** — default delegates to sync
     `try_grow`. `TrackConsumersPool` overrides it to walk registered
     reclaimers (priority desc, size desc) on OOM, retry the grow after
     each, then fall through to `inner.try_grow_async` so a wrapped
     reclaim-aware pool isn't shadowed.
   - **Operator-side state machine** (`SortExec`): a channel-based
     `ExternalSorterReclaimer` hands a oneshot to the partition's stream
     loop; `tokio::select! biased { reclaim_rx.recv() … ; input.next() … }`
     spills end-to-end before replying with the freed-byte count. The
     stream loop is the sole owner of the sorter's batches, so the spill
     is ordered before the report — the bytes the pool sees are bytes
     already on disk.
   
   ## How this differs from #21425
   
   - Async trait + `try_grow_async` instead of sync `pool.reclaim(...)` —
     matches the channel hand-off pattern needed for cooperative spill
     inside DataFusion's async execution.
   - Auto-triggered on OOM rather than caller-driven.
   - Includes operator wiring for `SortExec` to demonstrate the full flow.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Cooperative memory reclaim via async `MemoryReclaimer` [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #22043: Cooperative memory reclaim via async `MemoryReclaimer`
URL: https://github.com/apache/datafusion/pull/22043


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Cooperative memory reclaim via async `MemoryReclaimer` [datafusion]

Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on PR #22043:
URL: https://github.com/apache/datafusion/pull/22043#issuecomment-4394157165

   If I use this branch to query a larger-than-memory dataset, I get:
   ```
   Error: Resources exhausted: Additional allocation failed for ExternalSorterMerge[3] with top memory consumers (across reservations) as:
     ExternalSorter[6]#13(can spill: true) consumed 597.0 MB, peak 597.0 MB,
     ExternalSorter[12]#138(can spill: true) consumed 597.0 MB, peak 597.0 MB,
     ExternalSorter[2]#11(can spill: true) consumed 596.6 MB, peak 596.6 MB.
   Error: Failed to allocate additional 192.0 KB for ExternalSorterMerge[3] with 277.7 MB already allocated for this reservation - 2.2 KB remain available for the total memory pool: greedy(used: 8.0 GB, pool_size: 8.0 GB)
   ```
   With vanilla datafusion the ExternalSorter fails to allocate memory. So it looks like it solves the memory reclaiming issue for a single operator. However, now the next operator ExternalSorterMerge fails. So this solution doesn't handle cross operator reclamations.
   
   I think we need a hierarchical design with a MemoryPool and Reclaimer tree such that we have full control. I think the Velox design would be really great.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Cooperative memory reclaim via async `MemoryReclaimer` [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22043:
URL: https://github.com/apache/datafusion/pull/22043#issuecomment-4953831421

   Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Cooperative memory reclaim via async `MemoryReclaimer` [datafusion]

Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on PR #22043:
URL: https://github.com/apache/datafusion/pull/22043#issuecomment-4394157165

   If I use this branch to query a larger-than-memory dataset, I get:
   ```
   Error: Resources exhausted: Additional allocation failed for ExternalSorterMerge[3] with top memory consumers (across reservations) as:
     ExternalSorter[6]#13(can spill: true) consumed 597.0 MB, peak 597.0 MB,
     ExternalSorter[12]#138(can spill: true) consumed 597.0 MB, peak 597.0 MB,
     ExternalSorter[2]#11(can spill: true) consumed 596.6 MB, peak 596.6 MB.
   Error: Failed to allocate additional 192.0 KB for ExternalSorterMerge[3] with 277.7 MB already allocated for this reservation - 2.2 KB remain available for the total memory pool: greedy(used: 8.0 GB, pool_size: 8.0 GB)
   ```
   With vanilla datafusion the ExternalSorter fails to allocate memory. So it looks like it solves the memory reclaiming issue for a single operator. However, now the next operator ExternalSorterMerge fails. So this solution doesn't handle cross operator reclamations.
   
   I think we need a hierarchical design with a MemoryPool and Reclaimer tree such that we have full control. I think the Velox design would be really great.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Cooperative memory reclaim via async `MemoryReclaimer` [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22043:
URL: https://github.com/apache/datafusion/pull/22043#issuecomment-4390049705

   <!-- semver-check-comment -->
   Thank you for opening this pull request!
   
   Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
   
   <details>
   <summary>Details</summary>
   
   ```
        Cloning apache/main
       Building datafusion-execution v53.1.0 (current)
          Built [  30.084s] (current)
        Parsing datafusion-execution v53.1.0 (current)
         Parsed [   0.025s] (current)
       Building datafusion-execution v53.1.0 (baseline)
          Built [  29.348s] (baseline)
        Parsing datafusion-execution v53.1.0 (baseline)
         Parsed [   0.025s] (baseline)
       Checking datafusion-execution v53.1.0 -> v53.1.0 (no change; assume patch)
        Checked [   0.214s] 222 checks: 221 pass, 1 fail, 0 warn, 30 skip
   
   --- failure auto_trait_impl_removed: auto trait no longer implemented ---
   
   Description:
   A public type has stopped implementing one or more auto traits. This can break downstream code that depends on the traits being implemented.
           ref: https://doc.rust-lang.org/reference/special-types-and-traits.html#auto-traits
          impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/auto_trait_impl_removed.ron
   
   Failed in:
     type MemoryConsumer is no longer UnwindSafe, in /home/runner/work/datafusion/datafusion/datafusion/execution/src/memory_pool/mod.rs:263
     type MemoryConsumer is no longer RefUnwindSafe, in /home/runner/work/datafusion/datafusion/datafusion/execution/src/memory_pool/mod.rs:263
     type TrackConsumersPool is no longer UnwindSafe, in /home/runner/work/datafusion/datafusion/datafusion/execution/src/memory_pool/pool.rs:410
   
        Summary semver requires new major version: 1 major and 0 minor checks failed
       Finished [  61.434s] datafusion-execution
       Building datafusion-physical-plan v53.1.0 (current)
          Built [  32.161s] (current)
        Parsing datafusion-physical-plan v53.1.0 (current)
         Parsed [   0.122s] (current)
       Building datafusion-physical-plan v53.1.0 (baseline)
          Built [  31.738s] (baseline)
        Parsing datafusion-physical-plan v53.1.0 (baseline)
         Parsed [   0.122s] (baseline)
       Checking datafusion-physical-plan v53.1.0 -> v53.1.0 (no change; assume patch)
        Checked [   0.596s] 222 checks: 222 pass, 30 skip
        Summary no semver update required
       Finished [  66.300s] datafusion-physical-plan
   ```
   
   </details>
   <!-- semver-check-comment -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org