You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "ywskycn (via GitHub)" <gi...@apache.org> on 2026/08/28 01:06:09 UTC
[PR] Fix bounded shuffle allocator memory usage reporting [datafusion-comet]
ywskycn opened a new pull request, #5516:
URL: https://github.com/apache/datafusion-comet/pull/5516
## Which issue does this PR close?
Solving sub-issue 11 in #5212.
## Rationale for this change
`CometBoundedShuffleMemoryAllocator` tracks allocations through its own `allocatedMemory` counter because its allocations bypass Spark's memory manager. However, it inherited `MemoryConsumer.getUsed()`, which reports Spark's unused internal counter and always returned `0`.
This caused Comet shuffle memory usage logging to report zero even when the bounded allocator held pages or pointer arrays.
## What changes are included in this PR?
- Override `getUsed()` in `CometBoundedShuffleMemoryAllocator` to return the synchronized `allocatedMemory` counter.
- Document that the value represents allocator-wide usage shared across tasks.
- Add regression tests for page allocations, pointer arrays, repeated frees, failed allocations, exhausted budgets, and page table exhaustion.
- Register the new test suite in Linux and macOS CI.
## How are these changes tested?
- Added `CometBoundedShuffleMemoryAllocatorSuite`.
- Verified all five new tests failed before the fix because `getUsed()` returned `0`.
- Verified the allocator suite and existing `SpillSorterSuite` pass after the fix:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: report bounded shuffle allocator memory usage [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5516:
URL: https://github.com/apache/datafusion-comet/pull/5516#issuecomment-5448077714
Merged, thanks @ywskycn !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: report bounded shuffle allocator memory usage [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao merged PR #5516:
URL: https://github.com/apache/datafusion-comet/pull/5516
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org