You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/02/13 19:36:09 UTC

[I] Possible regression in memory usage with DataFusion 52 upgrade [datafusion-comet]

andygrove opened a new issue, #3517:
URL: https://github.com/apache/datafusion-comet/issues/3517

   ### Describe the bug
   
   I frequently run benchmarks in k8s, and noticed that on the PR for upgrading to DF 52, the pod is being killed due to OOM:
   
   ```
   comet-pr-3470-c3886859859-9c2jd       0/1     OOMKilled
   ```
   
   I grep'd the log for errors and warnings and saw this output:
   
   ```
   26/02/11 21:01:14 WARN DAGScheduler: Broadcasting large task binary with size 1077.4 KiB
   26/02/11 21:01:21 WARN DAGScheduler: Broadcasting large task binary with size 1077.5 KiB
   26/02/11 21:01:38 WARN DAGScheduler: Broadcasting large task binary with size 1204.7 KiB
   ```
   
   The final few lines in the log were:
   
   ```
   26/02/11 21:02:02 INFO CometExecIterator: memoryPoolType=fair_unified, offHeapSize=24576 MB, memoryFraction=1.0, memoryLimit=24576 MB, memoryLimitPerTask=3072 MB
   26/02/11 21:02:02 INFO CometExecIterator: memoryPoolType=fair_unified, offHeapSize=24576 MB, memoryFraction=1.0, memoryLimit=24576 MB, memoryLimitPerTask=3072 MB
   26/02/11 21:02:02 INFO CometExecIterator: memoryPoolType=fair_unified, offHeapSize=24576 MB, memoryFraction=1.0, memoryLimit=24576 MB, memoryLimitPerTask=3072 MB
   26/02/11 21:02:02 INFO CometExecIterator: memoryPoolType=fair_unified, offHeapSize=24576 MB, memoryFraction=1.0, memoryLimit=24576 MB, memoryLimitPerTask=3072 MB
   26/02/11 21:02:02 INFO ShuffleBlockFetcherIterator: Getting 48 (1843.0 KiB) non-empty blocks including 48 (1843.0 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
   26/02/11 21:02:02 INFO ShuffleBlockFetcherIterator: Getting 48 (1846.8 KiB) non-empty blocks including 48 (1846.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
   26/02/11 21:02:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
   26/02/11 21:02:02 INFO ShuffleBlockFetcherIterator: Getting 48 (1850.7 KiB) non-empty blocks including 48 (1850.7 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
   26/02/11 21:02:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
   26/02/11 21:02:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
   26/02/11 21:02:02 INFO ShuffleBlockFetcherIterator: Getting 48 (1846.8 KiB) non-empty blocks including 48 (1846.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
   26/02/11 21:02:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
   ```
   
   In this instance, it was running q21 when the OOM happened.
   
   ### Steps to reproduce
   
   _No response_
   
   ### Expected behavior
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Possible regression in memory usage with DataFusion 52 upgrade [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #3517:
URL: https://github.com/apache/datafusion-comet/issues/3517#issuecomment-3920971215

   I am no longer seeing this issue, but there were changes to memory accounting in DF 52 that could change how we tune memory for jobs. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Possible regression in memory usage with DataFusion 52 upgrade [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #3517:
URL: https://github.com/apache/datafusion-comet/issues/3517#issuecomment-3899632557

   If I run the benchmarks directly on my Linux desktop, they work fine.
   
   We could probably run in Docker with memory constraints to reproduce this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Possible regression in memory usage with DataFusion 52 upgrade [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #3517: Possible regression in memory usage with DataFusion 52 upgrade
URL: https://github.com/apache/datafusion-comet/issues/3517


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org