You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/02/13 19:36:09 UTC
[I] Possible regression in memory usage with DataFusion 52 upgrade [datafusion-comet]
andygrove opened a new issue, #3517:
URL: https://github.com/apache/datafusion-comet/issues/3517
### Describe the bug
I frequently run benchmarks in k8s, and noticed that on the PR for upgrading to DF 52, the pod is being killed due to OOM:
```
comet-pr-3470-c3886859859-9c2jd 0/1 OOMKilled
```
I grep'd the log for errors and warnings and saw this output:
```
26/02/11 21:01:14 WARN DAGScheduler: Broadcasting large task binary with size 1077.4 KiB
26/02/11 21:01:21 WARN DAGScheduler: Broadcasting large task binary with size 1077.5 KiB
26/02/11 21:01:38 WARN DAGScheduler: Broadcasting large task binary with size 1204.7 KiB
```
The final few lines in the log were:
```
26/02/11 21:02:02 INFO CometExecIterator: memoryPoolType=fair_unified, offHeapSize=24576 MB, memoryFraction=1.0, memoryLimit=24576 MB, memoryLimitPerTask=3072 MB
26/02/11 21:02:02 INFO CometExecIterator: memoryPoolType=fair_unified, offHeapSize=24576 MB, memoryFraction=1.0, memoryLimit=24576 MB, memoryLimitPerTask=3072 MB
26/02/11 21:02:02 INFO CometExecIterator: memoryPoolType=fair_unified, offHeapSize=24576 MB, memoryFraction=1.0, memoryLimit=24576 MB, memoryLimitPerTask=3072 MB
26/02/11 21:02:02 INFO CometExecIterator: memoryPoolType=fair_unified, offHeapSize=24576 MB, memoryFraction=1.0, memoryLimit=24576 MB, memoryLimitPerTask=3072 MB
26/02/11 21:02:02 INFO ShuffleBlockFetcherIterator: Getting 48 (1843.0 KiB) non-empty blocks including 48 (1843.0 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
26/02/11 21:02:02 INFO ShuffleBlockFetcherIterator: Getting 48 (1846.8 KiB) non-empty blocks including 48 (1846.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
26/02/11 21:02:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
26/02/11 21:02:02 INFO ShuffleBlockFetcherIterator: Getting 48 (1850.7 KiB) non-empty blocks including 48 (1850.7 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
26/02/11 21:02:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
26/02/11 21:02:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
26/02/11 21:02:02 INFO ShuffleBlockFetcherIterator: Getting 48 (1846.8 KiB) non-empty blocks including 48 (1846.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
26/02/11 21:02:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
```
In this instance, it was running q21 when the OOM happened.
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Possible regression in memory usage with DataFusion 52 upgrade [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #3517:
URL: https://github.com/apache/datafusion-comet/issues/3517#issuecomment-3920971215
I am no longer seeing this issue, but there were changes to memory accounting in DF 52 that could change how we tune memory for jobs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Possible regression in memory usage with DataFusion 52 upgrade [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #3517:
URL: https://github.com/apache/datafusion-comet/issues/3517#issuecomment-3899632557
If I run the benchmarks directly on my Linux desktop, they work fine.
We could probably run in Docker with memory constraints to reproduce this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Possible regression in memory usage with DataFusion 52 upgrade [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #3517: Possible regression in memory usage with DataFusion 52 upgrade
URL: https://github.com/apache/datafusion-comet/issues/3517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org