You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/04/14 20:46:21 UTC
[I] Investigate unstable benchmark results on macOS [datafusion-comet]
andygrove opened a new issue, #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648
### Describe the bug
I am running TPC-H benchmarks on macOS using the instructions in https://github.com/apache/datafusion-comet/pull/1647.
q13 sometimes takes 7 seconds and sometimes takes more than 5 minutes when Comet is enabled. I do not see any spilling. I am testing with Comet with the DataFusion 47 upgrade from https://github.com/apache/datafusion-comet/pull/1563
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2819735531
That's a great find. Arrow Cpp discussion: https://lists.apache.org/thread/dts9ggvkthczfpmd25wrz449mxod76o2
Also, I always thought we were using mimalloc (but never actually checked)!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2821858616
I've also only ever used jemalloc in the past. I'm not sure what the discussion was at the time to pursue mimalloc for Comet. @mdcallag has some recent writing on the topic, althrough RocksDB is a different workload than what we're doing:
https://smalldatum.blogspot.com/2025/04/battle-of-mallocators.html
https://smalldatum.blogspot.com/2025/04/battle-of-mallocators-part-2.html
The Germans :de: did a bake-off a few years ago and they chose jemalloc for Umbra (and likely CedarDB): https://www.adms-conf.org/2019-camera-ready/durner_adms19.pdf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2806279850
thermal management (CPU throttling) could also be a factor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2821878876
> Does LD_PRELOAD also change the allocator for the JVM?
Memory allocated by `Unsafe_AllocateMemory0` (Arrow native memory, Spark off-heap memory) uses the allocator. The large JVM heap regions seems to be directly allocated using syscalls such as mmap, which is not affected by allocator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2803232601
Why wouldn't we see the issue every time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2802988404
This comparison of two runs suggests that the issue is with a hash aggregate.
<img width="1035" alt="Image" src="https://github.com/user-attachments/assets/f0d5cecb-6481-4f00-969b-798c26077124" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2806812264
I switched from 100 GB to 10 GB data set and the situation is even worse. q13 hangs and there are no stats at all in the plan.
<img width="611" alt="Image" src="https://github.com/user-attachments/assets/7376ab65-725b-4cc3-adc7-8edff145db59" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2838915277
Related discord thread: https://discord.com/channels/885562378132000778/1363995762182193373
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2819405668
I profiled it and we're getting crushed in OS mutexes in the allocator. I noticed there's some support for mimalloc in Comet already that's not really documented. DF enables it by default for its benchmarks. I am testing a build with `make release COMET_FEATURES=mimalloc` to see if the issue goes away.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2821457688
For my local Linux benchmark, I saw performance improve from ~275 s to ~266 s when using mimalloc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2821819183
We prefer to use jemalloc with Comet by setting LD_PRELOAD, as it returns freed memory to the operating system more aggressively than ptmalloc, resulting in a more predictable RSS for the Spark executor process. Additionally, jemalloc's memory profiler is helpful for diagnosing OOM issues.
However, I'm uncertain whether jemalloc can still be used with LD_PRELOAD when Comet is compiled with mimalloc. The last time I attempted to dynamically override malloc it didn't work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2821860775
> However, I'm uncertain whether jemalloc can still be used with LD_PRELOAD when Comet is compiled with mimalloc. The last time I attempted to dynamically override malloc it didn't work.
Yeah I don't think you want to mix and match like that.
Does LD_PRELOAD also change the allocator for the JVM?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2821590325
FYI: We found mimalloc to be unstable for running long term (keeps memory allocated but seems doesn't release it over time => running into OOMs much more easily). Switching to jemalloc fixed this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2802991599
I wonder if it's related to the Vec resize issues that @Kontinuation was seeing in https://github.com/apache/datafusion-comet/pull/1511#issuecomment-2745079065
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #1648: Investigate unstable benchmark results on macOS
URL: https://github.com/apache/datafusion-comet/issues/1648
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2819483674
Anecdotally, my performance issues on macOS disappear when using mimalloc. We should have a larger discussion about where third-party allocators fit into the picture of optimizing Comet (off-heap, etc.).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2813944661
The plan is to put a disclaimer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2806260921
The issue seems at least partly due to a mix of performance and efficiency cores being used, which we do not have any control over.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org