You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/04/14 20:46:21 UTC

[I] Investigate unstable benchmark results on macOS [datafusion-comet]

andygrove opened a new issue, #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648

   ### Describe the bug
   
   I am running TPC-H benchmarks on macOS using the instructions in https://github.com/apache/datafusion-comet/pull/1647.
   
   q13 sometimes takes 7 seconds and sometimes takes more than 5 minutes when Comet is enabled. I do not see any spilling. I am testing with Comet with the DataFusion 47 upgrade from https://github.com/apache/datafusion-comet/pull/1563
   
   ### Steps to reproduce
   
   _No response_
   
   ### Expected behavior
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2819735531

   That's a great find. Arrow Cpp discussion: https://lists.apache.org/thread/dts9ggvkthczfpmd25wrz449mxod76o2
   Also, I always thought we were using mimalloc (but never actually checked)!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]

Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2821858616

   I've also only ever used jemalloc in the past. I'm not sure what the discussion was at the time to pursue mimalloc for Comet. @mdcallag has some recent writing on the topic, althrough RocksDB is a different workload than what we're doing:
   
   https://smalldatum.blogspot.com/2025/04/battle-of-mallocators.html
   https://smalldatum.blogspot.com/2025/04/battle-of-mallocators-part-2.html
   
   The Germans :de: did a bake-off a few years ago and they chose jemalloc for Umbra (and likely CedarDB): https://www.adms-conf.org/2019-camera-ready/durner_adms19.pdf


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2806279850

   thermal management (CPU throttling) could also be a factor


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]

Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2821878876

   > Does LD_PRELOAD also change the allocator for the JVM?
   
   Memory allocated by `Unsafe_AllocateMemory0` (Arrow native memory, Spark off-heap memory) uses the allocator. The large JVM heap regions seems to be directly allocated using syscalls such as mmap, which is not affected by allocator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2803232601

   Why wouldn't we see the issue every time? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2802988404

   This comparison of two runs suggests that the issue is with a hash aggregate.
   
   <img width="1035" alt="Image" src="https://github.com/user-attachments/assets/f0d5cecb-6481-4f00-969b-798c26077124" />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2806812264

   I switched from 100 GB to 10 GB data set and the situation is even worse. q13 hangs and there are no stats at all in the plan.
   
   <img width="611" alt="Image" src="https://github.com/user-attachments/assets/7376ab65-725b-4cc3-adc7-8edff145db59" />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2838915277

   Related discord thread: https://discord.com/channels/885562378132000778/1363995762182193373


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]

Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2819405668

   I profiled it and we're getting crushed in OS mutexes in the allocator. I noticed there's some support for mimalloc in Comet already that's not really documented. DF enables it by default for its benchmarks. I am testing a build with `make release COMET_FEATURES=mimalloc` to see if the issue goes away.
   
   ![Image](https://github.com/user-attachments/assets/4efe0caa-8d93-4a20-822a-603477a6c0dd)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2821457688

   For my local Linux benchmark, I saw performance improve from ~275 s to ~266 s when using mimalloc


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]

Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2821819183

   We prefer to use jemalloc with Comet by setting LD_PRELOAD, as it returns freed memory to the operating system more aggressively than ptmalloc, resulting in a more predictable RSS for the Spark executor process. Additionally, jemalloc's memory profiler is helpful for diagnosing OOM issues.
   
   However, I'm uncertain whether jemalloc can still be used with LD_PRELOAD when Comet is compiled with mimalloc. The last time I attempted to dynamically override malloc it didn't work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]

Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2821860775

   > However, I'm uncertain whether jemalloc can still be used with LD_PRELOAD when Comet is compiled with mimalloc. The last time I attempted to dynamically override malloc it didn't work.
   
   Yeah I don't think you want to mix and match like that.
   
   Does LD_PRELOAD also change the allocator for the JVM?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2821590325

   FYI: We found mimalloc to be unstable for running long term (keeps memory allocated but seems doesn't release it over time => running into OOMs much more easily). Switching to jemalloc fixed this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]

Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2802991599

   I wonder if it's related to the Vec resize issues that @Kontinuation was seeing in https://github.com/apache/datafusion-comet/pull/1511#issuecomment-2745079065


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #1648: Investigate unstable benchmark results on macOS
URL: https://github.com/apache/datafusion-comet/issues/1648


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]

Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2819483674

   Anecdotally, my performance issues on macOS disappear when using mimalloc. We should have a larger discussion about where third-party allocators fit into the picture of optimizing Comet (off-heap, etc.).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]

Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2813944661

   The plan is to put a disclaimer


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate unstable benchmark results on macOS [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1648:
URL: https://github.com/apache/datafusion-comet/issues/1648#issuecomment-2806260921

   The issue seems at least partly due to a mix of performance and efficiency cores being used, which we do not have any control over.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org