You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "blaginin (via GitHub)" <gi...@apache.org> on 2026/03/03 21:37:59 UTC
[PR] `MiMalloc` for `sqllogictests` [datafusion]
blaginin opened a new pull request, #20682:
URL: https://github.com/apache/datafusion/pull/20682
related to https://github.com/apache/datafusion/issues/20052
https://github.com/apache/datafusion/actions/runs/22638279915/job/65619265884
-10 minutes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] `MiMalloc` for `sqllogictests` [datafusion]
Posted by "blaginin (via GitHub)" <gi...@apache.org>.
blaginin commented on PR #20682:
URL: https://github.com/apache/datafusion/pull/20682#issuecomment-3999266197
hey @comphead! In this pr i change just sqlogictests, and they don't assert mem usage - just correctness. They're pretty high level so I'd be surprised if the allocator interferes in any way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] `MiMalloc` for `sqllogictests` [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on PR #20682:
URL: https://github.com/apache/datafusion/pull/20682#issuecomment-4012851126
> Thanks @blaginin however I'm not sure about `mimalloc` for tests tbh, we do `mimalloc` for benchmarks, but tests asserts correctness and mem usage for default user environment which is glibc malloc or musl malloc if we speaking for Linux and providing results from another mem allocator as a source of truth, WDYT?
Do people actually use the default allocator for their df based apps? I've found in my app that MiMalloc gave the best overall behaviour and was significantly better than glibc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] `MiMalloc` for `sqllogictests` [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #20682:
URL: https://github.com/apache/datafusion/pull/20682#issuecomment-4010971345
> Thanks @blaginin however I'm not sure about `mimalloc` for tests tbh, we do `mimalloc` for benchmarks, but tests asserts correctness and mem usage for default user environment which is glibc malloc or musl malloc if we speaking for Linux and providing results from another mem allocator as a source of truth, WDYT?
I am not sure tests should depend on the allocator? If they would it might be possible to change the allocator just for those tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] `MiMalloc` for `sqllogictests` [datafusion]
Posted by "AdamGS (via GitHub)" <gi...@apache.org>.
AdamGS commented on PR #20682:
URL: https://github.com/apache/datafusion/pull/20682#issuecomment-4013505506
what's production for DataFusion? Rust only allows you to set the global allocator once, setting it in a crate is a faux pas, and specifically here will block the use of the dhat allocator elsewhere
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] `MiMalloc` for `sqllogictests` [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #20682:
URL: https://github.com/apache/datafusion/pull/20682#issuecomment-4007856184
> hey @comphead! In this pr i change just sqlogictests, and they don't assert mem usage - just correctness. They're pretty high level so I'd be surprised if the allocator interferes in any way
IMO I would still expect tests to be in the same environment, otherwise it would be a debugging nightmare in case of false positives or false negatives 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] `MiMalloc` for `sqllogictests` [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20682:
URL: https://github.com/apache/datafusion/pull/20682#issuecomment-4514397170
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] `MiMalloc` for `sqllogictests` [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #20682:
URL: https://github.com/apache/datafusion/pull/20682#issuecomment-4013453890
Thanks @Omega359 and @Dandandan — the motivation mostly comes from DuckDB, which uses jemalloc by default.
In practice this means:
- tests run with jemalloc
- the shipped DuckDB distribution also uses jemalloc
Docs:
https://duckdb.org/docs/stable/guides/performance/environment#memory-allocator
https://duckdb.org/docs/stable/core_extensions/jemalloc
The advantage is that tests and the shipped binary run with the same allocator. If we think mimalloc works better (we also use it in Comet), it might make sense to make it the default allocator everywhere so tests and production match, also update documentation.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] `MiMalloc` for `sqllogictests` [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #20682:
URL: https://github.com/apache/datafusion/pull/20682#issuecomment-4105720036
I think for tests it should be OK to switch to a different allocator.
They shouldn't fail when a different allocator would be used (we also expect now tests do not fail on user machines which use different system allocators).
You could always write tests that switch to a different allocator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] `MiMalloc` for `sqllogictests` [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #20682: `MiMalloc` for `sqllogictests`
URL: https://github.com/apache/datafusion/pull/20682
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org