You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Vyas Ramasubramani <vy...@nvidia.com.INVALID> on 2025/08/19 19:22:55 UTC

[DISCUSS][C++] Removing jemalloc support

A few days ago I opened https://github.com/apache/arrow/issues/47309<https://github.com/apache/arrow/issues/47309#issuecomment-3196114599> where it was suggested that I bring this question to the mailing list:
Given that jemalloc development has stopped<https://jasone.github.io/2025/06/12/jemalloc-postmortem/> and the repo is archived<https://github.com/jemalloc/jemalloc> I wonder if Arrow should consider removing that support as well. Arrow has already moved to mimalloc as the default allocator, and there are a number of open issues that link back to jemalloc issues (I was brought here from a comment on #44342<https://github.com/apache/arrow/issues/44342>). It might help reduce some noise on this repo's issue tracker to remove jemalloc support altogether. I don't know if there is a desire to support any particular system where jemalloc is still preferred, though. If so, then delaying jemalloc support removal could make sense too. Just thought I'd raise the issue.
If developers would prefer to keep jemalloc support in Arrow until someone finds an intractable issue to resolve, that would be fine. I have seen enough other jemalloc-related issues on the issue board to suggest that there are sufficient edge cases to consider remove jemalloc as an option regardless, though. Given that jemalloc is no longer the default in Arrow, I would not be surprised to see bit rot in jemalloc support going unnoticed for multiple releases, and it could be worth nipping that in the bud.

Re: [DISCUSS][C++] Removing jemalloc support

Posted by Antoine Pitrou <an...@python.org>.
Hello Vyas,

I will state again what I said on the issue tracker:

As for which allocator is better, there is no simple, clear-cut answer, 
because it depends on the workload, the platform and the metrics you 
care about (is the user more sensitive to memory footprint or execution 
time?).

Sometimes people are unhappy about the behavior of the default allocator 
on their workload and it turns out they can work around their issues 
simply by changing the allocator, which is rather nice.

I will add that while mimalloc is maintained, it currently has a single 
maintainer and he is not always active (you can easily see that from the 
GitHub repository's statistics). We also have at least one recent report 
of memory consumption issues with mimalloc:
https://github.com/apache/arrow/issues/47266

So for now I'm -0.5 on removing jemalloc.

Regards

Antoine.


Le 19/08/2025 à 21:22, Vyas Ramasubramani a écrit :
> A few days ago I opened https://github.com/apache/arrow/issues/47309<https://github.com/apache/arrow/issues/47309#issuecomment-3196114599> where it was suggested that I bring this question to the mailing list:
> Given that jemalloc development has stopped<https://jasone.github.io/2025/06/12/jemalloc-postmortem/> and the repo is archived<https://github.com/jemalloc/jemalloc> I wonder if Arrow should consider removing that support as well. Arrow has already moved to mimalloc as the default allocator, and there are a number of open issues that link back to jemalloc issues (I was brought here from a comment on #44342<https://github.com/apache/arrow/issues/44342>). It might help reduce some noise on this repo's issue tracker to remove jemalloc support altogether. I don't know if there is a desire to support any particular system where jemalloc is still preferred, though. If so, then delaying jemalloc support removal could make sense too. Just thought I'd raise the issue.
> If developers would prefer to keep jemalloc support in Arrow until someone finds an intractable issue to resolve, that would be fine. I have seen enough other jemalloc-related issues on the issue board to suggest that there are sufficient edge cases to consider remove jemalloc as an option regardless, though. Given that jemalloc is no longer the default in Arrow, I would not be surprised to see bit rot in jemalloc support going unnoticed for multiple releases, and it could be worth nipping that in the bud.
> 


Re: [DISCUSS][C++] Removing jemalloc support

Posted by Sutou Kouhei <ko...@clear-code.com>.
Hi,

I agree with this proposal but how about creating a
deprecation period? For example: We will deprecate jemalloc
support in 22.0.0 release and announce it in the release
announce. We'll drop support for jemalloc in 24.0.0 release.

Thanks,
-- 
kou

In 
 <IA...@IA0PR12MB7649.namprd12.prod.outlook.com>
  "[DISCUSS][C++] Removing jemalloc support" on Tue, 19 Aug 2025 19:22:55 +0000,
  Vyas Ramasubramani <vy...@nvidia.com.INVALID> wrote:

> A few days ago I opened https://github.com/apache/arrow/issues/47309<https://github.com/apache/arrow/issues/47309#issuecomment-3196114599> where it was suggested that I bring this question to the mailing list:
> Given that jemalloc development has stopped<https://jasone.github.io/2025/06/12/jemalloc-postmortem/> and the repo is archived<https://github.com/jemalloc/jemalloc> I wonder if Arrow should consider removing that support as well. Arrow has already moved to mimalloc as the default allocator, and there are a number of open issues that link back to jemalloc issues (I was brought here from a comment on #44342<https://github.com/apache/arrow/issues/44342>). It might help reduce some noise on this repo's issue tracker to remove jemalloc support altogether. I don't know if there is a desire to support any particular system where jemalloc is still preferred, though. If so, then delaying jemalloc support removal could make sense too. Just thought I'd raise the issue.
> If developers would prefer to keep jemalloc support in Arrow until someone finds an intractable issue to resolve, that would be fine. I have seen enough other jemalloc-related issues on the issue board to suggest that there are sufficient edge cases to consider remove jemalloc as an option regardless, though. Given that jemalloc is no longer the default in Arrow, I would not be surprised to see bit rot in jemalloc support going unnoticed for multiple releases, and it could be worth nipping that in the bud.

Re: [DISCUSS][C++] Removing jemalloc support

Posted by kekronbekron <ke...@protonmail.com.INVALID>.
Sorry, I haven't checked if anyone has already replied with this.

Facebook has forked it, and I believe they're going to keep the lights on, at least.

https://github.com/facebook/jemalloc

- KB


On Wednesday, August 20th, 2025 at 07:38, Vyas Ramasubramani <vy...@nvidia.com.INVALID> wrote:

> A few days ago I opened https://github.com/apache/arrow/issues/47309https://github.com/apache/arrow/issues/47309#issuecomment-3196114599 where it was suggested that I bring this question to the mailing list:
> 
> Given that jemalloc development has stoppedhttps://jasone.github.io/2025/06/12/jemalloc-postmortem/ and the repo is archivedhttps://github.com/jemalloc/jemalloc I wonder if Arrow should consider removing that support as well. Arrow has already moved to mimalloc as the default allocator, and there are a number of open issues that link back to jemalloc issues (I was brought here from a comment on #44342https://github.com/apache/arrow/issues/44342). It might help reduce some noise on this repo's issue tracker to remove jemalloc support altogether. I don't know if there is a desire to support any particular system where jemalloc is still preferred, though. If so, then delaying jemalloc support removal could make sense too. Just thought I'd raise the issue.
> 
> If developers would prefer to keep jemalloc support in Arrow until someone finds an intractable issue to resolve, that would be fine. I have seen enough other jemalloc-related issues on the issue board to suggest that there are sufficient edge cases to consider remove jemalloc as an option regardless, though. Given that jemalloc is no longer the default in Arrow, I would not be surprised to see bit rot in jemalloc support going unnoticed for multiple releases, and it could be worth nipping that in the bud.