You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "tobim (via GitHub)" <gi...@apache.org> on 2025/11/14 17:28:34 UTC
[I] Add a way to programatically set the default memory pool [arrow]
tobim opened a new issue, #48135:
URL: https://github.com/apache/arrow/issues/48135
### Describe the enhancement requested
Arrow currently allows setting the default memory pool only via the `ARROW_DEFAULT_MEMORY_POOL` environment variable. It would be useful for applications to set the default memory pool programmatically, especially when they provide custom memory pools that can not be selected via the environment variable or know that a specific implementation is optimal for their use case.
Two possible implementation approaches:
1. Mark default_memory_pool() as a weak symbol
Annotating default_memory_pool() with [[gnu::weak]] would let client code override the default memory pool at link/load time.
Pros: avoids static initialization ordering issues and avoids internal thread-synchronization requirements.
2. Provide a setter API
Introduce a function such as SetDefaultMemoryPool(MemoryPool*) for applications to configure the default memory pool directly.
Pros: explicit and easy to understand; could allow changes during runtime if desired.
I'd like to Arrow to provide a way to override the default memory pool, and I'm willing to contribute an implementation, but I'd like to get some feedback on the direction from the maintainers first.
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Add a way to programatically set the default memory pool [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #48135:
URL: https://github.com/apache/arrow/issues/48135#issuecomment-3536206807
> especially when they provide custom memory pools that can not be selected via the environment variable or know that a specific implementation is optimal for their use case
Can they specify a specific memory pool instead of using the default memory pool?
Do you have any more concrete use case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Add a way to programatically set the default memory pool [arrow]
Posted by "tobim (via GitHub)" <gi...@apache.org>.
tobim commented on issue #48135:
URL: https://github.com/apache/arrow/issues/48135#issuecomment-3536252249
We want to make use of our own memory pool across our entire application, passing that to all functions and constructors that take one is error prone, because it is very easy to miss some.
For example, we had to implement our own clang-tidy rewrite operations to get it done.
https://github.com/tenzir/tenzir/commit/08bd1c462652a4df7ec60c68d630cf70a6da1d9d
https://github.com/tenzir/tenzir/commit/1aeed3496a29fd7056542b37dd3cf032a19e9be3
Setting the environment variable does not work with client provided implementations, and even if that mechanism would be extended to support that use case, it seems like the wrong approach.
Also, In case any third-party libraries call into arrow functionality that uses the allocator, we'd have to patch those as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Add a way to programatically set the default memory pool [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #48135:
URL: https://github.com/apache/arrow/issues/48135#issuecomment-3551018155
OK. Could you start a discussion about this on `dev@arrow.apache.org` https://arrow.apache.org/community/#mailing-lists to collect more opinions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Add a way to programatically set the default memory pool [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #48135:
URL: https://github.com/apache/arrow/issues/48135#issuecomment-3559041412
I would favor a setter API since it's cross-platform and it doesn't require usage of an obscure GNU feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org