[jira] [Created] (ARROW-11228) [C++] Allow find tuning of memory pool from environment variable(s)

["Weston Pace (Jira)" <ji...@apache.org>]

Weston Pace created ARROW-11228:
-----------------------------------

             Summary: [C++] Allow find tuning of memory pool from environment variable(s)
                 Key: ARROW-11228
                 URL: https://issues.apache.org/jira/browse/ARROW-11228
             Project: Apache Arrow
          Issue Type: Improvement
            Reporter: Weston Pace


Both jemalloc and mimalloc allow for some amount of fine tuning from environment variables.  However, in some cases, the settings entered into these environment variables are overridden by arrow.

For example, jemalloc docs describe MALLOC_CONF, an environment variable that can be used to tune any jemalloc parameter.  But, the global variable `malloc_conf` (which arrow uses) takes precedence.

Mimalloc also exposes some tuning parameters ([https://microsoft.github.io/mimalloc/environment.html)].  I have not tested which of these work and which do not work so it may be that all of them are accessible already.

Tuning may be useful in certain environments (for example, if someone wanted to use jemalloc over mimalloc in an environment where overcommit is disabled and so they wanted to set the retain:true option) and so it would be good to allow for these tuning parameters to be set at runtime.  One potential solution may be to expose our own environment variable (e.g. ARROW_MALLOC_CONF).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)