[jira] [Created] (ARROW-18164) [C++][Python] Dataset scanner does not follow default memory pool setting

["Joris Van den Bossche (Jira)" <ji...@apache.org>]

Joris Van den Bossche created ARROW-18164:
---------------------------------------------

             Summary: [C++][Python] Dataset scanner does not follow default memory pool setting
                 Key: ARROW-18164
                 URL: https://issues.apache.org/jira/browse/ARROW-18164
             Project: Apache Arrow
          Issue Type: Bug
          Components: C++, Python
            Reporter: Joris Van den Bossche


Even if I set the system memory pool as default, it still uses the jemalloc one (running this on Ubuntu where jemalloc is the default if not set by the user):

{code}
import pyarrow as pa
import pyarrow.dataset as ds
import pyarrow.parquet as pq
pq.write_table(pa.table({'a': [1, 2, 3]}), "test.parquet")

In [2]: pa.set_memory_pool(pa.system_memory_pool())

In [3]: pa.total_allocated_bytes()
Out[3]: 0

In [4]: table = ds.dataset("test.parquet").to_table()

In [5]: pa.total_allocated_bytes()
Out[5]: 0

In [6]: pa.set_memory_pool(pa.jemalloc_memory_pool())

In [7]: pa.total_allocated_bytes()
Out[7]: 128
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)