[jira] [Updated] (ARROW-16037) Possible memory leak in compute.take

["Ziheng Wang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-16037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ziheng Wang updated ARROW-16037:
--------------------------------
    Priority: Blocker  (was: Major)

> Possible memory leak in compute.take
> ------------------------------------
>
>                 Key: ARROW-16037
>                 URL: https://issues.apache.org/jira/browse/ARROW-16037
>             Project: Apache Arrow
>          Issue Type: Bug
>    Affects Versions: 6.0.1
>         Environment: Ubuntu
>            Reporter: Ziheng Wang
>            Priority: Blocker
>
> If you run the following code, the memory usage of the process goes up to 1GB even though the pyarrow allocated bytes is always at ~80MB. The process memory comes down after a while to 800 MB, but is still way more than what is necessary.
> '''
> import pyarrow as pa
> import numpy as np
> import pandas as pd
> import os, psutil
> import pyarrow.compute as compute
> import gc
> my_table = pa.Table.from_pandas(pd.DataFrame(np.random.normal(size=(10000,1000))))
> process = psutil.Process(os.getpid())
> print("mem usage", process.memory_info().rss, pa.total_allocated_bytes())
> for i in range(100):
>     print("mem usage", process.memory_info().rss, pa.total_allocated_bytes())
>     temp = compute.sort_indices(my_table['0'], sort_keys = [('0','ascending')])
>     my_table = my_table.take(temp)
>     gc.collect()
> '''



--
This message was sent by Atlassian Jira
(v8.20.1#820001)