[jira] [Created] (ARROW-16037) Possible memory leak in compute.take

["Ziheng Wang (Jira)" <ji...@apache.org>]

Ziheng Wang created ARROW-16037:
-----------------------------------

             Summary: Possible memory leak in compute.take
                 Key: ARROW-16037
                 URL: https://issues.apache.org/jira/browse/ARROW-16037
             Project: Apache Arrow
          Issue Type: Bug
    Affects Versions: 6.0.1
         Environment: Ubuntu
            Reporter: Ziheng Wang


If you run the following code, the memory usage of the process goes up to 1GB even though the pyarrow allocated bytes is always at ~80MB. The process memory comes down after a while to 800 MB, but is still way more than what is necessary.

'''

import pyarrow as pa
import numpy as np
import pandas as pd
import os, psutil
import pyarrow.compute as compute
import gc
my_table = pa.Table.from_pandas(pd.DataFrame(np.random.normal(size=(10000,1000))))

process = psutil.Process(os.getpid())
print("mem usage", process.memory_info().rss, pa.total_allocated_bytes())

for i in range(100):
    print("mem usage", process.memory_info().rss, pa.total_allocated_bytes())
    temp = compute.sort_indices(my_table['0'], sort_keys = [('0','ascending')])
    my_table = my_table.take(temp)
    gc.collect()

'''



--
This message was sent by Atlassian Jira
(v8.20.1#820001)