[jira] [Comment Edited] (ARROW-17590) Lower memory usage with filters

["Yin (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-17590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17599104#comment-17599104 ] 

Yin edited comment on ARROW-17590 at 9/1/22 7:07 PM:
-----------------------------------------------------

Hi Weston,  Just saw your comment. Will try it in the sample code. Thanks 

Update:

Printed out pyarrow.total_allocated_bytes and table.nbytes.
Below is in the updated sample code.

In the case A: reading all columns with the filter, 
total_allocated_bytes is 289 MB and dt.nbytes is very small.

case B reads one column with the filter.
case C reads all columns without filter.

# total_allocated_bytes  and table.nbytes
# pyarrow 7.0.0, pandas 1.4.4 numpy 1.23.2
# A: 289 MB 0.00115 MB B: 3.5 MB 9.53-e06 MB C: 289.72 MB 288.38 MB
# pyarrow 9.0.0, pandas 1.4.4 numpy 1.23.2
# A: 289 MB 0.0014 MB B: 3.5 MB 1.049-e05 MB C: 289.72 MB 288.38 MB

# rss memory after read_table
# pyarrow 7.0.0, pandas 1.4.4 numpy 1.23.2
# A: 1008 MB B: 88 MB C: 1008 MB
# pyarrow 9.0.0 pandas 1.4.4 numpy 1.23.2
# A: 394 MB B: 85 MB C: 393 MB


was (Author: JIRAUSER285415):
Hi Weston,  Just saw your comment. Will try it in the sample code. Thanks 

> Lower memory usage with filters
> -------------------------------
>
>                 Key: ARROW-17590
>                 URL: https://issues.apache.org/jira/browse/ARROW-17590
>             Project: Apache Arrow
>          Issue Type: Improvement
>            Reporter: Yin
>            Priority: Major
>         Attachments: sample-1.py, sample.py
>
>
> Hi,
> When I read a parquet file (about 23MB with 250K rows and 600 object/string columns with lots of None) with filter on a not null column for a small number of rows (e.g. 1 to 500), the memory usage is pretty high (around 900MB to 1GB). The result table and dataframe have only a few rows (1 row 20kb, 500 rows 20MB). Looks like it scans/loads many rows from the parquet file. Not only the footprint or watermark of memory usage is high, but also it seems not releasing the memory in time (such as after GC in Python, but may get used for subsequent read).
> When reading the same parquet file for all columns without filtering, the memory usage is about the same at 900MB. It goes up to 2.3GB after to_pandas dataframe,. df.info(memory_usage='deep') shows 4.3GB maybe double counting something.
> It helps to limit the number of columns read. Read 1 column with filter for 1 row or more or without filter, it takes about 10MB, which is quite smaller and better, but still bigger than the size of table or data frame with 1 or 500 rows of 1 columns (under 1MB)
> The filtered column is not a partition key, which functionally works to get the correct rows. But the memory usage is quite high even when the parquet file is not really large, partitioned or not. There were some references similar to this issue, for example: [https://github.com/apache/arrow/issues/7338]
> Related classes/methods in (pyarrow 9.0.0) 
> _ParquetDatasetV2.read
>     self._dataset.to_table(columns=columns, filter=self._filter_expression, use_threads=use_threads)
> pyarrow._dataset.FileSystemDatase.to_table
> I played with pyarrow._dataset.Scanner.to_table
>     self._dataset.scanner(columns=columns, filter=self._filter_expression).to_table()
> The memory usage is small to construct the scanner but then goes up after the to_table call materializes it.
> Is there some way or workaround to reduce the memory usage with read filtering? 
> If not supported yet, can it be fixed/improved with priority? 
> This is a blocking issue for us when we need to load all or many columns. 
> I am not sure what improvement is possible with respect to how the parquet columnar format works, and if it can be patched somehow in the Pyarrow Python code, or need to change and build the arrow C++ code.
> Thanks!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)