You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@arrow.apache.org by "Daniel Haviv (JIRA)" <ji...@apache.org> on 2019/07/21 11:45:00 UTC

[jira] [Created] (ARROW-5993) Reading a dicitionary column results in a disproportinate memory usage

Daniel Haviv created ARROW-5993:
-----------------------------------

             Summary: Reading a dicitionary column results in a disproportinate memory usage
                 Key: ARROW-5993
                 URL: https://issues.apache.org/jira/browse/ARROW-5993
             Project: Apache Arrow
          Issue Type: Bug
          Components: Python
    Affects Versions: 0.14.0
            Reporter: Daniel Haviv


I'm using pyarrow to read a 40MB parquet file.

When reading all of the columns besides the "body" columns, the process peaks at 170MB.

Reading only the "body" column results in over 6GB of memory used.

I made the file publicly accessible: s3://dhavivresearch/pyarrow/demofile.parquet

 

 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)