You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@arrow.apache.org by "Wes McKinney (Jira)" <ji...@apache.org> on 2019/08/19 22:35:00 UTC

[jira] [Updated] (ARROW-5993) [Python] Reading a dictionary column from Parquet results in disproportionate memory usage

     [ https://issues.apache.org/jira/browse/ARROW-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wes McKinney updated ARROW-5993:
--------------------------------
    Fix Version/s:     (was: 1.0.0)
                   0.15.0

> [Python] Reading a dictionary column from Parquet results in disproportionate memory usage
> ------------------------------------------------------------------------------------------
>
>                 Key: ARROW-5993
>                 URL: https://issues.apache.org/jira/browse/ARROW-5993
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>    Affects Versions: 0.14.0
>            Reporter: Daniel Haviv
>            Assignee: Wes McKinney
>            Priority: Major
>              Labels: memory, parquet
>             Fix For: 0.15.0
>
>
> I'm using pyarrow to read a 40MB parquet file.
> When reading all of the columns besides the "body" columns, the process peaks at 170MB.
> Reading only the "body" column results in over 6GB of memory used.
> I made the file publicly accessible: s3://dhavivresearch/pyarrow/demofile.parquet
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)