You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Neal Richardson (JIRA)" <ji...@apache.org> on 2019/07/22 16:29:00 UTC
[jira] [Updated] (ARROW-5993) [Python] Reading a dicitionary column
from Parquet results in disproportionate memory usage
[ https://issues.apache.org/jira/browse/ARROW-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neal Richardson updated ARROW-5993:
-----------------------------------
Summary: [Python] Reading a dicitionary column from Parquet results in disproportionate memory usage (was: Reading a dicitionary column results in disproportionate memory usage)
> [Python] Reading a dicitionary column from Parquet results in disproportionate memory usage
> -------------------------------------------------------------------------------------------
>
> Key: ARROW-5993
> URL: https://issues.apache.org/jira/browse/ARROW-5993
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.14.0
> Reporter: Daniel Haviv
> Priority: Major
> Labels: memory
>
> I'm using pyarrow to read a 40MB parquet file.
> When reading all of the columns besides the "body" columns, the process peaks at 170MB.
> Reading only the "body" column results in over 6GB of memory used.
> I made the file publicly accessible: s3://dhavivresearch/pyarrow/demofile.parquet
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)