You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "shadowdsp (Jira)" <ji...@apache.org> on 2021/02/04 01:40:00 UTC
[jira] [Commented] (ARROW-6976) Possible memory leak in pyarrow
read_parquet
[ https://issues.apache.org/jira/browse/ARROW-6976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17278463#comment-17278463 ]
shadowdsp commented on ARROW-6976:
----------------------------------
[~jesse_ventura] I have the same problem, do you have any ideas?
> Possible memory leak in pyarrow read_parquet
> --------------------------------------------
>
> Key: ARROW-6976
> URL: https://issues.apache.org/jira/browse/ARROW-6976
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.15.0
> Environment: linux ubuntu 18.04
> Reporter: david cottrell
> Priority: Critical
> Attachments: image-2019-10-23-16-17-20-739.png, pyarrow-master.png, pyarrow_0150.png
>
>
>
> Version and repro info in the gist below.
> Not sure if I'm not understanding something from this [https://arrow.apache.org/blog/2019/02/05/python-string-memory-0.12/]
> but there seems to be memory accumulation when that is exacerbated with higher arity objects like strings and dates (not datetimes).
>
> I was not able to reproduce the issue on MacOS. Downgrading to 0.14.1 seemed to "fix" or lessen the problem.
>
> [https://gist.github.com/cottrell/a3f95aa59408d87f925ec606d8783e62]
>
> Let me know if this post should go elsewhere.
> !image-2019-10-23-16-17-20-739.png!
>
> {code:java}
>
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)