You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Valery Meleshkin (JIRA)" <ji...@apache.org> on 2019/02/21 23:42:01 UTC
[jira] [Comment Edited] (PARQUET-1537) [C++] The patch for
PARQUET-1508 leads to infinite loop and infinite memory allocation when
reading very sparse ByteArray columns
[ https://issues.apache.org/jira/browse/PARQUET-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16774598#comment-16774598 ]
Valery Meleshkin edited comment on PARQUET-1537 at 2/21/19 11:41 PM:
---------------------------------------------------------------------
I can't attach the original file, but I was able to reproduce it by writing something like the following with pyarrow:
{{df3 = pd.DataFrame(}}{{{'id': [x for x in range(1000000)], 'str': [''] + [None for _ in range(1000000 - 1)]}}}{{)}}
{{ table3 = pa.Table.from_pandas(df3)}}{{with pq.ParquetWriter('example.parquet', table3.schema, use_dictionary=False) as writer:}}
{{ writer.write_table(table3)}}
was (Author: sumerman):
I can't attach the original file, but I was able to reproduce it by writing something like the following with pyarrow:
{{df3 = pd.DataFrame({'id': [x for x in range(1000000)],
'str': [''] + [None for _ in range(1000000 - 1)]})
table3 = pa.Table.from_pandas(df3)
with pq.ParquetWriter('example.parquet', table3.schema, use_dictionary=False) as writer:
writer.write_table(table3)
}}
> [C++] The patch for PARQUET-1508 leads to infinite loop and infinite memory allocation when reading very sparse ByteArray columns
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: PARQUET-1537
> URL: https://issues.apache.org/jira/browse/PARQUET-1537
> Project: Parquet
> Issue Type: Bug
> Components: parquet-cpp
> Reporter: Valery Meleshkin
> Assignee: Hatem Helal
> Priority: Major
> Fix For: cpp-1.6.0
>
>
> In this loop
> [https://github.com/apache/arrow/commit/3d435e4f8d5fb7a54a4a9d285e1a42d60186d8dc#diff-47fe879cb9baad6c633c55f0a34a09c3R739]
> The branch of if dealing with null values does not increment variable 'i'. Therefore on chunks containing only NULLs once a thread enters the loop, it stays in that loop forever. I'm not entirely sure whether 'num_values' variable was meant to be the number of non-NULL values, yet the total number of values is passed here [https://github.com/apache/arrow/blob/3d435e4f8d5fb7a54a4a9d285e1a42d60186d8dc/cpp/src/parquet/arrow/record_reader.cc#L528]
>
> On my local machine adding `++i` to the NULL-handling branch seems to fix the problem. Unfortunately, I'm not familiar with the codebase enough to be certain it's a proper fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)