You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@arrow.apache.org by "Valery Meleshkin (JIRA)" <ji...@apache.org> on 2019/02/21 15:57:00 UTC

[jira] [Created] (ARROW-4650) The patch for PARQUET-1508 leads to infinite loop and infinite memory allocation when reading very sparse ByteArray columns

Valery Meleshkin created ARROW-4650:
---------------------------------------

             Summary: The patch for PARQUET-1508 leads to infinite loop and infinite memory allocation when reading very sparse ByteArray columns
                 Key: ARROW-4650
                 URL: https://issues.apache.org/jira/browse/ARROW-4650
             Project: Apache Arrow
          Issue Type: Bug
          Components: C++
            Reporter: Valery Meleshkin


In this loop

[https://github.com/apache/arrow/commit/3d435e4f8d5fb7a54a4a9d285e1a42d60186d8dc#diff-47fe879cb9baad6c633c55f0a34a09c3R739]

The branch of if dealing with null values does not increment variable 'i'. Therefore on chunks containing only NULLs once a thread enters the loop, it stays in that loop forever. I'm not entirely sure whether 'num_values' variable was meant to be the number of non-NULL values, yet the total number of values is passed here [https://github.com/apache/arrow/blob/3d435e4f8d5fb7a54a4a9d285e1a42d60186d8dc/cpp/src/parquet/arrow/record_reader.cc#L528]

 

On my local machine adding `++i` to the NULL-handling branch seems to fix the problem. Unfortunately, I'm not familiar with the codebase enough to be certain it's a proper fix.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)