[jira] [Created] (ARROW-13024) Decoding byte stream split encoded parquet columns fails when file has nulls

["Roman Karlstetter (Jira)" <ji...@apache.org>]

Roman Karlstetter created ARROW-13024:
-----------------------------------------

             Summary: Decoding byte stream split encoded parquet columns fails when file has nulls
                 Key: ARROW-13024
                 URL: https://issues.apache.org/jira/browse/ARROW-13024
             Project: Apache Arrow
          Issue Type: Bug
          Components: C++, Parquet
    Affects Versions: 4.0.0, 3.0.0, 2.0.0
            Reporter: Roman Karlstetter


Reading from a parquet file fails with the following error

{{Data size too small for number of values (corrupted file?)}}.

This happens for the case when there is a {{BYTE_STREAM_SPLIT}}-encoded column which has less values stored than number of rows, which is the case when the column has null values (definition levels are present).

The problematic part is the condition checked in {{ByteStreamSplitDecoder<DType>::SetData}}, which raises the error if the number of values does not match the size of the data array.

I'm unsure whether I have enough experience with the internals of the encoding/decoding part of this implementation to fix this issue, but my suggestion would be to initialize {{num_values_in_buffer_}} with {{len/static_cast<int64_t>(sizeof(T))}}.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)