[jira] [Created] (ARROW-8127) [C++} [Parquet] Incorrect column chunk metadata for multipage batch writes

["TP Boudreau (Jira)" <ji...@apache.org>]

TP Boudreau created ARROW-8127:
----------------------------------

             Summary: [C++} [Parquet] Incorrect column chunk metadata for multipage batch writes
                 Key: ARROW-8127
                 URL: https://issues.apache.org/jira/browse/ARROW-8127
             Project: Apache Arrow
          Issue Type: Bug
          Components: C++
            Reporter: TP Boudreau
            Assignee: TP Boudreau
         Attachments: multipage-batch-write.cc

When writing to a buffered column writer using PLAIN encoding, if the size of the batch supplied for writing exceeds the page size for the writer, the resulting file has an incorrect data_page_offset set in its column chunk metadata.  This causes an exception to be thrown when reading the file (file appears to be too short to the reader).

For example, the attached code, which attempts to write a batch of 262145 Int32's (= 1048576 + 4 bytes) using the default page size of 1048576 bytes (with buffered writer, PLAIN encoding), fails on reading, throwing the error: "Tried reading 1048678 bytes starting at position 1048633 from file but only got 333".

The error is caused by the second page write tripping the conditional here https://github.com/apache/arrow/blob/master/cpp/src/parquet/column_writer.cc#L302, in the serialized in-memory writer wrapped by the buffered writer.

The fix builds the metadata with offsets from the terminal sink rather than the in memory buffered sink.  A PR is coming.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)