[jira] [Commented] (ARROW-13941) [C++] Parquet writes incorrect file_offset

["Micah Kornfield (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-13941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17414693#comment-17414693 ] 

Micah Kornfield commented on ARROW-13941:
-----------------------------------------

Moving to parquet metadata

> [C++] Parquet writes incorrect file_offset 
> -------------------------------------------
>
>                 Key: ARROW-13941
>                 URL: https://issues.apache.org/jira/browse/ARROW-13941
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Parquet
>            Reporter: Chao Sun
>            Priority: Critical
>
> Currently the Parquet writer sets {{file_offset}} in the following way (from {{metadata.cc}})
> {code:cpp}
>     if (dictionary_page_offset > 0) {
>       column_chunk_->meta_data.__set_dictionary_page_offset(dictionary_page_offset);
>       column_chunk_->__set_file_offset(dictionary_page_offset + compressed_size);
>     } else {
>       column_chunk_->__set_file_offset(data_page_offset + compressed_size);
>     }{code}
> This doesn't look correct, as it shouldn't take {{compressed_size}} into consideration.
> The {{file_offset}} is used when filtering row groups, and the above could cause correctness issue. See SPARK-36696.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)