You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Chao Sun (Jira)" <ji...@apache.org> on 2021/09/08 22:47:00 UTC
[jira] [Updated] (ARROW-13941) [C++] Parquet writes incorrect
file_offset
[ https://issues.apache.org/jira/browse/ARROW-13941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chao Sun updated ARROW-13941:
-----------------------------
Description:
Currently the Parquet writer sets {{file_offset}} in the following way:
{code:cpp}
if (dictionary_page_offset > 0) {
column_chunk_->meta_data.__set_dictionary_page_offset(dictionary_page_offset);
column_chunk_->__set_file_offset(dictionary_page_offset + compressed_size);
} else {
column_chunk_->__set_file_offset(data_page_offset + compressed_size);
}{code}
This doesn't look correct, as it shouldn't take {{compressed_size}} into consideration.
The {{file_offset}} is used when filtering row groups, and the above could cause correctness issue. See SPARK-36696.
was:
Currently the Parquet writer set {{file_offset}} in the following way:
{code:cpp}
if (dictionary_page_offset > 0) {
column_chunk_->meta_data.__set_dictionary_page_offset(dictionary_page_offset);
column_chunk_->__set_file_offset(dictionary_page_offset + compressed_size);
} else {
column_chunk_->__set_file_offset(data_page_offset + compressed_size);
}{code}
This doesn't look correct, as it should not take {{compressed_size}} into consideration.
The {{file_offset}} is used when filtering row groups, and the above could cause correctness issue. See SPARK-36696.
> [C++] Parquet writes incorrect file_offset
> -------------------------------------------
>
> Key: ARROW-13941
> URL: https://issues.apache.org/jira/browse/ARROW-13941
> Project: Apache Arrow
> Issue Type: Bug
> Components: Parquet
> Reporter: Chao Sun
> Priority: Critical
>
> Currently the Parquet writer sets {{file_offset}} in the following way:
> {code:cpp}
> if (dictionary_page_offset > 0) {
> column_chunk_->meta_data.__set_dictionary_page_offset(dictionary_page_offset);
> column_chunk_->__set_file_offset(dictionary_page_offset + compressed_size);
> } else {
> column_chunk_->__set_file_offset(data_page_offset + compressed_size);
> }{code}
> This doesn't look correct, as it shouldn't take {{compressed_size}} into consideration.
> The {{file_offset}} is used when filtering row groups, and the above could cause correctness issue. See SPARK-36696.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)