You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Shan Huang (Jira)" <ji...@apache.org> on 2021/09/23 10:17:00 UTC
[jira] [Commented] (ARROW-14089) [C++] Read DELTA_BYTE_ARRAY data
written with bug PARQUET-246
[ https://issues.apache.org/jira/browse/ARROW-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17419109#comment-17419109 ]
Shan Huang commented on ARROW-14089:
------------------------------------
I'm not sure whether to choose solution #1 because the bug PARQUET-246 has been fixed in 2015. We cannot produce file with this bug with hive after version 2.0.0
Besides, in solution #1, I think maybe we should pass a parameter(maybe the file version or a flag) down to DeltaByteArrayDecoder. Code would be modified in RowGroupReader, ColumnReader and DeltaByteArrayDecoder.
> [C++] Read DELTA_BYTE_ARRAY data written with bug PARQUET-246
> -------------------------------------------------------------
>
> Key: ARROW-14089
> URL: https://issues.apache.org/jira/browse/ARROW-14089
> Project: Apache Arrow
> Issue Type: Improvement
> Reporter: Shan Huang
> Priority: Major
>
> If a parquet file is written by parquet-mr before version 1.8.0, the bug [PARQUET-246|https://issues.apache.org/jira/browse/PARQUET-246] will produce data inconsistent with the encoding of DELTA_BYTE_ARRAY.
> There are two solutions:
> 1. We should update the read side to recover data written with this bug.
> 2. We need to throw an exception if the parquet file is written by parquet-mr before version 1.8.0 and has DELTA_BYTE_ARRAY encoding data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)