[jira] [Resolved] (PARQUET-1977) Invalid data_page_offset

["Gabor Szadovszky (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PARQUET-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gabor Szadovszky resolved PARQUET-1977.
---------------------------------------
    Fix Version/s: 1.12.0
       Resolution: Fixed

> Invalid data_page_offset
> ------------------------
>
>                 Key: PARQUET-1977
>                 URL: https://issues.apache.org/jira/browse/PARQUET-1977
>             Project: Parquet
>          Issue Type: Bug
>          Components: parquet-mr
>    Affects Versions: 1.12.0
>            Reporter: Gabor Szadovszky
>            Assignee: Gabor Szadovszky
>            Priority: Major
>             Fix For: 1.12.0
>
>
> The thrift field [{{data_page_offset}}|https://github.com/gszadovszky/parquet-format/blob/master/src/main/thrift/parquet.thrift#L733] is filled with incorrect value. Currently, it always points to the beginning of the column chunk which is not correct according to the spec in case there is a dictionary page. This is not a regression as it was written incorrectly since the beginning of parquet-mr.
> Meanwhile PARQUET-1850 fixed that we never wrote the field [{{dictionary_page_offset}}|https://github.com/gszadovszky/parquet-format/blob/master/src/main/thrift/parquet.thrift#L739]. After the fix we correctly write this field if there is a dictionary filter. The problem is we are using the same value to fill both fields. So there are two possibilities:
> * There is no dictionary page in the column chunk so {{data_page_offset}} is filled with the correct value while {{dictionary_page_offset}} is not filled which is still correct. We are good.
> * There is a dictionary page at the beginning of the column chunk so {{data_page_offset}} and {{dictionary_page_offset}} are both contains the same value. This is not only a regression but it causes issues in other implementations (e.g. Impala) where footer validation is more strict than in parquet-mr because {{dictionary_page_offset}} shall be less than {{data_page_offset}} at all time if it is filled.
> So, we need to fill {{data_page_offset}} correctly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)