[jira] [Created] (PARQUET-1977) Invalid data_page_offset

["Gabor Szadovszky (Jira)" <ji...@apache.org>]

Gabor Szadovszky created PARQUET-1977:
-----------------------------------------

             Summary: Invalid data_page_offset
                 Key: PARQUET-1977
                 URL: https://issues.apache.org/jira/browse/PARQUET-1977
             Project: Parquet
          Issue Type: Bug
          Components: parquet-mr
    Affects Versions: 1.12.0
            Reporter: Gabor Szadovszky
            Assignee: Gabor Szadovszky


The thrift field [{{data_page_offset}}|https://github.com/gszadovszky/parquet-format/blob/master/src/main/thrift/parquet.thrift#L733] is filled with incorrect value. Currently, it always points to the beginning of the column chunk which is not correct according to the spec in case there is a dictionary page. This is not a regression as it was written incorrectly since the beginning of parquet-mr.
Meanwhile PARQUET-1850 fixed that we never wrote the field [{{dictionary_page_offset}}|https://github.com/gszadovszky/parquet-format/blob/master/src/main/thrift/parquet.thrift#L739]. After the fix we correctly write this field if there is a dictionary filter. The problem is we are using the same value to fill both fields. So there are two possibilities:
* There is no dictionary page in the column chunk so {{data_page_offset}} is filled with the correct value while {{dictionary_page_offset}} is not filled which is still correct. We are good.
* There is a dictionary page at the beginning of the column chunk so {{data_page_offset}} and {{dictionary_page_offset}} are both contains the same value. This is not only a regression but it causes issues in other implementations (e.g. Impala) where footer validation is more strict than in parquet-mr because {{dictionary_page_offset}} shall be less than {{data_page_offset}} at all time if it is filled.

So, we need to fill {{data_page_offset}} correctly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)