You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@arrow.apache.org by "Pierre Belzile (Jira)" <ji...@apache.org> on 2020/04/30 17:00:13 UTC

[jira] [Created] (ARROW-8657) Distinguish parquet version 2 logical type vs DataPageV2

Pierre Belzile created ARROW-8657:
-------------------------------------

             Summary: Distinguish parquet version 2 logical type vs DataPageV2
                 Key: ARROW-8657
                 URL: https://issues.apache.org/jira/browse/ARROW-8657
             Project: Apache Arrow
          Issue Type: Bug
          Components: C++, Python
    Affects Versions: 0.17.0
            Reporter: Pierre Belzile


With the recent release of 0.17, the ParquetVersion is used to define the logical type interpretation of fields and the selection of the DataPage format.

As a result all parquet files that were created with ParquetVersion::V2 to get features such as unsigned int32s, timestamps with nanosecond resolution, etc are now unreadable. That's TBs of data in my case.

Those two concerns should be separated. Given that that DataPageV2 pages were not written prior to 0.17 and in order to allow reading existing files, the existing version property should continue to operate as in 0.16 and inform the logical type mapping.

Some consideration should be given to issue a release 0.17.1.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)