You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@arrow.apache.org by "Joris Van den Bossche (JIRA)" <ji...@apache.org> on 2019/07/11 20:39:00 UTC

[jira] [Updated] (ARROW-5915) [C++] [Python] Set up testing for backwards compatibility of the parquet reader

     [ https://issues.apache.org/jira/browse/ARROW-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joris Van den Bossche updated ARROW-5915:
-----------------------------------------
    Description: 
Given the recent parquet compat problems, we should have better testing for this.

For easy testing of backwards compatibility, we could add some files (with different types) written with older versions, and ensure they are read correctly with the current version.

Similarly as what Kartothek is doing: https://github.com/JDASoftwareGroup/kartothek/tree/master/reference-data/arrow-compat

An easy way would be to do that in pyarrow and add them to /pyarrow/tests/data/parquet (we already have some files from 0.7 there). 

  was:
Given the recent parquet compat problems, we should have better testing for this.

For easy testing of backwards compatibility, we could add some files (with different types) written with older versions, add them to /pyarrow/tests/data/parquet (we already have some files from 0.7 there) and ensure they are read correctly with the current version.

Similarly as what Kartothek is doing: https://github.com/JDASoftwareGroup/kartothek/tree/master/reference-data/arrow-compat




> [C++] [Python] Set up testing for backwards compatibility of the parquet reader
> -------------------------------------------------------------------------------
>
>                 Key: ARROW-5915
>                 URL: https://issues.apache.org/jira/browse/ARROW-5915
>             Project: Apache Arrow
>          Issue Type: Test
>          Components: C++, Python
>            Reporter: Joris Van den Bossche
>            Priority: Major
>              Labels: parquet
>
> Given the recent parquet compat problems, we should have better testing for this.
> For easy testing of backwards compatibility, we could add some files (with different types) written with older versions, and ensure they are read correctly with the current version.
> Similarly as what Kartothek is doing: https://github.com/JDASoftwareGroup/kartothek/tree/master/reference-data/arrow-compat
> An easy way would be to do that in pyarrow and add them to /pyarrow/tests/data/parquet (we already have some files from 0.7 there). 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)