[jira] [Closed] (ARROW-12904) [Rust] Unable to load Feather v2 files created by pyarrow and pandas.

["Gert Hulselmans (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-12904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gert Hulselmans closed ARROW-12904.
-----------------------------------
    Resolution: Information Provided

> [Rust] Unable to load Feather v2 files created by pyarrow and pandas.
> ---------------------------------------------------------------------
>
>                 Key: ARROW-12904
>                 URL: https://issues.apache.org/jira/browse/ARROW-12904
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Rust
>    Affects Versions: 4.0.1
>         Environment: Ubuntu 20.04
>            Reporter: Gert Hulselmans
>            Assignee: Joris Van den Bossche
>            Priority: Major
>
> arrow-rs seems unable to load Feather v2 files created by pyarrow (and pandas), while it can read Feather v2 created by itself.
> More info at:
> [https://github.com/apache/arrow-rs/issues/286]
>  
> Any idea what is missing in the Rust implementation (missing part of the spec?)?
>  
> {code:java}
> More details: in both files, I am getting the following:
> Reading Utf8
> field_node: FieldNode { length: 7, null_count: 0 }
> offset buffer: Buffer { offset: 200, length: 55 }
> offsets: [32, 0, 407708164, 545407072, 8388608, 67108864, 134217728, 201326592]
> values buffer: Buffer { offset: 256, length: 51 }
> offsets[0] != 0 indicates a problem: offsets are expected to start from zero on any array with offsets.
> offsets[i+1] < offsets[i+1] for some i, which indicates a problem: offsets are expected to be monotonically increasing
> I do not have a root cause yet, these are just observations.
> {code}
> https://github.com/apache/arrow-rs/issues/286#issuecomment-839524898
>  
>  In the attachment the following files can be found.
> {{}}
> {code:java}
> test_pandas.feather: Original Feather file
> test_arrow.feather: loading test_pandas.feather with pyarrow and saving with pyarrow: df_pa = pa.feather.read_feather('test_pandas.feather')
> test_polars.feather:  Loading test_pandas.feather with pyarrow and saving with polars (only this one can be read by arrow-rs)
> test_pandas_from_polars.feather: Loading test_polars.feather with polars and using the to_pandas option.
>  
>  
> {code}
>  
> [^test_feather_file.zip]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)