[jira] [Created] (ARROW-12904) [Rust] Unable to load Feather v2 files created by pyarrow and pandas.

["Gert Hulselmans (Jira)" <ji...@apache.org>]

Gert Hulselmans created ARROW-12904:
---------------------------------------

             Summary: [Rust] Unable to load Feather v2 files created by pyarrow and pandas.
                 Key: ARROW-12904
                 URL: https://issues.apache.org/jira/browse/ARROW-12904
             Project: Apache Arrow
          Issue Type: Bug
          Components: Rust
    Affects Versions: 4.0.1
         Environment: Ubuntu 20.04
            Reporter: Gert Hulselmans
            Assignee: Joris Van den Bossche


arrow-rs seems unable to load Feather v2 files created by pyarrow (and pandas), while it can read Feather v2 created by itself.

More info at:

[https://github.com/apache/arrow-rs/issues/286]

 

Any idea what is missing in the Rust implementation (missing part of the spec?)?

 
{code:java}
More details: in both files, I am getting the following:
Reading Utf8
field_node: FieldNode { length: 7, null_count: 0 }
offset buffer: Buffer { offset: 200, length: 55 }
offsets: [32, 0, 407708164, 545407072, 8388608, 67108864, 134217728, 201326592]
values buffer: Buffer { offset: 256, length: 51 }



offsets[0] != 0 indicates a problem: offsets are expected to start from zero on any array with offsets.
offsets[i+1] < offsets[i+1] for some i, which indicates a problem: offsets are expected to be monotonically increasing

I do not have a root cause yet, these are just observations.
{code}
https://github.com/apache/arrow-rs/issues/286#issuecomment-839524898

 

 In the attachment the following files can be found.

{{}}
{code:java}
test_pandas.feather: Original Feather file
test_arrow.feather: loading test_pandas.feather with pyarrow and saving with pyarrow: df_pa = pa.feather.read_feather('test_pandas.feather')
test_polars.feather:  Loading test_pandas.feather with pyarrow and saving with polars (only this one can be read by arrow-rs)
test_pandas_from_polars.feather: Loading test_polars.feather with polars and using the to_pandas option.
 
 
{code}
 

[^test_feather_file.zip]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)