[jira] [Commented] (ARROW-15642) [Python] [JavaScript] Arrow IPC file output by apache-arrow tableToIPC method cannot be read by pyarrow

["Weston Pace (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17504707#comment-17504707 ] 

Weston Pace commented on ARROW-15642:
-------------------------------------

You are creating a table in the Arrow IPC streaming format and then attempting to read it using the Arrow IPC file format.

You can either change your JS to write the file format:

https://arrow.apache.org/docs/js/modules/Arrow_dom.html#tableToIPC

{noformat}
const outputTable = tableToIPC(rainfall, 'file');
{noformat}

or you can change your python to read the stream format:

{noformat}
with pa.ipc.RecordBatchStreamReader('jsarrow.arrow') as reader:
    tab = reader.read_all()
    print(tab.to_pandas())
{noformat}

> [Python] [JavaScript] Arrow IPC file output by apache-arrow tableToIPC method cannot be read by pyarrow
> -------------------------------------------------------------------------------------------------------
>
>                 Key: ARROW-15642
>                 URL: https://issues.apache.org/jira/browse/ARROW-15642
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: JavaScript, Python
>    Affects Versions: 7.0.0
>            Reporter: Dan Coates
>            Priority: Major
>
> IPC files created by the node library `apache-arrow` don't seem to be able to be read by pyarrow. There is an example of this issue here: [https://github.com/dancoates/pyarrow-jsarrow-test |https://github.com/dancoates/pyarrow-jsarrow-test]
>  
> writing the arrow file from js
> {code:javascript}
> import {tableToIPC, tableFromArrays} from 'apache-arrow';
> import fs from 'fs';
> const LENGTH = 2000;
> const rainAmounts = Float32Array.from(
>     { length: LENGTH },
>     () => Number((Math.random() * 20).toFixed(1)));
> const rainDates = Array.from(
>     { length: LENGTH },
>     (_, i) => new Date(Date.now() - 1000 * 60 * 60 * 24 * i));
> const rainfall = tableFromArrays({
>     precipitation: rainAmounts,
>     date: rainDates
> });
> const outputTable = tableToIPC(rainfall);
> fs.writeFileSync('jsarrow.arrow', outputTable); {code}
>  
> reading in python
> {code:python}
> import pyarrow as pa
> with open('jsarrow.arrow', 'rb') as f:
>     with pa.ipc.open_file(f) as reader:
>         df = reader.read_pandas()
>         print(df.head())
>  {code}
>  
> produces the error:
> {code:java}
> pyarrow.lib.ArrowInvalid: Not an Arrow file {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)