pipe Feather bytes between processes

[Rares Vernica <rv...@gmail.com>]

Hi,

I am having trouble piping Feather structures between two processes. On the
receiving-process side, I get: pyarrow.lib.ArrowIOError: [Errno 29] Illegal
seek


I have process A and process B which communicate via pipes. Process A sends
the bytes of a Feather structure to process B. Process A could send one or
more Feather structures, but, after each one, it waits for a response from
B. The pipe is to remain open, so there is no EOF after each Feather
structure.

The receiving process is a Python process and I do something like:

pyarrow.feather.read_feather(sys.stdin)

The error I receive it:

File "/usr/local/lib/python2.7/dist-packages/pyarrow/feather.py", line 143,
in read_feather
    reader = FeatherReader(source)
File "/usr/local/lib/python2.7/dist-packages/pyarrow/feather.py", line 43,
in __init__
    self.open(source)
File "pyarrow/feather.pxi", line 83, in pyarrow.lib.FeatherReader.open
(/arrow/python/build/temp.linux-x86_64-2.7/lib.cxx:60120)
File "pyarrow/error.pxi", line 72, in pyarrow.lib.check_status
(/arrow/python/build/temp.linux-x86_64-2.7/lib.cxx:7495)
pyarrow.lib.ArrowIOError: [Errno 29] Illegal seek

If instead of process A, I redirect a file, it works fine:

# python -uc "import pyarrow.feather, sys;
print(pyarrow.feather.read_feather(sys.stdin))" < a.feather
    id
0  1.0
1  2.0
2  3.0
3  NaN
4  5.0
5  6.0
6  7.0
7  8.0

The difference between process A and redirecting a file is that process A
does not send EOF or close the pipe. Does read_feather need EOF?

I also tried reading the bytes from process A in memory, writing them to a
file, and then reading the file with read_feather. This works fine. So, I
believe process A sends a complete Feather structure.

Any thoughts? Thanks!
--
Rares

Re: pipe Feather bytes between processes

[Wes McKinney <we...@gmail.com>]

I suggest that you use the Arrow streaming format for this purpose
instead (http://arrow.apache.org/docs/python/ipc.html). The Feather
APIs exist for backwards compatibility, and require an IO interface
that supports random access.

- Wes

On Sun, Sep 10, 2017 at 10:05 PM, Rares Vernica <rv...@gmail.com> wrote:
> Hi,
>
> I am having trouble piping Feather structures between two processes. On the
> receiving-process side, I get: pyarrow.lib.ArrowIOError: [Errno 29] Illegal
> seek
>
>
> I have process A and process B which communicate via pipes. Process A sends
> the bytes of a Feather structure to process B. Process A could send one or
> more Feather structures, but, after each one, it waits for a response from
> B. The pipe is to remain open, so there is no EOF after each Feather
> structure.
>
> The receiving process is a Python process and I do something like:
>
> pyarrow.feather.read_feather(sys.stdin)
>
> The error I receive it:
>
> File "/usr/local/lib/python2.7/dist-packages/pyarrow/feather.py", line 143,
> in read_feather
>     reader = FeatherReader(source)
> File "/usr/local/lib/python2.7/dist-packages/pyarrow/feather.py", line 43,
> in __init__
>     self.open(source)
> File "pyarrow/feather.pxi", line 83, in pyarrow.lib.FeatherReader.open
> (/arrow/python/build/temp.linux-x86_64-2.7/lib.cxx:60120)
> File "pyarrow/error.pxi", line 72, in pyarrow.lib.check_status
> (/arrow/python/build/temp.linux-x86_64-2.7/lib.cxx:7495)
> pyarrow.lib.ArrowIOError: [Errno 29] Illegal seek
>
> If instead of process A, I redirect a file, it works fine:
>
> # python -uc "import pyarrow.feather, sys;
> print(pyarrow.feather.read_feather(sys.stdin))" < a.feather
>     id
> 0  1.0
> 1  2.0
> 2  3.0
> 3  NaN
> 4  5.0
> 5  6.0
> 6  7.0
> 7  8.0
>
> The difference between process A and redirecting a file is that process A
> does not send EOF or close the pipe. Does read_feather need EOF?
>
> I also tried reading the bytes from process A in memory, writing them to a
> file, and then reading the file with read_feather. This works fine. So, I
> believe process A sends a complete Feather structure.
>
> Any thoughts? Thanks!
> --
> Rares