You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@arrow.apache.org by seshu yamajala <sy...@gmail.com> on 2019/10/18 04:07:40 UTC

pyarrow and pyzmq no copy

I would like to use pyarrow with pyzmq no copy to send dicts of arrays
across the network without having to make copies of the arrays stored
in the dicts.

I've come up with an example, here: 
https://gist.github.com/syamajala/51d52f5e326ff719bf6231546091991d

However, I'm having trouble with deserializing and am seeing this:

Traceback (most recent call last):
  File "zmq_pyarrow.py", line 99, in <module>
    col.run_collector()
  File "zmq_pyarrow.py", line 74, in run_collector
    msg = pa.deserialize_components(comp)
  File "pyarrow/serialization.pxi", line 450, in
pyarrow.lib.deserialize_components
  File "pyarrow/serialization.pxi", line 307, in
pyarrow.lib.SerializedPyObject.from_components
  File "pyarrow/error.pxi", line 78, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Could not unwrap Buffer from the passed
Python object.

Does anyone have any tips?

Re: pyarrow and pyzmq no copy

Posted by Antoine Pitrou <an...@python.org>.

Hi,

Your example works here with the latest development version of Arrow, 
under Python 3.7.

Regards

Antoine.


Le 18/10/2019 à 00:07, seshu yamajala a écrit :
> I would like to use pyarrow with pyzmq no copy to send dicts of arrays
> across the network without having to make copies of the arrays stored
> in the dicts.
> 
> I've come up with an example, here:
> https://gist.github.com/syamajala/51d52f5e326ff719bf6231546091991d
> 
> However, I'm having trouble with deserializing and am seeing this:
> 
> Traceback (most recent call last):
>    File "zmq_pyarrow.py", line 99, in <module>
>      col.run_collector()
>    File "zmq_pyarrow.py", line 74, in run_collector
>      msg = pa.deserialize_components(comp)
>    File "pyarrow/serialization.pxi", line 450, in
> pyarrow.lib.deserialize_components
>    File "pyarrow/serialization.pxi", line 307, in
> pyarrow.lib.SerializedPyObject.from_components
>    File "pyarrow/error.pxi", line 78, in pyarrow.lib.check_status
> pyarrow.lib.ArrowInvalid: Could not unwrap Buffer from the passed
> Python object.
> 
> Does anyone have any tips?
>