You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by Chris Zheng <z...@caudate.me> on 2020/06/06 04:00:43 UTC
Re: Using 'zero copy' for interop with python from java
Hi there,
I’m coming here from clojure, where we are hoping to use more of Arrow for https://github.com/clj-python/libpython-clj <https://github.com/clj-python/libpython-clj>, a library that gives very easy python interop from the JVM.
Here is the link to the initial question:
https://clojurians.zulipchat.com/#narrow/stream/151924-data-science/topic/dplyr-like.20API.20to.20tech.2Eml.2Edataset/near/199914955 <https://clojurians.zulipchat.com/#narrow/stream/151924-data-science/topic/dplyr-like.20API.20to.20tech.2Eml.2Edataset/near/199914955>
The use case would be to load from a database on the Java side and then convert it to arrow, allowing it to be consumed by a python ml library (with the whole zero copy thing).
Once there is a Java Arrow Object created in the JVM, how would it be accessible by the Python Library?
I was hoping to get some input or a simple example on how it might be done
Chris.
Re: Using 'zero copy' for interop with python from java
Posted by "Uwe L. Korn" <uw...@xhochy.com>.
Yes, this is zero-copy and is probably the thing you are looking for. In future, it would be nice to migrate this codebase to the C-interface but for now, this should do the job.
On Mon, Jun 8, 2020, at 5:24 AM, Micah Kornfield wrote:
> Uwe wrote a blog post [1] on how to do this with PY4J a while ago. I think this ends up being zero copy but not 100% sure.
>
> [1] https://uwekorn.com/2019/11/17/fast-jdbc-access-in-python-using-pyarrow-jvm.html
>
> On Fri, Jun 5, 2020 at 9:01 PM Chris Zheng <z...@caudate.me> wrote:
>>
>> Hi there,
>>
>> I’m coming here from clojure, where we are hoping to use more of Arrow for https://github.com/clj-python/libpython-clj, a library that gives very easy python interop from the JVM.
>>
>> Here is the link to the initial question:
>> https://clojurians.zulipchat.com/#narrow/stream/151924-data-science/topic/dplyr-like.20API.20to.20tech.2Eml.2Edataset/near/199914955
>>
>> The use case would be to load from a database on the Java side and then convert it to arrow, allowing it to be consumed by a python ml library (with the whole zero copy thing).
>>
>> Once there is a Java Arrow Object created in the JVM, how would it be accessible by the Python Library?
>>
>> I was hoping to get some input or a simple example on how it might be done
>>
>> Chris.
>>
>>
Re: Using 'zero copy' for interop with python from java
Posted by Chris Zheng <z...@caudate.me>.
Hi Micah,
Thanks for the fantastic summary of what to do.
I’ll have a play with it in the next few weeks.
Will keep you posted.
Chris
> On 12 Jun 2020, at 2:05 pm, Micah Kornfield <em...@gmail.com> wrote:
>
> Hi Chris,
> There isn't anything prepackaged for this use-case as far as I know. As Uwe mentioned it would probably be nice to build something using the C interface for this purpose, but I think you should be able to do it today as described below.
>
> I think you can pass ArrowBuf pointers to python via foreign_buffer [1], but as far as I know, you would probably have to do some amount manual reconstructions of arrays from buffers. The rough steps would be:
> 1. Serialize the schema on the java side side [2] and obtain a memory address from it to share with python (via foreign_buffer) .
> 2. Deserialize the schema on the python side using pyarrow.ipc.read_schema [3]
> 3. Extract the buffer address/lengths in java (example from Gandiva [4]) and reconstruct with foreign_object
> 4. Traverse DataTypes the pyarrow schema to reconstruct the arrays [5] based on number of buffers required [6].
>
> If you do end up doing this, then I think #4 might make a nice contribution to the project.
>
> Thanks,
> Micah
>
> [1] https://arrow.apache.org/docs/python/generated/pyarrow.foreign_buffer.html <https://arrow.apache.org/docs/python/generated/pyarrow.foreign_buffer.html>
> [2] https://arrow.apache.org/docs/java/org/apache/arrow/vector/ipc/message/MessageSerializer.html#serializeMetadata-org.apache.arrow.vector.types.pojo.Schema <https://arrow.apache.org/docs/java/org/apache/arrow/vector/ipc/message/MessageSerializer.html#serializeMetadata-org.apache.arrow.vector.types.pojo.Schema>
> [3] https://github.com/apache/arrow/blob/1164079d5442c3910c18549bfcd2e68d4554b909/python/pyarrow/ipc.pxi#L577 <https://github.com/apache/arrow/blob/1164079d5442c3910c18549bfcd2e68d4554b909/python/pyarrow/ipc.pxi#L577>
> [4] https://github.com/apache/arrow/blob/17bdb5af9b3c63f6cbef57e88a6d2513e781b532/java/gandiva/src/main/java/org/apache/arrow/gandiva/evaluator/Projector.java#L139 <https://github.com/apache/arrow/blob/17bdb5af9b3c63f6cbef57e88a6d2513e781b532/java/gandiva/src/main/java/org/apache/arrow/gandiva/evaluator/Projector.java#L139> <https://github.com/apache/arrow/blob/17bdb5af9b3c63f6cbef57e88a6d2513e781b532/java/gandiva/src/main/java/org/apache/arrow/gandiva/evaluator/Projector.java#L139>
> [5] https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array.from_buffers <https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array.from_buffers>
> [6] https://arrow.apache.org/docs/python/generated/pyarrow.DataType.html#pyarrow.DataType.num_buffers <https://arrow.apache.org/docs/python/generated/pyarrow.DataType.html#pyarrow.DataType.num_buffers>
>
>
> On Mon, Jun 8, 2020 at 12:55 AM Chris Zheng <z@caudate.me <ma...@caudate.me>> wrote:
> That blog post is really good. However, I’d like to do this in a running JVM as opposed to a python program.
>
>
>> On 8 Jun 2020, at 11:24 am, Micah Kornfield <emkornfield@gmail.com <ma...@gmail.com>> wrote:
>>
>> Uwe wrote a blog post [1] on how to do this with PY4J a while ago. I think this ends up being zero copy but not 100% sure.
>>
>> [1] https://uwekorn.com/2019/11/17/fast-jdbc-access-in-python-using-pyarrow-jvm.html <https://uwekorn.com/2019/11/17/fast-jdbc-access-in-python-using-pyarrow-jvm.html>
Re: Using 'zero copy' for interop with python from java
Posted by Micah Kornfield <em...@gmail.com>.
Hi Chris,
There isn't anything prepackaged for this use-case as far as I know. As
Uwe mentioned it would probably be nice to build something using the C
interface for this purpose, but I think you should be able to do it today
as described below.
I think you can pass ArrowBuf pointers to python via foreign_buffer [1],
but as far as I know, you would probably have to do some amount manual
reconstructions of arrays from buffers. The rough steps would be:
1. Serialize the schema on the java side side [2] and obtain a memory
address from it to share with python (via foreign_buffer) .
2. Deserialize the schema on the python side using pyarrow.ipc.read_schema
[3]
3. Extract the buffer address/lengths in java (example from Gandiva [4])
and reconstruct with foreign_object
4. Traverse DataTypes the pyarrow schema to reconstruct the arrays [5]
based on number of buffers required [6].
If you do end up doing this, then I think #4 might make a nice contribution
to the project.
Thanks,
Micah
[1]
https://arrow.apache.org/docs/python/generated/pyarrow.foreign_buffer.html
[2]
https://arrow.apache.org/docs/java/org/apache/arrow/vector/ipc/message/MessageSerializer.html#serializeMetadata-org.apache.arrow.vector.types.pojo.Schema
[3]
https://github.com/apache/arrow/blob/1164079d5442c3910c18549bfcd2e68d4554b909/python/pyarrow/ipc.pxi#L577
[4]
https://github.com/apache/arrow/blob/17bdb5af9b3c63f6cbef57e88a6d2513e781b532/java/gandiva/src/main/java/org/apache/arrow/gandiva/evaluator/Projector.java#L139
<https://github.com/apache/arrow/blob/17bdb5af9b3c63f6cbef57e88a6d2513e781b532/java/gandiva/src/main/java/org/apache/arrow/gandiva/evaluator/Projector.java#L139>
[5]
https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array.from_buffers
[6]
https://arrow.apache.org/docs/python/generated/pyarrow.DataType.html#pyarrow.DataType.num_buffers
On Mon, Jun 8, 2020 at 12:55 AM Chris Zheng <z...@caudate.me> wrote:
> That blog post is really good. However, I’d like to do this in a running
> JVM as opposed to a python program.
>
>
> On 8 Jun 2020, at 11:24 am, Micah Kornfield <em...@gmail.com> wrote:
>
> Uwe wrote a blog post [1] on how to do this with PY4J a while ago. I think
> this ends up being zero copy but not 100% sure.
>
> [1]
> https://uwekorn.com/2019/11/17/fast-jdbc-access-in-python-using-pyarrow-jvm.html
>
>
>
Re: Using 'zero copy' for interop with python from java
Posted by Chris Zheng <z...@caudate.me>.
That blog post is really good. However, I’d like to do this in a running JVM as opposed to a python program.
> On 8 Jun 2020, at 11:24 am, Micah Kornfield <em...@gmail.com> wrote:
>
> Uwe wrote a blog post [1] on how to do this with PY4J a while ago. I think this ends up being zero copy but not 100% sure.
>
> [1] https://uwekorn.com/2019/11/17/fast-jdbc-access-in-python-using-pyarrow-jvm.html <https://uwekorn.com/2019/11/17/fast-jdbc-access-in-python-using-pyarrow-jvm.html>
Re: Using 'zero copy' for interop with python from java
Posted by Micah Kornfield <em...@gmail.com>.
Uwe wrote a blog post [1] on how to do this with PY4J a while ago. I think
this ends up being zero copy but not 100% sure.
[1]
https://uwekorn.com/2019/11/17/fast-jdbc-access-in-python-using-pyarrow-jvm.html
On Fri, Jun 5, 2020 at 9:01 PM Chris Zheng <z...@caudate.me> wrote:
>
> Hi there,
>
> I’m coming here from clojure, where we are hoping to use more of Arrow for
> https://github.com/clj-python/libpython-clj, a library that gives very
> easy python interop from the JVM.
>
> Here is the link to the initial question:
>
> https://clojurians.zulipchat.com/#narrow/stream/151924-data-science/topic/dplyr-like.20API.20to.20tech.2Eml.2Edataset/near/199914955
>
> The use case would be to load from a database on the Java side and then
> convert it to arrow, allowing it to be consumed by a python ml library
> (with the whole zero copy thing).
>
> Once there is a Java Arrow Object created in the JVM, how would it be
> accessible by the Python Library?
>
> I was hoping to get some input or a simple example on how it might be done
>
> Chris.
>
>
>
>