You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by Chang She <ch...@eto.ai> on 2022/10/23 18:42:13 UTC

Arrow in jupyter

Hi, just curious if anyone’s played with using Arrow to ship data between
languages in jupyter/ipython, specifically python and js (that doesn’t
require a flight server or writing the data to disk).

Eg, can I call do something like IPython.kernel.execute(“table.filter(…)”)
and read the results directly into arrowjs instead of converting from plain
text / json?

Could find an existing issue and figure I’d ask before I go chase this down.

Thanks!

Chang

RE: Arrow in jupyter

Posted by "Lee, David" <Da...@blackrock.com>.
I’ve done some testing using pyarrow.jvm as part of jdbc optimization.

https://arrow.apache.org/docs/python/integration/python_java.html

Pull a dataset using a JVM environment with JDBC.

Map JDBC arrow result set to a python pyarrow result set.

ra = jpype.JPackage("org").apache.arrow.memory.RootAllocator(sys.maxsize)

batch = jpype.JPackage("org").apache.arrow.adapter.jdbc.JdbcToArrow.sqlToArrow(
    jdbc_connection,
    sql,
    ra
)

arrow_table = pyarrow.jvm.record_batch(batch)


From: Chang She <ch...@eto.ai>
Sent: Monday, October 24, 2022 11:44 AM
To: user@arrow.apache.org
Subject: Re: Arrow in jupyter


External Email: Use caution with links and attachments
Awesome, I will continue on that JIRA.

Thanks!

On Mon, Oct 24, 2022 at 6:23 AM Neal Richardson <ne...@gmail.com>> wrote:
Hi Chang,
I don't recall seeing an Arrow issue about this specifically, or any examples of doing that, but that sounds nice. https://issues.apache.org/jira/browse/ARROW-17608<https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/ARROW-17608__;!!KSjYCgUGsB4!fr_nJ6s_jfTijlw-O2aBexeefqDuRbFByff-p8Zi9Y8VGZrzsqSy-tCzdikV_lCdBGz9Yl-nhYFCIrA$> (C interface support in JS) is probably a prerequisite.

Neal

On Sun, Oct 23, 2022 at 2:45 PM Chang She <ch...@eto.ai>> wrote:
couldn’t*

On Sun, Oct 23, 2022 at 11:42 AM Chang She <ch...@eto.ai>> wrote:
Hi, just curious if anyone’s played with using Arrow to ship data between languages in jupyter/ipython, specifically python and js (that doesn’t require a flight server or writing the data to disk).

Eg, can I call do something like IPython.kernel.execute(“table.filter(…)”) and read the results directly into arrowjs instead of converting from plain text / json?

Could find an existing issue and figure I’d ask before I go chase this down.

Thanks!

Chang

This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See http://www.blackrock.com/corporate/compliance/email-disclaimers for further information.  Please refer to http://www.blackrock.com/corporate/compliance/privacy-policy for more information about BlackRock’s Privacy Policy.


For a list of BlackRock's office addresses worldwide, see http://www.blackrock.com/corporate/about-us/contacts-locations.

© 2022 BlackRock, Inc. All rights reserved.

Re: Arrow in jupyter

Posted by Chang She <ch...@eto.ai>.
Awesome, I will continue on that JIRA.

Thanks!

On Mon, Oct 24, 2022 at 6:23 AM Neal Richardson <ne...@gmail.com>
wrote:

> Hi Chang,
> I don't recall seeing an Arrow issue about this specifically, or any
> examples of doing that, but that sounds nice.
> https://issues.apache.org/jira/browse/ARROW-17608 (C interface support in
> JS) is probably a prerequisite.
>
> Neal
>
> On Sun, Oct 23, 2022 at 2:45 PM Chang She <ch...@eto.ai> wrote:
>
>> couldn’t*
>>
>> On Sun, Oct 23, 2022 at 11:42 AM Chang She <ch...@eto.ai> wrote:
>>
>>> Hi, just curious if anyone’s played with using Arrow to ship data
>>> between languages in jupyter/ipython, specifically python and js (that
>>> doesn’t require a flight server or writing the data to disk).
>>>
>>> Eg, can I call do something like
>>> IPython.kernel.execute(“table.filter(…)”) and read the results directly
>>> into arrowjs instead of converting from plain text / json?
>>>
>>> Could find an existing issue and figure I’d ask before I go chase this
>>> down.
>>>
>>> Thanks!
>>>
>>> Chang
>>>
>>

Re: Arrow in jupyter

Posted by Neal Richardson <ne...@gmail.com>.
Hi Chang,
I don't recall seeing an Arrow issue about this specifically, or any
examples of doing that, but that sounds nice.
https://issues.apache.org/jira/browse/ARROW-17608 (C interface support in
JS) is probably a prerequisite.

Neal

On Sun, Oct 23, 2022 at 2:45 PM Chang She <ch...@eto.ai> wrote:

> couldn’t*
>
> On Sun, Oct 23, 2022 at 11:42 AM Chang She <ch...@eto.ai> wrote:
>
>> Hi, just curious if anyone’s played with using Arrow to ship data between
>> languages in jupyter/ipython, specifically python and js (that doesn’t
>> require a flight server or writing the data to disk).
>>
>> Eg, can I call do something like
>> IPython.kernel.execute(“table.filter(…)”) and read the results directly
>> into arrowjs instead of converting from plain text / json?
>>
>> Could find an existing issue and figure I’d ask before I go chase this
>> down.
>>
>> Thanks!
>>
>> Chang
>>
>

Re: Arrow in jupyter

Posted by Chang She <ch...@eto.ai>.
couldn’t*

On Sun, Oct 23, 2022 at 11:42 AM Chang She <ch...@eto.ai> wrote:

> Hi, just curious if anyone’s played with using Arrow to ship data between
> languages in jupyter/ipython, specifically python and js (that doesn’t
> require a flight server or writing the data to disk).
>
> Eg, can I call do something like IPython.kernel.execute(“table.filter(…)”)
> and read the results directly into arrowjs instead of converting from plain
> text / json?
>
> Could find an existing issue and figure I’d ask before I go chase this
> down.
>
> Thanks!
>
> Chang
>