Streaming Arrow files in Python to JS

[Akshay Verma <ak...@tuta.io>]

Hello,

I am very excited about the Arrow project and want to introduce it in the new application that we are developing. The first place where we can easily introduce is the communication between backend process and frontend visualisation. Currently we are doing it as JSON. We can currently convert that JSON to an arrow Table/RecordBatch and save it locally in Python. 

Ideally, we would like to stream the big JSON data from Python to the JS application and update the visualisation as the streamed data comes in.

Is there any example, or an intended approach we could take to implement this in our project?

Thanks for the project.

Kind Regards,
Akshay Verma

Re: Streaming Arrow files in Python to JS

[Wes McKinney <we...@gmail.com>]

hi,

We have a well-defined stream format which can be sent over any
network transport

http://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format

Once you decide on your network transport, you will want to use the
stream writer class from Python to send the byte stream to a reader in
JavaScript. I don't know any examples to point you to to get started
though.

- Wes

On Thu, Oct 31, 2019 at 3:19 AM Akshay Verma <ak...@tuta.io> wrote:
>
>
> Hello,
>
> I am very excited about the Arrow project and want to introduce it in the new application that we are developing. The first place where we can easily introduce is the communication between backend process and frontend visualisation. Currently we are doing it as JSON. We can currently convert that JSON to an arrow Table/RecordBatch and save it locally in Python.
>
> Ideally, we would like to stream the big JSON data from Python to the JS application and update the visualisation as the streamed data comes in.
>
> Is there any example, or an intended approach we could take to implement this in our project?
>
> Thanks for the project.
>
> Kind Regards,
> Akshay Verma