You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Samrat Batth <sa...@gmail.com> on 2019/11/01 02:58:27 UTC
re: Arrow for low latency IPC
Hi,
I am new to Apache Arrow and want to consider it for low latency IPC stream. I want to see if I can use Arrow for low latency IPC via processes written in multiple languages.
As far as what I can understand about current version of Arrow, the messages are passed in a batch via IPC, ie the RX will get the msgs in batch and not as soon as they are sent by the TX.
Any pointers to examples / code for this? Or any benchmarks on IPC?
Thanks,
Sam
Re: Arrow for low latency IPC
Posted by Wes McKinney <we...@gmail.com>.
hi Sam -- could you clarify a bit more about your use case?
Flight is used for passing batches efficiently over TCP (via gRPC). We
have shared memory IPC utilities as well, but coordination and
signaling / semaphores between processes are up to you. Plasma might
be worth exploring as a tool to help
- Wes
On Fri, Nov 1, 2019 at 3:32 AM Fan Liya <li...@gmail.com> wrote:
>
> Hi Samrat,
>
> Arrow has flexible support for IPC through grpc.
>
> The cpp benchmark can be found in:
> https://github.com/apache/arrow/blob/master/cpp/src/arrow/flight/flight_benchmark.cc
>
> The java benchmark can be found in:
> https://github.com/apache/arrow/blob/master/java/flight/src/test/java/org/apache/arrow/flight/perf/TestPerf.java
>
> About the latency problem, I am not clear about your scenario. Can you
> reduce the latency by making the batch small enough?
>
> Best,
> Liya Fan
>
> On Fri, Nov 1, 2019 at 10:58 AM Samrat Batth <sa...@gmail.com> wrote:
>
> > Hi,
> >
> > I am new to Apache Arrow and want to consider it for low latency IPC
> > stream. I want to see if I can use Arrow for low latency IPC via processes
> > written in multiple languages.
> >
> > As far as what I can understand about current version of Arrow, the
> > messages are passed in a batch via IPC, ie the RX will get the msgs in
> > batch and not as soon as they are sent by the TX.
> >
> > Any pointers to examples / code for this? Or any benchmarks on IPC?
> >
> > Thanks,
> > Sam
Re: Arrow for low latency IPC
Posted by Wes McKinney <we...@gmail.com>.
hi Sam -- could you clarify a bit more about your use case?
Flight is used for passing batches efficiently over TCP (via gRPC). We
have shared memory IPC utilities as well, but coordination and
signaling / semaphores between processes are up to you. Plasma might
be worth exploring as a tool to help
- Wes
On Fri, Nov 1, 2019 at 3:32 AM Fan Liya <li...@gmail.com> wrote:
>
> Hi Samrat,
>
> Arrow has flexible support for IPC through grpc.
>
> The cpp benchmark can be found in:
> https://github.com/apache/arrow/blob/master/cpp/src/arrow/flight/flight_benchmark.cc
>
> The java benchmark can be found in:
> https://github.com/apache/arrow/blob/master/java/flight/src/test/java/org/apache/arrow/flight/perf/TestPerf.java
>
> About the latency problem, I am not clear about your scenario. Can you
> reduce the latency by making the batch small enough?
>
> Best,
> Liya Fan
>
> On Fri, Nov 1, 2019 at 10:58 AM Samrat Batth <sa...@gmail.com> wrote:
>
> > Hi,
> >
> > I am new to Apache Arrow and want to consider it for low latency IPC
> > stream. I want to see if I can use Arrow for low latency IPC via processes
> > written in multiple languages.
> >
> > As far as what I can understand about current version of Arrow, the
> > messages are passed in a batch via IPC, ie the RX will get the msgs in
> > batch and not as soon as they are sent by the TX.
> >
> > Any pointers to examples / code for this? Or any benchmarks on IPC?
> >
> > Thanks,
> > Sam
Re: Arrow for low latency IPC
Posted by Fan Liya <li...@gmail.com>.
Hi Samrat,
Arrow has flexible support for IPC through grpc.
The cpp benchmark can be found in:
https://github.com/apache/arrow/blob/master/cpp/src/arrow/flight/flight_benchmark.cc
The java benchmark can be found in:
https://github.com/apache/arrow/blob/master/java/flight/src/test/java/org/apache/arrow/flight/perf/TestPerf.java
About the latency problem, I am not clear about your scenario. Can you
reduce the latency by making the batch small enough?
Best,
Liya Fan
On Fri, Nov 1, 2019 at 10:58 AM Samrat Batth <sa...@gmail.com> wrote:
> Hi,
>
> I am new to Apache Arrow and want to consider it for low latency IPC
> stream. I want to see if I can use Arrow for low latency IPC via processes
> written in multiple languages.
>
> As far as what I can understand about current version of Arrow, the
> messages are passed in a batch via IPC, ie the RX will get the msgs in
> batch and not as soon as they are sent by the TX.
>
> Any pointers to examples / code for this? Or any benchmarks on IPC?
>
> Thanks,
> Sam