Best ways to implement push notifications over Flight?

[Radu Teodorescu <ra...@yahoo.com.INVALID>]

Hi,
I am looking at the best way to push notifications from a server to clients over flight and I have a few questions on the approach:

A. Is there a standard way of doing it and/or does this fundamentally go against flight philosophy?

B. One approach is to run a doGet and then have the server push notifications as Flight Payloads on the corresponding FlightDataStream as they become available, Questions:
- is FlightStreamReader.Next() blocking and is it guarantee to wake up on each corresponding server side FlightDataStream.Next()?
- if I use this approach can initiate new flight calls from the same client while the same client, before the stream returned by get is closed? I guess in general is interesting to know whether one can process multiple Get streams in parallel?
- are there any async hooks for flight? Either for listening to a data stream or for processing responses to flight calls? 

Thank you
Radu

Re: Best ways to implement push notifications over Flight?

[David Li <li...@gmail.com>]

Hey Radu,

A. I don't think there's a standard way. You can also think about the
DoAction endpoint (which is a server-side stream) if you don't want to
abuse DoGet/if you want push notifications to be non-Arrow data.

B. This sounds fine. Reading from a stream is blocking, and it's
guaranteed to unblock (as far as network conditions allow :) ) once
the server sends data.

Clients can handle multiple parallel calls, so you're free to initiate
new calls while the push notification channel is ongoing.

As for async, there are some JIRAs outstanding, but it's not
implemented yet. In particular async in C++/Python is considerably
more difficult to implement; IMO it would be fairly trivial in Java
(gRPC in Java is async-first, the fact that Flight in Java is blocking
actually required quite a bit of work!)

Related JIRAs to async work are ARROW-4484, ARROW-6062, ARROW-1009.
There was some work done on the C++ API side (an async reader) in
ARROW-8311.

On 8/19/20, Radu Teodorescu <ra...@yahoo.com.invalid> wrote:
> Hi,
> I am looking at the best way to push notifications from a server to clients
> over flight and I have a few questions on the approach:
>
> A. Is there a standard way of doing it and/or does this fundamentally go
> against flight philosophy?
>
> B. One approach is to run a doGet and then have the server push
> notifications as Flight Payloads on the corresponding FlightDataStream as
> they become available, Questions:
> - is FlightStreamReader.Next() blocking and is it guarantee to wake up on
> each corresponding server side FlightDataStream.Next()?
> - if I use this approach can initiate new flight calls from the same client
> while the same client, before the stream returned by get is closed? I guess
> in general is interesting to know whether one can process multiple Get
> streams in parallel?
> - are there any async hooks for flight? Either for listening to a data
> stream or for processing responses to flight calls?
>
> Thank you
> Radu