[flight] how does the server terminate a flight with status code?

[Steve Kim <ch...@gmail.com>]

I am writing a Flight server using pyarrow.flight. I want the client to
receive INVALID_ARGUMENT status code when it sends an invalid ticket to the
server. I see that there are some FlightError subclasses that are
associated with status codes, but the set of covered status codes is
incomplete. It feels awkward to rely on exceptions to control the server's
response instead of explicitly aborting the Flight with a status code. Do I
misunderstand how this ought to work? If pyarrow.flight is missing
something, I am willing to contribute.

Thanks in advance for any help!
Steve

Re: [flight] how does the server terminate a flight with status code?

[David Li <li...@apache.org>]

1. In general we don't want to expose third party symbols in our public API. And gRPC and Protobuf especially make linking with them difficult.
2. Flight supports transports besides gRPC. This is still experimental, but the C++ implementation supports a UCX transport, and we may also want to consider something that will actually work with browsers (gRPC fails on this front, despite several workarounds).

You can absolutely reason about Flight as a gRPC service. We use the gRPC and Protobuf libraries, and we aren't rewriting or reimplementing those, so I'm not sure what you mean. Being able to have a 'native' implementation using just the Protobufs is a goal, and there are some StackOverflow questions that explore how that works (ideally this would also be in CI/documentation eventually). 

In C++ you can get access to the server builder during server configuration and make any changes there, including registering extra gRPC services. In Java, you can even "extract" the service implementation and use the regular gRPC APIs from there on out.

What would help interoperability more is probably just better support from the gRPC libraries themselves; even just having version macros so that we don't have to do manual version detection would help us maintain build time compatibility with multiple gRPC versions. Another frequent request is being able to interoperate with grpcio in Python; this just isn't possible with the way gRPC has designed those libraries.

On Thu, Oct 27, 2022, at 18:45, Steve Kim wrote:
> Thanks for the explanation!
> 
> On Fri, Oct 7, 2022, 06:52 David Li <li...@apache.org> wrote:
>> __
>> note that gRPC is not meant to be directly exposed so some gRPC codes do not have a direct equivalent.
> 
> This statement surprised me. What is reason for not exposing gRPC? A major advantage of Flight is that any gRPC client or server that knows the definitions in Flight.proto can speak Flight (without custom performance optimizations, but still correctly), and users can reason about Flight as just another gRPC service. I think that it would be unfortunate if Arrow Flight were to lose interoperability with the larger gRPC ecosystem.
> 
> Steve 
> 

Re: [flight] how does the server terminate a flight with status code?

[Steve Kim <ch...@gmail.com>]

Thanks for the explanation!

On Fri, Oct 7, 2022, 06:52 David Li <li...@apache.org> wrote:

> note that gRPC is not meant to be directly exposed so some gRPC codes do
> not have a direct equivalent.
>

This statement surprised me. What is reason for not exposing gRPC? A major
advantage of Flight is that any gRPC client or server that knows the
definitions in Flight.proto can speak Flight (without custom performance
optimizations, but still correctly), and users can reason about Flight as
just another gRPC service. I think that it would be unfortunate if Arrow
Flight were to lose interoperability with the larger gRPC ecosystem.

Steve

Re: [flight] how does the server terminate a flight with status code?

[David Li <li...@apache.org>]

Yes, a better API to differentiate the two is proposed in https://issues.apache.org/jira/browse/ARROW-16023

The 'missing' status codes are covered by standard exceptions (ValueError should map to INVALID_ARGUMENT), note that gRPC is not meant to be directly exposed so some gRPC codes do not have a direct equivalent.

-David

On Fri, Oct 7, 2022, at 00:28, Steve Kim wrote:
> I am writing a Flight server using pyarrow.flight. I want the client to receive INVALID_ARGUMENT status code when it sends an invalid ticket to the server. I see that there are some FlightError subclasses that are associated with status codes, but the set of covered status codes is incomplete. It feels awkward to rely on exceptions to control the server's response instead of explicitly aborting the Flight with a status code. Do I misunderstand how this ought to work? If pyarrow.flight is missing something, I am willing to contribute.
> 
> Thanks in advance for any help!
> Steve