[GitHub] [arrow] lidavidm opened a new issue, #37937: [C++][FlightRPC] Investigate using generic callback stub to bypass Protobuf serialization

["lidavidm (via GitHub)" <gi...@apache.org>]

lidavidm opened a new issue, #37937:
URL: https://github.com/apache/arrow/issues/37937

   ### Describe the enhancement requested
   
   Flight in C++ bypass the Protobuf serializer by specializing a gRPC template and doing what is almost certainly an illegal cast to trick gRPC into using our specialization. However, gRPC supports a ["generic" API](https://github.com/grpc/grpc/issues/33035#issuecomment-1544701100) that lets you call methods by name and get back the gRPC byte buffers, which should be a safe, officially sanctioned way of doing what we want. The API linked there is only applicable to async gRPC, so it would only help our new async implementation.
   
   This has a few other benefits:
   - The gRPC template we specialize technically lets us return (de)serialization errors, but in practice gRPC crashes if you error. This new API would let us handle the error gracefully in Flight code.
   - We could more easily pass in other arguments, like a memory allocator, to the (de)serialization code. (gRPC still controls memory allocation, though.)
   - We could implement more complex deserialization code (e.g. optionally trying to align buffers for #32276).
   
   ### Component(s)
   
   C++, FlightRPC


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [C++][FlightRPC] Investigate using generic callback stub to bypass Protobuf serialization [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on issue #37937:
URL: https://github.com/apache/arrow/issues/37937#issuecomment-1766615140

   IIUC, the idea is to use gRPC async APIs to handle serialization and deserialization ourselves ?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [C++][FlightRPC] Investigate using generic callback stub to bypass Protobuf serialization [arrow]

["lidavidm (via GitHub)" <gi...@apache.org>]

lidavidm commented on issue #37937:
URL: https://github.com/apache/arrow/issues/37937#issuecomment-1766703436

   Yes:
   
   - Right now, we do a cast that is definitely UB to try to hook into the gRPC serialization machinery
   - gRPC offers a "generic" stub that gives us the raw byte buffers directly without that hack, but it is only available for the async API
   - The gRPC serialization machinery is known to be broken (e.g. if you raise an error gRPC will just abort instead) so being able to bypass it entirely is a benefit
   
   So when we implement the async version of DoGet etc we should just try to use the generic stub and avoid all this in the first place (requires some refactoring, though)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org