[GitHub] [pinot] siddharthteotia commented on issue #6921: Arrow Flight Endpoint for Pinot

[GitBox <gi...@apache.org>]

siddharthteotia commented on issue #6921:
URL: https://github.com/apache/pinot/issues/6921#issuecomment-976049441


   Sorry, I somehow missed this discussion thread. 
   
   @kishoreg @mayankshriv @lfeagan - Arrow is a specification rather than implementation (has mulitple programming lang impls).
   
   My main intention behind this issue was from database connectivity point of view. So like JDBC, ODBC, this will be another connector to Pinot and the binary protocol will be based on Arrow spec and the transport will be via GRPC. This was not necessarily for server to broker communication but instead client to broker transport where the clients can just get the PinotResultSet as Arrow buffers from the wire into memory without copy / SerDe and continue further processing on Arrow data -- common scenario in Python / Data Science world from what I can see.
   
   Moving server to broker DataTable format to Arrow spec can also be considered
   
   PS - this is also not related to underlying disk/memory format being based on Arrow. As you mentioned, our format is already well optimized for the engine.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org