[jira] [Comment Edited] (ARROW-249) [Format] Define GRPC IDL / wire protocol for messaging with Arrow data

["Wes McKinney (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009556#comment-16009556 ] 

Wes McKinney edited comment on ARROW-249 at 5/14/17 12:15 AM:
--------------------------------------------------------------

I'd like to prototype a GRPC-based client and server that sends components of an Arrow stream as a series of RPC messages according to the Arrow streaming format. See this discussion about some issues around this

https://github.com/grpc/grpc-java/issues/1054#issuecomment-301282237

In ARROW-1008, I've refactored the "stream reader" and "stream writer" concepts to be abstract interfaces, so now we could define {{GrpcStreamWriter}} and {{GrpcStreamReader}} that would be a middleman to the underlying GRPC client/server machinery, presenting a standard stream iterator to the client side. I figure we could prototype this using protobuf messages that hold each piece of the Arrow stream (first RPC message is the schema) and see how the perf compares with a "raw" stream. 

cc [~nongli]


was (Author: wesmckinn):
I'd like to prototype a GRPC-based client and serve that sends components of an Arrow stream as a series of RPC messages according to the Arrow streaming format. See this discussion about some issues around this

https://github.com/grpc/grpc-java/issues/1054#issuecomment-301282237

In ARROW-1008, I've refactored the "stream reader" and "stream writer" concepts to be abstract interfaces, so now we could define {{GrpcStreamWriter}} and {{GrpcStreamReader}} that would be a middleman to the underlying GRPC client/server machinery, presenting a standard stream iterator to the client side. I figure we could prototype this using protobuf messages that hold each piece of the Arrow stream (first RPC message is the schema) and see how the perf compares with a "raw" stream. 

cc [~nongli]

> [Format] Define GRPC IDL / wire protocol for messaging with Arrow data
> ----------------------------------------------------------------------
>
>                 Key: ARROW-249
>                 URL: https://issues.apache.org/jira/browse/ARROW-249
>             Project: Apache Arrow
>          Issue Type: New Feature
>          Components: Format
>            Reporter: Wes McKinney
>             Fix For: 0.4.0
>
>
> In addition to memory maps / shared memory, we should be able to assemble one or more Arrow arrays in a memory block suitable for sending over the wire with GRPC (http://www.grpc.io/). This can be similarly adapted to other messaging / RPC frameworks. 
> We can continue to use Flatbuffers for the metadata serialization. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)