[jira] [Commented] (AVRO-2187) Add RPC Streaming constructs/keywords to Avro IDL or schema

["Srujan Narkedamalli (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AVRO-2187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16530314#comment-16530314 ] 

Srujan Narkedamalli commented on AVRO-2187:
-------------------------------------------

Thanks for your response and very sorry for the delay in getting back.

Avro's one-way is quite useful and unique, but the streaming that I was referring for to is a bit different and more in line with what gRPC offers.

What I mentioned as server streaming rpc is a single rpc request by client that yeilds a repeated response from server (of defined type). An example of this be a request by client for fetching a list of files from a storage like s3 and each of file is sent as response of for rpc request. Similarly for client-streaming rpc, multiple requests (of defined type) can be sent for single requests and sever sends back a single response. In bi-directional streaming both multiple requests and responses can be sent but all in context of single RPC. The difference/improvement from one-way calls is being able to receive multiple responses or send multiple requests in context of a single RPC.

In order to use such streaming calls provided by gRPC in Avro's IDL, we are currently annotating the calls with our custom annotations and modified the velocity templates for stub code generation for these. Instead of these annotations we would prefer to have this information specified as a Avro keyword to make sure its compatible for different languages and info is in the schema.

One way I see for this change to be backward compatible is, every rpc marked as streaming yeilds two rpcs: regular rpc call and in addition a streaming call with call name appended streaming probably and has the property streaming. Generating the additional streaming rpc could be an option.

Our avro avdl file for a server streaming RPC looks like this:
{code:java}
@(io.grpc.ServerStreaming)
File searchByNameAndStreamFiles(string name);

//our custom modified generated code is as below, (StreamObserver is gRPC API):
void searchByNameAndStreamFiles(string name, StreamObserver<File> name); {code}
 

We would prefer to have the above schema be represented as:
{code:java}
File searchByNameAndStreamFiles(string name) serverstreaming;
{code}
For comparison: gRPC's preferred IDL- Protobuf, represent a similar server streaming call as:
{code:java}
rpc searchByNameAndStreamFiles (SearchRequest) returns (stream File) {}
{code}

> Add RPC Streaming constructs/keywords to Avro IDL or schema
> -----------------------------------------------------------
>
>                 Key: AVRO-2187
>                 URL: https://issues.apache.org/jira/browse/AVRO-2187
>             Project: Avro
>          Issue Type: New Feature
>            Reporter: Srujan Narkedamalli
>            Priority: Major
>
> Motivation:
> We recently added support for transporting Avro serialization and IDL over gRPC for Java. In order to use the streaming features of gRPC or any other transport that supports streaming we need to be able to specify them IDL and schema.
> Details:
> Currently, gRPC supports 3 types of streaming calls:
>  # server streaming (server can send multiple responses for a single request)
>  # client streaming (client can multiple requests and server sends a single response)
>  # bi-directional streaming call (on going rpc with multiple requests and responses)
> We would want a way to represent these types on calls in Avro's IDL similar to one-way calls using a keywords. Usually in gRPC with other IDLs a streaming request or response is repeated payload of same type. For client streaming and bi-directional streaming it would be simpler to have a single request argument when representing their type in callbacks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)