[jira] [Updated] (RATIS-759) Support stream APIs to send large messages

["Tsz-wo Sze (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/RATIS-759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz-wo Sze updated RATIS-759:
-----------------------------
    Description: 
It is inefficient to send a large message using send(Message)/sendAsync(Message) in RaftClient.  We already have RaftOutputStream implemented with sendAsync(..).  We propose adding the following new APIs
{code}
  /** Create a stream to send a large message. */
  MessageOutputStream stream();

  /** Send the given message using a stream. */
  CompletableFuture<RaftClientReply> streamAsync(Message message);
{code} 

Why it is inefficient? Protobuf is not designed to handle large messages; see
https://developers.google.com/protocol-buffers/docs/techniques#large-data

Some details: Support we create a 100MB protobuf message.  Protobuf
and Grpc internally create multiple 100MB-buffers.  These buffers
won't be released until the entire message is processed.  If we stream
the message using one hundred 1MB-sub-messages.  The buffer for the
earlier sub-messages can be released when processing the later
sub-messages.

  was:
It is inefficient to send a large message using send(Message)/sendAsync(Message) in RaftClient.  We already have RaftOutputStream implemented with sendAsync(..).  We propose adding the following new APIs
{code}
  /** Create a stream to send a large message. */
  MessageOutputStream stream();

  /** Send the given message using a stream. */
  CompletableFuture<RaftClientReply> streamAsync(Message message);
{code} 


> Support stream APIs to send large messages
> ------------------------------------------
>
>                 Key: RATIS-759
>                 URL: https://issues.apache.org/jira/browse/RATIS-759
>             Project: Ratis
>          Issue Type: New Feature
>          Components: client, server
>            Reporter: Tsz-wo Sze
>            Assignee: Tsz-wo Sze
>            Priority: Major
>             Fix For: 0.5.0
>
>         Attachments: r759_20200115.patch, r759_20200123.patch, r759_20200204.patch, r759_20200206.patch
>
>
> It is inefficient to send a large message using send(Message)/sendAsync(Message) in RaftClient.  We already have RaftOutputStream implemented with sendAsync(..).  We propose adding the following new APIs
> {code}
>   /** Create a stream to send a large message. */
>   MessageOutputStream stream();
>   /** Send the given message using a stream. */
>   CompletableFuture<RaftClientReply> streamAsync(Message message);
> {code} 
> Why it is inefficient? Protobuf is not designed to handle large messages; see
> https://developers.google.com/protocol-buffers/docs/techniques#large-data
> Some details: Support we create a 100MB protobuf message.  Protobuf
> and Grpc internally create multiple 100MB-buffers.  These buffers
> won't be released until the entire message is processed.  If we stream
> the message using one hundred 1MB-sub-messages.  The buffer for the
> earlier sub-messages can be released when processing the later
> sub-messages.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)