[jira] [Created] (FLINK-4399) Add support for oversized messages

["Stephan Ewen (JIRA)" <ji...@apache.org>]

Stephan Ewen created FLINK-4399:
-----------------------------------

             Summary: Add support for oversized messages
                 Key: FLINK-4399
                 URL: https://issues.apache.org/jira/browse/FLINK-4399
             Project: Flink
          Issue Type: Sub-task
          Components: Distributed Coordination
         Environment: FLIP-6 feature branch
            Reporter: Stephan Ewen
             Fix For: 1.2.0


Currently, messages larger than the maximum Akka Framesize cause an error when being transported. We should add a way to pass messages that are larger than the Framesize, as may happen for:

  - {{collect()}} calls that collect large data sets (via accumulators)
  - Job submissions and operator deployments where the functions closures are large (for example because it contains large pre-loaded data)
  - Function restore in cases where restored state is larger than checkpointed state (union state)


I suggest to use the {{BlobManager}} to transfer large payload.

  - On the sender side, oversized messages are stored under a transient blob (which is deleted after first retrieval, or after a certain number of minutes)
  - The sender sends a "pointer to blob message" instead.
  - The receiver grabs the message from the blob upon receiving the pointer message


The RPC Service should be optionally initializable with a "large message handler" which is internally the {{BlobManager}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)