You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Leanken.Lin (JIRA)" <ji...@apache.org> on 2018/09/28 02:22:00 UTC
[jira] [Comment Edited] (FLINK-4399) Add support for oversized
messages
[ https://issues.apache.org/jira/browse/FLINK-4399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16631275#comment-16631275 ]
Leanken.Lin edited comment on FLINK-4399 at 9/28/18 2:21 AM:
-------------------------------------------------------------
Hi. [~till.rohrmann]
Do we have some update on this issue?
was (Author: leanken):
Hi. [~till.rohrmann]
> Add support for oversized messages
> ----------------------------------
>
> Key: FLINK-4399
> URL: https://issues.apache.org/jira/browse/FLINK-4399
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination
> Environment: FLIP-6 feature branch
> Reporter: Stephan Ewen
> Priority: Major
> Labels: flip-6
> Fix For: 1.7.0
>
>
> Currently, messages larger than the maximum Akka Framesize cause an error when being transported. We should add a way to pass messages that are larger than the Framesize, as may happen for:
> - {{collect()}} calls that collect large data sets (via accumulators)
> - Job submissions and operator deployments where the functions closures are large (for example because it contains large pre-loaded data)
> - Function restore in cases where restored state is larger than checkpointed state (union state)
> I suggest to use the {{BlobManager}} to transfer large payload.
> - On the sender side, oversized messages are stored under a transient blob (which is deleted after first retrieval, or after a certain number of minutes)
> - The sender sends a "pointer to blob message" instead.
> - The receiver grabs the message from the blob upon receiving the pointer message
> The RPC Service should be optionally initializable with a "large message handler" which is internally the {{BlobManager}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)