[jira] [Commented] (FLINK-6046) Add support for oversized messages during deployment

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679204#comment-16679204 ] 

ASF GitHub Bot commented on FLINK-6046:
---------------------------------------

TisonKun opened a new pull request #7050: [hotfix] Follow up FLINK-6046 to clean unused variable
URL: https://github.com/apache/flink/pull/7050
 
 
   ## What is the purpose of the change
   
   After FLINK-6046 5ff07e63d1e9a98959e5edf66872222b847d23d5 resolved, we replaced `serializedTaskInformation` and `taskInformationBlobKey` with `taskInformationOrBlobKey`. This pull requests acts as a follow up to clean the unused variables and update document.
   
   ## Verifying this change
   
   This change is a code cleanup without any test coverage.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (**no**)
     - The public API, i.e., is any changed class annotated with `(Evolving)`: (**no**)
     - The serializers: (**no**)
     - The runtime per-record code paths (performance sensitive): (**no**)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (**no**)
     - The S3 file system connector: (**no**)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (**no**)
     - If yes, how is the feature documented? (**not applicable**)
   
   cc @tillrohrmann @zentol 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Add support for oversized messages during deployment
> ----------------------------------------------------
>
>                 Key: FLINK-6046
>                 URL: https://issues.apache.org/jira/browse/FLINK-6046
>             Project: Flink
>          Issue Type: New Feature
>          Components: Distributed Coordination, Network
>            Reporter: Nico Kruber
>            Assignee: Nico Kruber
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.4.0
>
>
> This is the non-FLIP6 version of FLINK-4346, restricted to deployment messages:
> Currently, messages larger than the maximum Akka Framesize cause an error when being transported. We should add a way to pass messages that are larger than {{akka.framesize}} as may happen for task deployments via the {{TaskDeploymentDescriptor}}.
> We should use the {{BlobServer}} to offload big data items (if possible) and make use of any potential distributed file system behind. This way, not only do we avoid the akka framesize restriction, but may also be able to speed up deployment.
> I suggest the following changes:
>   - the sender, i.e. the {{Execution}} class, tries to store the serialized job information and serialized task information (if oversized) from the {{TaskDeploymentDescriptor}} (tdd) on the {{BlobServer}} as a single {{NAME_ADDRESSABLE}} blob under its job ID (if this does not work, we send the whole tdd as usual via akka)
>   - if stored in a blob, these data items are removed from the tdd
>   - the receiver, i.e. the {{TaskManager}} class, tries to retrieve any offloaded data after receiving the {{TaskDeploymentDescriptor}} from akka; it re-assembles the original tdd
>   - the stored blob may be deleted after re-assembly of the tdd
> Further (future) changes may include:
>   - separating the serialized job information and serialized task information into two files and re-use the first one for all tasks
>   - not re-deploying these two during job recovery (if possible)
>   - then, as all other {{NAME_ADDRESSABLE}} blobs, these offloaded blobs may be removed when the job enters a final state instead



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)