[jira] [Updated] (FLINK-13247) Implement external shuffle service for YARN

["Flink Jira Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-13247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Flink Jira Bot updated FLINK-13247:
-----------------------------------
    Labels: stale-major  (was: )

I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized.


> Implement external shuffle service for YARN
> -------------------------------------------
>
>                 Key: FLINK-13247
>                 URL: https://issues.apache.org/jira/browse/FLINK-13247
>             Project: Flink
>          Issue Type: New Feature
>          Components: Runtime / Network
>            Reporter: MalcolmSanders
>            Priority: Major
>              Labels: stale-major
>
> Flink batch job users could achieve better cluster utilization and job throughput throught external shuffle service because the producers of intermedia result partitions can be released once intermedia result partitions have been persisted on disks. In [FLINK-10653|https://issues.apache.org/jira/browse/FLINK-10653], [~zjwang] has introduced pluggable shuffle manager architecture which abstracts the process of data transfer between stages from flink runtime as shuffle service. I propose to YARN implementation for flink external shuffle service since YARN is widely used in various companies.
> The basic idea is as follows:
> (1) Producers write intermedia result partitions to local disks assigned by NodeManager;
> (2) Yarn shuffle servers, deployed on each NodeManager as an auxiliary service, are acknowledged of intermedia result partition descriptions by producers;
> (3) Consumers fetch intermedia result partition from yarn shuffle servers;



--
This message was sent by Atlassian Jira
(v8.3.4#803005)