[jira] [Created] (FLINK-1709) Add support for programs with higher-than-slot parallelism

["Ufuk Celebi (JIRA)" <ji...@apache.org>]

Ufuk Celebi created FLINK-1709:
----------------------------------

             Summary: Add support for programs with higher-than-slot parallelism
                 Key: FLINK-1709
                 URL: https://issues.apache.org/jira/browse/FLINK-1709
             Project: Flink
          Issue Type: Improvement
    Affects Versions: master
            Reporter: Ufuk Celebi


Currently, we can't run programs with higher parallelism than available slots.

For example, if we currently have a map-reduce program and 4 task slots configured (e.g. 2 task managers with 2 slots per task manager), the map and reduce tasks will be scheduled with pipelined results and the same parallelism in shared slots. Setting the parallelism to more than available slots will result in a NoResourcesAvailableException.

As a first step to support these kinds of programs, we can add initial support for this when running in batch mode (after https://github.com/apache/flink/pull/471 is merged).

This is easier than the original pipelined scenario, because the map tasks can be deployed after each other to produce the blocking result. The blocking result can then be consumed after all map tasks produced their result. The mechanism in #471 to deploy result receivers can be used for this and should not need any modifications.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)