[jira] [Created] (FLINK-10643) Bubble execution: Resource aware job execution

["JIN SUN (JIRA)" <ji...@apache.org>]

JIN SUN created FLINK-10643:
-------------------------------

             Summary: Bubble execution: Resource aware job execution
                 Key: FLINK-10643
                 URL: https://issues.apache.org/jira/browse/FLINK-10643
             Project: Flink
          Issue Type: New Feature
          Components: JobManager
            Reporter: JIN SUN
            Assignee: JIN SUN
             Fix For: 1.8.0
         Attachments: image-2018-10-22-16-28-32-355.png

Today Flink support various channels such as pipelined channel and blocking channel. Blocking channel indicate that data need to be persistent in a batch and then it can be consumed later, it also indicate that the downstream task cannot start to process data unless its producer finished and also downstream task will only depends on this intermediate partition instead of upstream tasks. 

By leverage this characteristic, Flink already support fine grain-failover which will build a failover region has reduce failover cost.  However, we can leverage this characteristic even more. As described by this [paper|http://www.vldb.org/pvldb/vol11/p746-yin.pdf] (VLDB 2018), *_Bubble Execution_* not only use this characteristic to implement fine-grain failover, but also use this to balance the resource utilization and job performance. As shown in the paper (also in the following chart), with 50% of the resource, it get 25% (0.75 speedup) average slow down for TPCH benchmark.

!image-2018-10-22-16-28-32-355.png!

This JIRA here is umbrella that try to apply the idea of this paper to FLINK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)