You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by "Andrew Chung (JIRA)" <ji...@apache.org> on 2015/12/08 22:07:10 UTC

[jira] [Created] (REEF-1052) A DAG task scheduler on REEF

Andrew Chung created REEF-1052:
----------------------------------

             Summary: A DAG task scheduler on REEF
                 Key: REEF-1052
                 URL: https://issues.apache.org/jira/browse/REEF-1052
             Project: REEF
          Issue Type: New Feature
          Components: REEF
            Reporter: Andrew Chung


Big data computation workloads can often be constructed as a directed acyclic graph (DAG) of interdependent stages/modules. An application that exposes a task scheduling API for such DAGs will be immensely useful such that the DAG can be run as a single REEF job instead of as multiple jobs that  input from/output to a shared distributed storage and are scheduled independently of each other to the same cluster/resource manager. With a single task scheduler that is entirely aware of the DAG structure and tasks scheduled by each stage in the DAG, many things can be optimized.

For example a few benefits, among others, are:
# Cost of starting a job and managing job resources are reduced
# Resource locality is improved and access costs are reduced
# Stage/task parallelism is fully exploited

This can be further integrated with the work done in REEF-580 to improve task scheduling properties.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)