You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Hitesh Shah (JIRA)" <ji...@apache.org> on 2013/12/27 18:18:00 UTC
[jira] [Comment Edited] (TEZ-684) Uber-mode for Tez
[ https://issues.apache.org/jira/browse/TEZ-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13857568#comment-13857568 ]
Hitesh Shah edited comment on TEZ-684 at 12/27/13 5:17 PM:
-----------------------------------------------------------
[~airbots] LocalContainerLauncher seems to be something which MR used. I doubt that concept will cleanly apply to Tez.
Local resources, environment(classspath, etc) are different for each vertex - how would these be handled?
Tasks in different vertices communicate via events - have you looked into how you plan to handle their routing?
Have you have made the decision of running tasks serially or in parallel?
In the future, there is a plan to support running inlined tasks within the AM i.e. a mix of inlined and remote tasks - the design should account for this. The MR uber AM made a call of either doing everything in one AM or all in remote tasks - this should definitely not be how the tez implementation should work.
was (Author: hitesh):
[~airbots] LocalContainerLauncher seems to be something which MR used. I doubt that concept will cleanly apply to Tez.
Local resources, environment(classspath, etc) are different for each vertex - how would these be handled?
Tasks in different vertices communicate via events - have you looked into how you plan to handle their routing?
Looks like you have made the decision of running tasks serially. Any reason they cannot be run in parallel?
In the future, there is a plan to support running inlined tasks within the AM i.e. a mix of inlined and remote tasks - the design should account for this. The MR uber AM made a call of either doing everything in one AM or all in remote tasks - this should definitely not be how the tez implementation should work.
> Uber-mode for Tez
> -----------------
>
> Key: TEZ-684
> URL: https://issues.apache.org/jira/browse/TEZ-684
> Project: Apache Tez
> Issue Type: New Feature
> Reporter: Chen He
>
> Similarly to MapReduce Uber-mode in Yarn, we plan to create the Uber-mode for Tez. It runs all tasks in local in one process.
> Our target is to start DAGAppMaster in local JVM and let it run all tasks in one process.
> Here is my design:
> Once user submits a DAG, Tez starts a instance of DAGAppMaster. This DAGAppMaster will check TezConfiguration before instantiate ContainerLauncher. If "is_Uber" is true, DAGAppMaster creates a LocalContainerLauncher. LocalTaskScheduler and LocalTaskSchedulerEventHandler will call LocalContainerLauncher to run all tasks one by one in a single JVM. Communications between ResourceManager and local classes (DAGAppMaster, LocalContainerLauncher, LocalTaskScheduler, and LocalTaskSchedulerEventHandler) are muted.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)