[jira] [Comment Edited] (TEZ-707) Create LocalContainerLauncher

["Chen He (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14075775#comment-14075775 ] 

Chen He edited comment on TEZ-707 at 7/27/14 10:31 PM:
-------------------------------------------------------

I am still confusing about why we need future type of tasks running in local mode after I investigate the Callback and ListeningExecutorService mechanism. 

Callback mechanism allows us to get return of task, ListeningExecutorService provides manageable async way to run tasks. We can get return of each running task by using above mechanisms. However, In current version of  LocalContainerLauncher, threadspool is used to control the parallelism of running tasks. We do blocking if running tasks reach the poolsize and sleep TezSubTaskRunner. Actually, tasks come and go, we care the parallelism more than the return of each task. Since a task failed, it will record failure in log (since we are running in a single process). 

Please correct me if there is misunderstanding. What you feel, [~sseth]?


was (Author: airbots):
I am still confusing about why we need future type of tasks running in local mode after I investigate the Callback and ListeningServiceFuture mechanism. 

Callback mechanism allows us to get return of task, ListeningServiceFuture provides manageable async way to run tasks. We can get return of each running task by using above mechanisms. However, In current version of  LocalContainerLauncher, threadspool is used to control the parallelism of running tasks. We do blocking if running tasks reach the poolsize and sleep TezSubTaskRunner. Actually, tasks come and go, we care the parallelism more than the return of each task. Since a task failed, it will record failure in log (since we are running in a single process). 

Please correct me if there is misunderstanding. What you feel, [~sseth]?

> Create LocalContainerLauncher
> -----------------------------
>
>                 Key: TEZ-707
>                 URL: https://issues.apache.org/jira/browse/TEZ-707
>             Project: Apache Tez
>          Issue Type: Sub-task
>    Affects Versions: 0.3.0
>            Reporter: Chen He
>            Assignee: Chen He
>            Priority: Blocker
>         Attachments: TEZ-707-2014-7015.patch, TEZ-707-2014-7015.patch.review, TEZ-707-v3.patch, TEZ-707-v4.patch, TEZ-707.patch, TEZ-707.patch, Tez-707.patch.v2, tez-707.patch
>
>
> Create LocalContainerLauncher and make it work for a single stage DAG. The TaskSchedulerEventHandler still asks RM for new container but LocalContainerLauncher will run TezTask in form of thread instead of using this container from yarn. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)