[jira] [Updated] (TEZ-235) Tez Local Mode

["Achal Soni (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TEZ-235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Achal Soni updated TEZ-235:
---------------------------

    Attachment: new_local_mode.patch

Here is the patch for the new local mode in Tez. 

IDConverters had to be changed to handle local mode. For now I have local mode identified by an id of -1 (not sure what good practice would be, so I will leave it as is for now). 

Some configurations had to be set such as TezJobConfig.JOB_LOCAL_DIR and TezJobConfig.TASK_LOCAL_RESOURCE_DIR. 

I changed LocalOnFileSorter to no longer rename the output to task_0. For now this is commented this out. It was breaking the logic of copying the output of the map into the reducer. I feel that this is a better approach because in the future, we may need to copy the output of the vertices in different ways and this responsibility should not lie in LocalOutputFileSorter. It seems to me that if this is acceptable, then LocalOutputFileSorter is identical to OutputFileSorter. Along with this change was changing the TestMapProcessor and TestReduceProcessor to copy the outputs appropriately.

I also ensured the OutputCommitter was working, so now the part-r-00000 file is in the actual directory. This was a very small change (OutputCommitter was looking in the wrong directory, so had to point it to the correct one, but took some digging to find this mistake). 

I also changed the code to allow for the MRR cases. This assumes that everything is submitted via the JobClient, and uses the Convert Tez to Linear conf files and all that. I will swap out this approach when I work out the more generic local runner in the future, as that is better. For the time being, this implementation will do. Also changed some stuff up to use TezTaskAttempts and all instead of having to use IDConverters. 


                
> Tez Local Mode 
> ---------------
>
>                 Key: TEZ-235
>                 URL: https://issues.apache.org/jira/browse/TEZ-235
>             Project: Apache Tez
>          Issue Type: Improvement
>    Affects Versions: 0.2.0
>            Reporter: Achal Soni
>             Fix For: 0.2.0
>
>         Attachments: new_local_mode.patch
>
>
> The Tez local mode has problems in several places. Certain configurations aren't set, IDConverter fails on local mode for the Job Application id, the reduce processor cannot find the input of the map processor, and the OutputCommitter does not work so all output is sitting in the _temporary dir. 
> Additionally local mode is restricted to only MR jobs. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira