You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Siddharth Seth (JIRA)" <ji...@apache.org> on 2013/06/20 22:46:20 UTC
[jira] [Commented] (TEZ-235) Tez Local Mode
[ https://issues.apache.org/jira/browse/TEZ-235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689641#comment-13689641 ]
Siddharth Seth commented on TEZ-235:
------------------------------------
Achal, thanks for working on this. The current patch includes changes from several other commits. Could you please upload a new patch which is limited to the LocalJobRunner changes.
Some comments on the patch itself
- We should avoid changing the defaults for the existing WordCountMRRTest. Can be replaced with a new class or parameters specifying stages.
- There's some references to MRRDag. The class itself is missing. What is this class for
- I think using LocalOutputFileSorter is acceptable - which is linked to LocalMergedInput/LocalShuffle. The producer-consumer will be used together. This is for a linear chain. If this is extended at a future point to handle DAGs, we can re-look at how the output is moved. My guess is a custom Ouput could be written for that case as well.
- There's some additional imports for JobClient - not sure why that is required.
- A unit test for the LJR will be helpful as well - since it's very easy to break with runtime changes.
> Tez Local Mode
> ---------------
>
> Key: TEZ-235
> URL: https://issues.apache.org/jira/browse/TEZ-235
> Project: Apache Tez
> Issue Type: Improvement
> Affects Versions: 0.2.0
> Reporter: Achal Soni
> Fix For: 0.2.0
>
> Attachments: 1.txt, new_local_mode.patch
>
>
> The Tez local mode has problems in several places. Certain configurations aren't set, IDConverter fails on local mode for the Job Application id, the reduce processor cannot find the input of the map processor, and the OutputCommitter does not work so all output is sitting in the _temporary dir.
> Additionally local mode is restricted to only MR jobs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira