[jira] [Comment Edited] (TEZ-1272) Change YARNRunner to make use of EdgeConfigurations

["Bikas Saha (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-1272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14059562#comment-14059562 ] 

Bikas Saha edited comment on TEZ-1272 at 7/12/14 1:06 AM:
----------------------------------------------------------

For the following replacement.
{code}-    MultiStageMRConfToTezTranslator.translateVertexConfToTez(finalReduceConf,
-        iReduceStageConf);
+    MRHelpers.translateVertexConfToTez(finalReduceConf);{code}
The new method is not calling the following and this actually does something regardless of predecessorConf. So the new code may be missing that. I am guessing you are doing that deliberately. Mind describing the reasoning? Is it because that was only handling the input/output duplication which is not needed now?
{code}    translateMultiStageWithPredecessor(vertexConf, predecessorConf) {code}

How did this ever work?
{code}-    registerMRToRuntimeKeyTranslation(TezJobConfig.TEZ_CREDENTIALS_PATH, MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY);
+    registerMRToRuntimeKeyTranslation(MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY, TezJobConfig.TEZ_CREDENTIALS_PATH);
{code}

This one and others like it, there is probably no need to INTERMEDIATE in the name, right? e.g. OnFileUnorderedOuput could write final output if so desired?
{code}TEZ_RUNTIME_INTERMEDIATE_INPUT_COMPRESS_CODEC{code}

Orthogonally, does OrderedPartitionedKVEdgeConfigurator sound better than OrderedPartitionedKVEdgeConfiguration. Reasons, its the entity that configures instead of being the configuration itself. It actually does more like create edge properties. And finally, it disconnects the name from Configuration.

Does this unblock TEZ-696?

Looks good.




was (Author: bikassaha):
For the following replacement.
{code}-    MultiStageMRConfToTezTranslator.translateVertexConfToTez(finalReduceConf,
-        iReduceStageConf);
+    MRHelpers.translateVertexConfToTez(finalReduceConf);{code}
The new method is not calling the following and this actually does something regardless of predecessorConf. So the new code may be missing that. I am guessing you are doing that deliberately. Mind describing the reasoning?
{code}    translateMultiStageWithPredecessor(vertexConf, predecessorConf) {code}

How did this ever work?
{code}-    registerMRToRuntimeKeyTranslation(TezJobConfig.TEZ_CREDENTIALS_PATH, MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY);
+    registerMRToRuntimeKeyTranslation(MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY, TezJobConfig.TEZ_CREDENTIALS_PATH);
{code}

This one and others like it, there is probably no need to INTERMEDIATE in the name, right? e.g. OnFileUnorderedOuput could write final output if so desired?
{code}TEZ_RUNTIME_INTERMEDIATE_INPUT_COMPRESS_CODEC{code}

Orthogonally, does OrderedPartitionedKVEdgeConfigurator sound better than OrderedPartitionedKVEdgeConfiguration. Reasons, its the entity that configures instead of being the configuration itself. It actually does more like create edge properties. And finally, it disconnects the name from Configuration.

Looks good.



> Change YARNRunner to make use of EdgeConfigurations
> ---------------------------------------------------
>
>                 Key: TEZ-1272
>                 URL: https://issues.apache.org/jira/browse/TEZ-1272
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Siddharth Seth
>            Assignee: Siddharth Seth
>         Attachments: TEZ-1272.1.txt
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)