[jira] Commented: (HIVE-69) genMapRedTasks does not use the tree walker and uses implicit state which makes it difficult to enhance

["Ashish Thusoo (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-69?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648792#action_12648792 ] 

Ashish Thusoo commented on HIVE-69:
-----------------------------------

General comment - need a lot of javadocs and documentation of how the state machine works. I think it works correctly but lets document as I have indicated below.

Also the state should not be maintained in the plan but should be in the treewalker. The state is never really used in the execution time and we should only try to put things that are used in the execution time in the plan.

Otherwise, this looks much cleaner than the pervious monolithic blob that we had. Thanks...
Inline Comments
ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java:32	Is this just a place holder right now?
ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java:112	Lets add proper javadocs!!
ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java:132	javadocs. What is this used for?
ql/src/java/org/apache/hadoop/hive/ql/parse/GenMapRedTaskProcessor.java:57	Move the new into the constructor?
ql/src/java/org/apache/hadoop/hive/ql/parse/GenMapRedTaskProcessor.java:1	javadocs needed.
ql/src/java/org/apache/hadoop/hive/ql/parse/GenMapRedTaskProcessor.java:117	Please put a comment on how this state machine works. What are the transitions, how the transitions are made and what are the actions taken by each transition.
ql/src/java/org/apache/hadoop/hive/ql/plan/mapredWork.java:28	This is purely compile time information that is used to generate the plan (cut the plan into multiple map reduce plans), so we should not be storing this in the plan. This seems to be specific to the GenMapRedPlanWalker, so I think we should be maintaining this state there in a hashmap.
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:2986	Looks much cleaner with the walker stuff... This is cool...
ql/src/java/org/apache/hadoop/hive/ql/parse/GenMapRedWalker.java:44	I thought we were going to move this logic into the dispatcher? no?
ql/src/java/org/apache/hadoop/hive/ql/parse/GenMapRedTaskProcessor.java:116	Also a description of the call stack that leads to there transitions would be very helpful.


> genMapRedTasks does not use the tree walker and uses implicit state which makes it difficult to enhance
> -------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-69
>                 URL: https://issues.apache.org/jira/browse/HIVE-69
>             Project: Hadoop Hive
>          Issue Type: Bug
>            Reporter: Namit Jain
>            Assignee: Namit Jain
>
> In SemanticAnalyzer, genmapredtasks() does not use a tree walker. For map-side joins, the taskplan needs to be enhanced to be possibly
> broken at MapSink also. Basically, the code is very difficult to enhance since there are implicit assumptions that reduce sink is the only
> operator where the plan breaks.
> This should be enhanced so that the user can implement their own task generation logic which is independent of the tree walking.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.