[jira] Commented: (HIVE-1100) Make it possible for users to retry map-reduce jobs in a single Hive query

["Zheng Shao (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804827#action_12804827 ] 

Zheng Shao commented on HIVE-1100:
----------------------------------

Goal:
There are 2 types of resume that we want:
1. Resume within the same session
2. Resume in a different session

Plan:
1. Currently the hive query compilation uses "random numbers" for intermediate data file directories. We should move all the indeterministic parts to "Driver" class.
2. Driver class will generate such a "random number" per session per query (let's call it ExecutionId), and store it in the HiveConf.
3. Driver.compile() should pass ExecutionId onto query compilation so that temp directories are deterministic given the ExecutionId.
4. Driver.execute() should record the completion of each of the tasks by "touching" a file in HDFS.
5. We should add a command "continue" that will continue the execution of the failed query within the session
6. We should be able to get the ExecutionId by "set" command so that we can continue it later.
7. We should add a command "clear" to remove the temporaries files in HDFS given the "ExecutionId".

 

> Make it possible for users to retry map-reduce jobs in a single Hive query
> --------------------------------------------------------------------------
>
>                 Key: HIVE-1100
>                 URL: https://issues.apache.org/jira/browse/HIVE-1100
>             Project: Hadoop Hive
>          Issue Type: New Feature
>    Affects Versions: 0.6.0
>            Reporter: Zheng Shao
>            Assignee: Zheng Shao
>
> Sometimes a single hive query get compiled into several map-reduce jobs, and one of the jobs failed because of some transient errors.
> Users would need to start from scratch.
> We should allow the user to start from the point of failure to continue the query.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.