You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Zheng Shao (JIRA)" <ji...@apache.org> on 2010/01/26 01:06:34 UTC
[jira] Created: (HIVE-1100) retry
retry
------
Key: HIVE-1100
URL: https://issues.apache.org/jira/browse/HIVE-1100
Project: Hadoop Hive
Issue Type: New Feature
Reporter: Zheng Shao
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1100) Make it possible for users to retry
map-reduce jobs in a single Hive query
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804848#action_12804848 ]
Zheng Shao commented on HIVE-1100:
----------------------------------
bq. Is it a timeout based approach? When does the cleanup happen if the client never issues a clear/continue ?
Actually we also have this problem today - if Hive failed before it cleans up the temporary data, but with this feature the problem will be much more obvious.
We can add the date and time into "ExecutionId", so that an external process can come in and clean things up.
bq. Do we need to enhance explain plan or some other tool so that the client knows which tasks will the query execute ?
That's a good idea.
8. Add the "done" information for each task to explain plan so that users can know the status of the query.
> Make it possible for users to retry map-reduce jobs in a single Hive query
> --------------------------------------------------------------------------
>
> Key: HIVE-1100
> URL: https://issues.apache.org/jira/browse/HIVE-1100
> Project: Hadoop Hive
> Issue Type: New Feature
> Affects Versions: 0.6.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
>
> Sometimes a single hive query get compiled into several map-reduce jobs, and one of the jobs failed because of some transient errors.
> Users would need to start from scratch.
> We should allow the user to start from the point of failure to continue the query.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1100) Make it possible for users to retry
map-reduce jobs in a single Hive query
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804827#action_12804827 ]
Zheng Shao commented on HIVE-1100:
----------------------------------
Goal:
There are 2 types of resume that we want:
1. Resume within the same session
2. Resume in a different session
Plan:
1. Currently the hive query compilation uses "random numbers" for intermediate data file directories. We should move all the indeterministic parts to "Driver" class.
2. Driver class will generate such a "random number" per session per query (let's call it ExecutionId), and store it in the HiveConf.
3. Driver.compile() should pass ExecutionId onto query compilation so that temp directories are deterministic given the ExecutionId.
4. Driver.execute() should record the completion of each of the tasks by "touching" a file in HDFS.
5. We should add a command "continue" that will continue the execution of the failed query within the session
6. We should be able to get the ExecutionId by "set" command so that we can continue it later.
7. We should add a command "clear" to remove the temporaries files in HDFS given the "ExecutionId".
> Make it possible for users to retry map-reduce jobs in a single Hive query
> --------------------------------------------------------------------------
>
> Key: HIVE-1100
> URL: https://issues.apache.org/jira/browse/HIVE-1100
> Project: Hadoop Hive
> Issue Type: New Feature
> Affects Versions: 0.6.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
>
> Sometimes a single hive query get compiled into several map-reduce jobs, and one of the jobs failed because of some transient errors.
> Users would need to start from scratch.
> We should allow the user to start from the point of failure to continue the query.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1100) Make it possible for users to retry
map-reduce jobs in a single Hive query
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805365#action_12805365 ]
Zheng Shao commented on HIVE-1100:
----------------------------------
What will be the semantics of pre-execution and post-execution hooks?
Shall we re-execute pre-execution hooks every time we resume the job?
Shall we fail the query if the post-execution hooks failed?
My preference is to treat the hooks as a task in the query plan - once it succeeds then we don't run it again when resuming the query.
> Make it possible for users to retry map-reduce jobs in a single Hive query
> --------------------------------------------------------------------------
>
> Key: HIVE-1100
> URL: https://issues.apache.org/jira/browse/HIVE-1100
> Project: Hadoop Hive
> Issue Type: New Feature
> Affects Versions: 0.6.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
>
> Sometimes a single hive query get compiled into several map-reduce jobs, and one of the jobs failed because of some transient errors.
> Users would need to start from scratch.
> We should allow the user to start from the point of failure to continue the query.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1100) Make it possible for users to retry
map-reduce jobs in a single Hive query
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804838#action_12804838 ]
Namit Jain commented on HIVE-1100:
----------------------------------
Is it a timeout based approach ? When does the cleanup happen if the client never issues a clear/continue ?
Do we need to enhance explain plan or some other tool so that the client knows which tasks will the query execute ?
> Make it possible for users to retry map-reduce jobs in a single Hive query
> --------------------------------------------------------------------------
>
> Key: HIVE-1100
> URL: https://issues.apache.org/jira/browse/HIVE-1100
> Project: Hadoop Hive
> Issue Type: New Feature
> Affects Versions: 0.6.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
>
> Sometimes a single hive query get compiled into several map-reduce jobs, and one of the jobs failed because of some transient errors.
> Users would need to start from scratch.
> We should allow the user to start from the point of failure to continue the query.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1100) Make it possible for users to retry
map-reduce jobs in a single Hive query
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805577#action_12805577 ]
Namit Jain commented on HIVE-1100:
----------------------------------
I think both pre-execution and post-execution hooks should be fired exactly once.
Pre hooks can be fired the first time, and post once only if the query succeeds
> Make it possible for users to retry map-reduce jobs in a single Hive query
> --------------------------------------------------------------------------
>
> Key: HIVE-1100
> URL: https://issues.apache.org/jira/browse/HIVE-1100
> Project: Hadoop Hive
> Issue Type: New Feature
> Affects Versions: 0.6.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
>
> Sometimes a single hive query get compiled into several map-reduce jobs, and one of the jobs failed because of some transient errors.
> Users would need to start from scratch.
> We should allow the user to start from the point of failure to continue the query.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1100) Make it possible for users to retry
map-reduce jobs in a single Hive query
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836867#action_12836867 ]
John Sichi commented on HIVE-1100:
----------------------------------
Mentioned to Zheng that we need to watch out for the case where there's a software upgrade in the middle and the old serialized plan is not compatible with the new software. We could require developers to take this into account when changing serialized plan structures (to make sure the new code can handle old structures or fail with a clean explanation), but it may be simpler just to stamp the serialized plan with the version of the code that generated it and then check this when deserializing it, failing with an incompatibility error if it differs.
> Make it possible for users to retry map-reduce jobs in a single Hive query
> --------------------------------------------------------------------------
>
> Key: HIVE-1100
> URL: https://issues.apache.org/jira/browse/HIVE-1100
> Project: Hadoop Hive
> Issue Type: New Feature
> Affects Versions: 0.6.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
>
> Sometimes a single hive query get compiled into several map-reduce jobs, and one of the jobs failed because of some transient errors.
> Users would need to start from scratch.
> We should allow the user to start from the point of failure to continue the query.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1100) Make it possible for users to retry
map-reduce jobs in a single Hive query
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-1100:
-----------------------------
Description:
Sometimes a single hive query get compiled into several map-reduce jobs, and one of the jobs failed because of some transient errors.
Users would need to start from scratch.
We should allow the user to start from the point of failure to continue the query.
Affects Version/s: 0.6.0
Assignee: Zheng Shao
Summary: Make it possible for users to retry map-reduce jobs in a single Hive query (was: retry )
> Make it possible for users to retry map-reduce jobs in a single Hive query
> --------------------------------------------------------------------------
>
> Key: HIVE-1100
> URL: https://issues.apache.org/jira/browse/HIVE-1100
> Project: Hadoop Hive
> Issue Type: New Feature
> Affects Versions: 0.6.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
>
> Sometimes a single hive query get compiled into several map-reduce jobs, and one of the jobs failed because of some transient errors.
> Users would need to start from scratch.
> We should allow the user to start from the point of failure to continue the query.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.