You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Amar Kamat (JIRA)" <ji...@apache.org> on 2009/10/13 18:39:31 UTC
[jira] Created: (MAPREDUCE-1102) Job gets killed even when the
cleanup completes
Job gets killed even when the cleanup completes
-----------------------------------------------
Key: MAPREDUCE-1102
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1102
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.20.1
Reporter: Amar Kamat
Fix For: 0.22.0
When the cleanup completes at the tasktracker and the job is killed by the user, the cleanup runs to completion but the job fails. Ideally if the cleanup is completed then the job should not be killed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1102) Job gets killed even when the
cleanup completes
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770595#action_12770595 ]
Devaraj Das commented on MAPREDUCE-1102:
----------------------------------------
The behavior of having the commitJob create a file is actually application logic (the FileOutputCommitter in this case). So yes, there is a small window of time when the job is considered successful, but maybe we can make that explicit in the FileOutputCommitter contract.
On a side note, longer term, we should probably make the tasks write to a temp JOB directory, and rename the temp job directory to the final output directory AFTER the job is successful. That way we can avoid the special files in the output directory of the job.
> Job gets killed even when the cleanup completes
> -----------------------------------------------
>
> Key: MAPREDUCE-1102
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1102
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.1
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Fix For: 0.22.0
>
>
> When the cleanup completes at the tasktracker and the job is killed by the user, the cleanup runs to completion but the job fails. Ideally if the cleanup is completed then the job should not be killed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1102) Job gets killed even when the
cleanup completes
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765378#action_12765378 ]
Vinod K V commented on MAPREDUCE-1102:
--------------------------------------
bq. When the cleanup completes at the tasktracker and the job is killed by the user, the cleanup runs to completion but the job fails.
No, it is marked as KILLED and not FAILED.
If the kill is issued by the user, then the state of the job *should* be killed, no? Kill by user means the user doesn't care as to whatever happens to the job's tasks.
> Job gets killed even when the cleanup completes
> -----------------------------------------------
>
> Key: MAPREDUCE-1102
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1102
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.1
> Reporter: Amar Kamat
> Fix For: 0.22.0
>
>
> When the cleanup completes at the tasktracker and the job is killed by the user, the cleanup runs to completion but the job fails. Ideally if the cleanup is completed then the job should not be killed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1102) Job gets killed even when the
cleanup completes
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770420#action_12770420 ]
Amar Kamat commented on MAPREDUCE-1102:
---------------------------------------
bq. Even if commitJob has started (or finished). We should also make the FileOutputFormat abortJob delete _SUCCESS to handle this case. This would also handle the case where the job commit task fails part way through.
Wouldn't the _SUCCESS file be available/visible for a small duration of time. External observers might consider the job as successful.
> Job gets killed even when the cleanup completes
> -----------------------------------------------
>
> Key: MAPREDUCE-1102
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1102
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.1
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Fix For: 0.22.0
>
>
> When the cleanup completes at the tasktracker and the job is killed by the user, the cleanup runs to completion but the job fails. Ideally if the cleanup is completed then the job should not be killed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1102) Job gets killed even when the
cleanup completes
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770798#action_12770798 ]
Amar Kamat commented on MAPREDUCE-1102:
---------------------------------------
bq. On a side note, longer term, we should probably make the tasks write to a temp JOB directory, and rename the temp job directory to the final output directory AFTER the job is successful. That way we can avoid the special files in the output directory of the job.
[comment|https://issues.apache.org/jira/browse/MAPREDUCE-947?focusedCommentId=12762146&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12762146].
> Job gets killed even when the cleanup completes
> -----------------------------------------------
>
> Key: MAPREDUCE-1102
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1102
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.1
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Fix For: 0.22.0
>
>
> When the cleanup completes at the tasktracker and the job is killed by the user, the cleanup runs to completion but the job fails. Ideally if the cleanup is completed then the job should not be killed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1102) Job gets killed even when the
cleanup completes
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770071#action_12770071 ]
Owen O'Malley commented on MAPREDUCE-1102:
------------------------------------------
Rather than blocking kill-job, I think we are better off guaranteeing that if the job fails, we will always call abortJob. Even if commitJob has started (or finished). We should also make the FileOutputFormat abortJob delete _SUCCESS to handle this case. This would also handle the case where the job commit task fails part way through.
I agree that all output committers may not be able to unroll their commit. However, I think that we need to give them the ability to do the right thing.
> Job gets killed even when the cleanup completes
> -----------------------------------------------
>
> Key: MAPREDUCE-1102
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1102
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.1
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Fix For: 0.22.0
>
>
> When the cleanup completes at the tasktracker and the job is killed by the user, the cleanup runs to completion but the job fails. Ideally if the cleanup is completed then the job should not be killed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1102) Job gets killed even when the
cleanup completes
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769979#action_12769979 ]
Amar Kamat commented on MAPREDUCE-1102:
---------------------------------------
One simple approach would be not to honour 'kill-job' when the cleanup is launched in which case the job can either move to FAILED or SUCCESSFUL state. The job can fail (after cleanup is launched) if all the cleanup attempts fail. The only corner case we need to take care if the case where the FileOutputCommitter.commitJob() creates _SUCCESS and fails. In such a case the job will fail with a _SUCCESS file.
> Job gets killed even when the cleanup completes
> -----------------------------------------------
>
> Key: MAPREDUCE-1102
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1102
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.1
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Fix For: 0.22.0
>
>
> When the cleanup completes at the tasktracker and the job is killed by the user, the cleanup runs to completion but the job fails. Ideally if the cleanup is completed then the job should not be killed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1102) Job gets killed even when the
cleanup completes
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770415#action_12770415 ]
Amar Kamat commented on MAPREDUCE-1102:
---------------------------------------
I think the same applies for failed jobs too. Consider a case where the job is marked to fail and the cleanup attempt (for failed job) is launched. Now the user issued kill should be respected i.e the cleanup-attempt for failed job should be killed and another cleanup-attempt (for killed job) should be launched. Note that two (or more) cleanup attempts should not run in parallel else there would be a race condition. Thoughts?
> Job gets killed even when the cleanup completes
> -----------------------------------------------
>
> Key: MAPREDUCE-1102
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1102
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.1
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Fix For: 0.22.0
>
>
> When the cleanup completes at the tasktracker and the job is killed by the user, the cleanup runs to completion but the job fails. Ideally if the cleanup is completed then the job should not be killed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1102) Job gets killed even when the
cleanup completes
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat reassigned MAPREDUCE-1102:
-------------------------------------
Assignee: Amar Kamat
> Job gets killed even when the cleanup completes
> -----------------------------------------------
>
> Key: MAPREDUCE-1102
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1102
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.1
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Fix For: 0.22.0
>
>
> When the cleanup completes at the tasktracker and the job is killed by the user, the cleanup runs to completion but the job fails. Ideally if the cleanup is completed then the job should not be killed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1102) Job gets killed even when the
cleanup completes
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765390#action_12765390 ]
Amar Kamat commented on MAPREDUCE-1102:
---------------------------------------
bq. No, it is marked as KILLED and not FAILED.
Ahh .. I meant killed. Thanks Vinod for pointing it out ;)
bq. If the kill is issued by the user, then the state of the job should be killed, no? Kill by user means the user doesn't care as to whatever happens to the job's tasks.
Note that the completed jobs cannot be killed by the user. So in the above mentioned case the job is actually complete but the state change hasn't happened. I think we should treat this case similar to "kill on completed jobs". Note that this is tricky as the cleanup can be in the following states
# UNASSIGNED
# running (in midway i.e some part of the cleanup has executed)
# cleanup has completed but the process has not exited
# cleanup task has died but the tasktracker is waiting to send the heartbeat or job-kill and cleanup-success both are in the ipc queue back to back.
Ideally we expect the job to be killed in cases 1 and 2 but succeeded in 3,4 and 5.
> Job gets killed even when the cleanup completes
> -----------------------------------------------
>
> Key: MAPREDUCE-1102
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1102
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.1
> Reporter: Amar Kamat
> Fix For: 0.22.0
>
>
> When the cleanup completes at the tasktracker and the job is killed by the user, the cleanup runs to completion but the job fails. Ideally if the cleanup is completed then the job should not be killed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1102) Job gets killed even when the
cleanup completes
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770079#action_12770079 ]
Owen O'Malley commented on MAPREDUCE-1102:
------------------------------------------
Naturally, the fact that it may be invoked after the commitJob method has been called should be called out in the JavaDoc for the abortJob method.
> Job gets killed even when the cleanup completes
> -----------------------------------------------
>
> Key: MAPREDUCE-1102
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1102
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.1
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Fix For: 0.22.0
>
>
> When the cleanup completes at the tasktracker and the job is killed by the user, the cleanup runs to completion but the job fails. Ideally if the cleanup is completed then the job should not be killed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.