You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Eli Collins (JIRA)" <ji...@apache.org> on 2011/09/08 23:04:09 UTC
[jira] [Created] (MAPREDUCE-2957) The TT should not re-init if it
has no good local dirs
The TT should not re-init if it has no good local dirs
------------------------------------------------------
Key: MAPREDUCE-2957
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: tasktracker
Affects Versions: 0.20.204.0
Reporter: Eli Collins
Fix For: 0.20.205.0
The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2957) The TT should not re-init if it
has no good local dirs
Posted by "Eli Collins (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins resolved MAPREDUCE-2957.
------------------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Thanks for the review Ravi! I've committed this.
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
> Key: MAPREDUCE-2957
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: mapreduce-2957-1.patch, mapreduce-2957-2.patch, mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2957) The TT should not re-init if it
has no good local dirs
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins reassigned MAPREDUCE-2957:
--------------------------------------
Assignee: Eli Collins
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
> Key: MAPREDUCE-2957
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.20.205.0
>
> Attachments: mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2957) The TT should not re-init if it
has no good local dirs
Posted by "Ravi Gummadi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133853#comment-13133853 ]
Ravi Gummadi commented on MAPREDUCE-2957:
-----------------------------------------
Looks good to me.
+1
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
> Key: MAPREDUCE-2957
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: mapreduce-2957-1.patch, mapreduce-2957-2.patch, mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2957) The TT should not re-init if it
has no good local dirs
Posted by "Ravi Gummadi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132324#comment-13132324 ]
Ravi Gummadi commented on MAPREDUCE-2957:
-----------------------------------------
Patch looks fine to me.
1 minor nit: If DEE is seen, then close() is called twice (once in finally block and 2nd time in shutDown()). If possible, we should avoid this (though calling close() twice may not be harmful except some more cpu-cycles-usage) ?
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
> Key: MAPREDUCE-2957
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: mapreduce-2957-1.patch, mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2957) The TT should not re-init if it
has no good local dirs
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105132#comment-13105132 ]
Ravi Gummadi commented on MAPREDUCE-2957:
-----------------------------------------
+1 to avoid calling TaskTracker.initialize() if there are no good local dirs.
Even with this patch, i.e. though State.DENIED is returned by offerService() when DiskCheckerException is seen, initialize() is getting called once more and is getting DiskCheckerException from TT.initialize() -> DiskChecker.checkLocalDirs(). This behavior is similar to what was happening without this patch. This needs to be improved somehow and we need to avoid the call to TT.initialize() in this case.
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
> Key: MAPREDUCE-2957
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.20.205.0
>
> Attachments: mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2957) The TT should not re-init if it
has no good local dirs
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100704#comment-13100704 ]
Arun C Murthy commented on MAPREDUCE-2957:
------------------------------------------
Won't MAPREDUCE-2924 handle this?
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
> Key: MAPREDUCE-2957
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Eli Collins
> Fix For: 0.20.205.0
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (MAPREDUCE-2957) The TT should not
re-init if it has no good local dirs
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105066#comment-13105066 ]
Eli Collins edited comment on MAPREDUCE-2957 at 9/15/11 2:06 AM:
-----------------------------------------------------------------
The TT trys to shut down due to the DiskErrorException thrown from checkLocalDirs, however because offerService returns STALE in this case run tries to reinitialize instead of shutdown.
was (Author: eli):
It's buggy, the TT shuts down due to the DiskErrorException thrown from checkLocalDirs, however because we return STALE from offerService run tries to reinitialized instead of shutdown.
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
> Key: MAPREDUCE-2957
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.20.205.0
>
> Attachments: mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2957) The TT should not re-init if it
has no good local dirs
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132183#comment-13132183 ]
Eli Collins commented on MAPREDUCE-2957:
----------------------------------------
Good point, the TT shouldn't re-initialize if offerService returns DENIED, both for this new use of DENIED and the current one. I'll update the patch.
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
> Key: MAPREDUCE-2957
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2957) The TT should not re-init if it
has no good local dirs
Posted by "Eli Collins (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated MAPREDUCE-2957:
-----------------------------------
Attachment: mapreduce-2957-1.patch
Updated patch attached. Previous patch plus the TT bails from the run loop before re-initializing if we were denied since we're going to shutdown anyway.
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
> Key: MAPREDUCE-2957
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: mapreduce-2957-1.patch, mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2957) The TT should not re-init if it
has no good local dirs
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101022#comment-13101022 ]
Ravi Gummadi commented on MAPREDUCE-2957:
-----------------------------------------
The original implementation in MR-2413 already handles this case of all-disks-failed by throwing Exception from checkLocalDirs(). No ?
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
> Key: MAPREDUCE-2957
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Eli Collins
> Fix For: 0.20.205.0
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2957) The TT should not re-init if it
has no good local dirs
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated MAPREDUCE-2957:
----------------------------------
Fix Version/s: 1.1.0
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
> Key: MAPREDUCE-2957
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 1.1.0
>
> Attachments: mapreduce-2957-1.patch, mapreduce-2957-2.patch, mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2957) The TT should not re-init if it
has no good local dirs
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105066#comment-13105066 ]
Eli Collins commented on MAPREDUCE-2957:
----------------------------------------
It's buggy, the TT shuts down due to the DiskErrorException thrown from checkLocalDirs, however because we return STALE from offerService run tries to reinitialized instead of shutdown.
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
> Key: MAPREDUCE-2957
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Eli Collins
> Fix For: 0.20.205.0
>
> Attachments: mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2957) The TT should not re-init if it
has no good local dirs
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated MAPREDUCE-2957:
-----------------------------------
Attachment: mapreduce-2957.patch
Patch attached.
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
> Key: MAPREDUCE-2957
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Eli Collins
> Fix For: 0.20.205.0
>
> Attachments: mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2957) The TT should not re-init if it
has no good local dirs
Posted by "Matt Foley (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated MAPREDUCE-2957:
----------------------------------
Target Version/s: 0.20.206.0
Fix Version/s: (was: 0.20.205.0)
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
> Key: MAPREDUCE-2957
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2957) The TT should not re-init if it
has no good local dirs
Posted by "Eli Collins (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated MAPREDUCE-2957:
-----------------------------------
Attachment: mapreduce-2957-2.patch
Yea the double close isn't just for a DEE, we always execute the close in the finally block. In the case where shuttingDown is true this isn't a duplicate call as shuttingDown is set (by heckJettyPort) w/o calling shutdown. This is a duplicate call for all the denied paths though so we don't need to close here if denied is true.
Upated patch attached.
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
> Key: MAPREDUCE-2957
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: mapreduce-2957-1.patch, mapreduce-2957-2.patch, mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira