You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Eli Collins (JIRA)" <ji...@apache.org> on 2011/09/08 23:04:09 UTC

[jira] [Created] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs

The TT should not re-init if it has no good local dirs
------------------------------------------------------

                 Key: MAPREDUCE-2957
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
             Project: Hadoop Map/Reduce
          Issue Type: Sub-task
          Components: tasktracker
    Affects Versions: 0.20.204.0
            Reporter: Eli Collins
             Fix For: 0.20.205.0


The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Resolved] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs

Posted by "Eli Collins (Resolved) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins resolved MAPREDUCE-2957.
------------------------------------

      Resolution: Fixed
    Hadoop Flags: Reviewed

Thanks for the review Ravi! I've committed this.
                
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-2957
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: tasktracker
>    Affects Versions: 0.20.204.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: mapreduce-2957-1.patch, mapreduce-2957-2.patch, mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Assigned] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs

Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins reassigned MAPREDUCE-2957:
--------------------------------------

    Assignee: Eli Collins

> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-2957
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: tasktracker
>    Affects Versions: 0.20.204.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.20.205.0
>
>         Attachments: mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs

Posted by "Ravi Gummadi (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133853#comment-13133853 ] 

Ravi Gummadi commented on MAPREDUCE-2957:
-----------------------------------------

Looks good to me.
+1
                
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-2957
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: tasktracker
>    Affects Versions: 0.20.204.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: mapreduce-2957-1.patch, mapreduce-2957-2.patch, mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs

Posted by "Ravi Gummadi (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132324#comment-13132324 ] 

Ravi Gummadi commented on MAPREDUCE-2957:
-----------------------------------------

Patch looks fine to me.
1 minor nit: If DEE is seen, then close() is called twice (once in finally block and 2nd time in shutDown()). If possible, we should avoid this (though calling close() twice may not be harmful except some more cpu-cycles-usage) ?
                
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-2957
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: tasktracker
>    Affects Versions: 0.20.204.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: mapreduce-2957-1.patch, mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs

Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105132#comment-13105132 ] 

Ravi Gummadi commented on MAPREDUCE-2957:
-----------------------------------------

+1 to avoid calling TaskTracker.initialize() if there are no good local dirs.

 Even with this patch, i.e. though State.DENIED is returned by offerService() when DiskCheckerException is seen, initialize() is getting called once more and is getting DiskCheckerException from TT.initialize() -> DiskChecker.checkLocalDirs(). This behavior is similar to what was happening without this patch. This needs to be improved somehow and we need to avoid the call to TT.initialize() in this case.

> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-2957
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: tasktracker
>    Affects Versions: 0.20.204.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.20.205.0
>
>         Attachments: mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100704#comment-13100704 ] 

Arun C Murthy commented on MAPREDUCE-2957:
------------------------------------------

Won't MAPREDUCE-2924 handle this?

> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-2957
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: tasktracker
>    Affects Versions: 0.20.204.0
>            Reporter: Eli Collins
>             Fix For: 0.20.205.0
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Issue Comment Edited] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs

Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105066#comment-13105066 ] 

Eli Collins edited comment on MAPREDUCE-2957 at 9/15/11 2:06 AM:
-----------------------------------------------------------------

The TT trys to shut down due to the DiskErrorException thrown from checkLocalDirs, however because offerService returns STALE in this case run tries to reinitialize instead of shutdown.

      was (Author: eli):
    It's buggy, the TT shuts down due to the DiskErrorException thrown from checkLocalDirs, however because we return STALE from offerService run tries to reinitialized instead of shutdown.
  
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-2957
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: tasktracker
>    Affects Versions: 0.20.204.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.20.205.0
>
>         Attachments: mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs

Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132183#comment-13132183 ] 

Eli Collins commented on MAPREDUCE-2957:
----------------------------------------

Good point, the TT shouldn't re-initialize if offerService returns DENIED, both for this new use of DENIED and the current one. I'll update the patch.
                
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-2957
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: tasktracker
>    Affects Versions: 0.20.204.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs

Posted by "Eli Collins (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated MAPREDUCE-2957:
-----------------------------------

    Attachment: mapreduce-2957-1.patch

Updated patch attached. Previous patch plus the TT bails from the run loop before re-initializing if we were denied since we're going to shutdown anyway.
                
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-2957
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: tasktracker
>    Affects Versions: 0.20.204.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: mapreduce-2957-1.patch, mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs

Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101022#comment-13101022 ] 

Ravi Gummadi commented on MAPREDUCE-2957:
-----------------------------------------

The original implementation in MR-2413 already handles this case of all-disks-failed by throwing Exception from checkLocalDirs(). No ?

> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-2957
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: tasktracker
>    Affects Versions: 0.20.204.0
>            Reporter: Eli Collins
>             Fix For: 0.20.205.0
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs

Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated MAPREDUCE-2957:
----------------------------------

    Fix Version/s: 1.1.0
    
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-2957
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: tasktracker
>    Affects Versions: 0.20.204.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 1.1.0
>
>         Attachments: mapreduce-2957-1.patch, mapreduce-2957-2.patch, mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs

Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105066#comment-13105066 ] 

Eli Collins commented on MAPREDUCE-2957:
----------------------------------------

It's buggy, the TT shuts down due to the DiskErrorException thrown from checkLocalDirs, however because we return STALE from offerService run tries to reinitialized instead of shutdown.

> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-2957
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: tasktracker
>    Affects Versions: 0.20.204.0
>            Reporter: Eli Collins
>             Fix For: 0.20.205.0
>
>         Attachments: mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs

Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated MAPREDUCE-2957:
-----------------------------------

    Attachment: mapreduce-2957.patch

Patch attached.

> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-2957
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: tasktracker
>    Affects Versions: 0.20.204.0
>            Reporter: Eli Collins
>             Fix For: 0.20.205.0
>
>         Attachments: mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs

Posted by "Matt Foley (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated MAPREDUCE-2957:
----------------------------------

    Target Version/s: 0.20.206.0
       Fix Version/s:     (was: 0.20.205.0)
    
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-2957
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: tasktracker
>    Affects Versions: 0.20.204.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs

Posted by "Eli Collins (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated MAPREDUCE-2957:
-----------------------------------

    Attachment: mapreduce-2957-2.patch

Yea the double close isn't just for a DEE, we always execute the close in the finally block. In the case where shuttingDown is true this isn't a duplicate call as shuttingDown is set (by heckJettyPort) w/o calling shutdown. This is a duplicate call for all the denied paths though so we don't need to close here if denied is true.

Upated patch attached.
                
> The TT should not re-init if it has no good local dirs
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-2957
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: tasktracker
>    Affects Versions: 0.20.204.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: mapreduce-2957-1.patch, mapreduce-2957-2.patch, mapreduce-2957.patch
>
>
> The TT will currently try to re-init itself on disk failure even if it has no good local dirs. It should shutdown instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira