You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Vinod K V (JIRA)" <ji...@apache.org> on 2009/02/27 13:41:12 UTC

[jira] Created: (HADOOP-5349) When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.

When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.
--------------------------------------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-5349
                 URL: https://issues.apache.org/jira/browse/HADOOP-5349
             Project: Hadoop Core
          Issue Type: Bug
          Components: mapred
    Affects Versions: 0.20.0
            Reporter: Vinod K V




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-5349) When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.

Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-5349:
------------------------------

    Attachment: HADOOP-5349.txt

Seeing this issue many times on our clusters because of many tasks are getting failed.

Attaching patch that should fix this issue. The patch doesn't include any tests - as of now there are no tests verifying the algorithm of round-robin disk selection; writing these tests is a bit involving and can be done as part of another JIRA issue.

> When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5349
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Vinod K V
>         Attachments: HADOOP-5349.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5349) When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.

Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707397#action_12707397 ] 

Nigel Daley commented on HADOOP-5349:
-------------------------------------

Vinod, please reference the Jira here that will test this.

> When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5349
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-5349.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (HADOOP-5349) When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.

Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V reassigned HADOOP-5349:
---------------------------------

    Assignee: Vinod K V

> When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5349
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>         Attachments: HADOOP-5349.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5349) When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707495#action_12707495 ] 

Hudson commented on HADOOP-5349:
--------------------------------

Integrated in Hadoop-trunk #830 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/830/])
    . Fixes a problem in LocalDirAllocator to check for the return path value that is returned for the case where the file we want to write is of an unknown size. Contributed by Vinod Kumar Vavilapalli.


> When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5349
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-5349.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-5349) When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.

Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-5349:
------------------------------

    Status: Patch Available  (was: Open)

Patch up for review. Running it through Hudson.

> When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5349
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Vinod K V
>         Attachments: HADOOP-5349.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5349) When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.

Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707895#action_12707895 ] 

Vinod K V commented on HADOOP-5349:
-----------------------------------

bq. Vinod, please reference the Jira here that will test this. 
HADOOP-5799

> When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5349
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-5349.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5349) When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.

Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12706984#action_12706984 ] 

Vinod K V commented on HADOOP-5349:
-----------------------------------


ant test-patch results:
{code}
     [exec] -1 overall.
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
     [exec]                         Please justify why no tests are needed for this patch.
     [exec]
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec]
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]
     [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
     [exec]
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
{code}

As said above, tests for this fix will be added as part of another issue. Core and contrib tests also passed.

> When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5349
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>         Attachments: HADOOP-5349.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5349) When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.

Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677349#action_12677349 ] 

Vinod K V commented on HADOOP-5349:
-----------------------------------

When the size required is unknown, LocalDirAllocator selects one target disk in a round-robin fashion. Before returning this path, it tries to create the path. If the selected disk turns about to be bad, then LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException. The fix for this is to make getLocalPathWrite to try other disks till it finds one, similar to the case when required size of the path is known.

> When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5349
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Vinod K V
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-5349) When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.

Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-5349:
--------------------------------

       Resolution: Fixed
    Fix Version/s: 0.20.1
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I just committed this. Thanks, Vinod!

> When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5349
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-5349.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.