You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Ramkumar Vadali (JIRA)" <ji...@apache.org> on 2010/09/30 23:19:34 UTC

[jira] Created: (MAPREDUCE-2099) RaidNode should recreate outdated parity HARs

RaidNode should recreate outdated parity HARs
---------------------------------------------

                 Key: MAPREDUCE-2099
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2099
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: contrib/raid
            Reporter: Ramkumar Vadali
            Assignee: Ramkumar Vadali


After parity files are archived into a parity HAR, a change in the source file does not cause the HAR to be recreated. Instead, individual parity files are created for the modified files but the HAR is not touched. This causes increased disk usage for parity data.

The parity HAR could be recreated if a certain percentage of files in the HAR are determined to be outdated.



-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-2099) RaidNode should recreate outdated parity HARs

Posted by "Ramkumar Vadali (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925549#action_12925549 ] 

Ramkumar Vadali commented on MAPREDUCE-2099:
--------------------------------------------

Test Results:

ant test-patch:
{code}
     [exec] 
     [exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 2 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec] 
     [exec]     +1 system tests framework.  The patch passed system tests framework compile.
     [exec] 
     [exec] 
     [exec] 
     [exec] 
     [exec] ======================================================================
     [exec] ======================================================================
     [exec]     Finished build.
     [exec] ======================================================================
     [exec] ======================================================================
{code}

ant test under src/contrib/raid:

{code}
test-junit:
    [junit] WARNING: multiple versions of ant detected in path for junit
    [junit]          jar:file:/home/rvadali/local/external/ant/lib/ant.jar!/org/apache/tools/ant/Project.class
    [junit]      and jar:file:/home/rvadali/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class
    [junit] Running org.apache.hadoop.hdfs.TestRaidDfs
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 44.295 sec
    [junit] Running org.apache.hadoop.raid.TestDirectoryTraversal
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 8.839 sec
    [junit] Running org.apache.hadoop.raid.TestErasureCodes
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 26.307 sec
    [junit] Running org.apache.hadoop.raid.TestGaloisField
    [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 0.393 sec
    [junit] Running org.apache.hadoop.raid.TestHarIndexParser
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.052 sec
    [junit] Running org.apache.hadoop.raid.TestRaidHar
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 72.067 sec
    [junit] Running org.apache.hadoop.raid.TestRaidNode
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 403.134 sec
    [junit] Running org.apache.hadoop.raid.TestRaidPurge
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 107.381 sec
    [junit] Running org.apache.hadoop.raid.TestRaidShell
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 23.601 sec

test:

BUILD SUCCESSFUL

{code}

> RaidNode should recreate outdated parity HARs
> ---------------------------------------------
>
>                 Key: MAPREDUCE-2099
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2099
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/raid
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>         Attachments: MAPREDUCE-2099.2.patch, MAPREDUCE-2099.patch
>
>
> After parity files are archived into a parity HAR, a change in the source file does not cause the HAR to be recreated. Instead, individual parity files are created for the modified files but the HAR is not touched. This causes increased disk usage for parity data.
> The parity HAR could be recreated if a certain percentage of files in the HAR are determined to be outdated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-2099) RaidNode should recreate outdated parity HARs

Posted by "Ramkumar Vadali (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ramkumar Vadali updated MAPREDUCE-2099:
---------------------------------------

    Attachment: MAPREDUCE-2099.patch

This patch allows RaidNode to delete parity file HARs that are useless, thus freeing up space.

> RaidNode should recreate outdated parity HARs
> ---------------------------------------------
>
>                 Key: MAPREDUCE-2099
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2099
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/raid
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>         Attachments: MAPREDUCE-2099.patch
>
>
> After parity files are archived into a parity HAR, a change in the source file does not cause the HAR to be recreated. Instead, individual parity files are created for the modified files but the HAR is not touched. This causes increased disk usage for parity data.
> The parity HAR could be recreated if a certain percentage of files in the HAR are determined to be outdated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-2099) RaidNode should recreate outdated parity HARs

Posted by "Ramkumar Vadali (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ramkumar Vadali updated MAPREDUCE-2099:
---------------------------------------

    Attachment: MAPREDUCE-2099.2.patch

Patch after svn up.

> RaidNode should recreate outdated parity HARs
> ---------------------------------------------
>
>                 Key: MAPREDUCE-2099
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2099
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/raid
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>         Attachments: MAPREDUCE-2099.2.patch, MAPREDUCE-2099.patch
>
>
> After parity files are archived into a parity HAR, a change in the source file does not cause the HAR to be recreated. Instead, individual parity files are created for the modified files but the HAR is not touched. This causes increased disk usage for parity data.
> The parity HAR could be recreated if a certain percentage of files in the HAR are determined to be outdated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-2099) RaidNode should recreate outdated parity HARs

Posted by "Ramkumar Vadali (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ramkumar Vadali updated MAPREDUCE-2099:
---------------------------------------

    Attachment: MAPREDUCE-2099.3.patch

Changed isDir() -> isDirectory().

Test results:

ant test-patch:
{code}
     [exec]
     [exec]
     [exec] +1 overall.
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     +1 tests included.  The patch appears to include 2 new or modified tests.
     [exec]
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec]
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec]
     [exec]     +1 system tests framework.  The patch passed system tests framework compile.
     [exec]
     [exec]
     [exec]
     [exec]
     [exec] ======================================================================
     [exec] ======================================================================
     [exec]     Finished build.
     [exec] ======================================================================
     [exec] ======================================================================
     [exec]
     [exec]

BUILD SUCCESSFUL
Total time: 16 minutes 57 seconds

{code}

ant test under src/contrib/raid:
{code}
test-junit:
    [junit] WARNING: multiple versions of ant detected in path for junit
    [junit]          jar:file:/home/rvadali/local/external/ant/lib/ant.jar!/org/apache/tools/ant/Project.class
    [junit]      and jar:file:/home/rvadali/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class
    [junit] Running org.apache.hadoop.hdfs.TestRaidDfs
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 44.602 sec
    [junit] Running org.apache.hadoop.raid.TestDirectoryTraversal
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 11.837 sec
    [junit] Running org.apache.hadoop.raid.TestErasureCodes
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 25.407 sec
    [junit] Running org.apache.hadoop.raid.TestGaloisField
    [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 0.399 sec
    [junit] Running org.apache.hadoop.raid.TestHarIndexParser
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.057 sec
    [junit] Running org.apache.hadoop.raid.TestRaidHar
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 70.775 sec
    [junit] Running org.apache.hadoop.raid.TestRaidNode
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 521.892 sec
    [junit] Running org.apache.hadoop.raid.TestRaidPurge
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 107.62 sec
    [junit] Running org.apache.hadoop.raid.TestRaidShell
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 26.355 sec

test:

BUILD SUCCESSFUL
Total time: 13 minutes 46 seconds

{code}

> RaidNode should recreate outdated parity HARs
> ---------------------------------------------
>
>                 Key: MAPREDUCE-2099
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2099
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/raid
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>         Attachments: MAPREDUCE-2099.2.patch, MAPREDUCE-2099.3.patch, MAPREDUCE-2099.patch
>
>
> After parity files are archived into a parity HAR, a change in the source file does not cause the HAR to be recreated. Instead, individual parity files are created for the modified files but the HAR is not touched. This causes increased disk usage for parity data.
> The parity HAR could be recreated if a certain percentage of files in the HAR are determined to be outdated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-2099) RaidNode should recreate outdated parity HARs

Posted by "Ramkumar Vadali (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ramkumar Vadali updated MAPREDUCE-2099:
---------------------------------------

    Attachment: MAPREDUCE-2099.4.patch

Removed .swp and .orig file that were not supposed to be part of the patch.

> RaidNode should recreate outdated parity HARs
> ---------------------------------------------
>
>                 Key: MAPREDUCE-2099
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2099
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/raid
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>         Attachments: MAPREDUCE-2099.2.patch, MAPREDUCE-2099.3.patch, MAPREDUCE-2099.4.patch, MAPREDUCE-2099.patch
>
>
> After parity files are archived into a parity HAR, a change in the source file does not cause the HAR to be recreated. Instead, individual parity files are created for the modified files but the HAR is not touched. This causes increased disk usage for parity data.
> The parity HAR could be recreated if a certain percentage of files in the HAR are determined to be outdated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-2099) RaidNode should recreate outdated parity HARs

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926045#action_12926045 ] 

Hudson commented on MAPREDUCE-2099:
-----------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/])
    MAPREDUCE-2099.  RaidNode recreates outdated parity HARs. (Ramkumar Vadali via
schen)


> RaidNode should recreate outdated parity HARs
> ---------------------------------------------
>
>                 Key: MAPREDUCE-2099
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2099
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/raid
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>             Fix For: 0.22.0
>
>         Attachments: MAPREDUCE-2099.2.patch, MAPREDUCE-2099.3.patch, MAPREDUCE-2099.4.patch, MAPREDUCE-2099.patch
>
>
> After parity files are archived into a parity HAR, a change in the source file does not cause the HAR to be recreated. Instead, individual parity files are created for the modified files but the HAR is not touched. This causes increased disk usage for parity data.
> The parity HAR could be recreated if a certain percentage of files in the HAR are determined to be outdated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-2099) RaidNode should recreate outdated parity HARs

Posted by "Ramkumar Vadali (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ramkumar Vadali updated MAPREDUCE-2099:
---------------------------------------

    Status: Patch Available  (was: Open)

> RaidNode should recreate outdated parity HARs
> ---------------------------------------------
>
>                 Key: MAPREDUCE-2099
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2099
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/raid
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>         Attachments: MAPREDUCE-2099.patch
>
>
> After parity files are archived into a parity HAR, a change in the source file does not cause the HAR to be recreated. Instead, individual parity files are created for the modified files but the HAR is not touched. This causes increased disk usage for parity data.
> The parity HAR could be recreated if a certain percentage of files in the HAR are determined to be outdated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-2099) RaidNode should recreate outdated parity HARs

Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925619#action_12925619 ] 

Scott Chen commented on MAPREDUCE-2099:
---------------------------------------

+1 Besides the minor thing. The patch looks good for me.

> RaidNode should recreate outdated parity HARs
> ---------------------------------------------
>
>                 Key: MAPREDUCE-2099
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2099
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/raid
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>         Attachments: MAPREDUCE-2099.2.patch, MAPREDUCE-2099.patch
>
>
> After parity files are archived into a parity HAR, a change in the source file does not cause the HAR to be recreated. Instead, individual parity files are created for the modified files but the HAR is not touched. This causes increased disk usage for parity data.
> The parity HAR could be recreated if a certain percentage of files in the HAR are determined to be outdated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-2099) RaidNode should recreate outdated parity HARs

Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925615#action_12925615 ] 

Scott Chen commented on MAPREDUCE-2099:
---------------------------------------

{code}
+      if (dest.isDir()) {
{code}
Can we use isDirectory()?

> RaidNode should recreate outdated parity HARs
> ---------------------------------------------
>
>                 Key: MAPREDUCE-2099
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2099
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/raid
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>         Attachments: MAPREDUCE-2099.2.patch, MAPREDUCE-2099.patch
>
>
> After parity files are archived into a parity HAR, a change in the source file does not cause the HAR to be recreated. Instead, individual parity files are created for the modified files but the HAR is not touched. This causes increased disk usage for parity data.
> The parity HAR could be recreated if a certain percentage of files in the HAR are determined to be outdated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-2099) RaidNode should recreate outdated parity HARs

Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Scott Chen updated MAPREDUCE-2099:
----------------------------------

       Resolution: Fixed
    Fix Version/s: 0.22.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I just committed this. Thanks Ram.

> RaidNode should recreate outdated parity HARs
> ---------------------------------------------
>
>                 Key: MAPREDUCE-2099
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2099
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/raid
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>             Fix For: 0.22.0
>
>         Attachments: MAPREDUCE-2099.2.patch, MAPREDUCE-2099.3.patch, MAPREDUCE-2099.4.patch, MAPREDUCE-2099.patch
>
>
> After parity files are archived into a parity HAR, a change in the source file does not cause the HAR to be recreated. Instead, individual parity files are created for the modified files but the HAR is not touched. This causes increased disk usage for parity data.
> The parity HAR could be recreated if a certain percentage of files in the HAR are determined to be outdated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.