You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Scott Chen (JIRA)" <ji...@apache.org> on 2010/06/01 21:01:45 UTC
[jira] Created: (MAPREDUCE-1831) Delete the replica on the most
concentrated node when raiding file
Delete the replica on the most concentrated node when raiding file
------------------------------------------------------------------
Key: MAPREDUCE-1831
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: contrib/raid
Affects Versions: 0.22.0
Reporter: Scott Chen
Assignee: Scott Chen
Fix For: 0.22.0
In raid, it is good to have the blocks on the same stripe located on different machine.
This way when one machine is down, it does not broke two blocks on the stripe.
By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement.
One way to do this is that we can add a new BlockPlacementPolicy which delete the replicas that are co-located.
So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1831:
----------------------------------
Status: Patch Available (was: Open)
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.20100610.txt, MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement (where p is the replica missing probability).
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1831:
----------------------------------
Status: Open (was: Patch Available)
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.20100610.txt, MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement (where p is the replica missing probability).
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874169#action_12874169 ]
Rodrigo Schmidt commented on MAPREDUCE-1831:
--------------------------------------------
This is great, Scott! Thanks for working on it.
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement.
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1831:
----------------------------------
Attachment: MAPREDUCE-1831.v1.1.txt
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement.
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1831:
----------------------------------
Description:
In raid, it is good to have the blocks on the same stripe located on different machine.
This way when one machine is down, it does not broke two blocks on the stripe.
By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement (where p is the replica missing probability).
One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
So when raiding the file, we can make the remaining replicas live on different machines.
was:
In raid, it is good to have the blocks on the same stripe located on different machine.
This way when one machine is down, it does not broke two blocks on the stripe.
By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement.
One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
So when raiding the file, we can make the remaining replicas live on different machines.
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement (where p is the replica missing probability).
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879211#action_12879211 ]
Hadoop QA commented on MAPREDUCE-1831:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12446833/MAPREDUCE-1831.20100610.txt
against trunk revision 954364.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/244/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/244/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/244/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/244/console
This message is automatically generated.
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.20100610.txt, MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement (where p is the replica missing probability).
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1831:
----------------------------------
Status: Patch Available (was: Open)
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement.
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876963#action_12876963 ]
Hadoop QA commented on MAPREDUCE-1831:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12446654/MAPREDUCE-1831.v1.1.txt
against trunk revision 952548.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/231/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/231/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/231/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/231/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/231/console
This message is automatically generated.
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement.
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877676#action_12877676 ]
dhruba borthakur commented on MAPREDUCE-1831:
---------------------------------------------
> Is there any block placement policy to make sure that parity does not land on the same node as its source file
This is work-inprogress and we will post it to a new JIRA soon.
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement.
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880339#action_12880339 ]
Rodrigo Schmidt commented on MAPREDUCE-1831:
--------------------------------------------
This still has the problem of probably leaving both replicas on the same switch, right?
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.20100610.txt, MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement (where p is the replica missing probability).
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1831:
----------------------------------
Status: Patch Available (was: Open)
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement.
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1831:
----------------------------------
Status: Patch Available (was: Open)
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.20100610.txt, MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement (where p is the replica missing probability).
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1831:
----------------------------------
Attachment: MAPREDUCE-1831.20100610.txt
Update. Looking at blocks on the same stripe only. Use FSNamesystem instead of FileSystem to get block information.
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.20100610.txt, MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement (where p is the replica missing probability).
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1831:
----------------------------------
Status: Patch Available (was: Open)
I think the failed contrib test may be TestSimulatorDeterministicReplay.testMain.
It is a know issue in MAPREDUCE-1834.
But the testReport is gone.
I am submitting this to hudson again.
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.20100610.txt, MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement (where p is the replica missing probability).
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882225#action_12882225 ]
Scott Chen commented on MAPREDUCE-1831:
---------------------------------------
Thanks, Dhruba. Yes, this one makes sense only when MAPREDUCE-1861 is available.
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.20100610.txt, MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement (where p is the replica missing probability).
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1831:
----------------------------------
Status: Open (was: Patch Available)
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.20100610.txt, MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement (where p is the replica missing probability).
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880073#action_12880073 ]
Hadoop QA commented on MAPREDUCE-1831:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12446833/MAPREDUCE-1831.20100610.txt
against trunk revision 955543.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/252/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/252/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/252/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/252/console
This message is automatically generated.
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.20100610.txt, MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement (where p is the replica missing probability).
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880312#action_12880312 ]
Scott Chen commented on MAPREDUCE-1831:
---------------------------------------
The failed contrib test is a known issue reported in MAPREDUCE-1834.
The patch is ready for review.
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.20100610.txt, MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement (where p is the replica missing probability).
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1831:
----------------------------------
Status: Open (was: Patch Available)
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement.
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1831:
----------------------------------
Status: Patch Available (was: Open)
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.20100610.txt, MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement (where p is the replica missing probability).
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Wittawat Tantisiriroj (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877674#action_12877674 ]
Wittawat Tantisiriroj commented on MAPREDUCE-1831:
--------------------------------------------------
Is there any block placement policy to make sure that parity does not land on the same node as its source file?
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement.
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876919#action_12876919 ]
Scott Chen commented on MAPREDUCE-1831:
---------------------------------------
Updated. Add the change to README file.
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement.
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1831:
----------------------------------
Attachment: MAPREDUCE-1831.txt
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement.
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880389#action_12880389 ]
Rodrigo Schmidt commented on MAPREDUCE-1831:
--------------------------------------------
Got it now. Thanks!
+1, patch looks good to me.
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.20100610.txt, MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement (where p is the replica missing probability).
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878189#action_12878189 ]
Hadoop QA commented on MAPREDUCE-1831:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12446833/MAPREDUCE-1831.20100610.txt
against trunk revision 953879.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/565/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/565/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/565/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/565/console
This message is automatically generated.
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.20100610.txt, MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement (where p is the replica missing probability).
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1831:
----------------------------------
Status: Open (was: Patch Available)
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.20100610.txt, MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement (where p is the replica missing probability).
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876917#action_12876917 ]
Scott Chen commented on MAPREDUCE-1831:
---------------------------------------
Uploaded patch. Add a new BlockPlacementPolicy called BlockPlacementPolicyRaid which overrides chooseReplicaToDelete.
Instead of choosing from the node with least free space, we choosing the replica that has largest number of co-located replicas in the same inode.
This eliminates the dependency of the replicas on the same stripe therefore decreases the file corruption probability.
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement.
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1831:
----------------------------------
Status: Open (was: Patch Available)
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.20100610.txt, MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement (where p is the replica missing probability).
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1831:
----------------------------------
Summary: Delete the co-located replicas when raiding file (was: Delete the replica on the most concentrated node when raiding file)
Description:
In raid, it is good to have the blocks on the same stripe located on different machine.
This way when one machine is down, it does not broke two blocks on the stripe.
By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement.
One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
So when raiding the file, we can make the remaining replicas live on different machines.
was:
In raid, it is good to have the blocks on the same stripe located on different machine.
This way when one machine is down, it does not broke two blocks on the stripe.
By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement.
One way to do this is that we can add a new BlockPlacementPolicy which delete the replicas that are co-located.
So when raiding the file, we can make the remaining replicas live on different machines.
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement.
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1831:
----------------------------------
Status: Open (was: Patch Available)
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement.
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1831) Delete the co-located replicas
when raiding file
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880388#action_12880388 ]
Scott Chen commented on MAPREDUCE-1831:
---------------------------------------
Rodrigo: Thanks for the comment. In
{code}
chooseReplicaToDelete(inode, block, replicationFactor, first, second);
{code}
We always go with the "first" set first. first set contains the replicas on the same rack.
In the normal case, the "local" replica will be in the second set.
In this patch, if we have a tie in the number of co-located replica, we compare the free space.
So this policy will just fall back to function as BlockPlacementPolicyDefault for the files with replication=3.
To solve this problem we need to create one more replica (make the replication=4) while raiding.
It is described in MAPREDUCE-1861.
After we have another off-rack replica, this policy kicks in to reduce the replication to 2.
This policy will first remove a replica in first set. And remove the replica co-located on the same stripe.
The final result will be two replica on different rack and no co-located replica on the same stripe.
We need both this JIRA and MAPREDUCE-1861 to achieve that result.
> Delete the co-located replicas when raiding file
> ------------------------------------------------
>
> Key: MAPREDUCE-1831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1831.20100610.txt, MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
>
>
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4) which can be a hugh improvement (where p is the replica missing probability).
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.