You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Patrick Kling <pk...@cs.uwaterloo.ca> on 2010/11/25 01:20:50 UTC
Review Request: Distributed block fixer for raid
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/120/
-----------------------------------------------------------
Review request for hadoop-mapreduce.
Summary
-------
This change adds a new implementation of the raid block fixer that uses map reduce jobs to fix missing/corrupt blocks, which allows us to fix a large number of blocks quickly. The user can specify via a configuration variable whether the old (local) or the new implementation should be used.
This addresses bug MAPREDUCE-2155.
https://issues.apache.org/jira/browse/MAPREDUCE-2155
Diffs
-----
http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/hdfs/RaidDFSUtil.java 1038358
http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java 1038358
http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/DistBlockFixer.java PRE-CREATION
http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/LocalBlockFixer.java PRE-CREATION
http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/RaidNode.java 1038358
http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/RaidShell.java 1038358
http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/test/org/apache/hadoop/raid/TestBlockFixer.java 1038358
http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/test/org/apache/hadoop/raid/TestRaidShell.java 1038358
http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/test/org/apache/hadoop/raid/TestReedSolomonDecoder.java 1038358
Diff: https://reviews.apache.org/r/120/diff
Testing
-------
TestBlockFixer, other raid unit tests, manual testing on test cluster
Thanks,
Patrick
Re: Review Request: Distributed block fixer for raid
Posted by Ramkumar Vadali <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/120/#review44
-----------------------------------------------------------
http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java
<https://reviews.apache.org/r/120/#comment34>
What changed here?
http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/DistBlockFixer.java
<https://reviews.apache.org/r/120/#comment35>
Can we remove this debugging helper?
- Ramkumar
On 2010-11-24 16:20:50, Patrick Kling wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/120/
> -----------------------------------------------------------
>
> (Updated 2010-11-24 16:20:50)
>
>
> Review request for hadoop-mapreduce.
>
>
> Summary
> -------
>
> This change adds a new implementation of the raid block fixer that uses map reduce jobs to fix missing/corrupt blocks, which allows us to fix a large number of blocks quickly. The user can specify via a configuration variable whether the old (local) or the new implementation should be used.
>
>
> This addresses bug MAPREDUCE-2155.
> https://issues.apache.org/jira/browse/MAPREDUCE-2155
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/hdfs/RaidDFSUtil.java 1038358
> http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java 1038358
> http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/DistBlockFixer.java PRE-CREATION
> http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/LocalBlockFixer.java PRE-CREATION
> http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/RaidNode.java 1038358
> http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/RaidShell.java 1038358
> http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/test/org/apache/hadoop/raid/TestBlockFixer.java 1038358
> http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/test/org/apache/hadoop/raid/TestRaidShell.java 1038358
> http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/test/org/apache/hadoop/raid/TestReedSolomonDecoder.java 1038358
>
> Diff: https://reviews.apache.org/r/120/diff
>
>
> Testing
> -------
>
> TestBlockFixer, other raid unit tests, manual testing on test cluster
>
>
> Thanks,
>
> Patrick
>
>
Re: Review Request: Distributed block fixer for raid
Posted by Ramkumar Vadali <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/120/#review45
-----------------------------------------------------------
Ship it!
- Ramkumar
On 2010-11-29 11:40:53, Patrick Kling wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/120/
> -----------------------------------------------------------
>
> (Updated 2010-11-29 11:40:53)
>
>
> Review request for hadoop-mapreduce.
>
>
> Summary
> -------
>
> This change adds a new implementation of the raid block fixer that uses map reduce jobs to fix missing/corrupt blocks, which allows us to fix a large number of blocks quickly. The user can specify via a configuration variable whether the old (local) or the new implementation should be used.
>
>
> This addresses bug MAPREDUCE-2155.
> https://issues.apache.org/jira/browse/MAPREDUCE-2155
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/hdfs/RaidDFSUtil.java 1038358
> http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java 1038358
> http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/DistBlockFixer.java PRE-CREATION
> http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/LocalBlockFixer.java PRE-CREATION
> http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/RaidNode.java 1038358
> http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/RaidShell.java 1038358
> http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/test/org/apache/hadoop/raid/TestBlockFixer.java 1038358
> http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/test/org/apache/hadoop/raid/TestRaidShell.java 1038358
> http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/test/org/apache/hadoop/raid/TestReedSolomonDecoder.java 1038358
>
> Diff: https://reviews.apache.org/r/120/diff
>
>
> Testing
> -------
>
> TestBlockFixer, other raid unit tests, manual testing on test cluster
>
>
> Thanks,
>
> Patrick
>
>
Re: Review Request: Distributed block fixer for raid
Posted by Patrick Kling <pk...@cs.uwaterloo.ca>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/120/
-----------------------------------------------------------
(Updated 2010-11-29 11:40:53.572161)
Review request for hadoop-mapreduce.
Changes
-------
- Reverted whitespace change
- Removed debugging helper method
Summary
-------
This change adds a new implementation of the raid block fixer that uses map reduce jobs to fix missing/corrupt blocks, which allows us to fix a large number of blocks quickly. The user can specify via a configuration variable whether the old (local) or the new implementation should be used.
This addresses bug MAPREDUCE-2155.
https://issues.apache.org/jira/browse/MAPREDUCE-2155
Diffs (updated)
-----
http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/hdfs/RaidDFSUtil.java 1038358
http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java 1038358
http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/DistBlockFixer.java PRE-CREATION
http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/LocalBlockFixer.java PRE-CREATION
http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/RaidNode.java 1038358
http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/java/org/apache/hadoop/raid/RaidShell.java 1038358
http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/test/org/apache/hadoop/raid/TestBlockFixer.java 1038358
http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/test/org/apache/hadoop/raid/TestRaidShell.java 1038358
http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid/src/test/org/apache/hadoop/raid/TestReedSolomonDecoder.java 1038358
Diff: https://reviews.apache.org/r/120/diff
Testing
-------
TestBlockFixer, other raid unit tests, manual testing on test cluster
Thanks,
Patrick