You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Patrick Kling <pk...@cs.uwaterloo.ca> on 2010/11/03 05:12:28 UTC
Review Request: DFSClient.getBlockLocations returns BlockLocations with no
indication that the corresponding blocks are corrupt
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27/
-----------------------------------------------------------
Review request for hadoop-hdfs.
Summary
-------
DFSClient.getBlockLocations returns BlockLocations with no indication that the corresponding blocks are corrupt
When there are no uncorrupted replicas of a block, FSNamesystem.getBlockLocations returns LocatedBlocks corresponding to corrupt blocks. When DFSClient converts these to BlockLocations, the information that the corresponding block is corrupt is lost. We should add a field to BlockLocation to indicate whether the corresponding block is corrupt in order to warn the client that reading this block will fail. This would be especially useful for tools such as RAID FSCK, which could then easily inspect whether data or parity blocks are corrupted without having to make direct RPC calls
This addresses bug HDFS-1483.
https://issues.apache.org/jira/browse/HDFS-1483
Diffs
-----
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DFSUtil.java 1028386
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSUtil.java PRE-CREATION
Diff: https://reviews.apache.org/r/27/diff
Testing
-------
TestDFSUtil
Thanks,
Patrick
Re: Review Request: DFSClient.getBlockLocations returns
BlockLocations with no indication that the corresponding blocks are corrupt
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Arun,
I filed https://issues.apache.org/jira/browse/INFRA-3153
<https://issues.apache.org/jira/browse/INFRA-3153>Thanks
-Todd
On Wed, Nov 3, 2010 at 9:47 PM, Arun C Murthy <ac...@yahoo-inc.com> wrote:
>
> On Nov 3, 2010, at 5:03 PM, Todd Lipcon wrote:
>
> I wrote such a procmail script for review.hbase.org and posted it for the
>> ASF Infra guys a few weeks ago. We can file a new INFRA JIRA to get them
>> to
>> install/configure it.
>>
>>
> +1, thanks Todd!
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Review Request: DFSClient.getBlockLocations returns BlockLocations with no indication that the corresponding blocks are corrupt
Posted by Arun C Murthy <ac...@yahoo-inc.com>.
On Nov 3, 2010, at 5:03 PM, Todd Lipcon wrote:
> I wrote such a procmail script for review.hbase.org and posted it
> for the
> ASF Infra guys a few weeks ago. We can file a new INFRA JIRA to get
> them to
> install/configure it.
>
+1, thanks Todd!
Re: Review Request: DFSClient.getBlockLocations returns
BlockLocations with no indication that the corresponding blocks are corrupt
Posted by Todd Lipcon <to...@cloudera.com>.
I wrote such a procmail script for review.hbase.org and posted it for the
ASF Infra guys a few weeks ago. We can file a new INFRA JIRA to get them to
install/configure it.
-Todd
On Wed, Nov 3, 2010 at 1:27 PM, Arun C Murthy <ac...@yahoo-inc.com> wrote:
> Can we get RB to send these to jira? Having comments here and on jira is
> very confusing...
>
>
> On Nov 3, 2010, at 11:11 AM, Ramkumar Vadali wrote:
>
>
>> -----------------------------------------------------------
>> This is an automatically generated e-mail. To reply, visit:
>> https://reviews.apache.org/r/27/#review27
>> -----------------------------------------------------------
>>
>>
>>
>>
>> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DFSUtil.java
>> <https://reviews.apache.org/r/27/#comment17>
>>
>> It should be sufficient to pass blk.isCorrupt() here. The client can
>> check the number of locations based on the remaining information.
>>
>>
>> - Ramkumar
>>
>>
>> On 2010-11-02 21:12:28, Patrick Kling wrote:
>>
>>>
>>> -----------------------------------------------------------
>>> This is an automatically generated e-mail. To reply, visit:
>>> https://reviews.apache.org/r/27/
>>> -----------------------------------------------------------
>>>
>>> (Updated 2010-11-02 21:12:28)
>>>
>>>
>>> Review request for hadoop-hdfs.
>>>
>>>
>>> Summary
>>> -------
>>>
>>> DFSClient.getBlockLocations returns BlockLocations with no indication
>>> that the corresponding blocks are corrupt
>>>
>>> When there are no uncorrupted replicas of a block,
>>> FSNamesystem.getBlockLocations returns LocatedBlocks corresponding to
>>> corrupt blocks. When DFSClient converts these to BlockLocations, the
>>> information that the corresponding block is corrupt is lost. We should add a
>>> field to BlockLocation to indicate whether the corresponding block is
>>> corrupt in order to warn the client that reading this block will fail. This
>>> would be especially useful for tools such as RAID FSCK, which could then
>>> easily inspect whether data or parity blocks are corrupted without having to
>>> make direct RPC calls
>>>
>>>
>>> This addresses bug HDFS-1483.
>>> https://issues.apache.org/jira/browse/HDFS-1483
>>>
>>>
>>> Diffs
>>> -----
>>>
>>>
>>> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DFSUtil.java1028386
>>>
>>> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSUtil.javaPRE-CREATION
>>>
>>> Diff: https://reviews.apache.org/r/27/diff
>>>
>>>
>>> Testing
>>> -------
>>>
>>> TestDFSUtil
>>>
>>>
>>> Thanks,
>>>
>>> Patrick
>>>
>>>
>>>
>>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Review Request: DFSClient.getBlockLocations returns BlockLocations with no indication that the corresponding blocks are corrupt
Posted by Arun C Murthy <ac...@yahoo-inc.com>.
Can we get RB to send these to jira? Having comments here and on jira
is very confusing...
On Nov 3, 2010, at 11:11 AM, Ramkumar Vadali wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27/#review27
> -----------------------------------------------------------
>
>
>
> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DFSUtil.java
> <https://reviews.apache.org/r/27/#comment17>
>
> It should be sufficient to pass blk.isCorrupt() here. The client
> can check the number of locations based on the remaining information.
>
>
> - Ramkumar
>
>
> On 2010-11-02 21:12:28, Patrick Kling wrote:
>>
>> -----------------------------------------------------------
>> This is an automatically generated e-mail. To reply, visit:
>> https://reviews.apache.org/r/27/
>> -----------------------------------------------------------
>>
>> (Updated 2010-11-02 21:12:28)
>>
>>
>> Review request for hadoop-hdfs.
>>
>>
>> Summary
>> -------
>>
>> DFSClient.getBlockLocations returns BlockLocations with no
>> indication that the corresponding blocks are corrupt
>>
>> When there are no uncorrupted replicas of a block,
>> FSNamesystem.getBlockLocations returns LocatedBlocks corresponding
>> to corrupt blocks. When DFSClient converts these to BlockLocations,
>> the information that the corresponding block is corrupt is lost. We
>> should add a field to BlockLocation to indicate whether the
>> corresponding block is corrupt in order to warn the client that
>> reading this block will fail. This would be especially useful for
>> tools such as RAID FSCK, which could then easily inspect whether
>> data or parity blocks are corrupted without having to make direct
>> RPC calls
>>
>>
>> This addresses bug HDFS-1483.
>> https://issues.apache.org/jira/browse/HDFS-1483
>>
>>
>> Diffs
>> -----
>>
>> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DFSUtil.java
>> 1028386
>> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSUtil.java
>> PRE-CREATION
>>
>> Diff: https://reviews.apache.org/r/27/diff
>>
>>
>> Testing
>> -------
>>
>> TestDFSUtil
>>
>>
>> Thanks,
>>
>> Patrick
>>
>>
>
Re: Review Request: DFSClient.getBlockLocations returns BlockLocations with
no indication that the corresponding blocks are corrupt
Posted by Ramkumar Vadali <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27/#review27
-----------------------------------------------------------
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DFSUtil.java
<https://reviews.apache.org/r/27/#comment17>
It should be sufficient to pass blk.isCorrupt() here. The client can check the number of locations based on the remaining information.
- Ramkumar
On 2010-11-02 21:12:28, Patrick Kling wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27/
> -----------------------------------------------------------
>
> (Updated 2010-11-02 21:12:28)
>
>
> Review request for hadoop-hdfs.
>
>
> Summary
> -------
>
> DFSClient.getBlockLocations returns BlockLocations with no indication that the corresponding blocks are corrupt
>
> When there are no uncorrupted replicas of a block, FSNamesystem.getBlockLocations returns LocatedBlocks corresponding to corrupt blocks. When DFSClient converts these to BlockLocations, the information that the corresponding block is corrupt is lost. We should add a field to BlockLocation to indicate whether the corresponding block is corrupt in order to warn the client that reading this block will fail. This would be especially useful for tools such as RAID FSCK, which could then easily inspect whether data or parity blocks are corrupted without having to make direct RPC calls
>
>
> This addresses bug HDFS-1483.
> https://issues.apache.org/jira/browse/HDFS-1483
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DFSUtil.java 1028386
> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSUtil.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/27/diff
>
>
> Testing
> -------
>
> TestDFSUtil
>
>
> Thanks,
>
> Patrick
>
>
Re: Review Request: DFSClient.getBlockLocations returns BlockLocations with
no indication that the corresponding blocks are corrupt
Posted by Patrick Kling <pk...@cs.uwaterloo.ca>.
> On 2010-11-03 11:41:39, Ramkumar Vadali wrote:
> > Looks good to me, but this diff depends on a hadoop-common change, right?
It depends on HADOOP-7013, which can be found here: https://reviews.apache.org/r/26/
- Patrick
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27/#review29
-----------------------------------------------------------
On 2010-11-03 11:33:39, Patrick Kling wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27/
> -----------------------------------------------------------
>
> (Updated 2010-11-03 11:33:39)
>
>
> Review request for hadoop-hdfs.
>
>
> Summary
> -------
>
> DFSClient.getBlockLocations returns BlockLocations with no indication that the corresponding blocks are corrupt
>
> When there are no uncorrupted replicas of a block, FSNamesystem.getBlockLocations returns LocatedBlocks corresponding to corrupt blocks. When DFSClient converts these to BlockLocations, the information that the corresponding block is corrupt is lost. We should add a field to BlockLocation to indicate whether the corresponding block is corrupt in order to warn the client that reading this block will fail. This would be especially useful for tools such as RAID FSCK, which could then easily inspect whether data or parity blocks are corrupted without having to make direct RPC calls
>
>
> This addresses bug HDFS-1483.
> https://issues.apache.org/jira/browse/HDFS-1483
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DFSUtil.java 1028386
> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSUtil.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/27/diff
>
>
> Testing
> -------
>
> TestDFSUtil
>
>
> Thanks,
>
> Patrick
>
>
Re: Review Request: DFSClient.getBlockLocations returns BlockLocations with
no indication that the corresponding blocks are corrupt
Posted by Ramkumar Vadali <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27/#review29
-----------------------------------------------------------
Looks good to me, but this diff depends on a hadoop-common change, right?
- Ramkumar
On 2010-11-03 11:33:39, Patrick Kling wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27/
> -----------------------------------------------------------
>
> (Updated 2010-11-03 11:33:39)
>
>
> Review request for hadoop-hdfs.
>
>
> Summary
> -------
>
> DFSClient.getBlockLocations returns BlockLocations with no indication that the corresponding blocks are corrupt
>
> When there are no uncorrupted replicas of a block, FSNamesystem.getBlockLocations returns LocatedBlocks corresponding to corrupt blocks. When DFSClient converts these to BlockLocations, the information that the corresponding block is corrupt is lost. We should add a field to BlockLocation to indicate whether the corresponding block is corrupt in order to warn the client that reading this block will fail. This would be especially useful for tools such as RAID FSCK, which could then easily inspect whether data or parity blocks are corrupted without having to make direct RPC calls
>
>
> This addresses bug HDFS-1483.
> https://issues.apache.org/jira/browse/HDFS-1483
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DFSUtil.java 1028386
> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSUtil.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/27/diff
>
>
> Testing
> -------
>
> TestDFSUtil
>
>
> Thanks,
>
> Patrick
>
>
Re: Review Request: DFSClient.getBlockLocations returns BlockLocations with
no indication that the corresponding blocks are corrupt
Posted by Patrick Kling <pk...@cs.uwaterloo.ca>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27/
-----------------------------------------------------------
(Updated 2010-11-03 11:33:39.415750)
Review request for hadoop-hdfs.
Changes
-------
Incorporated Ram's feedback. Thank you!
Summary
-------
DFSClient.getBlockLocations returns BlockLocations with no indication that the corresponding blocks are corrupt
When there are no uncorrupted replicas of a block, FSNamesystem.getBlockLocations returns LocatedBlocks corresponding to corrupt blocks. When DFSClient converts these to BlockLocations, the information that the corresponding block is corrupt is lost. We should add a field to BlockLocation to indicate whether the corresponding block is corrupt in order to warn the client that reading this block will fail. This would be especially useful for tools such as RAID FSCK, which could then easily inspect whether data or parity blocks are corrupted without having to make direct RPC calls
This addresses bug HDFS-1483.
https://issues.apache.org/jira/browse/HDFS-1483
Diffs (updated)
-----
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DFSUtil.java 1028386
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSUtil.java PRE-CREATION
Diff: https://reviews.apache.org/r/27/diff
Testing
-------
TestDFSUtil
Thanks,
Patrick