You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Steve Loughran (Created) (JIRA)" <ji...@apache.org> on 2011/10/21 17:12:33 UTC
[jira] [Created] (HDFS-2486) Review issues with
UnderReplicatedBlocks
Review issues with UnderReplicatedBlocks
----------------------------------------
Key: HDFS-2486
URL: https://issues.apache.org/jira/browse/HDFS-2486
Project: Hadoop HDFS
Issue Type: Task
Components: name-node
Affects Versions: 0.23.0
Reporter: Steve Loughran
Priority: Minor
Here are some things I've noted in the UnderReplicatedBlocks class that someone else should review and consider if the code is correct. If not, they are easy to fix.
remove(Block block, int priLevel) is not synchronized, and as the inner classes are not, there is a risk of race conditions there.
some of the code assumes that getPriority can return the value LEVEL, and if so does not attempt to queue the blocks. As this return value is not currently possible, those checks can be removed.
The queue gives priority to blocks whose replication count is less than a third of its expected count over those that are "normally under replicated". While this is good for ensuring that files scheduled for large replication are replicated fast, it may not be the best strategy for maintaining data integrity. For that it may be better to give whichever blocks have only two replicas priority over blocks that may, for example, already have 3 out of 10 copies in the filesystem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2486) Review issues with
UnderReplicatedBlocks
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205164#comment-13205164 ]
Hudson commented on HDFS-2486:
------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1718 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1718/])
HDFS-2486. Remove unnecessary priority level checks in UnderReplicatedBlocks. Contributed by Uma Maheswara Rao G
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1242636
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
> Review issues with UnderReplicatedBlocks
> ----------------------------------------
>
> Key: HDFS-2486
> URL: https://issues.apache.org/jira/browse/HDFS-2486
> Project: Hadoop HDFS
> Issue Type: Task
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Steve Loughran
> Assignee: Uma Maheswara Rao G
> Priority: Minor
> Fix For: 0.24.0
>
> Attachments: HDFS-2486.patch
>
>
> Here are some things I've noted in the UnderReplicatedBlocks class that someone else should review and consider if the code is correct. If not, they are easy to fix.
> remove(Block block, int priLevel) is not synchronized, and as the inner classes are not, there is a risk of race conditions there.
> some of the code assumes that getPriority can return the value LEVEL, and if so does not attempt to queue the blocks. As this return value is not currently possible, those checks can be removed.
> The queue gives priority to blocks whose replication count is less than a third of its expected count over those that are "normally under replicated". While this is good for ensuring that files scheduled for large replication are replicated fast, it may not be the best strategy for maintaining data integrity. For that it may be better to give whichever blocks have only two replicas priority over blocks that may, for example, already have 3 out of 10 copies in the filesystem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2486) Review issues with
UnderReplicatedBlocks
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205158#comment-13205158 ]
Hudson commented on HDFS-2486:
------------------------------
Integrated in Hadoop-Common-trunk-Commit #1707 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1707/])
HDFS-2486. Remove unnecessary priority level checks in UnderReplicatedBlocks. Contributed by Uma Maheswara Rao G
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1242636
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
> Review issues with UnderReplicatedBlocks
> ----------------------------------------
>
> Key: HDFS-2486
> URL: https://issues.apache.org/jira/browse/HDFS-2486
> Project: Hadoop HDFS
> Issue Type: Task
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Steve Loughran
> Assignee: Uma Maheswara Rao G
> Priority: Minor
> Fix For: 0.24.0
>
> Attachments: HDFS-2486.patch
>
>
> Here are some things I've noted in the UnderReplicatedBlocks class that someone else should review and consider if the code is correct. If not, they are easy to fix.
> remove(Block block, int priLevel) is not synchronized, and as the inner classes are not, there is a risk of race conditions there.
> some of the code assumes that getPriority can return the value LEVEL, and if so does not attempt to queue the blocks. As this return value is not currently possible, those checks can be removed.
> The queue gives priority to blocks whose replication count is less than a third of its expected count over those that are "normally under replicated". While this is good for ensuring that files scheduled for large replication are replicated fast, it may not be the best strategy for maintaining data integrity. For that it may be better to give whichever blocks have only two replicas priority over blocks that may, for example, already have 3 out of 10 copies in the filesystem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2486) Review issues with
UnderReplicatedBlocks
Posted by "Uma Maheswara Rao G (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-2486:
--------------------------------------
Assignee: Uma Maheswara Rao G
Target Version/s: 0.24.0
Status: Patch Available (was: Open)
removed the unnecessary priLevel != LEVEL checks from UnderReplicated class as getPriority API will never return LEVEL (5) .
Patch mainly tested with:
TestReplication: this can verify the update api
TestReplicationPolicy: this will verify add functionality.
Normal other tests not impacted due to this change.
> Review issues with UnderReplicatedBlocks
> ----------------------------------------
>
> Key: HDFS-2486
> URL: https://issues.apache.org/jira/browse/HDFS-2486
> Project: Hadoop HDFS
> Issue Type: Task
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Steve Loughran
> Assignee: Uma Maheswara Rao G
> Priority: Minor
> Attachments: HDFS-2486.patch
>
>
> Here are some things I've noted in the UnderReplicatedBlocks class that someone else should review and consider if the code is correct. If not, they are easy to fix.
> remove(Block block, int priLevel) is not synchronized, and as the inner classes are not, there is a risk of race conditions there.
> some of the code assumes that getPriority can return the value LEVEL, and if so does not attempt to queue the blocks. As this return value is not currently possible, those checks can be removed.
> The queue gives priority to blocks whose replication count is less than a third of its expected count over those that are "normally under replicated". While this is good for ensuring that files scheduled for large replication are replicated fast, it may not be the best strategy for maintaining data integrity. For that it may be better to give whichever blocks have only two replicas priority over blocks that may, for example, already have 3 out of 10 copies in the filesystem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2486) Review issues with
UnderReplicatedBlocks
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171577#comment-13171577 ]
Uma Maheswara Rao G commented on HDFS-2486:
-------------------------------------------
This patch addresses the second comment. After concluding the last comment, that can be handled as separate JIRA(subtask), if required and also depending on code change.
> Review issues with UnderReplicatedBlocks
> ----------------------------------------
>
> Key: HDFS-2486
> URL: https://issues.apache.org/jira/browse/HDFS-2486
> Project: Hadoop HDFS
> Issue Type: Task
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Steve Loughran
> Assignee: Uma Maheswara Rao G
> Priority: Minor
> Attachments: HDFS-2486.patch
>
>
> Here are some things I've noted in the UnderReplicatedBlocks class that someone else should review and consider if the code is correct. If not, they are easy to fix.
> remove(Block block, int priLevel) is not synchronized, and as the inner classes are not, there is a risk of race conditions there.
> some of the code assumes that getPriority can return the value LEVEL, and if so does not attempt to queue the blocks. As this return value is not currently possible, those checks can be removed.
> The queue gives priority to blocks whose replication count is less than a third of its expected count over those that are "normally under replicated". While this is good for ensuring that files scheduled for large replication are replicated fast, it may not be the best strategy for maintaining data integrity. For that it may be better to give whichever blocks have only two replicas priority over blocks that may, for example, already have 3 out of 10 copies in the filesystem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2486) Review issues with
UnderReplicatedBlocks
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202247#comment-13202247 ]
Uma Maheswara Rao G commented on HDFS-2486:
-------------------------------------------
Could some one please check this points and patch?
> Review issues with UnderReplicatedBlocks
> ----------------------------------------
>
> Key: HDFS-2486
> URL: https://issues.apache.org/jira/browse/HDFS-2486
> Project: Hadoop HDFS
> Issue Type: Task
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Steve Loughran
> Assignee: Uma Maheswara Rao G
> Priority: Minor
> Attachments: HDFS-2486.patch
>
>
> Here are some things I've noted in the UnderReplicatedBlocks class that someone else should review and consider if the code is correct. If not, they are easy to fix.
> remove(Block block, int priLevel) is not synchronized, and as the inner classes are not, there is a risk of race conditions there.
> some of the code assumes that getPriority can return the value LEVEL, and if so does not attempt to queue the blocks. As this return value is not currently possible, those checks can be removed.
> The queue gives priority to blocks whose replication count is less than a third of its expected count over those that are "normally under replicated". While this is good for ensuring that files scheduled for large replication are replicated fast, it may not be the best strategy for maintaining data integrity. For that it may be better to give whichever blocks have only two replicas priority over blocks that may, for example, already have 3 out of 10 copies in the filesystem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2486) Review issues with
UnderReplicatedBlocks
Posted by "Tsz Wo (Nicholas), SZE (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205151#comment-13205151 ]
Tsz Wo (Nicholas), SZE commented on HDFS-2486:
----------------------------------------------
@Steve, good observations!
@Uma
+1 patch looks good. Thanks for working on this.
> Review issues with UnderReplicatedBlocks
> ----------------------------------------
>
> Key: HDFS-2486
> URL: https://issues.apache.org/jira/browse/HDFS-2486
> Project: Hadoop HDFS
> Issue Type: Task
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Steve Loughran
> Assignee: Uma Maheswara Rao G
> Priority: Minor
> Attachments: HDFS-2486.patch
>
>
> Here are some things I've noted in the UnderReplicatedBlocks class that someone else should review and consider if the code is correct. If not, they are easy to fix.
> remove(Block block, int priLevel) is not synchronized, and as the inner classes are not, there is a risk of race conditions there.
> some of the code assumes that getPriority can return the value LEVEL, and if so does not attempt to queue the blocks. As this return value is not currently possible, those checks can be removed.
> The queue gives priority to blocks whose replication count is less than a third of its expected count over those that are "normally under replicated". While this is good for ensuring that files scheduled for large replication are replicated fast, it may not be the best strategy for maintaining data integrity. For that it may be better to give whichever blocks have only two replicas priority over blocks that may, for example, already have 3 out of 10 copies in the filesystem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2486) Review issues with
UnderReplicatedBlocks
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171586#comment-13171586 ]
Hadoop QA commented on HDFS-2486:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12507779/HDFS-2486.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javadoc. The javadoc tool appears to have generated 90 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1720//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1720//console
This message is automatically generated.
> Review issues with UnderReplicatedBlocks
> ----------------------------------------
>
> Key: HDFS-2486
> URL: https://issues.apache.org/jira/browse/HDFS-2486
> Project: Hadoop HDFS
> Issue Type: Task
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Steve Loughran
> Assignee: Uma Maheswara Rao G
> Priority: Minor
> Attachments: HDFS-2486.patch
>
>
> Here are some things I've noted in the UnderReplicatedBlocks class that someone else should review and consider if the code is correct. If not, they are easy to fix.
> remove(Block block, int priLevel) is not synchronized, and as the inner classes are not, there is a risk of race conditions there.
> some of the code assumes that getPriority can return the value LEVEL, and if so does not attempt to queue the blocks. As this return value is not currently possible, those checks can be removed.
> The queue gives priority to blocks whose replication count is less than a third of its expected count over those that are "normally under replicated". While this is good for ensuring that files scheduled for large replication are replicated fast, it may not be the best strategy for maintaining data integrity. For that it may be better to give whichever blocks have only two replicas priority over blocks that may, for example, already have 3 out of 10 copies in the filesystem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2486) Review issues with
UnderReplicatedBlocks
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169420#comment-13169420 ]
Uma Maheswara Rao G commented on HDFS-2486:
-------------------------------------------
Hi Steve,
I just verified your points here.
{quote}
remove(Block block, int priLevel) is not synchronized, and as the inner classes are not, there is a risk of race conditions there.
{quote}
there is an other remove method which is synchronized.
The non synchronized remove method will be invoked from
1) BlockManager#computeReplicationWorkForBlocks -- it has synchronized block on neededReplications (UnderReplicatedBlocks) and called remove from this block. So, should not be a problem.
2)synchronized UnderReplicatedBlocks#update
This is already synchronized on UnderReplicatedBlocks, should nt be a problem.
3)synchronized UnderReplicatedBlocks#remove
This is also synchronized on UnderReplicatedBlocks, should not be a problem.
{quote}
some of the code assumes that getPriority can return the value LEVEL, and if so does not attempt to queue the blocks. As this return value is not currently possible, those checks can be removed.
{quote}
I agree, there are unnecessary checks here, can be removed.
{quote}
The queue gives priority to blocks whose replication count is less than a third of its expected count over those that are "normally under replicated". While this is good for ensuring that files scheduled for large replication are replicated fast, it may not be the best strategy for maintaining data integrity. For that it may be better to give whichever blocks have only two replicas priority over blocks that may, for example, already have 3 out of 10 copies in the filesystem.
{quote}
I am not sure, whether i understud your argument correctly here.
We are consideing 'very under replicated' , if block are replicated at 1:3 ratio.
considering a case 10 replication factor and 3 blocks present, should be consider as just 'under replicated'? here my understanding about your argument is yes.
But i argue here is that, still we should consider that blocks as 'very under replicated' because, user set the replication factor as 10 means, that files might be very important/valuable info. Please corect me if i understud wrongly about this point.
Thanks
Uma
> Review issues with UnderReplicatedBlocks
> ----------------------------------------
>
> Key: HDFS-2486
> URL: https://issues.apache.org/jira/browse/HDFS-2486
> Project: Hadoop HDFS
> Issue Type: Task
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Steve Loughran
> Priority: Minor
>
> Here are some things I've noted in the UnderReplicatedBlocks class that someone else should review and consider if the code is correct. If not, they are easy to fix.
> remove(Block block, int priLevel) is not synchronized, and as the inner classes are not, there is a risk of race conditions there.
> some of the code assumes that getPriority can return the value LEVEL, and if so does not attempt to queue the blocks. As this return value is not currently possible, those checks can be removed.
> The queue gives priority to blocks whose replication count is less than a third of its expected count over those that are "normally under replicated". While this is good for ensuring that files scheduled for large replication are replicated fast, it may not be the best strategy for maintaining data integrity. For that it may be better to give whichever blocks have only two replicas priority over blocks that may, for example, already have 3 out of 10 copies in the filesystem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2486) Review issues with
UnderReplicatedBlocks
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205447#comment-13205447 ]
Hudson commented on HDFS-2486:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #985 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/985/])
HDFS-2486. Remove unnecessary priority level checks in UnderReplicatedBlocks. Contributed by Uma Maheswara Rao G
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1242636
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
> Review issues with UnderReplicatedBlocks
> ----------------------------------------
>
> Key: HDFS-2486
> URL: https://issues.apache.org/jira/browse/HDFS-2486
> Project: Hadoop HDFS
> Issue Type: Task
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Steve Loughran
> Assignee: Uma Maheswara Rao G
> Priority: Minor
> Fix For: 0.24.0
>
> Attachments: HDFS-2486.patch
>
>
> Here are some things I've noted in the UnderReplicatedBlocks class that someone else should review and consider if the code is correct. If not, they are easy to fix.
> remove(Block block, int priLevel) is not synchronized, and as the inner classes are not, there is a risk of race conditions there.
> some of the code assumes that getPriority can return the value LEVEL, and if so does not attempt to queue the blocks. As this return value is not currently possible, those checks can be removed.
> The queue gives priority to blocks whose replication count is less than a third of its expected count over those that are "normally under replicated". While this is good for ensuring that files scheduled for large replication are replicated fast, it may not be the best strategy for maintaining data integrity. For that it may be better to give whichever blocks have only two replicas priority over blocks that may, for example, already have 3 out of 10 copies in the filesystem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2486) Review issues with
UnderReplicatedBlocks
Posted by "Uma Maheswara Rao G (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-2486:
--------------------------------------
Attachment: HDFS-2486.patch
> Review issues with UnderReplicatedBlocks
> ----------------------------------------
>
> Key: HDFS-2486
> URL: https://issues.apache.org/jira/browse/HDFS-2486
> Project: Hadoop HDFS
> Issue Type: Task
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Steve Loughran
> Priority: Minor
> Attachments: HDFS-2486.patch
>
>
> Here are some things I've noted in the UnderReplicatedBlocks class that someone else should review and consider if the code is correct. If not, they are easy to fix.
> remove(Block block, int priLevel) is not synchronized, and as the inner classes are not, there is a risk of race conditions there.
> some of the code assumes that getPriority can return the value LEVEL, and if so does not attempt to queue the blocks. As this return value is not currently possible, those checks can be removed.
> The queue gives priority to blocks whose replication count is less than a third of its expected count over those that are "normally under replicated". While this is good for ensuring that files scheduled for large replication are replicated fast, it may not be the best strategy for maintaining data integrity. For that it may be better to give whichever blocks have only two replicas priority over blocks that may, for example, already have 3 out of 10 copies in the filesystem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2486) Review issues with
UnderReplicatedBlocks
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173186#comment-13173186 ]
Uma Maheswara Rao G commented on HDFS-2486:
-------------------------------------------
Test failures and javadoc comments are unrelated to this patch.
> Review issues with UnderReplicatedBlocks
> ----------------------------------------
>
> Key: HDFS-2486
> URL: https://issues.apache.org/jira/browse/HDFS-2486
> Project: Hadoop HDFS
> Issue Type: Task
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Steve Loughran
> Assignee: Uma Maheswara Rao G
> Priority: Minor
> Attachments: HDFS-2486.patch
>
>
> Here are some things I've noted in the UnderReplicatedBlocks class that someone else should review and consider if the code is correct. If not, they are easy to fix.
> remove(Block block, int priLevel) is not synchronized, and as the inner classes are not, there is a risk of race conditions there.
> some of the code assumes that getPriority can return the value LEVEL, and if so does not attempt to queue the blocks. As this return value is not currently possible, those checks can be removed.
> The queue gives priority to blocks whose replication count is less than a third of its expected count over those that are "normally under replicated". While this is good for ensuring that files scheduled for large replication are replicated fast, it may not be the best strategy for maintaining data integrity. For that it may be better to give whichever blocks have only two replicas priority over blocks that may, for example, already have 3 out of 10 copies in the filesystem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2486) Review issues with
UnderReplicatedBlocks
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205160#comment-13205160 ]
Hudson commented on HDFS-2486:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1782 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1782/])
HDFS-2486. Remove unnecessary priority level checks in UnderReplicatedBlocks. Contributed by Uma Maheswara Rao G
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1242636
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
> Review issues with UnderReplicatedBlocks
> ----------------------------------------
>
> Key: HDFS-2486
> URL: https://issues.apache.org/jira/browse/HDFS-2486
> Project: Hadoop HDFS
> Issue Type: Task
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Steve Loughran
> Assignee: Uma Maheswara Rao G
> Priority: Minor
> Fix For: 0.24.0
>
> Attachments: HDFS-2486.patch
>
>
> Here are some things I've noted in the UnderReplicatedBlocks class that someone else should review and consider if the code is correct. If not, they are easy to fix.
> remove(Block block, int priLevel) is not synchronized, and as the inner classes are not, there is a risk of race conditions there.
> some of the code assumes that getPriority can return the value LEVEL, and if so does not attempt to queue the blocks. As this return value is not currently possible, those checks can be removed.
> The queue gives priority to blocks whose replication count is less than a third of its expected count over those that are "normally under replicated". While this is good for ensuring that files scheduled for large replication are replicated fast, it may not be the best strategy for maintaining data integrity. For that it may be better to give whichever blocks have only two replicas priority over blocks that may, for example, already have 3 out of 10 copies in the filesystem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2486) Review issues with
UnderReplicatedBlocks
Posted by "Tsz Wo (Nicholas), SZE (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-2486:
-----------------------------------------
Resolution: Fixed
Fix Version/s: 0.24.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I have committed this. Thanks, Uma!
> Review issues with UnderReplicatedBlocks
> ----------------------------------------
>
> Key: HDFS-2486
> URL: https://issues.apache.org/jira/browse/HDFS-2486
> Project: Hadoop HDFS
> Issue Type: Task
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Steve Loughran
> Assignee: Uma Maheswara Rao G
> Priority: Minor
> Fix For: 0.24.0
>
> Attachments: HDFS-2486.patch
>
>
> Here are some things I've noted in the UnderReplicatedBlocks class that someone else should review and consider if the code is correct. If not, they are easy to fix.
> remove(Block block, int priLevel) is not synchronized, and as the inner classes are not, there is a risk of race conditions there.
> some of the code assumes that getPriority can return the value LEVEL, and if so does not attempt to queue the blocks. As this return value is not currently possible, those checks can be removed.
> The queue gives priority to blocks whose replication count is less than a third of its expected count over those that are "normally under replicated". While this is good for ensuring that files scheduled for large replication are replicated fast, it may not be the best strategy for maintaining data integrity. For that it may be better to give whichever blocks have only two replicas priority over blocks that may, for example, already have 3 out of 10 copies in the filesystem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2486) Review issues with
UnderReplicatedBlocks
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205410#comment-13205410 ]
Hudson commented on HDFS-2486:
------------------------------
Integrated in Hadoop-Hdfs-trunk #952 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/952/])
HDFS-2486. Remove unnecessary priority level checks in UnderReplicatedBlocks. Contributed by Uma Maheswara Rao G
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1242636
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
> Review issues with UnderReplicatedBlocks
> ----------------------------------------
>
> Key: HDFS-2486
> URL: https://issues.apache.org/jira/browse/HDFS-2486
> Project: Hadoop HDFS
> Issue Type: Task
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Steve Loughran
> Assignee: Uma Maheswara Rao G
> Priority: Minor
> Fix For: 0.24.0
>
> Attachments: HDFS-2486.patch
>
>
> Here are some things I've noted in the UnderReplicatedBlocks class that someone else should review and consider if the code is correct. If not, they are easy to fix.
> remove(Block block, int priLevel) is not synchronized, and as the inner classes are not, there is a risk of race conditions there.
> some of the code assumes that getPriority can return the value LEVEL, and if so does not attempt to queue the blocks. As this return value is not currently possible, those checks can be removed.
> The queue gives priority to blocks whose replication count is less than a third of its expected count over those that are "normally under replicated". While this is good for ensuring that files scheduled for large replication are replicated fast, it may not be the best strategy for maintaining data integrity. For that it may be better to give whichever blocks have only two replicas priority over blocks that may, for example, already have 3 out of 10 copies in the filesystem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira