You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Harsh J (Created) (JIRA)" <ji...@apache.org> on 2011/11/10 03:37:51 UTC
[jira] [Created] (HDFS-2547) Design doc is wrong about default
block placement policy.
Design doc is wrong about default block placement policy.
---------------------------------------------------------
Key: HDFS-2547
URL: https://issues.apache.org/jira/browse/HDFS-2547
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.20.1
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
Fix For: 0.24.0
Attachments: HDFS-2547.patch
bq. For the common case, when the replication factor is three, HDFS's placement policy is to put one replica on one node in the local rack, another on a node in a different (remote) rack, and the last on a different node in the same *remote* rack.
Should actually be: "and the last on a different node in the same *local* rack."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2547) Design doc is wrong about default
block placement policy.
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148176#comment-13148176 ]
Aaron T. Myers commented on HDFS-2547:
--------------------------------------
Hey Harsh, seems like we should fix the comments then. Want to re-title/re-open this JIRA for that purpose?
> Design doc is wrong about default block placement policy.
> ---------------------------------------------------------
>
> Key: HDFS-2547
> URL: https://issues.apache.org/jira/browse/HDFS-2547
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.20.1
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Fix For: 0.24.0
>
> Attachments: HDFS-2547.patch
>
>
> bq. For the common case, when the replication factor is three, HDFS's placement policy is to put one replica on one node in the local rack, another on a node in a different (remote) rack, and the last on a different node in the same *remote* rack.
> Should actually be: "and the last on a different node in the same *local* rack."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2547) ReplicationTargetChooser has
incorrect block placement comments
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148632#comment-13148632 ]
Aaron T. Myers commented on HDFS-2547:
--------------------------------------
+1, patch looks good to me.
> ReplicationTargetChooser has incorrect block placement comments
> ---------------------------------------------------------------
>
> Key: HDFS-2547
> URL: https://issues.apache.org/jira/browse/HDFS-2547
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.20.1
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Fix For: 0.20.206.0
>
> Attachments: HDFS-2547.patch, HDFS-2547.patch
>
>
> {code}
> /** The class is responsible for choosing the desired number of targets
> * for placing block replicas.
> * The replica placement strategy is that if the writer is on a datanode,
> * the 1st replica is placed on the local machine,
> * otherwise a random datanode. The 2nd replica is placed on a datanode
> * that is on a different rack. The 3rd replica is placed on a datanode
> * which is on the same rack as the **first replca**.
> */
> {code}
> That should read "second replica". The test cases confirm that this is the behavior, as well as the docs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2547) Design doc is wrong about default
block placement policy.
Posted by "Harsh J (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HDFS-2547:
--------------------------
Resolution: Invalid
Status: Resolved (was: Patch Available)
The ReplicationTargetChooser comments are incorrect, and lead to this confusion. Resolving as invalid. The documented behavior is correct for all of the common cases.
> Design doc is wrong about default block placement policy.
> ---------------------------------------------------------
>
> Key: HDFS-2547
> URL: https://issues.apache.org/jira/browse/HDFS-2547
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.20.1
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Fix For: 0.24.0
>
> Attachments: HDFS-2547.patch
>
>
> bq. For the common case, when the replication factor is three, HDFS's placement policy is to put one replica on one node in the local rack, another on a node in a different (remote) rack, and the last on a different node in the same *remote* rack.
> Should actually be: "and the last on a different node in the same *local* rack."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2547) ReplicationTargetChooser has incorrect
block placement comments
Posted by "Harsh J (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HDFS-2547:
--------------------------
Attachment: HDFS-2547.patch
New patch that changes comments and fixes a typo.
> ReplicationTargetChooser has incorrect block placement comments
> ---------------------------------------------------------------
>
> Key: HDFS-2547
> URL: https://issues.apache.org/jira/browse/HDFS-2547
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.20.1
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Fix For: 0.20.206.0
>
> Attachments: HDFS-2547.patch, HDFS-2547.patch
>
>
> {code}
> /** The class is responsible for choosing the desired number of targets
> * for placing block replicas.
> * The replica placement strategy is that if the writer is on a datanode,
> * the 1st replica is placed on the local machine,
> * otherwise a random datanode. The 2nd replica is placed on a datanode
> * that is on a different rack. The 3rd replica is placed on a datanode
> * which is on the same rack as the **first replca**.
> */
> {code}
> That should read "second replica". The test cases confirm that this is the behavior, as well as the docs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2547) Design doc is wrong about default
block placement policy.
Posted by "Harsh J (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148445#comment-13148445 ]
Harsh J commented on HDFS-2547:
-------------------------------
Aaron,
{{BlockPlacementPolicyDefault}} has proper comments in it, on trunk. It was 0.20's ReplicationTargetChooser that threw me off there. Reopening to fix against 0.20.
> Design doc is wrong about default block placement policy.
> ---------------------------------------------------------
>
> Key: HDFS-2547
> URL: https://issues.apache.org/jira/browse/HDFS-2547
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.20.1
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Fix For: 0.24.0
>
> Attachments: HDFS-2547.patch
>
>
> bq. For the common case, when the replication factor is three, HDFS's placement policy is to put one replica on one node in the local rack, another on a node in a different (remote) rack, and the last on a different node in the same *remote* rack.
> Should actually be: "and the last on a different node in the same *local* rack."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2547) ReplicationTargetChooser has incorrect
block placement comments
Posted by "Harsh J (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HDFS-2547:
--------------------------
Description:
{code}
/** The class is responsible for choosing the desired number of targets
* for placing block replicas.
* The replica placement strategy is that if the writer is on a datanode,
* the 1st replica is placed on the local machine,
* otherwise a random datanode. The 2nd replica is placed on a datanode
* that is on a different rack. The 3rd replica is placed on a datanode
* which is on the same rack as the **first replca**.
*/
{code}
That should read "second replica". The test cases confirm that this is the behavior, as well as the docs.
was:
bq. For the common case, when the replication factor is three, HDFS's placement policy is to put one replica on one node in the local rack, another on a node in a different (remote) rack, and the last on a different node in the same *remote* rack.
Should actually be: "and the last on a different node in the same *local* rack."
Fix Version/s: (was: 0.24.0)
0.20.206.0
Summary: ReplicationTargetChooser has incorrect block placement comments (was: Design doc is wrong about default block placement policy.)
Updating title/topic for 0.20
> ReplicationTargetChooser has incorrect block placement comments
> ---------------------------------------------------------------
>
> Key: HDFS-2547
> URL: https://issues.apache.org/jira/browse/HDFS-2547
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.20.1
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Fix For: 0.20.206.0
>
> Attachments: HDFS-2547.patch
>
>
> {code}
> /** The class is responsible for choosing the desired number of targets
> * for placing block replicas.
> * The replica placement strategy is that if the writer is on a datanode,
> * the 1st replica is placed on the local machine,
> * otherwise a random datanode. The 2nd replica is placed on a datanode
> * that is on a different rack. The 3rd replica is placed on a datanode
> * which is on the same rack as the **first replca**.
> */
> {code}
> That should read "second replica". The test cases confirm that this is the behavior, as well as the docs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2547) ReplicationTargetChooser has incorrect
block placement comments
Posted by "Harsh J (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HDFS-2547:
--------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks Aaron for your persistence on this one. Committed to branch-1.
> ReplicationTargetChooser has incorrect block placement comments
> ---------------------------------------------------------------
>
> Key: HDFS-2547
> URL: https://issues.apache.org/jira/browse/HDFS-2547
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.20.1
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Fix For: 1.1.0
>
> Attachments: HDFS-2547.patch, HDFS-2547.patch
>
>
> {code}
> /** The class is responsible for choosing the desired number of targets
> * for placing block replicas.
> * The replica placement strategy is that if the writer is on a datanode,
> * the 1st replica is placed on the local machine,
> * otherwise a random datanode. The 2nd replica is placed on a datanode
> * that is on a different rack. The 3rd replica is placed on a datanode
> * which is on the same rack as the **first replca**.
> */
> {code}
> That should read "second replica". The test cases confirm that this is the behavior, as well as the docs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2547) Design doc is wrong about default
block placement policy.
Posted by "Harsh J (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HDFS-2547:
--------------------------
Attachment: HDFS-2547.patch
> Design doc is wrong about default block placement policy.
> ---------------------------------------------------------
>
> Key: HDFS-2547
> URL: https://issues.apache.org/jira/browse/HDFS-2547
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.20.1
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Fix For: 0.24.0
>
> Attachments: HDFS-2547.patch
>
>
> bq. For the common case, when the replication factor is three, HDFS's placement policy is to put one replica on one node in the local rack, another on a node in a different (remote) rack, and the last on a different node in the same *remote* rack.
> Should actually be: "and the last on a different node in the same *local* rack."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2547) Design doc is wrong about default
block placement policy.
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147536#comment-13147536 ]
Hadoop QA commented on HDFS-2547:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12503161/HDFS-2547.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.TestFileAppend
org.apache.hadoop.hdfs.TestDFSRemove
org.apache.hadoop.hdfs.TestFileAppend2
org.apache.hadoop.hdfs.TestBalancerBandwidth
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1550//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1550//console
This message is automatically generated.
> Design doc is wrong about default block placement policy.
> ---------------------------------------------------------
>
> Key: HDFS-2547
> URL: https://issues.apache.org/jira/browse/HDFS-2547
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.20.1
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Fix For: 0.24.0
>
> Attachments: HDFS-2547.patch
>
>
> bq. For the common case, when the replication factor is three, HDFS's placement policy is to put one replica on one node in the local rack, another on a node in a different (remote) rack, and the last on a different node in the same *remote* rack.
> Should actually be: "and the last on a different node in the same *local* rack."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2547) ReplicationTargetChooser has incorrect
block placement comments
Posted by "Harsh J (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HDFS-2547:
--------------------------
Status: Patch Available (was: Reopened)
> ReplicationTargetChooser has incorrect block placement comments
> ---------------------------------------------------------------
>
> Key: HDFS-2547
> URL: https://issues.apache.org/jira/browse/HDFS-2547
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.20.1
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Fix For: 0.20.206.0
>
> Attachments: HDFS-2547.patch, HDFS-2547.patch
>
>
> {code}
> /** The class is responsible for choosing the desired number of targets
> * for placing block replicas.
> * The replica placement strategy is that if the writer is on a datanode,
> * the 1st replica is placed on the local machine,
> * otherwise a random datanode. The 2nd replica is placed on a datanode
> * that is on a different rack. The 3rd replica is placed on a datanode
> * which is on the same rack as the **first replca**.
> */
> {code}
> That should read "second replica". The test cases confirm that this is the behavior, as well as the docs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2547) Design doc is wrong about default
block placement policy.
Posted by "Harsh J (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HDFS-2547:
--------------------------
Status: Patch Available (was: Open)
> Design doc is wrong about default block placement policy.
> ---------------------------------------------------------
>
> Key: HDFS-2547
> URL: https://issues.apache.org/jira/browse/HDFS-2547
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.20.1
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Fix For: 0.24.0
>
> Attachments: HDFS-2547.patch
>
>
> bq. For the common case, when the replication factor is three, HDFS's placement policy is to put one replica on one node in the local rack, another on a node in a different (remote) rack, and the last on a different node in the same *remote* rack.
> Should actually be: "and the last on a different node in the same *local* rack."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2547) Design doc is wrong about default
block placement policy.
Posted by "Harsh J (Reopened) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J reopened HDFS-2547:
---------------------------
> Design doc is wrong about default block placement policy.
> ---------------------------------------------------------
>
> Key: HDFS-2547
> URL: https://issues.apache.org/jira/browse/HDFS-2547
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.20.1
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Fix For: 0.24.0
>
> Attachments: HDFS-2547.patch
>
>
> bq. For the common case, when the replication factor is three, HDFS's placement policy is to put one replica on one node in the local rack, another on a node in a different (remote) rack, and the last on a different node in the same *remote* rack.
> Should actually be: "and the last on a different node in the same *local* rack."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2547) ReplicationTargetChooser has
incorrect block placement comments
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148467#comment-13148467 ]
Hadoop QA commented on HDFS-2547:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12503369/HDFS-2547.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1553//console
This message is automatically generated.
> ReplicationTargetChooser has incorrect block placement comments
> ---------------------------------------------------------------
>
> Key: HDFS-2547
> URL: https://issues.apache.org/jira/browse/HDFS-2547
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.20.1
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Fix For: 0.20.206.0
>
> Attachments: HDFS-2547.patch, HDFS-2547.patch
>
>
> {code}
> /** The class is responsible for choosing the desired number of targets
> * for placing block replicas.
> * The replica placement strategy is that if the writer is on a datanode,
> * the 1st replica is placed on the local machine,
> * otherwise a random datanode. The 2nd replica is placed on a datanode
> * that is on a different rack. The 3rd replica is placed on a datanode
> * which is on the same rack as the **first replca**.
> */
> {code}
> That should read "second replica". The test cases confirm that this is the behavior, as well as the docs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira