You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Steve Loughran (Created) (JIRA)" <ji...@apache.org> on 2011/10/22 17:49:32 UTC
[jira] [Created] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
BlockManager cross-rack replication checks only work for ScriptBasedMapping
---------------------------------------------------------------------------
Key: HDFS-2492
URL: https://issues.apache.org/jira/browse/HDFS-2492
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.23.0, 0.24.0
Reporter: Steve Loughran
The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
This is because the BlockManager sets its rack checking flag if there is a filename key
{code}
shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
{code}
yet this filename key is only used if the topology mapper defined by
{code}
DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
{code}
is an instance of {{ScriptBasedMapping}}
If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Status: Open (was: Patch Available)
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.24.0, 0.23.3
>
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Status: Patch Available (was: Open)
patch is ready to be applied to trunk
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Priority: Blocker
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Attachment: HDFS-2492-blockmanager.patch
This is exactly the same patch as before, I am just trying to trigger jenkins to re-run this test as DistributedUpgrade is working happily locally on every test run
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.24.0, 0.23.1
>
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Fix Version/s: (was: 0.23.3)
(was: 0.24.0)
3.0.0
2.0.0
Affects Version/s: (was: 0.24.0)
(was: 0.23.0)
3.0.0
2.0.0
1.0.1
1.0.2
Status: Patch Available (was: Open)
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.2, 1.0.1, 2.0.0, 3.0.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133419#comment-13133419 ]
Steve Loughran commented on HDFS-2492:
--------------------------------------
The short term workaround is to document that regardless of how topology is determined, to make HDFS care about whether or not blocks are all on the same rack, you need to set this filename key.
The medium term workaround is to set the {{shouldCheckForEnoughRacks}} flag if the key is set or the switch mapping implementation class is anything other than an instance of {{ScriptBasedMapping}}. That is somewhat inelegant.
Ideally the {{DNSToSwitchMapping}} interface should be extended so that it can declare whether there are multiple racks, and if there is more than one rack then the block mapper should go rack aware. Having knowledge of the expected number of racks could be of use for making other parts of the system multi-rack for better availability.
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167613#comment-13167613 ]
Hadoop QA commented on HDFS-2492:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12505903/HDFS-2492-blockmanager.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javadoc. The javadoc tool appears to have generated 90 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 12 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.server.datanode.TestBlockReport
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
org.apache.hadoop.hdfs.TestDfsOverAvroRpc
org.apache.hadoop.hdfs.server.namenode.TestBlockUnderConstruction
org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1688//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1688//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs-httpfs.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1688//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1688//console
This message is automatically generated.
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Blocker
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157586#comment-13157586 ]
Hadoop QA commented on HDFS-2492:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12504094/HDFS-2492-blockmanager.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager
org.apache.hadoop.fs.viewfs.TestViewFsHdfs
org.apache.hadoop.hdfs.server.datanode.TestBlockReport
org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1609//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1609//console
This message is automatically generated.
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Priority: Blocker
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135781#comment-13135781 ]
Steve Loughran commented on HDFS-2492:
--------------------------------------
I'm not sure it's quite so serious, as for the normal script based mapper it works; it looks for the script file and assumes the presence of the file means cross-rack.
It's only for people who have written their own DNS mapper that the problem will arise, and the problem isn't topology mapping, its that the replication logic that tries to ensure that replicas are spread across multiple racks.
The risk then is: if you have a custom DNS mapper, all your blocks may end up on the same rack, making you less resilient to rack failures. Your data is still spread across machines, and the actual placement code itself will work to place the blocks in multiple racks. The only issue is that at a switch or rack failure is slightly more likely to take all replicas of a block offline until the switch comes back up.
It's easy to fix though, I will try and do something today.
Before then: who is running a custom DNS Mapper? Do you expect it to work unchanged on 0.23+, or is adding another method to an interface something you can handle?
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Priority: Blocker
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138647#comment-13138647 ]
Steve Loughran commented on HDFS-2492:
--------------------------------------
HADOOP-7777 contains all the hadoop-common code to add a new predicate to the switch mappers (via a base class and a query method that handles mappers that are not subclasses of that); once that is in the change to block manager will be trivial.
reviews of that patch would be welcome
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Priority: Blocker
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Priority: Minor (was: Blocker)
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Attachment: HDFS-2492-blockmanager.patch
This patch gets the ! operator the right way round.
Without HADOOP-7878 it should pass all but one test; {{TestBlocksWithNotEnoughRacks}} needs that fix.
No new tests.
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Blocker
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Status: Open (was: Patch Available)
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Status: Patch Available (was: In Progress)
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215721#comment-13215721 ]
Hadoop QA commented on HDFS-2492:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12515926/HDFS-2492-blockmanager.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1900//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1900//console
This message is automatically generated.
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.24.0, 0.23.1
>
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259887#comment-13259887 ]
Steve Loughran commented on HDFS-2492:
--------------------------------------
I'm +1 for this going in to trunk and 2.0; the test is that none of the minidfs tests fail.
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.1, 1.0.2, 2.0.0, 3.0.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Fix Version/s: 0.23.1
0.24.0
Status: Patch Available (was: Open)
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.24.0, 0.23.1
>
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Eli Collins (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-2492:
------------------------------
Priority: Blocker (was: Major)
Seems like this should be a blocker given that it can cause blocks to be miss-replicated.
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Priority: Blocker
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HDFS-2492:
--------------------------------
Fix Version/s: (was: 0.23.1)
0.23.3
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.24.0, 0.23.3
>
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Status: Open (was: Patch Available)
waiting for the next common snapshot to submit
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Blocker
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Attachment: HDFS-2492-blockmanager.patch
Merge of HA branch obsoleted this patch; updated version.
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.24.0, 0.23.3
>
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Status: Open (was: Patch Available)
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Status: Open (was: Patch Available)
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Status: Patch Available (was: Open)
retrying to see if TestDistributedUpgrade works this time ; it seems brittle
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134024#comment-13134024 ]
Steve Loughran commented on HDFS-2492:
--------------------------------------
an alternative to extending {{DNSToSwitchMapping}} would be to extend it with a {{MultiRackDNSToSwitchMapping}} interface that would have a boolean {{isMultiRack()}} condition to say whether or not the mapping was multirack; the script mapper would implement this and decide for itself whether or not a script existed and hence was multirack.
For backwards compatibility, any implementation that didn't implement the new interface would be viewed as some custom plug in and could be assumed to be multi-rack. why else would someone write a DNS mapper?
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Assignee: Steve Loughran
Status: Open (was: Patch Available)
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Blocker
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Attachment: HDFS-2492.patch
in sync w/ trunk
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.1, 1.0.2, 2.0.0, 3.0.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215772#comment-13215772 ]
Steve Loughran commented on HDFS-2492:
--------------------------------------
no tests because the tests for this are in hadoop common; the existing test suite are the regression test, and now that TestDistributedUpgrade is no longer (spuriously) failing, I plan to commit this unless anyone vetos it
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.24.0, 0.23.1
>
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257913#comment-13257913 ]
Hadoop QA commented on HDFS-2492:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12523415/HDFS-2492.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2305//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2305//console
This message is automatically generated.
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.1, 1.0.2, 2.0.0, 3.0.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Attachment: HDFS-2492-blockmanager.patch
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210971#comment-13210971 ]
Hadoop QA commented on HDFS-2492:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12515086/HDFS-2492-blockmanager.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1884//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1884//console
This message is automatically generated.
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226063#comment-13226063 ]
Steve Loughran commented on HDFS-2492:
--------------------------------------
+1 for this; it's needed to complete the roll out of the (still optional) topology base class; all tests are working.
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.24.0, 0.23.3
>
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Attachment: HDFS-2492-blockmanager.patch
patch against trunk; only valid with HADOOP-7777 committed
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Priority: Blocker
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HDFS-2492) BlockManager cross-rack
replication checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Work started) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HDFS-2492 started by Steve Loughran.
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Blocker
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222395#comment-13222395 ]
Hadoop QA commented on HDFS-2492:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12517085/HDFS-2492-blockmanager.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1950//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1950//console
This message is automatically generated.
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.24.0, 0.23.3
>
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Status: Patch Available (was: In Progress)
resubmit now that HADOOP-7878 is in
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Blocker
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Status: Patch Available (was: Open)
resubmit
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Status: Patch Available (was: Open)
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.24.0, 0.23.3
>
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Attachment: HDFS-2492-blockmanager.patch
This is the patch to the block manager that uses HADOOP-7777 to state when a DNS mapper is explicitly single switch.
It adds a public getter on the DatanodeManager to provide access to its DNS mapping; this could be downgraded to package-scope.
I'm not submitting this as a patch for automated review (yet) because of its dependencies.
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Priority: Blocker
> Attachments: HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HDFS-2492) BlockManager cross-rack
replication checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Work started) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HDFS-2492 started by Steve Loughran.
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Status: Open (was: Patch Available)
cancelling until I tweak some (new) method names in the topo classes
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha, 1.0.2, 1.0.1, 3.0.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2492) BlockManager cross-rack replication
checks only work for ScriptBasedMapping
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2492:
---------------------------------
Status: Open (was: Patch Available)
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
> Key: HDFS-2492
> URL: https://issues.apache.org/jira/browse/HDFS-2492
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.24.0, 0.23.3
>
> Attachments: HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch, HDFS-2492-blockmanager.patch
>
>
> The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;
> {code}
> yet this filename key is only used if the topology mapper defined by
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira