Redundant block searches in BlockManager.
-----------------------------------------
Key: HDFS-511
URL: https://issues.apache.org/jira/browse/HDFS-511
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Affects Versions: 0.21.0
Reporter: Konstantin Shvachko
Assignee: Konstantin Shvachko
Fix For: 0.21.0
Some searches for a block in {{blocksMap}} are redundant and should be eliminated.
{{BlockManager.addStoredBlock()}} contain unnecessary explicit and hidden (inside {{markBlockAsCorrupt()}}) calls of {{blocksMap.getStoredBlock()}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[ https://issues.apache.org/jira/browse/HDFS-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737682#action_12737682 ]
Hudson commented on HDFS-511:
-----------------------------
Integrated in Hadoop-Hdfs-trunk #35 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/35/])
. Remove redundant block searches in BlockManager. Contributed by Konstantin Shvachko.
> Redundant block searches in BlockManager.
> -----------------------------------------
>
> Key: HDFS-511
> URL: https://issues.apache.org/jira/browse/HDFS-511
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BlockManager.patch, BlockManager.patch, BlockManager.patch
>
>
> Some searches for a block in {{blocksMap}} are redundant and should be eliminated.
> {{BlockManager.addStoredBlock()}} contain unnecessary explicit and hidden (inside {{markBlockAsCorrupt()}}) calls of {{blocksMap.getStoredBlock()}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[ https://issues.apache.org/jira/browse/HDFS-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737352#action_12737352 ]
Hairong Kuang commented on HDFS-511:
------------------------------------
+1 This looks good to me.
> Redundant block searches in BlockManager.
> -----------------------------------------
>
> Key: HDFS-511
> URL: https://issues.apache.org/jira/browse/HDFS-511
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BlockManager.patch, BlockManager.patch, BlockManager.patch
>
>
> Some searches for a block in {{blocksMap}} are redundant and should be eliminated.
> {{BlockManager.addStoredBlock()}} contain unnecessary explicit and hidden (inside {{markBlockAsCorrupt()}}) calls of {{blocksMap.getStoredBlock()}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[ https://issues.apache.org/jira/browse/HDFS-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737818#action_12737818 ]
Hadoop QA commented on HDFS-511:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12415060/BlockManager.patch
against trunk revision 799769.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/41/console
This message is automatically generated.
> Redundant block searches in BlockManager.
> -----------------------------------------
>
> Key: HDFS-511
> URL: https://issues.apache.org/jira/browse/HDFS-511
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BlockManager.patch, BlockManager.patch, BlockManager.patch
>
>
> Some searches for a block in {{blocksMap}} are redundant and should be eliminated.
> {{BlockManager.addStoredBlock()}} contain unnecessary explicit and hidden (inside {{markBlockAsCorrupt()}}) calls of {{blocksMap.getStoredBlock()}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[ https://issues.apache.org/jira/browse/HDFS-511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HDFS-511:
-------------------------------------
Status: Patch Available (was: Open)
> Redundant block searches in BlockManager.
> -----------------------------------------
>
> Key: HDFS-511
> URL: https://issues.apache.org/jira/browse/HDFS-511
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BlockManager.patch
>
>
> Some searches for a block in {{blocksMap}} are redundant and should be eliminated.
> {{BlockManager.addStoredBlock()}} contain unnecessary explicit and hidden (inside {{markBlockAsCorrupt()}}) calls of {{blocksMap.getStoredBlock()}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[ https://issues.apache.org/jira/browse/HDFS-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737013#action_12737013 ]
Hairong Kuang commented on HDFS-511:
------------------------------------
> markBlockAsCorrupt() takes BlockInfo instead of Block as a parameter, thus there is no need to search for the block in the BlocksMap.
I would prefer to keep BlockManager#markBlockAsCorrupt(Block, ...) to make BlockManager APIs consistently to take Block as a parameter. But to have a private method BlockManager#markBlockAsCorrupt(BlockInfo...) to avoid redundant lookup.
> Redundant block searches in BlockManager.
> -----------------------------------------
>
> Key: HDFS-511
> URL: https://issues.apache.org/jira/browse/HDFS-511
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BlockManager.patch, BlockManager.patch
>
>
> Some searches for a block in {{blocksMap}} are redundant and should be eliminated.
> {{BlockManager.addStoredBlock()}} contain unnecessary explicit and hidden (inside {{markBlockAsCorrupt()}}) calls of {{blocksMap.getStoredBlock()}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[ https://issues.apache.org/jira/browse/HDFS-511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HDFS-511:
-------------------------------------
Attachment: BlockManager.patch
# {{markBlockAsCorrupt()}} takes {{BlockInfo}} instead of {{Block}} as a parameter, thus there is no need to search for the block in the {{BlocksMap}}.
# Usage of {{block}} variable in {{addStoredBlock()}} is replaced by {{storedBlock}} wherever appropriate, so there is no need to explicitly search for the block once more.
# {{FSDirectory.addBlock()}} should not call {{getStoredBlock(block)}} because {{addINode()}} just before that already returns the block from the block map.
# {{BlockManager.addBlock(Block)}} is renamed to {{getValidLocations()}} because it does not add anything anywhere it just returns the locations of the block which are not in {{recentInvalidateSets}}.
> Redundant block searches in BlockManager.
> -----------------------------------------
>
> Key: HDFS-511
> URL: https://issues.apache.org/jira/browse/HDFS-511
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BlockManager.patch
>
>
> Some searches for a block in {{blocksMap}} are redundant and should be eliminated.
> {{BlockManager.addStoredBlock()}} contain unnecessary explicit and hidden (inside {{markBlockAsCorrupt()}}) calls of {{blocksMap.getStoredBlock()}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[ https://issues.apache.org/jira/browse/HDFS-511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HDFS-511:
-------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
This has been committed.
> Redundant block searches in BlockManager.
> -----------------------------------------
>
> Key: HDFS-511
> URL: https://issues.apache.org/jira/browse/HDFS-511
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BlockManager.patch, BlockManager.patch, BlockManager.patch
>
>
> Some searches for a block in {{blocksMap}} are redundant and should be eliminated.
> {{BlockManager.addStoredBlock()}} contain unnecessary explicit and hidden (inside {{markBlockAsCorrupt()}}) calls of {{blocksMap.getStoredBlock()}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[ https://issues.apache.org/jira/browse/HDFS-511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HDFS-511:
-------------------------------------
Attachment: BlockManager.patch
Don't see any rational reasons to keep the old method in BlockManager, but here it is.
I had to rename it to {{findAndMarkBlockAsCorrupt()}} because if there are two methods with the same name it would be unclear which method is actually called for a stored block.
> Redundant block searches in BlockManager.
> -----------------------------------------
>
> Key: HDFS-511
> URL: https://issues.apache.org/jira/browse/HDFS-511
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BlockManager.patch, BlockManager.patch, BlockManager.patch
>
>
> Some searches for a block in {{blocksMap}} are redundant and should be eliminated.
> {{BlockManager.addStoredBlock()}} contain unnecessary explicit and hidden (inside {{markBlockAsCorrupt()}}) calls of {{blocksMap.getStoredBlock()}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[ https://issues.apache.org/jira/browse/HDFS-511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HDFS-511:
-------------------------------------
Attachment: BlockManager.patch
This fixes one findbug in the previous version of the patch.
The test-patch targets complains there are no new tests, which is correct since this does not introduce any new behavior.
{code}
[exec] There appear to be 154 release audit warnings before the patch and 154 release audit warnings after applying the patch.
[exec] -1 overall.
[exec] +1 @author. The patch does not contain any @author tags.
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no new tests are needed for this patch.
[exec] Also please list what manual steps were performed to verify this patch.
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec] ======================================================================
[exec] ======================================================================
[exec] Finished build.
[exec] ======================================================================
[exec] ======================================================================
{code}
> Redundant block searches in BlockManager.
> -----------------------------------------
>
> Key: HDFS-511
> URL: https://issues.apache.org/jira/browse/HDFS-511
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BlockManager.patch, BlockManager.patch
>
>
> Some searches for a block in {{blocksMap}} are redundant and should be eliminated.
> {{BlockManager.addStoredBlock()}} contain unnecessary explicit and hidden (inside {{markBlockAsCorrupt()}}) calls of {{blocksMap.getStoredBlock()}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.