You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Konstantin Shvachko (JIRA)" <ji...@apache.org> on 2009/07/29 01:31:15 UTC

[jira] Created: (HDFS-511) Redundant block searches in BlockManager.

Redundant block searches in BlockManager.
-----------------------------------------

                 Key: HDFS-511
                 URL: https://issues.apache.org/jira/browse/HDFS-511
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: name-node
    Affects Versions: 0.21.0
            Reporter: Konstantin Shvachko
            Assignee: Konstantin Shvachko
             Fix For: 0.21.0


Some searches for a block in {{blocksMap}} are redundant and should be eliminated.
{{BlockManager.addStoredBlock()}} contain unnecessary explicit and hidden (inside {{markBlockAsCorrupt()}}) calls of {{blocksMap.getStoredBlock()}}.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-511) Redundant block searches in BlockManager.

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737682#action_12737682 ] 

Hudson commented on HDFS-511:
-----------------------------

Integrated in Hadoop-Hdfs-trunk #35 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/35/])
    . Remove redundant block searches in BlockManager. Contributed by Konstantin Shvachko.


> Redundant block searches in BlockManager.
> -----------------------------------------
>
>                 Key: HDFS-511
>                 URL: https://issues.apache.org/jira/browse/HDFS-511
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.21.0
>
>         Attachments: BlockManager.patch, BlockManager.patch, BlockManager.patch
>
>
> Some searches for a block in {{blocksMap}} are redundant and should be eliminated.
> {{BlockManager.addStoredBlock()}} contain unnecessary explicit and hidden (inside {{markBlockAsCorrupt()}}) calls of {{blocksMap.getStoredBlock()}}.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-511) Redundant block searches in BlockManager.

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737352#action_12737352 ] 

Hairong Kuang commented on HDFS-511:
------------------------------------

+1 This looks good to me.

> Redundant block searches in BlockManager.
> -----------------------------------------
>
>                 Key: HDFS-511
>                 URL: https://issues.apache.org/jira/browse/HDFS-511
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.21.0
>
>         Attachments: BlockManager.patch, BlockManager.patch, BlockManager.patch
>
>
> Some searches for a block in {{blocksMap}} are redundant and should be eliminated.
> {{BlockManager.addStoredBlock()}} contain unnecessary explicit and hidden (inside {{markBlockAsCorrupt()}}) calls of {{blocksMap.getStoredBlock()}}.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-511) Redundant block searches in BlockManager.

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737818#action_12737818 ] 

Hadoop QA commented on HDFS-511:
--------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12415060/BlockManager.patch
  against trunk revision 799769.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/41/console

This message is automatically generated.

> Redundant block searches in BlockManager.
> -----------------------------------------
>
>                 Key: HDFS-511
>                 URL: https://issues.apache.org/jira/browse/HDFS-511
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.21.0
>
>         Attachments: BlockManager.patch, BlockManager.patch, BlockManager.patch
>
>
> Some searches for a block in {{blocksMap}} are redundant and should be eliminated.
> {{BlockManager.addStoredBlock()}} contain unnecessary explicit and hidden (inside {{markBlockAsCorrupt()}}) calls of {{blocksMap.getStoredBlock()}}.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-511) Redundant block searches in BlockManager.

Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HDFS-511:
-------------------------------------

    Status: Patch Available  (was: Open)

> Redundant block searches in BlockManager.
> -----------------------------------------
>
>                 Key: HDFS-511
>                 URL: https://issues.apache.org/jira/browse/HDFS-511
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.21.0
>
>         Attachments: BlockManager.patch
>
>
> Some searches for a block in {{blocksMap}} are redundant and should be eliminated.
> {{BlockManager.addStoredBlock()}} contain unnecessary explicit and hidden (inside {{markBlockAsCorrupt()}}) calls of {{blocksMap.getStoredBlock()}}.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-511) Redundant block searches in BlockManager.

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737013#action_12737013 ] 

Hairong Kuang commented on HDFS-511:
------------------------------------

> markBlockAsCorrupt() takes BlockInfo instead of Block as a parameter, thus there is no need to search for the block in the BlocksMap.
I would prefer to keep BlockManager#markBlockAsCorrupt(Block, ...) to make BlockManager APIs consistently to take Block as a parameter. But to have a private method BlockManager#markBlockAsCorrupt(BlockInfo...) to avoid redundant lookup.

> Redundant block searches in BlockManager.
> -----------------------------------------
>
>                 Key: HDFS-511
>                 URL: https://issues.apache.org/jira/browse/HDFS-511
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.21.0
>
>         Attachments: BlockManager.patch, BlockManager.patch
>
>
> Some searches for a block in {{blocksMap}} are redundant and should be eliminated.
> {{BlockManager.addStoredBlock()}} contain unnecessary explicit and hidden (inside {{markBlockAsCorrupt()}}) calls of {{blocksMap.getStoredBlock()}}.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-511) Redundant block searches in BlockManager.

Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HDFS-511:
-------------------------------------

    Attachment: BlockManager.patch

# {{markBlockAsCorrupt()}} takes {{BlockInfo}} instead of {{Block}} as a parameter, thus there is no need to search for the block in the {{BlocksMap}}.
# Usage of {{block}} variable in {{addStoredBlock()}} is replaced by {{storedBlock}} wherever appropriate, so there is no need to explicitly search for the block once more.
# {{FSDirectory.addBlock()}} should not call {{getStoredBlock(block)}} because {{addINode()}} just before that already returns the block from the block map.
# {{BlockManager.addBlock(Block)}} is renamed to {{getValidLocations()}} because it does not add anything anywhere it just returns the locations of the block which are not in {{recentInvalidateSets}}.

> Redundant block searches in BlockManager.
> -----------------------------------------
>
>                 Key: HDFS-511
>                 URL: https://issues.apache.org/jira/browse/HDFS-511
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.21.0
>
>         Attachments: BlockManager.patch
>
>
> Some searches for a block in {{blocksMap}} are redundant and should be eliminated.
> {{BlockManager.addStoredBlock()}} contain unnecessary explicit and hidden (inside {{markBlockAsCorrupt()}}) calls of {{blocksMap.getStoredBlock()}}.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-511) Redundant block searches in BlockManager.

Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HDFS-511:
-------------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

This has been committed.

> Redundant block searches in BlockManager.
> -----------------------------------------
>
>                 Key: HDFS-511
>                 URL: https://issues.apache.org/jira/browse/HDFS-511
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.21.0
>
>         Attachments: BlockManager.patch, BlockManager.patch, BlockManager.patch
>
>
> Some searches for a block in {{blocksMap}} are redundant and should be eliminated.
> {{BlockManager.addStoredBlock()}} contain unnecessary explicit and hidden (inside {{markBlockAsCorrupt()}}) calls of {{blocksMap.getStoredBlock()}}.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-511) Redundant block searches in BlockManager.

Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HDFS-511:
-------------------------------------

    Attachment: BlockManager.patch

Don't see any rational reasons to keep the old method in BlockManager, but here it is.
I had to rename it to {{findAndMarkBlockAsCorrupt()}} because if there are two methods with the same name it would be unclear which method is actually called for a stored block.

> Redundant block searches in BlockManager.
> -----------------------------------------
>
>                 Key: HDFS-511
>                 URL: https://issues.apache.org/jira/browse/HDFS-511
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.21.0
>
>         Attachments: BlockManager.patch, BlockManager.patch, BlockManager.patch
>
>
> Some searches for a block in {{blocksMap}} are redundant and should be eliminated.
> {{BlockManager.addStoredBlock()}} contain unnecessary explicit and hidden (inside {{markBlockAsCorrupt()}}) calls of {{blocksMap.getStoredBlock()}}.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-511) Redundant block searches in BlockManager.

Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HDFS-511:
-------------------------------------

    Attachment: BlockManager.patch

This fixes one findbug in the previous version of the patch.
The test-patch targets complains there are no new tests, which is correct since this does not introduce any new behavior.
{code}
[exec] There appear to be 154 release audit warnings before the patch and 154 release audit warnings after applying the patch.
[exec] -1 overall.  
[exec]     +1 @author.  The patch does not contain any @author tags.
[exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
[exec]                         Please justify why no new tests are needed for this patch.
[exec]                         Also please list what manual steps were performed to verify this patch.
[exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
[exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
[exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
[exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
[exec] ======================================================================
[exec] ======================================================================
[exec]     Finished build.
[exec] ======================================================================
[exec] ======================================================================
{code}

> Redundant block searches in BlockManager.
> -----------------------------------------
>
>                 Key: HDFS-511
>                 URL: https://issues.apache.org/jira/browse/HDFS-511
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.21.0
>
>         Attachments: BlockManager.patch, BlockManager.patch
>
>
> Some searches for a block in {{blocksMap}} are redundant and should be eliminated.
> {{BlockManager.addStoredBlock()}} contain unnecessary explicit and hidden (inside {{markBlockAsCorrupt()}}) calls of {{blocksMap.getStoredBlock()}}.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.