You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Jitendra Nath Pandey (JIRA)" <ji...@apache.org> on 2011/02/25 21:19:21 UTC
[jira] Created: (HDFS-1655) Federation: DatablockScanner should
scan blocks for all the block pools.
Federation: DatablockScanner should scan blocks for all the block pools.
------------------------------------------------------------------------
Key: HDFS-1655
URL: https://issues.apache.org/jira/browse/HDFS-1655
Project: Hadoop HDFS
Issue Type: New Feature
Affects Versions: Federation Branch
Reporter: Jitendra Nath Pandey
Assignee: Jitendra Nath Pandey
Fix For: Federation Branch
DataBlockScanner changes are needed to work with federation.
Goal is to have DataBlockScanner visit one volume at a time, scanning block pools under it one at a time.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1655) Federation: DatablockScanner should
scan blocks for all the block pools.
Posted by "Jitendra Nath Pandey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jitendra Nath Pandey updated HDFS-1655:
---------------------------------------
Attachment: HDFS-1655.1.patch
Patch uploaded.
> Federation: DatablockScanner should scan blocks for all the block pools.
> ------------------------------------------------------------------------
>
> Key: HDFS-1655
> URL: https://issues.apache.org/jira/browse/HDFS-1655
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: Federation Branch
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
> Fix For: Federation Branch
>
> Attachments: HDFS-1655.1.patch
>
>
> DataBlockScanner changes are needed to work with federation.
> Goal is to have DataBlockScanner visit one volume at a time, scanning block pools under it one at a time.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1655) Federation: DatablockScanner should
scan blocks for all the block pools.
Posted by "Jitendra Nath Pandey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jitendra Nath Pandey updated HDFS-1655:
---------------------------------------
Issue Type: Sub-task (was: New Feature)
Parent: HDFS-1052
> Federation: DatablockScanner should scan blocks for all the block pools.
> ------------------------------------------------------------------------
>
> Key: HDFS-1655
> URL: https://issues.apache.org/jira/browse/HDFS-1655
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: Federation Branch
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
> Fix For: Federation Branch
>
> Attachments: HDFS-1655.1.patch, HDFS-1655.2.patch
>
>
> DataBlockScanner changes are needed to work with federation.
> Goal is to have DataBlockScanner visit one volume at a time, scanning block pools under it one at a time.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HDFS-1655) Federation: DatablockScanner should
scan blocks for all the block pools.
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas resolved HDFS-1655.
-----------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
I committed the patch
> Federation: DatablockScanner should scan blocks for all the block pools.
> ------------------------------------------------------------------------
>
> Key: HDFS-1655
> URL: https://issues.apache.org/jira/browse/HDFS-1655
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: Federation Branch
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
> Fix For: Federation Branch
>
> Attachments: HDFS-1655.1.patch, HDFS-1655.2.patch
>
>
> DataBlockScanner changes are needed to work with federation.
> Goal is to have DataBlockScanner visit one volume at a time, scanning block pools under it one at a time.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1655) Federation: DatablockScanner should
scan blocks for all the block pools.
Posted by "Jitendra Nath Pandey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001026#comment-13001026 ]
Jitendra Nath Pandey commented on HDFS-1655:
--------------------------------------------
Changes in the patch.
- The block pools are scanned one at a time. Most of the original DataBlockScanner functionality is moved to BlockPoolScanner. The DataBlockScanner runs a scanner thread as before and selects next block pool to scan and invokes BlockPoolScanner to do the scan work.
- One BlockPoolScanner is instantiated for each block pool which maintains the state of the scan, blockMap and blockInfoSet similar to what DataBlockScanner had before.
- The current verification log file is created for each block pool and once the block pool scan is complete it is rolled into the previous log file, before processing the next block pool. Therefore, at any instant there should be only one current verification log file.
- To find the next block pool to scan, the datablock scanner iterates over the set of blockPools. However, if a datanode is restarted it needs to pick the block pool which was being scanned before. Therefore, first it looks for the block pool that contains the current verification log file and that is used as the starting block pool id. If no current files are found it starts with first block-pool.
> Federation: DatablockScanner should scan blocks for all the block pools.
> ------------------------------------------------------------------------
>
> Key: HDFS-1655
> URL: https://issues.apache.org/jira/browse/HDFS-1655
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: Federation Branch
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
> Fix For: Federation Branch
>
> Attachments: HDFS-1655.1.patch
>
>
> DataBlockScanner changes are needed to work with federation.
> Goal is to have DataBlockScanner visit one volume at a time, scanning block pools under it one at a time.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1655) Federation: DatablockScanner should
scan blocks for all the block pools.
Posted by "Jitendra Nath Pandey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jitendra Nath Pandey updated HDFS-1655:
---------------------------------------
Attachment: HDFS-1655.2.patch
Patch rebased against latest state of the branch.
> Federation: DatablockScanner should scan blocks for all the block pools.
> ------------------------------------------------------------------------
>
> Key: HDFS-1655
> URL: https://issues.apache.org/jira/browse/HDFS-1655
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: Federation Branch
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
> Fix For: Federation Branch
>
> Attachments: HDFS-1655.1.patch, HDFS-1655.2.patch
>
>
> DataBlockScanner changes are needed to work with federation.
> Goal is to have DataBlockScanner visit one volume at a time, scanning block pools under it one at a time.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1655) Federation: DatablockScanner should
scan blocks for all the block pools.
Posted by "Jitendra Nath Pandey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001193#comment-13001193 ]
Jitendra Nath Pandey commented on HDFS-1655:
--------------------------------------------
test-patch results.
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 5 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] -1 findbugs. The patch appears to introduce 1 new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
[exec] +1 system tests framework. The patch passed system tests framework compile.
The findbugs warning is for directory scanner, not related to this patch.
> Federation: DatablockScanner should scan blocks for all the block pools.
> ------------------------------------------------------------------------
>
> Key: HDFS-1655
> URL: https://issues.apache.org/jira/browse/HDFS-1655
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: Federation Branch
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
> Fix For: Federation Branch
>
> Attachments: HDFS-1655.1.patch, HDFS-1655.2.patch
>
>
> DataBlockScanner changes are needed to work with federation.
> Goal is to have DataBlockScanner visit one volume at a time, scanning block pools under it one at a time.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1655) Federation: DatablockScanner should
scan blocks for all the block pools.
Posted by "Jitendra Nath Pandey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014103#comment-13014103 ]
Jitendra Nath Pandey commented on HDFS-1655:
--------------------------------------------
This patch removes the REMOTE_READ scan type from the DataBlockScanner. Client still verifies the block and namenode is informed if any corruption detected, but the block scanner is not notified to update its scan status for that block. The reasons:
1. The datanode verification runs in background and should be an independent verification mechanism internal to the datanode.
2. It is more complicated with federation because block pools are scanned in round robin and a client might verify a block for a block pool scanner which may not be running.
3. The goal of the optimization was just to skip verification of the block which has been read by the client. This optimization doesn't seem to justify the complexity. Anyway the scanner doesn't verify a block again if it was recently verified.
> Federation: DatablockScanner should scan blocks for all the block pools.
> ------------------------------------------------------------------------
>
> Key: HDFS-1655
> URL: https://issues.apache.org/jira/browse/HDFS-1655
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: Federation Branch
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
> Fix For: Federation Branch
>
> Attachments: HDFS-1655.1.patch, HDFS-1655.2.patch
>
>
> DataBlockScanner changes are needed to work with federation.
> Goal is to have DataBlockScanner visit one volume at a time, scanning block pools under it one at a time.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1655) Federation: DatablockScanner should
scan blocks for all the block pools.
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001197#comment-13001197 ]
Suresh Srinivas commented on HDFS-1655:
---------------------------------------
+1 for the change.
> Federation: DatablockScanner should scan blocks for all the block pools.
> ------------------------------------------------------------------------
>
> Key: HDFS-1655
> URL: https://issues.apache.org/jira/browse/HDFS-1655
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: Federation Branch
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
> Fix For: Federation Branch
>
> Attachments: HDFS-1655.1.patch, HDFS-1655.2.patch
>
>
> DataBlockScanner changes are needed to work with federation.
> Goal is to have DataBlockScanner visit one volume at a time, scanning block pools under it one at a time.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira