You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Konstantin Shvachko (JIRA)" <ji...@apache.org> on 2008/05/09 01:56:55 UTC
[jira] Created: (HADOOP-3369) Fast block processing during
name-node startup.
Fast block processing during name-node startup.
-----------------------------------------------
Key: HADOOP-3369
URL: https://issues.apache.org/jira/browse/HADOOP-3369
Project: Hadoop Core
Issue Type: Improvement
Components: dfs
Affects Versions: 0.17.0
Reporter: Konstantin Shvachko
Assignee: Konstantin Shvachko
Fix For: 0.18.0
The block report processing during the startup period should be optimized.
As noted in HADOOP-3022 during cluster startup all blocks are under-replicated
because they have not been reported by name-nodes yet.
Currently, we routinely move blocks to the neededReplications queue when they
are first reported and then remove them from the list when other nodes report it.
In ideal situation we end up adding all blocks into neededReplications queue first
only in order to remove all of them in the end.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3369) Fast block processing during
name-node startup.
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595717#action_12595717 ]
Hairong Kuang commented on HADOOP-3369:
---------------------------------------
Nice change! Simple but great startup performance improvement.
In FSnmensystem.processMisReplicatedBlocks, it is better to reset all kinds of queues in the very beginning, then simply add blocks to each queue with no need of removal.
> Fast block processing during name-node startup.
> -----------------------------------------------
>
> Key: HADOOP-3369
> URL: https://issues.apache.org/jira/browse/HADOOP-3369
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.18.0
>
> Attachments: fastBlockReports.patch
>
>
> The block report processing during the startup period should be optimized.
> As noted in HADOOP-3022 during cluster startup all blocks are under-replicated
> because they have not been reported by name-nodes yet.
> Currently, we routinely move blocks to the neededReplications queue when they
> are first reported and then remove them from the list when other nodes report it.
> In ideal situation we end up adding all blocks into neededReplications queue first
> only in order to remove all of them in the end.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3369) Fast block processing during
name-node startup.
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595767#action_12595767 ]
Hairong Kuang commented on HADOOP-3369:
---------------------------------------
+1 The patch looks good.
> Fast block processing during name-node startup.
> -----------------------------------------------
>
> Key: HADOOP-3369
> URL: https://issues.apache.org/jira/browse/HADOOP-3369
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.18.0
>
> Attachments: fastBlockReports.patch, fastBlockReports.patch
>
>
> The block report processing during the startup period should be optimized.
> As noted in HADOOP-3022 during cluster startup all blocks are under-replicated
> because they have not been reported by name-nodes yet.
> Currently, we routinely move blocks to the neededReplications queue when they
> are first reported and then remove them from the list when other nodes report it.
> In ideal situation we end up adding all blocks into neededReplications queue first
> only in order to remove all of them in the end.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3369) Fast block processing during
name-node startup.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-3369:
----------------------------------------
Hadoop Flags: [Reviewed]
Status: Patch Available (was: Open)
> Fast block processing during name-node startup.
> -----------------------------------------------
>
> Key: HADOOP-3369
> URL: https://issues.apache.org/jira/browse/HADOOP-3369
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.18.0
>
> Attachments: fastBlockReports.patch, fastBlockReports.patch
>
>
> The block report processing during the startup period should be optimized.
> As noted in HADOOP-3022 during cluster startup all blocks are under-replicated
> because they have not been reported by name-nodes yet.
> Currently, we routinely move blocks to the neededReplications queue when they
> are first reported and then remove them from the list when other nodes report it.
> In ideal situation we end up adding all blocks into neededReplications queue first
> only in order to remove all of them in the end.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3369) Fast block processing during
name-node startup.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-3369:
----------------------------------------
Status: Patch Available (was: Open)
> Fast block processing during name-node startup.
> -----------------------------------------------
>
> Key: HADOOP-3369
> URL: https://issues.apache.org/jira/browse/HADOOP-3369
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.18.0
>
> Attachments: fastBlockReports.patch
>
>
> The block report processing during the startup period should be optimized.
> As noted in HADOOP-3022 during cluster startup all blocks are under-replicated
> because they have not been reported by name-nodes yet.
> Currently, we routinely move blocks to the neededReplications queue when they
> are first reported and then remove them from the list when other nodes report it.
> In ideal situation we end up adding all blocks into neededReplications queue first
> only in order to remove all of them in the end.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3369) Fast block processing during
name-node startup.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595501#action_12595501 ]
Hadoop QA commented on HADOOP-3369:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12381727/fastBlockReports.patch
against trunk revision 654315.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2436/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2436/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2436/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2436/console
This message is automatically generated.
> Fast block processing during name-node startup.
> -----------------------------------------------
>
> Key: HADOOP-3369
> URL: https://issues.apache.org/jira/browse/HADOOP-3369
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.18.0
>
> Attachments: fastBlockReports.patch
>
>
> The block report processing during the startup period should be optimized.
> As noted in HADOOP-3022 during cluster startup all blocks are under-replicated
> because they have not been reported by name-nodes yet.
> Currently, we routinely move blocks to the neededReplications queue when they
> are first reported and then remove them from the list when other nodes report it.
> In ideal situation we end up adding all blocks into neededReplications queue first
> only in order to remove all of them in the end.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3369) Fast block processing during
name-node startup.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595785#action_12595785 ]
Hadoop QA commented on HADOOP-3369:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12381809/fastBlockReports.patch
against trunk revision 654973.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2446/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2446/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2446/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2446/console
This message is automatically generated.
> Fast block processing during name-node startup.
> -----------------------------------------------
>
> Key: HADOOP-3369
> URL: https://issues.apache.org/jira/browse/HADOOP-3369
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.18.0
>
> Attachments: fastBlockReports.patch, fastBlockReports.patch
>
>
> The block report processing during the startup period should be optimized.
> As noted in HADOOP-3022 during cluster startup all blocks are under-replicated
> because they have not been reported by name-nodes yet.
> Currently, we routinely move blocks to the neededReplications queue when they
> are first reported and then remove them from the list when other nodes report it.
> In ideal situation we end up adding all blocks into neededReplications queue first
> only in order to remove all of them in the end.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3369) Fast block processing during
name-node startup.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-3369:
----------------------------------------
Status: Open (was: Patch Available)
> Fast block processing during name-node startup.
> -----------------------------------------------
>
> Key: HADOOP-3369
> URL: https://issues.apache.org/jira/browse/HADOOP-3369
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.18.0
>
> Attachments: fastBlockReports.patch
>
>
> The block report processing during the startup period should be optimized.
> As noted in HADOOP-3022 during cluster startup all blocks are under-replicated
> because they have not been reported by name-nodes yet.
> Currently, we routinely move blocks to the neededReplications queue when they
> are first reported and then remove them from the list when other nodes report it.
> In ideal situation we end up adding all blocks into neededReplications queue first
> only in order to remove all of them in the end.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3369) Fast block processing during
name-node startup.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12596050#action_12596050 ]
Hudson commented on HADOOP-3369:
--------------------------------
Integrated in Hadoop-trunk #488 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/488/])
> Fast block processing during name-node startup.
> -----------------------------------------------
>
> Key: HADOOP-3369
> URL: https://issues.apache.org/jira/browse/HADOOP-3369
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.18.0
>
> Attachments: fastBlockReports.patch, fastBlockReports.patch
>
>
> The block report processing during the startup period should be optimized.
> As noted in HADOOP-3022 during cluster startup all blocks are under-replicated
> because they have not been reported by name-nodes yet.
> Currently, we routinely move blocks to the neededReplications queue when they
> are first reported and then remove them from the list when other nodes report it.
> In ideal situation we end up adding all blocks into neededReplications queue first
> only in order to remove all of them in the end.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3369) Fast block processing during
name-node startup.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-3369:
----------------------------------------
Attachment: fastBlockReports.patch
This patch implements the following approach:
When the name-node is in safe-mode block reports do not cause modifications of the
queues of over- and under- replicated blocks.
Verification of replication of all blocks is rather performed right before exiting the safe mode.
Thus only those blocks that really have missing replicas will appear in the neededReplications.
In my tests this approach completes block processing almost 5 times faster than the existing one,
which substantially improves the total name-node startup time.
> Fast block processing during name-node startup.
> -----------------------------------------------
>
> Key: HADOOP-3369
> URL: https://issues.apache.org/jira/browse/HADOOP-3369
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.18.0
>
> Attachments: fastBlockReports.patch
>
>
> The block report processing during the startup period should be optimized.
> As noted in HADOOP-3022 during cluster startup all blocks are under-replicated
> because they have not been reported by name-nodes yet.
> Currently, we routinely move blocks to the neededReplications queue when they
> are first reported and then remove them from the list when other nodes report it.
> In ideal situation we end up adding all blocks into neededReplications queue first
> only in order to remove all of them in the end.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3369) Fast block processing during
name-node startup.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-3369:
----------------------------------------
Attachment: fastBlockReports.patch
Yes code got simpler if both under- and over- replicated queues are emptied in the beginning.
And this method can still be used for periodic block map checkeups if we ever decide to do that.
> Fast block processing during name-node startup.
> -----------------------------------------------
>
> Key: HADOOP-3369
> URL: https://issues.apache.org/jira/browse/HADOOP-3369
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.18.0
>
> Attachments: fastBlockReports.patch, fastBlockReports.patch
>
>
> The block report processing during the startup period should be optimized.
> As noted in HADOOP-3022 during cluster startup all blocks are under-replicated
> because they have not been reported by name-nodes yet.
> Currently, we routinely move blocks to the neededReplications queue when they
> are first reported and then remove them from the list when other nodes report it.
> In ideal situation we end up adding all blocks into neededReplications queue first
> only in order to remove all of them in the end.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3369) Fast block processing during
name-node startup.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-3369:
----------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> Fast block processing during name-node startup.
> -----------------------------------------------
>
> Key: HADOOP-3369
> URL: https://issues.apache.org/jira/browse/HADOOP-3369
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.18.0
>
> Attachments: fastBlockReports.patch, fastBlockReports.patch
>
>
> The block report processing during the startup period should be optimized.
> As noted in HADOOP-3022 during cluster startup all blocks are under-replicated
> because they have not been reported by name-nodes yet.
> Currently, we routinely move blocks to the neededReplications queue when they
> are first reported and then remove them from the list when other nodes report it.
> In ideal situation we end up adding all blocks into neededReplications queue first
> only in order to remove all of them in the end.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.