You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2009/12/27 08:15:29 UTC
[jira] Created: (HDFS-854) Datanode should scan devices in parallel
to generate block report
Datanode should scan devices in parallel to generate block report
-----------------------------------------------------------------
Key: HDFS-854
URL: https://issues.apache.org/jira/browse/HDFS-854
Project: Hadoop HDFS
Issue Type: Improvement
Components: data-node
Reporter: dhruba borthakur
A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-854) Datanode should scan devices in
parallel to generate block report
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853645#action_12853645 ]
Hudson commented on HDFS-854:
-----------------------------
Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #146 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/146/])
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854-2.patch, HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-854) Datanode should scan devices in
parallel to generate block report
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846268#action_12846268 ]
Hadoop QA commented on HDFS-854:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12438985/HDFS-854.patch.1
against trunk revision 923467.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/270/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/270/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/270/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/270/console
This message is automatically generated.
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-854) Datanode should scan devices in
parallel to generate block report
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849476#action_12849476 ]
Jakob Homan commented on HDFS-854:
----------------------------------
The test failure is not related in all likelihood. The patch overall looks good.
One change: dfs.datanode.directoryscan.threads should be defined and referenced via DFSConfigKeys.java.
One question: Is it appropriate to just log the error when compiling block reports (line 125 in patch), or should more drastic action be taken?
One request: Naming patches with a final number really confuses all my editors, how about HDFS-854-1.patch (although numbering patches is greatly appreciated!)
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-854) Datanode should scan devices in
parallel to generate block report
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849553#action_12849553 ]
Rodrigo Schmidt commented on HDFS-854:
--------------------------------------
Patch looks good to me. +1.
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854-2.patch, HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-854) Datanode should scan devices in
parallel to generate block report
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803501#action_12803501 ]
Lohit Vijayarenu commented on HDFS-854:
---------------------------------------
Or probably keep last added/deleted blocks in memory and send block report and doing disk scan once in a while?
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-854) Datanode should scan devices in
parallel to generate block report
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851177#action_12851177 ]
Dmytro Molkov commented on HDFS-854:
------------------------------------
The errors in the build system again look like they are not related to the actual patch. Two errors in BlockScanner are
java.lang.RuntimeException: java.util.zip.ZipException: error reading zip file
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1715)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1529)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1475)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:596)
at org.apache.hadoop.conf.Configuration.setLong(Configuration.java:731)
at org.apache.hadoop.hdfs.TestDatanodeBlockScanner.blockCorruptionRecoveryPolicy(TestDatanodeBlockScanner.java:277)
at org.apache.hadoop.hdfs.TestDatanodeBlockScanner.testBlockCorruptionRecoveryPolicy(TestDatanodeBlockScanner.java:269)
and there is one more error with HDFSProxy which cannot have anything to do with my code changes.
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854-2.patch, HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-854) Datanode should scan devices in parallel
to generate block report
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-854:
-------------------------------
Attachment: HDFS-854.patch.1
Added a shutdown to DirectoryScanner to close the thread pool on exit.
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-854) Datanode should scan devices in parallel
to generate block report
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-854:
-------------------------------
Attachment: HDFS-854.patch
Please have a look at the patch.
The problem we are trying to solve here is generating the first block report quicker after restart by scanning the volumes in parallel. This way instead of scanning 12 TB of data sequentially we scan 12 chunks of 1 TB in parallel. Since there is a lot of latency in IO we have an improvement of a few times in the time to generate the block report.
The test for this is just running the directory scanner test twice: with parallel execution and without it.
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Attachments: HDFS-854.patch
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-854) Datanode should scan devices in parallel
to generate block report
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan updated HDFS-854:
-----------------------------
Hadoop Flags: [Reviewed]
+1. TestDataNodeScanner passes with the patch on my box - Hudson's acting up again. Other test failure is unrelated. Unless there are more comments, I'll commit this this evening.
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854-2.patch, HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-854) Datanode should scan devices in parallel
to generate block report
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-854:
-------------------------------
Status: Patch Available (was: Open)
Submitting the latest version for Hudson
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854-2.patch, HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-854) Datanode should scan devices in
parallel to generate block report
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur reassigned HDFS-854:
-------------------------------------
Assignee: Dmytro Molkov
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-854) Datanode should scan devices in parallel
to generate block report
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan updated HDFS-854:
-----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I have committed this. Resolving as fixed. Thanks for the contribution, Dmytro!
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854-2.patch, HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-854) Datanode should scan devices in parallel
to generate block report
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-854:
-------------------------------
Fix Version/s: 0.22.0
Affects Version/s: 0.22.0
Status: Patch Available (was: Open)
Submitting it for Hudson to run the tests. Can someone review the patch in the meanwhile?
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854.patch
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-854) Datanode should scan devices in
parallel to generate block report
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849543#action_12849543 ]
Hadoop QA commented on HDFS-854:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12438985/HDFS-854.patch.1
against trunk revision 926469.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/283/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/283/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/283/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/283/console
This message is automatically generated.
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-854) Datanode should scan devices in
parallel to generate block report
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853942#action_12853942 ]
Hudson commented on HDFS-854:
-----------------------------
Integrated in Hadoop-Hdfs-trunk #275 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/275/])
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854-2.patch, HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-854) Datanode should scan devices in
parallel to generate block report
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853857#action_12853857 ]
Hudson commented on HDFS-854:
-----------------------------
Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #302 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/302/])
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854-2.patch, HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-854) Datanode should scan devices in
parallel to generate block report
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849388#action_12849388 ]
Dmytro Molkov commented on HDFS-854:
------------------------------------
The test failure org.apache.hadoop.hdfs.TestReadWhileWriting.pipeline_02_03 doesn't seem to have anything to do with this patch.
Can anyone review this one so that it can get commited?
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-854) Datanode should scan devices in parallel
to generate block report
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-854:
-------------------------------
Status: Patch Available (was: Open)
Going to submit it once again to see if the tests pass.
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-854) Datanode should scan devices in parallel
to generate block report
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-854:
-------------------------------
Status: Patch Available (was: Open)
Resubmitting the same patch, because I could not connect it with the failures in the tests.
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854.patch
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-854) Datanode should scan devices in parallel
to generate block report
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-854:
-------------------------------
Attachment: HDFS-854-2.patch
Addressing Jakob's comments:
Moved dfs.datanode.directoryscan.threads to DFSConfigKeys
Changed the name of the patch file :)
And we are now throwing RuntimeException in case of failure in compile report.
Previous behaviour was - nothing in the call stack throws Exception except for those runtime ones, if there is a runtime exception - shutdown the datanode.
Future abstraction in java throws an exception in case your subroutine throws and you want to catch it so we need to catch Exception and throw RuntimeException to save the semantics of the call.
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854-2.patch, HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-854) Datanode should scan devices in
parallel to generate block report
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851175#action_12851175 ]
Hadoop QA commented on HDFS-854:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12439737/HDFS-854-2.patch
against trunk revision 927999.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/287/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/287/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/287/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/287/console
This message is automatically generated.
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854-2.patch, HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-854) Datanode should scan devices in
parallel to generate block report
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849458#action_12849458 ]
Rodrigo Schmidt commented on HDFS-854:
--------------------------------------
Reviewed the patch and it looks good.
+1
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-854) Datanode should scan devices in parallel
to generate block report
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-854:
-------------------------------
Status: Open (was: Patch Available)
Found a small problem with it
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854.patch
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-854) Datanode should scan devices in
parallel to generate block report
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844657#action_12844657 ]
Hadoop QA commented on HDFS-854:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12438481/HDFS-854.patch
against trunk revision 921697.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/269/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/269/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/269/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/269/console
This message is automatically generated.
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854.patch
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-854) Datanode should scan devices in
parallel to generate block report
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851256#action_12851256 ]
Hudson commented on HDFS-854:
-----------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #227 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/227/])
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854-2.patch, HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-854) Datanode should scan devices in
parallel to generate block report
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843935#action_12843935 ]
dhruba borthakur commented on HDFS-854:
---------------------------------------
> Or probably keep last added/deleted blocks in memory and send block report and doing disk scan once in a while?
Lohit:This is already part of HDFS trunk, I think Suresh has earlier improved this to generate block reports from memory.
This patch improves the startup time of the cluster when the entire cluster is restarted. In this case, the datanodes are started newly and have to scan the disk to generate the first block report. The faster we can generate the block report, the shorter the time for the NN to exit safemode!
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Attachments: HDFS-854.patch
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-854) Datanode should scan devices in parallel
to generate block report
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-854:
-------------------------------
Status: Open (was: Patch Available)
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854-2.patch, HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-854) Datanode should scan devices in
parallel to generate block report
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849510#action_12849510 ]
Rodrigo Schmidt commented on HDFS-854:
--------------------------------------
I agree with Jakob about the patch name. :-)
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-854) Datanode should scan devices in
parallel to generate block report
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844422#action_12844422 ]
Hadoop QA commented on HDFS-854:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12438481/HDFS-854.patch
against trunk revision 921697.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/268/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/268/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/268/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/268/console
This message is automatically generated.
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854.patch
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-854) Datanode should scan devices in
parallel to generate block report
Posted by "Zlatin Balevsky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803480#action_12803480 ]
Zlatin Balevsky commented on HDFS-854:
--------------------------------------
If it is not possible to move the i/o operations listFiles() and length() outside of the lock, it would make sense to set a flag that a block report is in progress so that the rest of the datanode doesn't just hang. My 2c.
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: dhruba borthakur
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-854) Datanode should scan devices in parallel
to generate block report
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-854:
-------------------------------
Status: Open (was: Patch Available)
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-854) Datanode should scan devices in parallel
to generate block report
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-854:
-------------------------------
Status: Patch Available (was: Open)
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854.patch, HDFS-854.patch.1
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-854) Datanode should scan devices in parallel
to generate block report
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-854:
-------------------------------
Status: Open (was: Patch Available)
> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>
> Key: HDFS-854
> URL: https://issues.apache.org/jira/browse/HDFS-854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-854.patch
>
>
> A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.