You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org> on 2011/03/22 16:59:05 UTC
[jira] [Created] (HDFS-1774) Optimization in
org.apache.hadoop.hdfs.server.datanode.FSDataset class.
Optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.
-----------------------------------------------------------------------
Key: HDFS-1774
URL: https://issues.apache.org/jira/browse/HDFS-1774
Project: Hadoop HDFS
Issue Type: Improvement
Components: data-node
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
Consider a case: one directory has only one child directory and 10000 files.
1) First loop will get the number of children directories.
2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1774) Small optimization in
org.apache.hadoop.hdfs.server.datanode.FSDataset class.
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067198#comment-13067198 ]
Uma Maheswara Rao G commented on HDFS-1774:
-------------------------------------------
Hi Eli,
I am little bit confusing with the question :-(
You are talking about this line?
{code}
if (dirList.size() > 0) {
+ children = dirList.toArray(new FSDir[dirList.size()]);
}
{code}
if i am not wrong: this is because i can not assign directly list to array right?
Since i can not guess the children size, i have chosen list initially to collect the elements.
please correct me if i understood wrongly
> Small optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.
> -----------------------------------------------------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.23.0
>
> Attachments: HDFS-1774-1.patch, HDFS-1774-1.patch, HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1774) Small optimization to FSDataset
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1774:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I've committed this. Thanks Uma!
> Small optimization to FSDataset
> -------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HDFS-1774-1.patch, HDFS-1774-1.patch, HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1774) Small optimization in
org.apache.hadoop.hdfs.server.datanode.FSDataset class.
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065910#comment-13065910 ]
Uma Maheswara Rao G commented on HDFS-1774:
-------------------------------------------
Attached the Patch for Review.
> Small optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.
> -----------------------------------------------------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-1774-1.patch, HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1774) Small optimization in
org.apache.hadoop.hdfs.server.datanode.FSDataset class.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067214#comment-13067214 ]
Hadoop QA commented on HDFS-1774:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12486886/HDFS-1774-1.patch
against trunk revision 1147980.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/962//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/962//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/962//console
This message is automatically generated.
> Small optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.
> -----------------------------------------------------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.23.0
>
> Attachments: HDFS-1774-1.patch, HDFS-1774-1.patch, HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1774) Small optimization in
org.apache.hadoop.hdfs.server.datanode.FSDataset class.
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1774:
-----------------------------------------
Hadoop Flags: [Reviewed]
+1 change look good.
Eli, do you have further comments?
> Small optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.
> -----------------------------------------------------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.23.0
>
> Attachments: HDFS-1774-1.patch, HDFS-1774-1.patch, HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1774) Small optimization in
org.apache.hadoop.hdfs.server.datanode.FSDataset class.
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-1774:
--------------------------------------
Attachment: HDFS-1774-1.patch
Re submitted the same patch to trigger hudson.
> Small optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.
> -----------------------------------------------------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.20.1, 0.23.0
>
> Attachments: HDFS-1774-1.patch, HDFS-1774-1.patch, HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1774) Optimization in
org.apache.hadoop.hdfs.server.datanode.FSDataset class.
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-1774:
--------------------------------------
Attachment: HDFS-1774.patch
> Optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.
> -----------------------------------------------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1774) Small optimization in
org.apache.hadoop.hdfs.server.datanode.FSDataset class.
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1774:
------------------------------
Fix Version/s: (was: 0.20.1)
Status: Patch Available (was: Open)
> Small optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.
> -----------------------------------------------------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.23.0
>
> Attachments: HDFS-1774-1.patch, HDFS-1774-1.patch, HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1774) Small optimization in
org.apache.hadoop.hdfs.server.datanode.FSDataset class.
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-1774:
--------------------------------------
Fix Version/s: 0.20.1
0.23.0
> Small optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.
> -----------------------------------------------------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.20.1, 0.23.0
>
> Attachments: HDFS-1774-1.patch, HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1774) Small optimization in
org.apache.hadoop.hdfs.server.datanode.FSDataset class.
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067181#comment-13067181 ]
Eli Collins commented on HDFS-1774:
-----------------------------------
I understand the fix, I'm asking why you need to copy the array, seems like assigning it to child is sufficient no?
> Small optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.
> -----------------------------------------------------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.23.0
>
> Attachments: HDFS-1774-1.patch, HDFS-1774-1.patch, HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1774) Small optimization in
org.apache.hadoop.hdfs.server.datanode.FSDataset class.
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067174#comment-13067174 ]
Eli Collins commented on HDFS-1774:
-----------------------------------
Why not just assign dirList to children (vs copying it)? dirList isn't used anywhere, you add it in this patch.
> Small optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.
> -----------------------------------------------------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.23.0
>
> Attachments: HDFS-1774-1.patch, HDFS-1774-1.patch, HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1774) Optimization in
org.apache.hadoop.hdfs.server.datanode.FSDataset class.
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-1774:
--------------------------------------
Status: Patch Available (was: Open)
> Optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.
> -----------------------------------------------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-1774-1.patch, HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1774) Optimization in
org.apache.hadoop.hdfs.server.datanode.FSDataset class.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010805#comment-13010805 ]
Hadoop QA commented on HDFS-1774:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12474509/HDFS-1774.patch
against trunk revision 1083958.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.cli.TestHDFSCLI
org.apache.hadoop.hdfs.server.datanode.TestBlockReport
org.apache.hadoop.hdfs.server.datanode.TestTransferRbw
org.apache.hadoop.hdfs.TestDFSShell
-1 contrib tests. The patch failed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/282//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/282//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/282//console
This message is automatically generated.
> Optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.
> -----------------------------------------------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1774) Small optimization in
org.apache.hadoop.hdfs.server.datanode.FSDataset class.
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-1774:
--------------------------------------
Status: Open (was: Patch Available)
> Small optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.
> -----------------------------------------------------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.23.0, 0.20.1
>
> Attachments: HDFS-1774-1.patch, HDFS-1774-1.patch, HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1774) Small optimization in
org.apache.hadoop.hdfs.server.datanode.FSDataset class.
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067219#comment-13067219 ]
Uma Maheswara Rao G commented on HDFS-1774:
-------------------------------------------
This change is just kind of re-factoring. No tests required.
> Small optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.
> -----------------------------------------------------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.23.0
>
> Attachments: HDFS-1774-1.patch, HDFS-1774-1.patch, HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1774) Optimization in
org.apache.hadoop.hdfs.server.datanode.FSDataset class.
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-1774:
--------------------------------------
Status: Patch Available (was: Open)
> Optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.
> -----------------------------------------------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1774) Small optimization in
org.apache.hadoop.hdfs.server.datanode.FSDataset class.
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-1774:
--------------------------------------
Summary: Small optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class. (was: Optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.)
> Small optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.
> -----------------------------------------------------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-1774-1.patch, HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1774) Small optimization to FSDataset
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068571#comment-13068571 ]
Hudson commented on HDFS-1774:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #796 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/796/])
HDFS-1774. Small optimization to FSDataset. Contributed by Uma Maheswara Rao G
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1148894
Files :
* /hadoop/common/trunk/hdfs/CHANGES.txt
* /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
> Small optimization to FSDataset
> -------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HDFS-1774-1.patch, HDFS-1774-1.patch, HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1774) Optimization in
org.apache.hadoop.hdfs.server.datanode.FSDataset class.
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-1774:
--------------------------------------
Status: Open (was: Patch Available)
> Optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.
> -----------------------------------------------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-1774-1.patch, HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1774) Small optimization to FSDataset
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1774:
------------------------------
Priority: Minor (was: Major)
Summary: Small optimization to FSDataset (was: Small optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.)
Sorry, I forgot that children was an array, your makes sense.
I'm +1 on the latest patch as well.
> Small optimization to FSDataset
> -------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HDFS-1774-1.patch, HDFS-1774-1.patch, HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1774) Optimization in
org.apache.hadoop.hdfs.server.datanode.FSDataset class.
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-1774:
--------------------------------------
Attachment: HDFS-1774-1.patch
> Optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.
> -----------------------------------------------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-1774-1.patch, HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1774) Small optimization to FSDataset
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068564#comment-13068564 ]
Uma Maheswara Rao G commented on HDFS-1774:
-------------------------------------------
Thanks a lot for Nicholas and Eli for Review :-)
> Small optimization to FSDataset
> -------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HDFS-1774-1.patch, HDFS-1774-1.patch, HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1774) Small optimization in
org.apache.hadoop.hdfs.server.datanode.FSDataset class.
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067180#comment-13067180 ]
Uma Maheswara Rao G commented on HDFS-1774:
-------------------------------------------
Thanks a lot Eli, for taking a look on this patch.
Here since children is an array, i am just adding the elements into list and converting it to array if it has elements.
old code will just loop two times and perform isDirectory checks twice.
> Small optimization in org.apache.hadoop.hdfs.server.datanode.FSDataset class.
> -----------------------------------------------------------------------------
>
> Key: HDFS-1774
> URL: https://issues.apache.org/jira/browse/HDFS-1774
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.23.0
>
> Attachments: HDFS-1774-1.patch, HDFS-1774-1.patch, HDFS-1774.patch
>
>
> Inner class FSDir constructor is doing duplicate iterations over the listed files in the passed directory. We can optimize this to single loop and also we can avoid isDirectory check which will perform some native invocations.
> Consider a case: one directory has only one child directory and 10000 files.
> 1) First loop will get the number of children directories.
> 2) if (numChildren > 0) , This condition will satisfy and again it will iterate 10001 times and also will check isDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira