You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Suresh Srinivas (JIRA)" <ji...@apache.org> on 2008/11/04 01:39:44 UTC
[jira] Created: (HADOOP-4583) Code optimization/cleanup in HDFS
Code optimization/cleanup in HDFS
---------------------------------
Key: HADOOP-4583
URL: https://issues.apache.org/jira/browse/HADOOP-4583
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Affects Versions: 0.20.0
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
Fix For: 0.20.0
Some of the changes needed:
- FSNameSystem.allocateBlock creates a lot of object unnecessarily to find a blockID that is unused. It should reuse the Block object created by changing the blockID and reusing the GenerationStamp.
- InodeFile.addBlock should use System.arrayCopy instead of copying 1 at a time.
- Some of the for loops in FSEditLog.java can be optimized. Instead of getting the edit stream length in termination condition, a local variable should be used to do the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4583) Code optimization/cleanup in HDFS
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645998#action_12645998 ]
Hudson commented on HADOOP-4583:
--------------------------------
Integrated in Hadoop-trunk #655 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/655/])
. Several code optimizations in HDFS. (Suresh Srinivas via szetszwo)
> Code optimization/cleanup in HDFS
> ---------------------------------
>
> Key: HADOOP-4583
> URL: https://issues.apache.org/jira/browse/HADOOP-4583
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: 4583.patch, 4583.patch
>
>
> Some of the changes needed:
> - FSNameSystem.allocateBlock creates a lot of object unnecessarily to find a blockID that is unused. It should reuse the Block object created by changing the blockID and reusing the GenerationStamp.
> - InodeFile.addBlock should use System.arrayCopy instead of copying 1 at a time.
> - Some of the for loops in FSEditLog.java can be optimized. Instead of getting the edit stream length in termination condition, a local variable should be used to do the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4583) Code optimization/cleanup in HDFS
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-4583:
------------------------------------
Attachment: 4583.patch
New patch uploaded. Here are the results of testing the patch. No tests included as the functionality is tested by existing tests.
[exec] -1 overall.
[exec] +1 @author. The patch does not contain any @author tags.
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
> Code optimization/cleanup in HDFS
> ---------------------------------
>
> Key: HADOOP-4583
> URL: https://issues.apache.org/jira/browse/HADOOP-4583
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: 4583.patch, 4583.patch
>
>
> Some of the changes needed:
> - FSNameSystem.allocateBlock creates a lot of object unnecessarily to find a blockID that is unused. It should reuse the Block object created by changing the blockID and reusing the GenerationStamp.
> - InodeFile.addBlock should use System.arrayCopy instead of copying 1 at a time.
> - Some of the for loops in FSEditLog.java can be optimized. Instead of getting the edit stream length in termination condition, a local variable should be used to do the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4583) Code optimization/cleanup in HDFS
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-4583:
-------------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Suresh!
> Code optimization/cleanup in HDFS
> ---------------------------------
>
> Key: HADOOP-4583
> URL: https://issues.apache.org/jira/browse/HADOOP-4583
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: 4583.patch, 4583.patch
>
>
> Some of the changes needed:
> - FSNameSystem.allocateBlock creates a lot of object unnecessarily to find a blockID that is unused. It should reuse the Block object created by changing the blockID and reusing the GenerationStamp.
> - InodeFile.addBlock should use System.arrayCopy instead of copying 1 at a time.
> - Some of the for loops in FSEditLog.java can be optimized. Instead of getting the edit stream length in termination condition, a local variable should be used to do the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4583) Code optimization/cleanup in HDFS
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645939#action_12645939 ]
Hadoop QA commented on HADOOP-4583:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12393485/4583.patch
against trunk revision 712305.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3552/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3552/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3552/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3552/console
This message is automatically generated.
> Code optimization/cleanup in HDFS
> ---------------------------------
>
> Key: HADOOP-4583
> URL: https://issues.apache.org/jira/browse/HADOOP-4583
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: 4583.patch, 4583.patch
>
>
> Some of the changes needed:
> - FSNameSystem.allocateBlock creates a lot of object unnecessarily to find a blockID that is unused. It should reuse the Block object created by changing the blockID and reusing the GenerationStamp.
> - InodeFile.addBlock should use System.arrayCopy instead of copying 1 at a time.
> - Some of the for loops in FSEditLog.java can be optimized. Instead of getting the edit stream length in termination condition, a local variable should be used to do the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4583) Code optimization/cleanup in HDFS
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645942#action_12645942 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4583:
------------------------------------------------
The javadoc warning is unrelated. See HADOOP-4621.
> Code optimization/cleanup in HDFS
> ---------------------------------
>
> Key: HADOOP-4583
> URL: https://issues.apache.org/jira/browse/HADOOP-4583
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: 4583.patch, 4583.patch
>
>
> Some of the changes needed:
> - FSNameSystem.allocateBlock creates a lot of object unnecessarily to find a blockID that is unused. It should reuse the Block object created by changing the blockID and reusing the GenerationStamp.
> - InodeFile.addBlock should use System.arrayCopy instead of copying 1 at a time.
> - Some of the for loops in FSEditLog.java can be optimized. Instead of getting the edit stream length in termination condition, a local variable should be used to do the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4583) Code optimization/cleanup in HDFS
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-4583:
------------------------------------
Status: Patch Available (was: Open)
> Code optimization/cleanup in HDFS
> ---------------------------------
>
> Key: HADOOP-4583
> URL: https://issues.apache.org/jira/browse/HADOOP-4583
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: 4583.patch
>
>
> Some of the changes needed:
> - FSNameSystem.allocateBlock creates a lot of object unnecessarily to find a blockID that is unused. It should reuse the Block object created by changing the blockID and reusing the GenerationStamp.
> - InodeFile.addBlock should use System.arrayCopy instead of copying 1 at a time.
> - Some of the for loops in FSEditLog.java can be optimized. Instead of getting the edit stream length in termination condition, a local variable should be used to do the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4583) Code optimization/cleanup in HDFS
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645663#action_12645663 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4583:
------------------------------------------------
Codes look good but the patch is out dated. Need sync with trunk.
> Code optimization/cleanup in HDFS
> ---------------------------------
>
> Key: HADOOP-4583
> URL: https://issues.apache.org/jira/browse/HADOOP-4583
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: 4583.patch
>
>
> Some of the changes needed:
> - FSNameSystem.allocateBlock creates a lot of object unnecessarily to find a blockID that is unused. It should reuse the Block object created by changing the blockID and reusing the GenerationStamp.
> - InodeFile.addBlock should use System.arrayCopy instead of copying 1 at a time.
> - Some of the for loops in FSEditLog.java can be optimized. Instead of getting the edit stream length in termination condition, a local variable should be used to do the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4583) Code optimization/cleanup in HDFS
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-4583:
------------------------------------
Attachment: 4583.patch
> Code optimization/cleanup in HDFS
> ---------------------------------
>
> Key: HADOOP-4583
> URL: https://issues.apache.org/jira/browse/HADOOP-4583
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: 4583.patch
>
>
> Some of the changes needed:
> - FSNameSystem.allocateBlock creates a lot of object unnecessarily to find a blockID that is unused. It should reuse the Block object created by changing the blockID and reusing the GenerationStamp.
> - InodeFile.addBlock should use System.arrayCopy instead of copying 1 at a time.
> - Some of the for loops in FSEditLog.java can be optimized. Instead of getting the edit stream length in termination condition, a local variable should be used to do the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4583) Code optimization/cleanup in HDFS
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644909#action_12644909 ]
dhruba borthakur commented on HADOOP-4583:
------------------------------------------
>Instead of getting the edit stream length in termination condition, a local variable should be used to do the same
Won't the Java compiler do this on its own?
> Code optimization/cleanup in HDFS
> ---------------------------------
>
> Key: HADOOP-4583
> URL: https://issues.apache.org/jira/browse/HADOOP-4583
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
>
> Some of the changes needed:
> - FSNameSystem.allocateBlock creates a lot of object unnecessarily to find a blockID that is unused. It should reuse the Block object created by changing the blockID and reusing the GenerationStamp.
> - InodeFile.addBlock should use System.arrayCopy instead of copying 1 at a time.
> - Some of the for loops in FSEditLog.java can be optimized. Instead of getting the edit stream length in termination condition, a local variable should be used to do the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4583) Code optimization/cleanup in HDFS
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-4583:
-------------------------------------------
Status: Patch Available (was: Open)
> Code optimization/cleanup in HDFS
> ---------------------------------
>
> Key: HADOOP-4583
> URL: https://issues.apache.org/jira/browse/HADOOP-4583
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: 4583.patch, 4583.patch
>
>
> Some of the changes needed:
> - FSNameSystem.allocateBlock creates a lot of object unnecessarily to find a blockID that is unused. It should reuse the Block object created by changing the blockID and reusing the GenerationStamp.
> - InodeFile.addBlock should use System.arrayCopy instead of copying 1 at a time.
> - Some of the for loops in FSEditLog.java can be optimized. Instead of getting the edit stream length in termination condition, a local variable should be used to do the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4583) Code optimization/cleanup in HDFS
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645044#action_12645044 ]
Suresh Srinivas commented on HADOOP-4583:
-----------------------------------------
I am not sure if the method call in termination condition is optimized. I know that loops may be optimized from condition < len to condition <= 0. Based on the tests I ran, moving the method call out of terminating condition improves the performance.
> Code optimization/cleanup in HDFS
> ---------------------------------
>
> Key: HADOOP-4583
> URL: https://issues.apache.org/jira/browse/HADOOP-4583
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
>
> Some of the changes needed:
> - FSNameSystem.allocateBlock creates a lot of object unnecessarily to find a blockID that is unused. It should reuse the Block object created by changing the blockID and reusing the GenerationStamp.
> - InodeFile.addBlock should use System.arrayCopy instead of copying 1 at a time.
> - Some of the for loops in FSEditLog.java can be optimized. Instead of getting the edit stream length in termination condition, a local variable should be used to do the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4583) Code optimization/cleanup in HDFS
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-4583:
----------------------------------
Status: Open (was: Patch Available)
> Code optimization/cleanup in HDFS
> ---------------------------------
>
> Key: HADOOP-4583
> URL: https://issues.apache.org/jira/browse/HADOOP-4583
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: 4583.patch
>
>
> Some of the changes needed:
> - FSNameSystem.allocateBlock creates a lot of object unnecessarily to find a blockID that is unused. It should reuse the Block object created by changing the blockID and reusing the GenerationStamp.
> - InodeFile.addBlock should use System.arrayCopy instead of copying 1 at a time.
> - Some of the for loops in FSEditLog.java can be optimized. Instead of getting the edit stream length in termination condition, a local variable should be used to do the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4583) Code optimization/cleanup in HDFS
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644864#action_12644864 ]
Suresh Srinivas commented on HADOOP-4583:
-----------------------------------------
Thanks Nicholas for pointing out, GenerationStamp cannot be reused. But the Block object can be reused.
> Code optimization/cleanup in HDFS
> ---------------------------------
>
> Key: HADOOP-4583
> URL: https://issues.apache.org/jira/browse/HADOOP-4583
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
>
> Some of the changes needed:
> - FSNameSystem.allocateBlock creates a lot of object unnecessarily to find a blockID that is unused. It should reuse the Block object created by changing the blockID and reusing the GenerationStamp.
> - InodeFile.addBlock should use System.arrayCopy instead of copying 1 at a time.
> - Some of the for loops in FSEditLog.java can be optimized. Instead of getting the edit stream length in termination condition, a local variable should be used to do the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.