You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Robert Chansler (JIRA)" <ji...@apache.org> on 2007/10/27 00:39:50 UTC
[jira] Created: (HADOOP-2110) Block Report Optimization: Replace
class instance
Block Report Optimization: Replace class instance
-------------------------------------------------
Key: HADOOP-2110
URL: https://issues.apache.org/jira/browse/HADOOP-2110
Project: Hadoop
Issue Type: Improvement
Components: dfs
Affects Versions: 0.14.0
Reporter: Robert Chansler
Assignee: Robert Chansler
Fix For: 0.16.0
Block report processing can be more efficient if instances of the block ID class are replaced with a sequence of longs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2110) Block Report Optimization: Replace
class instance
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2110:
---------------------------------
Issue Type: Sub-task (was: Improvement)
Parent: HADOOP-2448
> Block Report Optimization: Replace class instance
> -------------------------------------------------
>
> Key: HADOOP-2110
> URL: https://issues.apache.org/jira/browse/HADOOP-2110
> Project: Hadoop
> Issue Type: Sub-task
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Robert Chansler
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
>
> Block report processing can be more efficient if instances of the block ID class are replaced with a sequence of longs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2110) Block Report Optimization: Replace
class instance
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2110:
---------------------------------
Status: Patch Available (was: Open)
Submitting my last patch: brAsLongsPatch5.txt
> Block Report Optimization: Replace class instance
> -------------------------------------------------
>
> Key: HADOOP-2110
> URL: https://issues.apache.org/jira/browse/HADOOP-2110
> Project: Hadoop
> Issue Type: Sub-task
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Robert Chansler
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: brAsLongsPatch2.txt, brAsLongsPatch3.txt, brAsLongsPatch5.txt, comaprison.txt
>
>
> Block report processing can be more efficient if instances of the block ID class are replaced with a sequence of longs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2110) Block Report Optimization: Replace
class instance
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12554091 ]
dhruba borthakur commented on HADOOP-2110:
------------------------------------------
1. In BlockListAsLongs() , you access iBlockList before checking whether it is null.
2. Also, BlockListAsLongs() constructor checks th condition "if (iBlockList.length%LONGS_PER_BLOCK != 0)" twice.
3. There should be no "public" methods in BlockListAsLongs. These methods should be package private.
All other portions of code looks good. +1.
> Block Report Optimization: Replace class instance
> -------------------------------------------------
>
> Key: HADOOP-2110
> URL: https://issues.apache.org/jira/browse/HADOOP-2110
> Project: Hadoop
> Issue Type: Sub-task
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Robert Chansler
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: brAsLongsPatch2.txt, brAsLongsPatch3.txt, comaprison.txt
>
>
> Block report processing can be more efficient if instances of the block ID class are replaced with a sequence of longs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2110) Block Report Optimization: Replace
class instance
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler reassigned HADOOP-2110:
---------------------------------------
Assignee: Sanjay Radia (was: Robert Chansler)
> Block Report Optimization: Replace class instance
> -------------------------------------------------
>
> Key: HADOOP-2110
> URL: https://issues.apache.org/jira/browse/HADOOP-2110
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Robert Chansler
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
>
> Block report processing can be more efficient if instances of the block ID class are replaced with a sequence of longs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2110) Block Report Optimization: Replace
class instance
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2110:
---------------------------------
Attachment: brAsLongsPatch5.txt
Updated - fixed Dhruba's sugestins.
> Block Report Optimization: Replace class instance
> -------------------------------------------------
>
> Key: HADOOP-2110
> URL: https://issues.apache.org/jira/browse/HADOOP-2110
> Project: Hadoop
> Issue Type: Sub-task
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Robert Chansler
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: brAsLongsPatch2.txt, brAsLongsPatch3.txt, brAsLongsPatch5.txt, comaprison.txt
>
>
> Block report processing can be more efficient if instances of the block ID class are replaced with a sequence of longs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2110) Block Report Optimization: Replace
class instance
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555713#action_12555713 ]
dhruba borthakur commented on HADOOP-2110:
------------------------------------------
+1. Code looks good.
> Block Report Optimization: Replace class instance
> -------------------------------------------------
>
> Key: HADOOP-2110
> URL: https://issues.apache.org/jira/browse/HADOOP-2110
> Project: Hadoop
> Issue Type: Sub-task
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Robert Chansler
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: brAsLongsPatch2.txt, brAsLongsPatch3.txt, brAsLongsPatch5.txt, comaprison.txt
>
>
> Block report processing can be more efficient if instances of the block ID class are replaced with a sequence of longs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2110) Block Report Optimization: Replace
class instance
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553487 ]
Doug Cutting commented on HADOOP-2110:
--------------------------------------
BlockListAsLongs should not be public.
DatanodeProtocol's version should be incremented.
> Block Report Optimization: Replace class instance
> -------------------------------------------------
>
> Key: HADOOP-2110
> URL: https://issues.apache.org/jira/browse/HADOOP-2110
> Project: Hadoop
> Issue Type: Sub-task
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Robert Chansler
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: brAsLongsPatch2.txt, comaprison.txt
>
>
> Block report processing can be more efficient if instances of the block ID class are replaced with a sequence of longs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2110) Block Report Optimization: Replace
class instance
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555873#action_12555873 ]
Hadoop QA commented on HADOOP-2110:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12372470/brAsLongsPatch5.txt
against trunk revision .
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1463/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1463/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1463/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1463/console
This message is automatically generated.
> Block Report Optimization: Replace class instance
> -------------------------------------------------
>
> Key: HADOOP-2110
> URL: https://issues.apache.org/jira/browse/HADOOP-2110
> Project: Hadoop
> Issue Type: Sub-task
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Robert Chansler
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: brAsLongsPatch2.txt, brAsLongsPatch3.txt, brAsLongsPatch5.txt, comaprison.txt
>
>
> Block report processing can be more efficient if instances of the block ID class are replaced with a sequence of longs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2110) Block Report Optimization: Replace
class instance
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2110:
---------------------------------
Attachment: comaprison.txt
Sorry the formating of the comaprison table got messed up.
Here the same in a attached file.
> Block Report Optimization: Replace class instance
> -------------------------------------------------
>
> Key: HADOOP-2110
> URL: https://issues.apache.org/jira/browse/HADOOP-2110
> Project: Hadoop
> Issue Type: Sub-task
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Robert Chansler
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: brAsLongsPatch2.txt, comaprison.txt
>
>
> Block report processing can be more efficient if instances of the block ID class are replaced with a sequence of longs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2110) Block Report Optimization: Replace
class instance
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2110:
---------------------------------
Attachment: brAsLongsPatch3.txt
Updated patch
- updated the protocol version for data node protocol
- BlockListAsLongs is not public
-
> Block Report Optimization: Replace class instance
> -------------------------------------------------
>
> Key: HADOOP-2110
> URL: https://issues.apache.org/jira/browse/HADOOP-2110
> Project: Hadoop
> Issue Type: Sub-task
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Robert Chansler
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: brAsLongsPatch2.txt, brAsLongsPatch3.txt, comaprison.txt
>
>
> Block report processing can be more efficient if instances of the block ID class are replaced with a sequence of longs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2110) Block Report Optimization: Replace
class instance
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2110:
---------------------------------
Attachment: brAsLongsPatch2.txt
The attached patch uses an array of longs instead of an array of Class Block
for the Block report (BR)
A block is 2 longs (block id and block length).
Java's overhead for an object is 16 bytes.
For an array of objects each array entry is a separately allocated object.
Comparison of the block report processing times.
All times are in millisec and are measured from the data node sending the BR and
receiving an acknowledgement from the NN that the BR was processed.
Approx 50 improvement in the BR processing time
BRs using Class Block || BRs using Longs
Num Time Time || Time Time
Blocks Total per 1K blocks || Total per 1K blocks
---------------------------------------------||-----------------------
40000 651ms 16ms || 230ms 5.75ms
100000 1258ms 12ms || 573ms 5.73ms
300000 3521ms 12ms || 1800ms 6.00ms
500000 5930ms 12ms || 2916ms 5.83ms
> Block Report Optimization: Replace class instance
> -------------------------------------------------
>
> Key: HADOOP-2110
> URL: https://issues.apache.org/jira/browse/HADOOP-2110
> Project: Hadoop
> Issue Type: Sub-task
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Robert Chansler
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: brAsLongsPatch2.txt
>
>
> Block report processing can be more efficient if instances of the block ID class are replaced with a sequence of longs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2110) Block Report Optimization: Replace
class instance
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-2110:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Sanjay!
> Block Report Optimization: Replace class instance
> -------------------------------------------------
>
> Key: HADOOP-2110
> URL: https://issues.apache.org/jira/browse/HADOOP-2110
> Project: Hadoop
> Issue Type: Sub-task
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Robert Chansler
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: brAsLongsPatch2.txt, brAsLongsPatch3.txt, brAsLongsPatch5.txt, comaprison.txt
>
>
> Block report processing can be more efficient if instances of the block ID class are replaced with a sequence of longs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.