You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2006/11/30 01:03:21 UTC
[jira] Created: (HADOOP-764) The memory consumption of
processReport() in the namenode can be reduced
The memory consumption of processReport() in the namenode can be reduced
------------------------------------------------------------------------
Key: HADOOP-764
URL: http://issues.apache.org/jira/browse/HADOOP-764
Project: Hadoop
Issue Type: Bug
Components: dfs
Reporter: dhruba borthakur
Assigned To: dhruba borthakur
The FSNamesystem.processReport() method converts the blocklist for a datanode into an array by calling node.getBlocks(). Although this memory allocation is transient, it could possibly require the garbage-collector to work that much harder.
The method Block.getBlocks() should be deprecated. Code that currently uses this method should instead iterate over the Collection.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-764) The memory consumption of
processReport() in the namenode can be reduced
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-764?page=comments#action_12456276 ]
Raghu Angadi commented on HADOOP-764:
-------------------------------------
As a related note, since we do a blocks.clear() in DatanodeDescriptor when we update blocks from , this results in one separate copy of a block exists for each node (i.e. one for each replica) and one coy in NameNode's blockMap(). Ideally blocks in DatanodeDescriptor should be references to blocks in global blockMap.
This patch decreases number of times blocks.clear() is invoked but over time there will be separate copies of blocks. Fix is not to call blocks.clear() but update blocks map inline when new blocks are removed or added inside processReport().
> The memory consumption of processReport() in the namenode can be reduced
> ------------------------------------------------------------------------
>
> Key: HADOOP-764
> URL: http://issues.apache.org/jira/browse/HADOOP-764
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: processBlockReport3.patch
>
>
> The FSNamesystem.processReport() method converts the blocklist for a datanode into an array by calling node.getBlocks(). Although this memory allocation is transient, it could possibly require the garbage-collector to work that much harder.
> The method Block.getBlocks() should be deprecated. Code that currently uses this method should instead iterate over the Collection.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-764) The memory consumption of
processReport() in the namenode can be reduced
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-764?page=comments#action_12456626 ]
dhruba borthakur commented on HADOOP-764:
-----------------------------------------
I agree with your observation. This is another level of refinement that we could do as a follow-on to this patch. Does it sound ok?
> The memory consumption of processReport() in the namenode can be reduced
> ------------------------------------------------------------------------
>
> Key: HADOOP-764
> URL: http://issues.apache.org/jira/browse/HADOOP-764
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: processBlockReport3.patch
>
>
> The FSNamesystem.processReport() method converts the blocklist for a datanode into an array by calling node.getBlocks(). Although this memory allocation is transient, it could possibly require the garbage-collector to work that much harder.
> The method Block.getBlocks() should be deprecated. Code that currently uses this method should instead iterate over the Collection.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-764) The memory consumption of
processReport() in the namenode can be reduced
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-764?page=all ]
dhruba borthakur updated HADOOP-764:
------------------------------------
Status: Patch Available (was: Open)
This patch reduces namenode's memory consumption while processing block reports.
> The memory consumption of processReport() in the namenode can be reduced
> ------------------------------------------------------------------------
>
> Key: HADOOP-764
> URL: http://issues.apache.org/jira/browse/HADOOP-764
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: processBlockReport3.patch
>
>
> The FSNamesystem.processReport() method converts the blocklist for a datanode into an array by calling node.getBlocks(). Although this memory allocation is transient, it could possibly require the garbage-collector to work that much harder.
> The method Block.getBlocks() should be deprecated. Code that currently uses this method should instead iterate over the Collection.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-764) The memory consumption of
processReport() in the namenode can be reduced
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-764?page=comments#action_12455500 ]
Konstantin Shvachko commented on HADOOP-764:
--------------------------------------------
If we use iterator for old report, then oldPos and oldsize become redundant
iter.hasNext() <=> oldPos < oldsize
> The memory consumption of processReport() in the namenode can be reduced
> ------------------------------------------------------------------------
>
> Key: HADOOP-764
> URL: http://issues.apache.org/jira/browse/HADOOP-764
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: processBlockReport.patch
>
>
> The FSNamesystem.processReport() method converts the blocklist for a datanode into an array by calling node.getBlocks(). Although this memory allocation is transient, it could possibly require the garbage-collector to work that much harder.
> The method Block.getBlocks() should be deprecated. Code that currently uses this method should instead iterate over the Collection.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-764) The memory consumption of
processReport() in the namenode can be reduced
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-764?page=comments#action_12456652 ]
Raghu Angadi commented on HADOOP-764:
-------------------------------------
Yes, I was going to comment saying the patch should be applied.
> The memory consumption of processReport() in the namenode can be reduced
> ------------------------------------------------------------------------
>
> Key: HADOOP-764
> URL: http://issues.apache.org/jira/browse/HADOOP-764
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: processBlockReport3.patch
>
>
> The FSNamesystem.processReport() method converts the blocklist for a datanode into an array by calling node.getBlocks(). Although this memory allocation is transient, it could possibly require the garbage-collector to work that much harder.
> The method Block.getBlocks() should be deprecated. Code that currently uses this method should instead iterate over the Collection.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-764) The memory consumption of
processReport() in the namenode can be reduced
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-764?page=all ]
dhruba borthakur updated HADOOP-764:
------------------------------------
Attachment: processBlockReport.patch
Request for code review.
> The memory consumption of processReport() in the namenode can be reduced
> ------------------------------------------------------------------------
>
> Key: HADOOP-764
> URL: http://issues.apache.org/jira/browse/HADOOP-764
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: processBlockReport.patch
>
>
> The FSNamesystem.processReport() method converts the blocklist for a datanode into an array by calling node.getBlocks(). Although this memory allocation is transient, it could possibly require the garbage-collector to work that much harder.
> The method Block.getBlocks() should be deprecated. Code that currently uses this method should instead iterate over the Collection.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-764) The memory consumption of
processReport() in the namenode can be reduced
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-764?page=comments#action_12456515 ]
Raghu Angadi commented on HADOOP-764:
-------------------------------------
Regd references:
Say Bx is a block in Namenode.blockMap and Bx_ref is in Dx.blocks as desired. Now when we update Dx.block in processReport(), we do Dx.blocks.clear() : i.e. Bx_ref is removed. Bx_new_ref in newReport array is added to Dx.block. So there are two separate objects referenced by namenode.blockMap and Dx.block.
newReport[] does not hold reference to blocks in namenode.blockMap (as far as I can see). What I am missing?
> The memory consumption of processReport() in the namenode can be reduced
> ------------------------------------------------------------------------
>
> Key: HADOOP-764
> URL: http://issues.apache.org/jira/browse/HADOOP-764
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: processBlockReport3.patch
>
>
> The FSNamesystem.processReport() method converts the blocklist for a datanode into an array by calling node.getBlocks(). Although this memory allocation is transient, it could possibly require the garbage-collector to work that much harder.
> The method Block.getBlocks() should be deprecated. Code that currently uses this method should instead iterate over the Collection.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-764) The memory consumption of
processReport() in the namenode can be reduced
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-764?page=comments#action_12454778 ]
Raghu Angadi commented on HADOOP-764:
-------------------------------------
Another memory overhead in processReport() is the fact that we seem to do blocks.clear() and rebuild the blocks tree each time.
> The memory consumption of processReport() in the namenode can be reduced
> ------------------------------------------------------------------------
>
> Key: HADOOP-764
> URL: http://issues.apache.org/jira/browse/HADOOP-764
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
>
> The FSNamesystem.processReport() method converts the blocklist for a datanode into an array by calling node.getBlocks(). Although this memory allocation is transient, it could possibly require the garbage-collector to work that much harder.
> The method Block.getBlocks() should be deprecated. Code that currently uses this method should instead iterate over the Collection.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-764) The memory consumption of
processReport() in the namenode can be reduced
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-764?page=comments#action_12456417 ]
dhruba borthakur commented on HADOOP-764:
-----------------------------------------
All those other places you mention actually contain references to the same block object. The block object is created when the blockReport RPC arrives at the namenode, and then references to it are stored in Namenode.blockMap as well as DatanodeDescriptor.blocks.
> The memory consumption of processReport() in the namenode can be reduced
> ------------------------------------------------------------------------
>
> Key: HADOOP-764
> URL: http://issues.apache.org/jira/browse/HADOOP-764
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: processBlockReport3.patch
>
>
> The FSNamesystem.processReport() method converts the blocklist for a datanode into an array by calling node.getBlocks(). Although this memory allocation is transient, it could possibly require the garbage-collector to work that much harder.
> The method Block.getBlocks() should be deprecated. Code that currently uses this method should instead iterate over the Collection.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-764) The memory consumption of
processReport() in the namenode can be reduced
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-764?page=all ]
dhruba borthakur updated HADOOP-764:
------------------------------------
Attachment: processBlockReport3.patch
Incorporated code review comments.
> The memory consumption of processReport() in the namenode can be reduced
> ------------------------------------------------------------------------
>
> Key: HADOOP-764
> URL: http://issues.apache.org/jira/browse/HADOOP-764
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: processBlockReport3.patch
>
>
> The FSNamesystem.processReport() method converts the blocklist for a datanode into an array by calling node.getBlocks(). Although this memory allocation is transient, it could possibly require the garbage-collector to work that much harder.
> The method Block.getBlocks() should be deprecated. Code that currently uses this method should instead iterate over the Collection.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-764) The memory consumption of
processReport() in the namenode can be reduced
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-764?page=comments#action_12456494 ]
Hadoop QA commented on HADOOP-764:
----------------------------------
+1, http://issues.apache.org/jira/secure/attachment/12346511/processBlockReport3.patch applied and successfully tested against trunk revision r483294
> The memory consumption of processReport() in the namenode can be reduced
> ------------------------------------------------------------------------
>
> Key: HADOOP-764
> URL: http://issues.apache.org/jira/browse/HADOOP-764
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: processBlockReport3.patch
>
>
> The FSNamesystem.processReport() method converts the blocklist for a datanode into an array by calling node.getBlocks(). Although this memory allocation is transient, it could possibly require the garbage-collector to work that much harder.
> The method Block.getBlocks() should be deprecated. Code that currently uses this method should instead iterate over the Collection.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-764) The memory consumption of
processReport() in the namenode can be reduced
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-764?page=all ]
Doug Cutting updated HADOOP-764:
--------------------------------
Status: Resolved (was: Patch Available)
Fix Version/s: 0.10.0
Resolution: Fixed
I just committed this. Thanks, Dhruba.
> The memory consumption of processReport() in the namenode can be reduced
> ------------------------------------------------------------------------
>
> Key: HADOOP-764
> URL: http://issues.apache.org/jira/browse/HADOOP-764
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Fix For: 0.10.0
>
> Attachments: processBlockReport3.patch
>
>
> The FSNamesystem.processReport() method converts the blocklist for a datanode into an array by calling node.getBlocks(). Although this memory allocation is transient, it could possibly require the garbage-collector to work that much harder.
> The method Block.getBlocks() should be deprecated. Code that currently uses this method should instead iterate over the Collection.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-764) The memory consumption of
processReport() in the namenode can be reduced
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-764?page=all ]
dhruba borthakur updated HADOOP-764:
------------------------------------
Attachment: (was: processBlockReport.patch)
> The memory consumption of processReport() in the namenode can be reduced
> ------------------------------------------------------------------------
>
> Key: HADOOP-764
> URL: http://issues.apache.org/jira/browse/HADOOP-764
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: processBlockReport3.patch
>
>
> The FSNamesystem.processReport() method converts the blocklist for a datanode into an array by calling node.getBlocks(). Although this memory allocation is transient, it could possibly require the garbage-collector to work that much harder.
> The method Block.getBlocks() should be deprecated. Code that currently uses this method should instead iterate over the Collection.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira