You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Raghu Angadi (JIRA)" <ji...@apache.org> on 2007/02/07 20:44:05 UTC
[jira] Created: (HADOOP-988) Namenode should use single map for
block to its meta data.
Namenode should use single map for block to its meta data.
----------------------------------------------------------
Key: HADOOP-988
URL: https://issues.apache.org/jira/browse/HADOOP-988
Project: Hadoop
Issue Type: Improvement
Components: dfs
Reporter: Raghu Angadi
Assigned To: Sameer Paranjpye
Fix For: 0.12.0
This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
1) blockMap : maps block to list of datanodes that contain the block
2) activeBlocks : maps block to INode that it blongs to.
Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
We could combine these two into something like block to { containingNodes, INode, etc }.
Another option is to get Move INode and list of dataNodes into Block object.
Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12482512 ]
Raghu Angadi commented on HADOOP-988:
-------------------------------------
> Please don't use inheritance for this. It creates lots of confusion. Please create a field that is the Block.
This is what current patch does.
I am planning to keep the inheritance hidden, just like how current BlockInfo is inside BlocksMap. Only advantage of inheritance is that it will further reduce the memory per block by 16 bytes.
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Attachments: HADOOP-988-1.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-988:
--------------------------------
Attachment: HADOOP-988-5.patch
Update patch for conflicts.
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Fix For: 0.13.0
>
> Attachments: HADOOP-988-1.patch, HADOOP-988-3.patch, HADOOP-988-4.patch, HADOOP-988-5.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-988:
--------------------------------
Attachment: HADOOP-988-6.patch
6.patch: Fixed again for conflicts.
Please commit this patch earlier than later. It changes some basic interface and many patches might interfere with it if this is not committed sooner.
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Fix For: 0.13.0
>
> Attachments: HADOOP-988-1.patch, HADOOP-988-3.patch, HADOOP-988-4.patch, HADOOP-988-5.patch, HADOOP-988-6.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12482504 ]
Owen O'Malley commented on HADOOP-988:
--------------------------------------
Please don't use inheritance for this. It creates lots of confusion. Please create a field that is the Block.
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Attachments: HADOOP-988-1.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-988:
--------------------------------
Fix Version/s: 0.13.0
Status: Patch Available (was: Open)
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Fix For: 0.13.0
>
> Attachments: HADOOP-988-1.patch, HADOOP-988-3.patch, HADOOP-988-4.patch, HADOOP-988-5.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-988:
--------------------------------
Attachment: HADOOP-988-1.patch
Attached 1.patch for review. I have done fair amount of testing.
> For each block : + BlockInfo object (24 bytes) - HashMap$Entry(24bytes in activeBlocks) - Block (24 ...
This becomes : + BlockInfo (24) - HashMap$Entry(24, in activeBlocks) - (datanode) ArrayList (24) - Block(24 for block existing before namenode restarts.
i.e. 48 byte per block.
Since this patch touches various places in FSNamesystem.java, I also did minor code changes and removed some allocations here and there (e.g. in open()).
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Attachments: HADOOP-988-1.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12482628 ]
Raghu Angadi commented on HADOOP-988:
-------------------------------------
Thanks Hairong. That check (done during safe mode) could go wrong with patch. I will submit an updated patch after talking to Konstantin.
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Attachments: HADOOP-988-1.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-988:
-----------------------------
Status: Open (was: Patch Available)
Raghu, this patch no longer cleanly applies to trunk due to conflicts in FSNamesystem. Could you regenerate it please? Thanks.
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Fix For: 0.13.0
>
> Attachments: HADOOP-988-1.patch, HADOOP-988-3.patch, HADOOP-988-4.patch, HADOOP-988-5.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486536 ]
Hadoop QA commented on HADOOP-988:
----------------------------------
+1, because http://issues.apache.org/jira/secure/attachment/12354892/HADOOP-988-5.patch applied and successfully tested against trunk revision http://svn.apache.org/repos/asf/lucene/hadoop/trunk/525290. Results are at http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Fix For: 0.13.0
>
> Attachments: HADOOP-988-1.patch, HADOOP-988-3.patch, HADOOP-988-4.patch, HADOOP-988-5.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12482519 ]
Raghu Angadi commented on HADOOP-988:
-------------------------------------
Ok, Owen clarified that inheritance will affect things like RPC which will end up serializing BlockInfo when users just want to send just a Block (among other confusions). HADOOP-989 aims to get rid of Block onjects.. we will get the same memory savings when that jira is fixed. For now we will keep Block and BlockInfo seperate.
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Attachments: HADOOP-988-1.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-988:
-----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I've just committed this. Thanks Raghu!
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Fix For: 0.13.0
>
> Attachments: HADOOP-988-1.patch, HADOOP-988-3.patch, HADOOP-988-4.patch, HADOOP-988-5.patch, HADOOP-988-6.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12482493 ]
Hairong Kuang commented on HADOOP-988:
--------------------------------------
This patch creates a class BlocksMap mapping a block to BlockInfo which contains a reference to the Block. It looks more natural to me to let class BlockInfo inherit from class Block.
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Attachments: HADOOP-988-1.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12475551 ]
Raghu Angadi commented on HADOOP-988:
-------------------------------------
now blocks map maps a Block to a new object BlockInfo. The effective savings are 24 bytes for each of pre-existing blocks :
For each block : + BlockInfo object (24 bytes) - HashMap$Entry(24bytes in activeBlocks) - Block (24 bytes for blocks that exist before namenode restarts)
If ArrayList used for datanodes containing a block can be changed to a java array, it will be 24 more bytes per block. I am implementing this part now. This ArrayList is moved into new BlockInfo object.
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Fix For: 0.12.0
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi reassigned HADOOP-988:
-----------------------------------
Assignee: Raghu Angadi (was: Sameer Paranjpye)
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Fix For: 0.12.0
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12482456 ]
Raghu Angadi commented on HADOOP-988:
-------------------------------------
removed 2.patch. Not necessary right now. I will include a different patch that fixes 'dfsadmin -report' also later.
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Attachments: HADOOP-988-1.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-988:
--------------------------------
Attachment: HADOOP-988-4.patch
4.patch: Just improves an assert comment.
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Attachments: HADOOP-988-1.patch, HADOOP-988-3.patch, HADOOP-988-4.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-988:
--------------------------------
Attachment: (was: HADOOP-988-2.patch)
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Attachments: HADOOP-988-1.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-988:
--------------------------------
Status: Patch Available (was: Open)
Incorporated Hairong's and Konstantin's comments
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Attachments: HADOOP-988-1.patch, HADOOP-988-3.patch, HADOOP-988-4.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12487295 ]
Hadoop QA commented on HADOOP-988:
----------------------------------
+1, because http://issues.apache.org/jira/secure/attachment/12355091/HADOOP-988-6.patch applied and successfully tested against trunk revision http://svn.apache.org/repos/asf/lucene/hadoop/trunk/526215. Results are at http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Fix For: 0.13.0
>
> Attachments: HADOOP-988-1.patch, HADOOP-988-3.patch, HADOOP-988-4.patch, HADOOP-988-5.patch, HADOOP-988-6.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12487398 ]
Hadoop QA commented on HADOOP-988:
----------------------------------
Integrated in Hadoop-Nightly #50 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/50/)
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Fix For: 0.13.0
>
> Attachments: HADOOP-988-1.patch, HADOOP-988-3.patch, HADOOP-988-4.patch, HADOOP-988-5.patch, HADOOP-988-6.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-988:
--------------------------------
Status: Patch Available (was: Open)
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Fix For: 0.13.0
>
> Attachments: HADOOP-988-1.patch, HADOOP-988-3.patch, HADOOP-988-4.patch, HADOOP-988-5.patch, HADOOP-988-6.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-988:
--------------------------------
Attachment: HADOOP-988-2.patch
attached updated patch. This includes a correction to datanode log message and an update Namenode's front page ( includes number of blocks and replication factor, HADOOP-620 ).
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Attachments: HADOOP-988-1.patch, HADOOP-988-2.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12474799 ]
Raghu Angadi commented on HADOOP-988:
-------------------------------------
Proposed chages:
create a new class 'BlockMap' class that internally maps a block to its metadata (currently INode and containing nodes). This class will provided member functions for various operations on containingNodes and current accesses to activeBlocks.
This will provide flexibility to use either ArrayList or a java array (saves an ArrayList object per block) for containing nodes.
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Fix For: 0.12.0
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12482616 ]
Hairong Kuang commented on HADOOP-988:
--------------------------------------
+1 The patch looks good. One comment is that the patch caculates the number of active blocks to be equal to the blockMap size, which is not true all the time. Active blocks are the blocks that belong to a file, but blockMap may contain blocks that do not belong to any file. Please check if this causes any problem.
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Attachments: HADOOP-988-1.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-988:
--------------------------------
Attachment: HADOOP-988-3.patch
Attached 3.patch fixes the bug Hairong found.
isConsistent() is a little costlier during initial safeMode and is moved into assert. Thanks to Konstantin for the suggestion.
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Attachments: HADOOP-988-1.patch, HADOOP-988-3.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-988) Namenode should use single map for
block to its meta data.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-988:
--------------------------------
Status: Open (was: Patch Available)
Will resubmit after HADOOP-702 is committed.
> Namenode should use single map for block to its meta data.
> ----------------------------------------------------------
>
> Key: HADOOP-988
> URL: https://issues.apache.org/jira/browse/HADOOP-988
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Attachments: HADOOP-988-1.patch, HADOOP-988-3.patch, HADOOP-988-4.patch
>
>
> This is a follow up from HADOOP-803. Currently there two maps which have similar functionality :
> 1) blockMap : maps block to list of datanodes that contain the block
> 2) activeBlocks : maps block to INode that it blongs to.
> Apart from simplifying, it saves 32 bytes per block and 24 bytes by avoid extra block object we currently have for files that exist before Namenode starts (see HADOOP-803).
> We could combine these two into something like block to { containingNodes, INode, etc }.
> Another option is to get Move INode and list of dataNodes into Block object.
> Another option that requires bigger change is not use Block object but just 64 bit BlockId. Then the map would be BlockId to all the block related info. I will file another Jira regd not using Block object in NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.