You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2007/03/30 23:33:25 UTC
[jira] Created: (HADOOP-1187) DFS Scalability: avoid scanning
entire list of datanodes in getAdditionalBlocks
DFS Scalability: avoid scanning entire list of datanodes in getAdditionalBlocks
-------------------------------------------------------------------------------
Key: HADOOP-1187
URL: https://issues.apache.org/jira/browse/HADOOP-1187
Project: Hadoop
Issue Type: Bug
Components: dfs
Reporter: dhruba borthakur
Assigned To: dhruba borthakur
A new block allocations for a file scans the list of all known datanodes to find if the client that is a also a cluster node. If so, then it tries to allocate a replica locally. This check consumes plenty of CPU, especially if the number of datanodes in a cluster is large.
An optimization: if the client is also a cluster node, then cache a reference to the corresponding DatanodeDescriptor from the entry in pendingCreate. The method getAdditionalBlock() uses the cached DatanodeDescriptor and thus avoids scanning the entire list of datanodes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1187) DFS Scalability: avoid scanning
entire list of datanodes in getAdditionalBlocks
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1187:
-------------------------------------
Status: Patch Available (was: Open)
> DFS Scalability: avoid scanning entire list of datanodes in getAdditionalBlocks
> -------------------------------------------------------------------------------
>
> Key: HADOOP-1187
> URL: https://issues.apache.org/jira/browse/HADOOP-1187
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: clientPendingCreate2.patch
>
>
> A new block allocations for a file scans the list of all known datanodes to find if the client that is a also a cluster node. If so, then it tries to allocate a replica locally. This check consumes plenty of CPU, especially if the number of datanodes in a cluster is large.
> An optimization: if the client is also a cluster node, then cache a reference to the corresponding DatanodeDescriptor from the entry in pendingCreate. The method getAdditionalBlock() uses the cached DatanodeDescriptor and thus avoids scanning the entire list of datanodes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1187) DFS Scalability: avoid scanning
entire list of datanodes in getAdditionalBlocks
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12485668 ]
Raghu Angadi commented on HADOOP-1187:
--------------------------------------
Hairong is adding map for hostname to datanode map. Would that help?
> DFS Scalability: avoid scanning entire list of datanodes in getAdditionalBlocks
> -------------------------------------------------------------------------------
>
> Key: HADOOP-1187
> URL: https://issues.apache.org/jira/browse/HADOOP-1187
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
>
> A new block allocations for a file scans the list of all known datanodes to find if the client that is a also a cluster node. If so, then it tries to allocate a replica locally. This check consumes plenty of CPU, especially if the number of datanodes in a cluster is large.
> An optimization: if the client is also a cluster node, then cache a reference to the corresponding DatanodeDescriptor from the entry in pendingCreate. The method getAdditionalBlock() uses the cached DatanodeDescriptor and thus avoids scanning the entire list of datanodes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1187) DFS Scalability: avoid scanning
entire list of datanodes in getAdditionalBlocks
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1187:
-------------------------------------
Attachment: clientPendingCreate.patch
A reference to the client's datanode descriptor is stored in pendingCreates. This avoids a hash-lookup in getAdditionalBlock.
> DFS Scalability: avoid scanning entire list of datanodes in getAdditionalBlocks
> -------------------------------------------------------------------------------
>
> Key: HADOOP-1187
> URL: https://issues.apache.org/jira/browse/HADOOP-1187
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: clientPendingCreate.patch
>
>
> A new block allocations for a file scans the list of all known datanodes to find if the client that is a also a cluster node. If so, then it tries to allocate a replica locally. This check consumes plenty of CPU, especially if the number of datanodes in a cluster is large.
> An optimization: if the client is also a cluster node, then cache a reference to the corresponding DatanodeDescriptor from the entry in pendingCreate. The method getAdditionalBlock() uses the cached DatanodeDescriptor and thus avoids scanning the entire list of datanodes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1187) DFS Scalability: avoid scanning
entire list of datanodes in getAdditionalBlocks
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12485679 ]
dhruba borthakur commented on HADOOP-1187:
------------------------------------------
It could help. But if we store it in the pendingCreates, then we avoid this lookup completely.
> DFS Scalability: avoid scanning entire list of datanodes in getAdditionalBlocks
> -------------------------------------------------------------------------------
>
> Key: HADOOP-1187
> URL: https://issues.apache.org/jira/browse/HADOOP-1187
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
>
> A new block allocations for a file scans the list of all known datanodes to find if the client that is a also a cluster node. If so, then it tries to allocate a replica locally. This check consumes plenty of CPU, especially if the number of datanodes in a cluster is large.
> An optimization: if the client is also a cluster node, then cache a reference to the corresponding DatanodeDescriptor from the entry in pendingCreate. The method getAdditionalBlock() uses the cached DatanodeDescriptor and thus avoids scanning the entire list of datanodes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1187) DFS Scalability: avoid scanning
entire list of datanodes in getAdditionalBlocks
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486927 ]
Hadoop QA commented on HADOOP-1187:
-----------------------------------
Integrated in Hadoop-Nightly #48 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/48/)
> DFS Scalability: avoid scanning entire list of datanodes in getAdditionalBlocks
> -------------------------------------------------------------------------------
>
> Key: HADOOP-1187
> URL: https://issues.apache.org/jira/browse/HADOOP-1187
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Fix For: 0.13.0
>
> Attachments: clientPendingCreate2.patch
>
>
> A new block allocations for a file scans the list of all known datanodes to find if the client that is a also a cluster node. If so, then it tries to allocate a replica locally. This check consumes plenty of CPU, especially if the number of datanodes in a cluster is large.
> An optimization: if the client is also a cluster node, then cache a reference to the corresponding DatanodeDescriptor from the entry in pendingCreate. The method getAdditionalBlock() uses the cached DatanodeDescriptor and thus avoids scanning the entire list of datanodes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1187) DFS Scalability: avoid scanning
entire list of datanodes in getAdditionalBlocks
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1187:
-------------------------------------
Attachment: clientPendingCreate2.patch
Merged with latest code.
> DFS Scalability: avoid scanning entire list of datanodes in getAdditionalBlocks
> -------------------------------------------------------------------------------
>
> Key: HADOOP-1187
> URL: https://issues.apache.org/jira/browse/HADOOP-1187
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: clientPendingCreate2.patch
>
>
> A new block allocations for a file scans the list of all known datanodes to find if the client that is a also a cluster node. If so, then it tries to allocate a replica locally. This check consumes plenty of CPU, especially if the number of datanodes in a cluster is large.
> An optimization: if the client is also a cluster node, then cache a reference to the corresponding DatanodeDescriptor from the entry in pendingCreate. The method getAdditionalBlock() uses the cached DatanodeDescriptor and thus avoids scanning the entire list of datanodes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1187) DFS Scalability: avoid scanning
entire list of datanodes in getAdditionalBlocks
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486522 ]
Hadoop QA commented on HADOOP-1187:
-----------------------------------
+1, because http://issues.apache.org/jira/secure/attachment/12354888/clientPendingCreate2.patch applied and successfully tested against trunk revision http://svn.apache.org/repos/asf/lucene/hadoop/trunk/525290. Results are at http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch
> DFS Scalability: avoid scanning entire list of datanodes in getAdditionalBlocks
> -------------------------------------------------------------------------------
>
> Key: HADOOP-1187
> URL: https://issues.apache.org/jira/browse/HADOOP-1187
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: clientPendingCreate2.patch
>
>
> A new block allocations for a file scans the list of all known datanodes to find if the client that is a also a cluster node. If so, then it tries to allocate a replica locally. This check consumes plenty of CPU, especially if the number of datanodes in a cluster is large.
> An optimization: if the client is also a cluster node, then cache a reference to the corresponding DatanodeDescriptor from the entry in pendingCreate. The method getAdditionalBlock() uses the cached DatanodeDescriptor and thus avoids scanning the entire list of datanodes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1187) DFS Scalability: avoid scanning
entire list of datanodes in getAdditionalBlocks
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486149 ]
Hairong Kuang commented on HADOOP-1187:
---------------------------------------
+1 The code looks good.
> DFS Scalability: avoid scanning entire list of datanodes in getAdditionalBlocks
> -------------------------------------------------------------------------------
>
> Key: HADOOP-1187
> URL: https://issues.apache.org/jira/browse/HADOOP-1187
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: clientPendingCreate.patch
>
>
> A new block allocations for a file scans the list of all known datanodes to find if the client that is a also a cluster node. If so, then it tries to allocate a replica locally. This check consumes plenty of CPU, especially if the number of datanodes in a cluster is large.
> An optimization: if the client is also a cluster node, then cache a reference to the corresponding DatanodeDescriptor from the entry in pendingCreate. The method getAdditionalBlock() uses the cached DatanodeDescriptor and thus avoids scanning the entire list of datanodes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1187) DFS Scalability: avoid scanning
entire list of datanodes in getAdditionalBlocks
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-1187:
------------------------------
Resolution: Fixed
Fix Version/s: 0.13.0
Status: Resolved (was: Patch Available)
I've just committed this. Thanks Dhruba!
> DFS Scalability: avoid scanning entire list of datanodes in getAdditionalBlocks
> -------------------------------------------------------------------------------
>
> Key: HADOOP-1187
> URL: https://issues.apache.org/jira/browse/HADOOP-1187
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Fix For: 0.13.0
>
> Attachments: clientPendingCreate2.patch
>
>
> A new block allocations for a file scans the list of all known datanodes to find if the client that is a also a cluster node. If so, then it tries to allocate a replica locally. This check consumes plenty of CPU, especially if the number of datanodes in a cluster is large.
> An optimization: if the client is also a cluster node, then cache a reference to the corresponding DatanodeDescriptor from the entry in pendingCreate. The method getAdditionalBlock() uses the cached DatanodeDescriptor and thus avoids scanning the entire list of datanodes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1187) DFS Scalability: avoid scanning
entire list of datanodes in getAdditionalBlocks
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1187:
-------------------------------------
Attachment: (was: clientPendingCreate.patch)
> DFS Scalability: avoid scanning entire list of datanodes in getAdditionalBlocks
> -------------------------------------------------------------------------------
>
> Key: HADOOP-1187
> URL: https://issues.apache.org/jira/browse/HADOOP-1187
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: clientPendingCreate2.patch
>
>
> A new block allocations for a file scans the list of all known datanodes to find if the client that is a also a cluster node. If so, then it tries to allocate a replica locally. This check consumes plenty of CPU, especially if the number of datanodes in a cluster is large.
> An optimization: if the client is also a cluster node, then cache a reference to the corresponding DatanodeDescriptor from the entry in pendingCreate. The method getAdditionalBlock() uses the cached DatanodeDescriptor and thus avoids scanning the entire list of datanodes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.