You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2007/03/24 00:00:32 UTC
[jira] Created: (HADOOP-1155) Additional performance improvement to
chooseTarget
Additional performance improvement to chooseTarget
--------------------------------------------------
Key: HADOOP-1155
URL: https://issues.apache.org/jira/browse/HADOOP-1155
Project: Hadoop
Issue Type: Improvement
Components: dfs
Affects Versions: 0.12.2
Reporter: Hairong Kuang
Assigned To: Hairong Kuang
Fix For: 0.13.0
A few additional thoughts to improve the performance of chooseTarget:
1. Reduce the # of calls to getDistance in sortedByDistance
2. Improve the performance of getNode by adding a rack name to rack node map
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1155) Additional performance improvement to
chooseTarget
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1155:
----------------------------------
Attachment: rackMap2.patch
This patch makes 4 changes:
1. add a rackname to its rack node map in NetworkTopology to speed up getNode
2. optimize sortedByDistance by taking Sameer's suggestion in HADOOP-1073:
> Do we need to sort datanodes by distance? Why not just do a linear scan for the on node and on rack instances, put them at the front of the pipeline and leave the rest in random order?
This suggestion allows us to reduce memeory allocation and the # of calls to getDistance.
3. add a test case to test sortedByDistance
4. change chooseRandom to return a list instead of an array. This allows us to reduce one memory allocation.
> Additional performance improvement to chooseTarget
> --------------------------------------------------
>
> Key: HADOOP-1155
> URL: https://issues.apache.org/jira/browse/HADOOP-1155
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: rackMap.patch, rackMap1.patch, rackMap2.patch
>
>
> A few additional thoughts to improve the performance of chooseTarget:
> 1. Reduce the # of calls to getDistance in sortedByDistance
> 2. Improve the performance of getNode by adding a rack name to rack node map
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1155) Additional performance improvement
to chooseTarget
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12488234 ]
Raghu Angadi commented on HADOOP-1155:
--------------------------------------
+1 for the patch.
But please don't take this as my +1 for the name "pseudoSortByDistance()" :-)
> Additional performance improvement to chooseTarget
> --------------------------------------------------
>
> Key: HADOOP-1155
> URL: https://issues.apache.org/jira/browse/HADOOP-1155
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: rackMap.patch, rackMap1.patch, rackMap2.patch, rackMap3.patch
>
>
> A few additional thoughts to improve the performance of chooseTarget:
> 1. Reduce the # of calls to getDistance in sortedByDistance
> 2. Improve the performance of getNode by adding a rack name to rack node map
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1155) Additional performance improvement to
chooseTarget
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1155:
----------------------------------
Attachment: rackMap.patch
This patch is intended for review. It depends on HADOOP-1073 and collides with HADOOP-988. I will resubmit one after both are resolved.
Besides the changes I suggested in the jira description, I modified the method in ReplicationTargetChooser "private DatanodeDescriptor[] chooseRandom(int numOfReplicas, String nodes, List<DatanodeDescriptor> excludedNodes)" to return a list.
> Additional performance improvement to chooseTarget
> --------------------------------------------------
>
> Key: HADOOP-1155
> URL: https://issues.apache.org/jira/browse/HADOOP-1155
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: rackMap.patch
>
>
> A few additional thoughts to improve the performance of chooseTarget:
> 1. Reduce the # of calls to getDistance in sortedByDistance
> 2. Improve the performance of getNode by adding a rack name to rack node map
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HADOOP-1155) Additional performance improvement
to chooseTarget
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting reopened HADOOP-1155:
----------------------------------
I just reverted this, since it really ought be be benchmarked before it's committed.
> Additional performance improvement to chooseTarget
> --------------------------------------------------
>
> Key: HADOOP-1155
> URL: https://issues.apache.org/jira/browse/HADOOP-1155
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: rackMap.patch, rackMap1.patch, rackMap2.patch, rackMap3.patch
>
>
> A few additional thoughts to improve the performance of chooseTarget:
> 1. Reduce the # of calls to getDistance in sortedByDistance
> 2. Improve the performance of getNode by adding a rack name to rack node map
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1155) Additional performance improvement
to chooseTarget
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12489364 ]
Hadoop QA commented on HADOOP-1155:
-----------------------------------
Integrated in Hadoop-Nightly #60 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/60/)
> Additional performance improvement to chooseTarget
> --------------------------------------------------
>
> Key: HADOOP-1155
> URL: https://issues.apache.org/jira/browse/HADOOP-1155
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: rackMap.patch, rackMap1.patch, rackMap2.patch, rackMap3.patch
>
>
> A few additional thoughts to improve the performance of chooseTarget:
> 1. Reduce the # of calls to getDistance in sortedByDistance
> 2. Improve the performance of getNode by adding a rack name to rack node map
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1155) Additional performance improvement to
chooseTarget
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1155:
----------------------------------
Status: Patch Available (was: Open)
> Additional performance improvement to chooseTarget
> --------------------------------------------------
>
> Key: HADOOP-1155
> URL: https://issues.apache.org/jira/browse/HADOOP-1155
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: rackMap.patch, rackMap1.patch, rackMap2.patch, rackMap3.patch
>
>
> A few additional thoughts to improve the performance of chooseTarget:
> 1. Reduce the # of calls to getDistance in sortedByDistance
> 2. Improve the performance of getNode by adding a rack name to rack node map
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1155) Additional performance improvement
to chooseTarget
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12489227 ]
Doug Cutting commented on HADOOP-1155:
--------------------------------------
Sorry, Tom, I committed this before I saw your comments. I agree with them, but I also didn't want this to get too stale, since Hairong just left for a vacation. Should I file a new issue to rename the method prior to 0.13 to something better than pseudoSortByDistance()? Or should I roll this one back until we get some benchmark data?
> Additional performance improvement to chooseTarget
> --------------------------------------------------
>
> Key: HADOOP-1155
> URL: https://issues.apache.org/jira/browse/HADOOP-1155
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: rackMap.patch, rackMap1.patch, rackMap2.patch, rackMap3.patch
>
>
> A few additional thoughts to improve the performance of chooseTarget:
> 1. Reduce the # of calls to getDistance in sortedByDistance
> 2. Improve the performance of getNode by adding a rack name to rack node map
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1155) Additional performance improvement
to chooseTarget
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12489265 ]
Sameer Paranjpye commented on HADOOP-1155:
------------------------------------------
I believe this problem showed up during Dhrubas tests on 2k nodes. If so, he'll probably have a before and after result for this patch. Dhruba?
> Additional performance improvement to chooseTarget
> --------------------------------------------------
>
> Key: HADOOP-1155
> URL: https://issues.apache.org/jira/browse/HADOOP-1155
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: rackMap.patch, rackMap1.patch, rackMap2.patch, rackMap3.patch
>
>
> A few additional thoughts to improve the performance of chooseTarget:
> 1. Reduce the # of calls to getDistance in sortedByDistance
> 2. Improve the performance of getNode by adding a rack name to rack node map
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1155) Additional performance improvement to
chooseTarget
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-1155:
---------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Hairong!
> Additional performance improvement to chooseTarget
> --------------------------------------------------
>
> Key: HADOOP-1155
> URL: https://issues.apache.org/jira/browse/HADOOP-1155
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: rackMap.patch, rackMap1.patch, rackMap2.patch, rackMap3.patch
>
>
> A few additional thoughts to improve the performance of chooseTarget:
> 1. Reduce the # of calls to getDistance in sortedByDistance
> 2. Improve the performance of getNode by adding a rack name to rack node map
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1155) Additional performance improvement
to chooseTarget
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12487963 ]
Raghu Angadi commented on HADOOP-1155:
--------------------------------------
> 2. optimize sortedByDistance by taking Sameer's suggestion in HADOOP-1073:
I am looking at the patch, in the mean while, I think should change the name of 'sortByDistance()' to something else, since it neither sorts the array nor uses distances any more. But I can't think of a good name.. suggestions are welcome.
> Additional performance improvement to chooseTarget
> --------------------------------------------------
>
> Key: HADOOP-1155
> URL: https://issues.apache.org/jira/browse/HADOOP-1155
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: rackMap.patch, rackMap1.patch, rackMap2.patch
>
>
> A few additional thoughts to improve the performance of chooseTarget:
> 1. Reduce the # of calls to getDistance in sortedByDistance
> 2. Improve the performance of getNode by adding a rack name to rack node map
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1155) Additional performance improvement to
chooseTarget
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1155:
----------------------------------
Attachment: rackMap3.patch
This patch incorprates many good suggestions given by Raghu. Thanks Raghu.
> Additional performance improvement to chooseTarget
> --------------------------------------------------
>
> Key: HADOOP-1155
> URL: https://issues.apache.org/jira/browse/HADOOP-1155
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: rackMap.patch, rackMap1.patch, rackMap2.patch, rackMap3.patch
>
>
> A few additional thoughts to improve the performance of chooseTarget:
> 1. Reduce the # of calls to getDistance in sortedByDistance
> 2. Improve the performance of getNode by adding a rack name to rack node map
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1155) Additional performance improvement to
chooseTarget
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1155:
----------------------------------
Attachment: rackMap1.patch
> Additional performance improvement to chooseTarget
> --------------------------------------------------
>
> Key: HADOOP-1155
> URL: https://issues.apache.org/jira/browse/HADOOP-1155
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: rackMap.patch, rackMap1.patch
>
>
> A few additional thoughts to improve the performance of chooseTarget:
> 1. Reduce the # of calls to getDistance in sortedByDistance
> 2. Improve the performance of getNode by adding a rack name to rack node map
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1155) Additional performance improvement
to chooseTarget
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12488435 ]
Hadoop QA commented on HADOOP-1155:
-----------------------------------
+1
http://issues.apache.org/jira/secure/attachment/12355366/rackMap3.patch applied and successfully tested against trunk revision r527711.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/40/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/40/console
> Additional performance improvement to chooseTarget
> --------------------------------------------------
>
> Key: HADOOP-1155
> URL: https://issues.apache.org/jira/browse/HADOOP-1155
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: rackMap.patch, rackMap1.patch, rackMap2.patch, rackMap3.patch
>
>
> A few additional thoughts to improve the performance of chooseTarget:
> 1. Reduce the # of calls to getDistance in sortedByDistance
> 2. Improve the performance of getNode by adding a rack name to rack node map
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1155) Additional performance improvement
to chooseTarget
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12489515 ]
dhruba borthakur commented on HADOOP-1155:
------------------------------------------
I only have a "before measurement". The method getNode() was consuming about 5% CPU on a 2000 node cluster. I have not yet measured this patch.
> Additional performance improvement to chooseTarget
> --------------------------------------------------
>
> Key: HADOOP-1155
> URL: https://issues.apache.org/jira/browse/HADOOP-1155
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: rackMap.patch, rackMap1.patch, rackMap2.patch, rackMap3.patch
>
>
> A few additional thoughts to improve the performance of chooseTarget:
> 1. Reduce the # of calls to getDistance in sortedByDistance
> 2. Improve the performance of getNode by adding a rack name to rack node map
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1155) Additional performance improvement
to chooseTarget
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12489222 ]
Tom White commented on HADOOP-1155:
-----------------------------------
Following the comments in HADOOP-1256, it would be good to have some benchmarks to show the performance gains for this optimization.
And on the naming point, I'm struggling to come up with a better name too - but you could justify sortByDistance as being perfectly acceptable for a certain definition of "distance". Mathematicians make up all kinds of distance metrics, and that's what's being done here too with the the local rack being given a privileged role. Furthermore one could imagine other optimizations that come up with a different sort order, so the name needs to describe the intent, not the algorithm too closely (disqualifying things like "swapLocalNodesToTop").
> Additional performance improvement to chooseTarget
> --------------------------------------------------
>
> Key: HADOOP-1155
> URL: https://issues.apache.org/jira/browse/HADOOP-1155
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: rackMap.patch, rackMap1.patch, rackMap2.patch, rackMap3.patch
>
>
> A few additional thoughts to improve the performance of chooseTarget:
> 1. Reduce the # of calls to getDistance in sortedByDistance
> 2. Improve the performance of getNode by adding a rack name to rack node map
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1155) Additional performance improvement
to chooseTarget
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12489230 ]
Tom White commented on HADOOP-1155:
-----------------------------------
No problem. I would guess that it's fine - Hairong is likely to have run it on a large cluster - so perhaps just file a Jira for a benchmark.
> Additional performance improvement to chooseTarget
> --------------------------------------------------
>
> Key: HADOOP-1155
> URL: https://issues.apache.org/jira/browse/HADOOP-1155
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: rackMap.patch, rackMap1.patch, rackMap2.patch, rackMap3.patch
>
>
> A few additional thoughts to improve the performance of chooseTarget:
> 1. Reduce the # of calls to getDistance in sortedByDistance
> 2. Improve the performance of getNode by adding a rack name to rack node map
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.