You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Ted Yu (JIRA)" <ji...@apache.org> on 2011/03/16 19:21:29 UTC
[jira] Created: (HBASE-3657) reduce copying of HRegionInfo's
reduce copying of HRegionInfo's
-------------------------------
Key: HBASE-3657
URL: https://issues.apache.org/jira/browse/HBASE-3657
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu
I looked at calls to this method in AssignmentManager:
public void assignUserRegions(List<HRegionInfo> regions, List<HServerInfo> servers)
Here is one from AssignmentManager:
assignUserRegions(Arrays.asList(regions.keySet().toArray(new HRegionInfo[0])), servers);
Here is another from Master:
this.assignmentManager.assignUserRegions(Arrays.asList(newRegions), servers);
I propose changing the first parameter to HRegionInfo[] so that extra copying can be avoided - considering that the number of regions in production is high.
A bigger task is to reduce memory footprint of AssignmentManager.
It would eliminate this copy:
165 public static <T> List<T> asList(T... array) {
166 return new ArrayList<T>(array);
167 }
168
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HBASE-3657) reduce copying of HRegionInfo's
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3657:
--------------------------
Attachment: hbase-3657.txt
Here is the patch
> reduce copying of HRegionInfo's
> -------------------------------
>
> Key: HBASE-3657
> URL: https://issues.apache.org/jira/browse/HBASE-3657
> Project: HBase
> Issue Type: Improvement
> Reporter: Ted Yu
> Assignee: Ted Yu
> Attachments: hbase-3657.txt
>
>
> I looked at calls to this method in AssignmentManager:
> public void assignUserRegions(List<HRegionInfo> regions, List<HServerInfo> servers)
> Here is one from AssignmentManager:
> assignUserRegions(Arrays.asList(regions.keySet().toArray(new HRegionInfo[0])), servers);
> Here is another from Master:
> this.assignmentManager.assignUserRegions(Arrays.asList(newRegions), servers);
> I propose changing the first parameter to HRegionInfo[] so that extra copying can be avoided - considering that the number of regions in production is high.
> A bigger task is to reduce memory footprint of AssignmentManager.
> It would eliminate this copy:
> 165 public static <T> List<T> asList(T... array) {
> 166 return new ArrayList<T>(array);
> 167 }
> 168
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HBASE-3657) reduce copying of HRegionInfo's
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008106#comment-13008106 ]
stack commented on HBASE-3657:
------------------------------
@Ted I got this when I tried to apply patch:
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Compilation failure
/Users/Stack/checkouts/clean_trunk/src/test/java/org/apache/hadoop/hbase/master/TestLoadBalancer.java:[279,20] roundRobinAssignment(org.apache.hadoop.hbase.HRegionInfo[],java.util.List<org.apache.hadoop.hbase.HServerInfo>) in org.apache.hadoop.hbase.master.LoadBalancer cannot be applied to (java.util.List<org.apache.hadoop.hbase.HRegionInfo>,java.util.List<org.apache.hadoop.hbase.HServerInfo>)
Does it work for you?
Thanks.
> reduce copying of HRegionInfo's
> -------------------------------
>
> Key: HBASE-3657
> URL: https://issues.apache.org/jira/browse/HBASE-3657
> Project: HBase
> Issue Type: Improvement
> Reporter: Ted Yu
> Assignee: Ted Yu
> Attachments: hbase-3657.txt
>
>
> I looked at calls to this method in AssignmentManager:
> public void assignUserRegions(List<HRegionInfo> regions, List<HServerInfo> servers)
> Here is one from AssignmentManager:
> assignUserRegions(Arrays.asList(regions.keySet().toArray(new HRegionInfo[0])), servers);
> Here is another from Master:
> this.assignmentManager.assignUserRegions(Arrays.asList(newRegions), servers);
> I propose changing the first parameter to HRegionInfo[] so that extra copying can be avoided - considering that the number of regions in production is high.
> A bigger task is to reduce memory footprint of AssignmentManager.
> It would eliminate this copy:
> 165 public static <T> List<T> asList(T... array) {
> 166 return new ArrayList<T>(array);
> 167 }
> 168
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HBASE-3657) reduce copying of HRegionInfo's
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3657:
--------------------------
Attachment: hbase-3657-with-unit-test.txt
Forgot to include TestLoadBalancer which passes on my machine.
> reduce copying of HRegionInfo's
> -------------------------------
>
> Key: HBASE-3657
> URL: https://issues.apache.org/jira/browse/HBASE-3657
> Project: HBase
> Issue Type: Improvement
> Reporter: Ted Yu
> Assignee: Ted Yu
> Attachments: hbase-3657-with-unit-test.txt, hbase-3657.txt
>
>
> I looked at calls to this method in AssignmentManager:
> public void assignUserRegions(List<HRegionInfo> regions, List<HServerInfo> servers)
> Here is one from AssignmentManager:
> assignUserRegions(Arrays.asList(regions.keySet().toArray(new HRegionInfo[0])), servers);
> Here is another from Master:
> this.assignmentManager.assignUserRegions(Arrays.asList(newRegions), servers);
> I propose changing the first parameter to HRegionInfo[] so that extra copying can be avoided - considering that the number of regions in production is high.
> A bigger task is to reduce memory footprint of AssignmentManager.
> It would eliminate this copy:
> 165 public static <T> List<T> asList(T... array) {
> 166 return new ArrayList<T>(array);
> 167 }
> 168
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HBASE-3657) reduce copying of HRegionInfo's
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-3657.
--------------------------
Resolution: Fixed
Fix Version/s: 0.92.0
Hadoop Flags: [Reviewed]
Applied to TRUNK. Thank you for the patch Ted.
> reduce copying of HRegionInfo's
> -------------------------------
>
> Key: HBASE-3657
> URL: https://issues.apache.org/jira/browse/HBASE-3657
> Project: HBase
> Issue Type: Improvement
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: hbase-3657-with-unit-test.txt, hbase-3657.txt
>
>
> I looked at calls to this method in AssignmentManager:
> public void assignUserRegions(List<HRegionInfo> regions, List<HServerInfo> servers)
> Here is one from AssignmentManager:
> assignUserRegions(Arrays.asList(regions.keySet().toArray(new HRegionInfo[0])), servers);
> Here is another from Master:
> this.assignmentManager.assignUserRegions(Arrays.asList(newRegions), servers);
> I propose changing the first parameter to HRegionInfo[] so that extra copying can be avoided - considering that the number of regions in production is high.
> A bigger task is to reduce memory footprint of AssignmentManager.
> It would eliminate this copy:
> 165 public static <T> List<T> asList(T... array) {
> 166 return new ArrayList<T>(array);
> 167 }
> 168
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HBASE-3657) reduce copying of HRegionInfo's
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3657:
--------------------------
Attachment: (was: hbase-3657.txt)
> reduce copying of HRegionInfo's
> -------------------------------
>
> Key: HBASE-3657
> URL: https://issues.apache.org/jira/browse/HBASE-3657
> Project: HBase
> Issue Type: Improvement
> Reporter: Ted Yu
> Assignee: Ted Yu
> Attachments: hbase-3657.txt
>
>
> I looked at calls to this method in AssignmentManager:
> public void assignUserRegions(List<HRegionInfo> regions, List<HServerInfo> servers)
> Here is one from AssignmentManager:
> assignUserRegions(Arrays.asList(regions.keySet().toArray(new HRegionInfo[0])), servers);
> Here is another from Master:
> this.assignmentManager.assignUserRegions(Arrays.asList(newRegions), servers);
> I propose changing the first parameter to HRegionInfo[] so that extra copying can be avoided - considering that the number of regions in production is high.
> A bigger task is to reduce memory footprint of AssignmentManager.
> It would eliminate this copy:
> 165 public static <T> List<T> asList(T... array) {
> 166 return new ArrayList<T>(array);
> 167 }
> 168
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HBASE-3657) reduce copying of HRegionInfo's
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3657:
--------------------------
Attachment: hbase-3657.txt
Adopted Benoit's suggestion.
> reduce copying of HRegionInfo's
> -------------------------------
>
> Key: HBASE-3657
> URL: https://issues.apache.org/jira/browse/HBASE-3657
> Project: HBase
> Issue Type: Improvement
> Reporter: Ted Yu
> Assignee: Ted Yu
> Attachments: hbase-3657.txt
>
>
> I looked at calls to this method in AssignmentManager:
> public void assignUserRegions(List<HRegionInfo> regions, List<HServerInfo> servers)
> Here is one from AssignmentManager:
> assignUserRegions(Arrays.asList(regions.keySet().toArray(new HRegionInfo[0])), servers);
> Here is another from Master:
> this.assignmentManager.assignUserRegions(Arrays.asList(newRegions), servers);
> I propose changing the first parameter to HRegionInfo[] so that extra copying can be avoided - considering that the number of regions in production is high.
> A bigger task is to reduce memory footprint of AssignmentManager.
> It would eliminate this copy:
> 165 public static <T> List<T> asList(T... array) {
> 166 return new ArrayList<T>(array);
> 167 }
> 168
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HBASE-3657) reduce copying of HRegionInfo's
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008267#comment-13008267 ]
Hudson commented on HBASE-3657:
-------------------------------
Integrated in HBase-TRUNK #1794 (See [https://hudson.apache.org/hudson/job/HBase-TRUNK/1794/])
HBASE-3657 reduce copying of HRegionInfo's
> reduce copying of HRegionInfo's
> -------------------------------
>
> Key: HBASE-3657
> URL: https://issues.apache.org/jira/browse/HBASE-3657
> Project: HBase
> Issue Type: Improvement
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: hbase-3657-with-unit-test.txt, hbase-3657.txt
>
>
> I looked at calls to this method in AssignmentManager:
> public void assignUserRegions(List<HRegionInfo> regions, List<HServerInfo> servers)
> Here is one from AssignmentManager:
> assignUserRegions(Arrays.asList(regions.keySet().toArray(new HRegionInfo[0])), servers);
> Here is another from Master:
> this.assignmentManager.assignUserRegions(Arrays.asList(newRegions), servers);
> I propose changing the first parameter to HRegionInfo[] so that extra copying can be avoided - considering that the number of regions in production is high.
> A bigger task is to reduce memory footprint of AssignmentManager.
> It would eliminate this copy:
> 165 public static <T> List<T> asList(T... array) {
> 166 return new ArrayList<T>(array);
> 167 }
> 168
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HBASE-3657) reduce copying of HRegionInfo's
Posted by "Benoit Sigoure (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008015#comment-13008015 ]
Benoit Sigoure commented on HBASE-3657:
---------------------------------------
{code}
assignUserRegions(allRegions.keySet().toArray(new HRegionInfo[0]), servers);
{code}
Instead of creating a useless 0-length array that will be discarded almost immediately, can you please do this instead:
{code}
assignUserRegions(allRegions.keySet().toArray(new HRegionInfo[allRegions.size()]), servers);
{code}
Thanks.
> reduce copying of HRegionInfo's
> -------------------------------
>
> Key: HBASE-3657
> URL: https://issues.apache.org/jira/browse/HBASE-3657
> Project: HBase
> Issue Type: Improvement
> Reporter: Ted Yu
> Assignee: Ted Yu
> Attachments: hbase-3657.txt
>
>
> I looked at calls to this method in AssignmentManager:
> public void assignUserRegions(List<HRegionInfo> regions, List<HServerInfo> servers)
> Here is one from AssignmentManager:
> assignUserRegions(Arrays.asList(regions.keySet().toArray(new HRegionInfo[0])), servers);
> Here is another from Master:
> this.assignmentManager.assignUserRegions(Arrays.asList(newRegions), servers);
> I propose changing the first parameter to HRegionInfo[] so that extra copying can be avoided - considering that the number of regions in production is high.
> A bigger task is to reduce memory footprint of AssignmentManager.
> It would eliminate this copy:
> 165 public static <T> List<T> asList(T... array) {
> 166 return new ArrayList<T>(array);
> 167 }
> 168
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira