You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Karthik Ranganathan (JIRA)" <ji...@apache.org> on 2012/11/15 00:14:11 UTC
[jira] [Created] (HBASE-7163) Change cachedRegionsLocations in
HConnectionManager from SoftValueSortedMap to ConcurrentSkipListMap
Karthik Ranganathan created HBASE-7163:
------------------------------------------
Summary: Change cachedRegionsLocations in HConnectionManager from SoftValueSortedMap to ConcurrentSkipListMap
Key: HBASE-7163
URL: https://issues.apache.org/jira/browse/HBASE-7163
Project: HBase
Issue Type: Sub-task
Reporter: Karthik Ranganathan
Assignee: Karthik Ranganathan
This change saves 15% CPU on the client side per profiling. In using the ConcurrentSkipListMap, we can do:
tableLocations.floorEntry(row).getValue()
instead of doing:
SortedMap<byte[], HRegionLocation> matchingRegions =
tableLocations.floorEntry(row).getValue();
if (!matchingRegions.isEmpty()) {
HRegionLocation possibleRegion =
matchingRegions.get(matchingRegions.lastKey());
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7163) Low-hanging perf improvements in
HBase client
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498466#comment-13498466 ]
Lars Hofhansl commented on HBASE-7163:
--------------------------------------
Presumably SoftValueSortedMap was used to allow GC'ing cached entries when memory is tight.
Not that I am a big fan of that, but I just want to point out that we loose this with ConcurrentSkipListMap.
> Low-hanging perf improvements in HBase client
> ----------------------------------------------
>
> Key: HBASE-7163
> URL: https://issues.apache.org/jira/browse/HBASE-7163
> Project: HBase
> Issue Type: Sub-task
> Reporter: Karthik Ranganathan
> Assignee: Karthik Ranganathan
>
> 1. Change cachedRegionsLocations in HConnectionManager from SoftValueSortedMap to ConcurrentSkipListMap:
> This change saves 15% CPU on the client side per profiling. In using the ConcurrentSkipListMap, we can do:
> tableLocations.floorEntry(row).getValue()
> instead of doing:
> SortedMap<byte[], HRegionLocation> matchingRegions =
> tableLocations.floorEntry(row).getValue();
> if (!matchingRegions.isEmpty()) {
> HRegionLocation possibleRegion =
> matchingRegions.get(matchingRegions.lastKey());
> }
> 2. NetUtils.getDefaultSocketFactory is very inefficient, use
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7163) Low-hanging perf improvements in
HBase client
Posted by "Karthik Ranganathan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498501#comment-13498501 ]
Karthik Ranganathan commented on HBASE-7163:
--------------------------------------------
Yes, thanks for explicitly mentioning, forgot to mention that point. The thought was that the overhead of caching all regions would not be too large.
> Low-hanging perf improvements in HBase client
> ----------------------------------------------
>
> Key: HBASE-7163
> URL: https://issues.apache.org/jira/browse/HBASE-7163
> Project: HBase
> Issue Type: Sub-task
> Reporter: Karthik Ranganathan
> Assignee: Karthik Ranganathan
>
> 1. Change cachedRegionsLocations in HConnectionManager from SoftValueSortedMap to ConcurrentSkipListMap:
> This change saves 15% CPU on the client side per profiling. In using the ConcurrentSkipListMap, we can do:
> tableLocations.floorEntry(row).getValue()
> instead of doing:
> SortedMap<byte[], HRegionLocation> matchingRegions =
> tableLocations.floorEntry(row).getValue();
> if (!matchingRegions.isEmpty()) {
> HRegionLocation possibleRegion =
> matchingRegions.get(matchingRegions.lastKey());
> }
> 2. NetUtils.getDefaultSocketFactory is very inefficient, use
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7163) Low-hanging perf improvements in
HBase client
Posted by "Karthik Ranganathan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498416#comment-13498416 ]
Karthik Ranganathan commented on HBASE-7163:
--------------------------------------------
@Ted - yes that's the part, fix has another component to it though. Also, changed this task to add one more perf improvement.
> Low-hanging perf improvements in HBase client
> ----------------------------------------------
>
> Key: HBASE-7163
> URL: https://issues.apache.org/jira/browse/HBASE-7163
> Project: HBase
> Issue Type: Sub-task
> Reporter: Karthik Ranganathan
> Assignee: Karthik Ranganathan
>
> 1. Change cachedRegionsLocations in HConnectionManager from SoftValueSortedMap to ConcurrentSkipListMap:
> This change saves 15% CPU on the client side per profiling. In using the ConcurrentSkipListMap, we can do:
> tableLocations.floorEntry(row).getValue()
> instead of doing:
> SortedMap<byte[], HRegionLocation> matchingRegions =
> tableLocations.floorEntry(row).getValue();
> if (!matchingRegions.isEmpty()) {
> HRegionLocation possibleRegion =
> matchingRegions.get(matchingRegions.lastKey());
> }
> 2. NetUtils.getDefaultSocketFactory is very inefficient, use
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7163) Low-hanging perf improvements in
HBase client
Posted by "Karthik Ranganathan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Ranganathan updated HBASE-7163:
---------------------------------------
Description:
1. Change cachedRegionsLocations in HConnectionManager from SoftValueSortedMap to ConcurrentSkipListMap:
This change saves 15% CPU on the client side per profiling. In using the ConcurrentSkipListMap, we can do:
tableLocations.floorEntry(row).getValue()
instead of doing:
SortedMap<byte[], HRegionLocation> matchingRegions =
tableLocations.floorEntry(row).getValue();
if (!matchingRegions.isEmpty()) {
HRegionLocation possibleRegion =
matchingRegions.get(matchingRegions.lastKey());
}
2. NetUtils.getDefaultSocketFactory is very inefficient, use
was:
This change saves 15% CPU on the client side per profiling. In using the ConcurrentSkipListMap, we can do:
tableLocations.floorEntry(row).getValue()
instead of doing:
SortedMap<byte[], HRegionLocation> matchingRegions =
tableLocations.floorEntry(row).getValue();
if (!matchingRegions.isEmpty()) {
HRegionLocation possibleRegion =
matchingRegions.get(matchingRegions.lastKey());
}
> Low-hanging perf improvements in HBase client
> ----------------------------------------------
>
> Key: HBASE-7163
> URL: https://issues.apache.org/jira/browse/HBASE-7163
> Project: HBase
> Issue Type: Sub-task
> Reporter: Karthik Ranganathan
> Assignee: Karthik Ranganathan
>
> 1. Change cachedRegionsLocations in HConnectionManager from SoftValueSortedMap to ConcurrentSkipListMap:
> This change saves 15% CPU on the client side per profiling. In using the ConcurrentSkipListMap, we can do:
> tableLocations.floorEntry(row).getValue()
> instead of doing:
> SortedMap<byte[], HRegionLocation> matchingRegions =
> tableLocations.floorEntry(row).getValue();
> if (!matchingRegions.isEmpty()) {
> HRegionLocation possibleRegion =
> matchingRegions.get(matchingRegions.lastKey());
> }
> 2. NetUtils.getDefaultSocketFactory is very inefficient, use
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7163) Low-hanging perf improvements in
HBase client
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498639#comment-13498639 ]
stack commented on HBASE-7163:
------------------------------
Sounds right [~lhofhansl]
> Low-hanging perf improvements in HBase client
> ----------------------------------------------
>
> Key: HBASE-7163
> URL: https://issues.apache.org/jira/browse/HBASE-7163
> Project: HBase
> Issue Type: Sub-task
> Reporter: Karthik Ranganathan
> Assignee: Karthik Ranganathan
>
> 1. Change cachedRegionsLocations in HConnectionManager from SoftValueSortedMap to ConcurrentSkipListMap:
> This change saves 15% CPU on the client side per profiling. In using the ConcurrentSkipListMap, we can do:
> tableLocations.floorEntry(row).getValue()
> instead of doing:
> SortedMap<byte[], HRegionLocation> matchingRegions =
> tableLocations.floorEntry(row).getValue();
> if (!matchingRegions.isEmpty()) {
> HRegionLocation possibleRegion =
> matchingRegions.get(matchingRegions.lastKey());
> }
> 2. NetUtils.getDefaultSocketFactory is very inefficient, use
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7163) Low-hanging perf improvements in
HBase client
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498635#comment-13498635 ]
Lars Hofhansl commented on HBASE-7163:
--------------------------------------
Probably right.
A region name is what 20 bytes, start and stop key in HRegionInfo can be long, let's say 40, plus region name again and encoded regionname 20 each, plus tablename 20, plus hostname 20. That's 20+40*2+20*2+20+20 = 180 bytes, let's round up to 200 bytes.
(I hope that is roughly right)
Even with 100k regions that'd only be ~19MB.
> Low-hanging perf improvements in HBase client
> ----------------------------------------------
>
> Key: HBASE-7163
> URL: https://issues.apache.org/jira/browse/HBASE-7163
> Project: HBase
> Issue Type: Sub-task
> Reporter: Karthik Ranganathan
> Assignee: Karthik Ranganathan
>
> 1. Change cachedRegionsLocations in HConnectionManager from SoftValueSortedMap to ConcurrentSkipListMap:
> This change saves 15% CPU on the client side per profiling. In using the ConcurrentSkipListMap, we can do:
> tableLocations.floorEntry(row).getValue()
> instead of doing:
> SortedMap<byte[], HRegionLocation> matchingRegions =
> tableLocations.floorEntry(row).getValue();
> if (!matchingRegions.isEmpty()) {
> HRegionLocation possibleRegion =
> matchingRegions.get(matchingRegions.lastKey());
> }
> 2. NetUtils.getDefaultSocketFactory is very inefficient, use
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7163) Change cachedRegionsLocations in
HConnectionManager from SoftValueSortedMap to ConcurrentSkipListMap
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497584#comment-13497584 ]
Ted Yu commented on HBASE-7163:
-------------------------------
@Karthik:
The second snippet contains your proposed change.
Were you referring to the following code in getCachedLocation() ?
{code}
// Cut the cache so that we only get the part that could contain
// regions that match our key
SortedMap<byte[], HRegionLocation> matchingRegions =
tableLocations.headMap(row);
{code}
Thanks
> Change cachedRegionsLocations in HConnectionManager from SoftValueSortedMap to ConcurrentSkipListMap
> ----------------------------------------------------------------------------------------------------
>
> Key: HBASE-7163
> URL: https://issues.apache.org/jira/browse/HBASE-7163
> Project: HBase
> Issue Type: Sub-task
> Reporter: Karthik Ranganathan
> Assignee: Karthik Ranganathan
>
> This change saves 15% CPU on the client side per profiling. In using the ConcurrentSkipListMap, we can do:
> tableLocations.floorEntry(row).getValue()
> instead of doing:
> SortedMap<byte[], HRegionLocation> matchingRegions =
> tableLocations.floorEntry(row).getValue();
> if (!matchingRegions.isEmpty()) {
> HRegionLocation possibleRegion =
> matchingRegions.get(matchingRegions.lastKey());
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7163) Low-hanging perf improvements in
HBase client
Posted by "Karthik Ranganathan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Ranganathan updated HBASE-7163:
---------------------------------------
Summary: Low-hanging perf improvements in HBase client (was: Change cachedRegionsLocations in HConnectionManager from SoftValueSortedMap to ConcurrentSkipListMap)
> Low-hanging perf improvements in HBase client
> ----------------------------------------------
>
> Key: HBASE-7163
> URL: https://issues.apache.org/jira/browse/HBASE-7163
> Project: HBase
> Issue Type: Sub-task
> Reporter: Karthik Ranganathan
> Assignee: Karthik Ranganathan
>
> This change saves 15% CPU on the client side per profiling. In using the ConcurrentSkipListMap, we can do:
> tableLocations.floorEntry(row).getValue()
> instead of doing:
> SortedMap<byte[], HRegionLocation> matchingRegions =
> tableLocations.floorEntry(row).getValue();
> if (!matchingRegions.isEmpty()) {
> HRegionLocation possibleRegion =
> matchingRegions.get(matchingRegions.lastKey());
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira