You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2011/01/27 19:49:44 UTC
[jira] Created: (HBASE-3484) Replace memstore's
ConcurrentSkipListMap with our own implementation
Replace memstore's ConcurrentSkipListMap with our own implementation
--------------------------------------------------------------------
Key: HBASE-3484
URL: https://issues.apache.org/jira/browse/HBASE-3484
Project: HBase
Issue Type: Improvement
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Fix For: 0.92.0
By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements to it for our use case in MemStore:
- add an iterator.replace() method which should allow us to do upsert much more cheaply
- implement a Set directly without having to do Map<KeyValue,KeyValue> to save one reference per entry
It turns out CSLM is in public domain from its development as part of JSR 166, so we should be OK with licenses.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Updated] (HBASE-3484) Replace memstore's
ConcurrentSkipListMap with our own implementation
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3484:
-------------------------
Component/s: performance
> Replace memstore's ConcurrentSkipListMap with our own implementation
> --------------------------------------------------------------------
>
> Key: HBASE-3484
> URL: https://issues.apache.org/jira/browse/HBASE-3484
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.92.0
>
>
> By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements to it for our use case in MemStore:
> - add an iterator.replace() method which should allow us to do upsert much more cheaply
> - implement a Set directly without having to do Map<KeyValue,KeyValue> to save one reference per entry
> It turns out CSLM is in public domain from its development as part of JSR 166, so we should be OK with licenses.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3484) Replace memstore's
ConcurrentSkipListMap with our own implementation
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214179#comment-13214179 ]
Todd Lipcon commented on HBASE-3484:
------------------------------------
bq. How you think Todd? Because of the tiering cost more or is it something to do w/ mslab allocations?
Extra container costs with the fact that we have extra CSLM objects for each row. I haven't measured but I bet there is some map-wide overhead that we're paying.
There are some other things I noticed that could be improved, though. In particular, CSLM optimizes for Comparable keys, so if you specify a custom comparator, then it has to wrap every key you insert with a wrapper object. Specializing CSLM for our purposes would easily save 64 bytes per entry on this.
Another thought I had was to do the following:
- have the actual entries in rowMap be Object-typed, rather than CSLMs.
- when the first insert happens, just insert the KeyValue itself (optimization for the case where each row has only one cell)
- when more inserts happen, swap it out for a proper container type
The proper container type's also interesting to consider here. We never have contention on update within a row, since the updates happen under a row lock, right? So, we can consider any map type that supports single-writer multiple-reader efficiently, which is a wider range of data structures than support multi-writer multi-reader. One possibility is snap trees or even copy-on-write sorted array lists.
bq. What would you like to see test-wise proving this direction better than what we currently have? I could work up some tests?
Would be great if we had a benchmark focused on memstore-only which allowed a mix of the following operations from different threads:
- full scans
- range scans
- updates to existing rows which just touch 1 or a few columns
- updates to existing rows which touch lots of columns
- inserts of new rows (few or lots of columns)
But it's a bit of work to do all that. So, a microbenchmark which just timed something like having 20 threads each do a bunch of inserts with multi-column rows would at least show whether there's promise here.
> Replace memstore's ConcurrentSkipListMap with our own implementation
> --------------------------------------------------------------------
>
> Key: HBASE-3484
> URL: https://issues.apache.org/jira/browse/HBASE-3484
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Priority: Critical
> Attachments: hierarchical-map.txt
>
>
> By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements to it for our use case in MemStore:
> - add an iterator.replace() method which should allow us to do upsert much more cheaply
> - implement a Set directly without having to do Map<KeyValue,KeyValue> to save one reference per entry
> It turns out CSLM is in public domain from its development as part of JSR 166, so we should be OK with licenses.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-3484) Replace memstore's
ConcurrentSkipListMap with our own implementation
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon reassigned HBASE-3484:
----------------------------------
Assignee: (was: Todd Lipcon)
> Replace memstore's ConcurrentSkipListMap with our own implementation
> --------------------------------------------------------------------
>
> Key: HBASE-3484
> URL: https://issues.apache.org/jira/browse/HBASE-3484
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Priority: Critical
>
> By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements to it for our use case in MemStore:
> - add an iterator.replace() method which should allow us to do upsert much more cheaply
> - implement a Set directly without having to do Map<KeyValue,KeyValue> to save one reference per entry
> It turns out CSLM is in public domain from its development as part of JSR 166, so we should be OK with licenses.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3484) Replace memstore's
ConcurrentSkipListMap with our own implementation
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214132#comment-13214132 ]
stack commented on HBASE-3484:
------------------------------
bq. It probably has negative memory effects in its current incarnation.
How you think Todd? Because of the tiering cost more or is it something to do w/ mslab allocations?
What would you like to see test-wise proving this direction better than what we currently have? I could work up some tests?
> Replace memstore's ConcurrentSkipListMap with our own implementation
> --------------------------------------------------------------------
>
> Key: HBASE-3484
> URL: https://issues.apache.org/jira/browse/HBASE-3484
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Priority: Critical
> Attachments: hierarchical-map.txt
>
>
> By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements to it for our use case in MemStore:
> - add an iterator.replace() method which should allow us to do upsert much more cheaply
> - implement a Set directly without having to do Map<KeyValue,KeyValue> to save one reference per entry
> It turns out CSLM is in public domain from its development as part of JSR 166, so we should be OK with licenses.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3484) Replace memstore's
ConcurrentSkipListMap with our own implementation
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HBASE-3484:
-------------------------------
Attachment: hierarchical-map.txt
Here's something I hacked together tonight which maps the memstore maps hierarchical. It should save a bit of CPU especially when doing wide puts, but I haven't done any serious benchmarking. It probably has negative memory effects in its current incarnation. Seems to kind-of work.
> Replace memstore's ConcurrentSkipListMap with our own implementation
> --------------------------------------------------------------------
>
> Key: HBASE-3484
> URL: https://issues.apache.org/jira/browse/HBASE-3484
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Priority: Critical
> Attachments: hierarchical-map.txt
>
>
> By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements to it for our use case in MemStore:
> - add an iterator.replace() method which should allow us to do upsert much more cheaply
> - implement a Set directly without having to do Map<KeyValue,KeyValue> to save one reference per entry
> It turns out CSLM is in public domain from its development as part of JSR 166, so we should be OK with licenses.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3484) Replace memstore's
ConcurrentSkipListMap with our own implementation
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3484:
-------------------------
Priority: Critical (was: Major)
Upping this to critical. It keeps coming up as an issue.
> Replace memstore's ConcurrentSkipListMap with our own implementation
> --------------------------------------------------------------------
>
> Key: HBASE-3484
> URL: https://issues.apache.org/jira/browse/HBASE-3484
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.92.0
>
>
> By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements to it for our use case in MemStore:
> - add an iterator.replace() method which should allow us to do upsert much more cheaply
> - implement a Set directly without having to do Map<KeyValue,KeyValue> to save one reference per entry
> It turns out CSLM is in public domain from its development as part of JSR 166, so we should be OK with licenses.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3484) Replace memstore's
ConcurrentSkipListMap with our own implementation
Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13409901#comment-13409901 ]
Otis Gospodnetic commented on HBASE-3484:
-----------------------------------------
@JD - what would/should the ideal graph look like, roughly?
> Replace memstore's ConcurrentSkipListMap with our own implementation
> --------------------------------------------------------------------
>
> Key: HBASE-3484
> URL: https://issues.apache.org/jira/browse/HBASE-3484
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Priority: Critical
> Attachments: hierarchical-map.txt, memstore_drag.png
>
>
> By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements to it for our use case in MemStore:
> - add an iterator.replace() method which should allow us to do upsert much more cheaply
> - implement a Set directly without having to do Map<KeyValue,KeyValue> to save one reference per entry
> It turns out CSLM is in public domain from its development as part of JSR 166, so we should be OK with licenses.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3484) Replace memstore's
ConcurrentSkipListMap with our own implementation
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410818#comment-13410818 ]
Jean-Daniel Cryans commented on HBASE-3484:
-------------------------------------------
Something that's not log(n), so a straight line would be ideal :)
> Replace memstore's ConcurrentSkipListMap with our own implementation
> --------------------------------------------------------------------
>
> Key: HBASE-3484
> URL: https://issues.apache.org/jira/browse/HBASE-3484
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Priority: Critical
> Attachments: hierarchical-map.txt, memstore_drag.png
>
>
> By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements to it for our use case in MemStore:
> - add an iterator.replace() method which should allow us to do upsert much more cheaply
> - implement a Set directly without having to do Map<KeyValue,KeyValue> to save one reference per entry
> It turns out CSLM is in public domain from its development as part of JSR 166, so we should be OK with licenses.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3484) Replace memstore's
ConcurrentSkipListMap with our own implementation
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410944#comment-13410944 ]
Zhihong Ted Yu commented on HBASE-3484:
---------------------------------------
+1 on the above suggestion.
We can trade some complexity for better compression rate.
> Replace memstore's ConcurrentSkipListMap with our own implementation
> --------------------------------------------------------------------
>
> Key: HBASE-3484
> URL: https://issues.apache.org/jira/browse/HBASE-3484
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Priority: Critical
> Attachments: hierarchical-map.txt, memstore_drag.png
>
>
> By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements to it for our use case in MemStore:
> - add an iterator.replace() method which should allow us to do upsert much more cheaply
> - implement a Set directly without having to do Map<KeyValue,KeyValue> to save one reference per entry
> It turns out CSLM is in public domain from its development as part of JSR 166, so we should be OK with licenses.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3484) Replace memstore's
ConcurrentSkipListMap with our own implementation
Posted by "Matt Corgan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410934#comment-13410934 ]
Matt Corgan commented on HBASE-3484:
------------------------------------
I've been pondering how to better compact the data in the memstore. Sometimes we see a 100MB memstore flush that is really 10MB of KeyValues, which gzips to like 2MB, meaning there is a ton of pointer overhead.
One thing that came to mind was splitting each memstore into "regions" of consecutive cell ranges and fronting these regions with an index of some sort. Instead of Set<KeyValue> the memstore is Set<Set<KeyValue>>. When an internal region crosses a certain size we split it in half. With a good index structure in front of the memstore blocks, it might get closer to a linear performance/size curve. It's comparable with hbase splitting a table into regions.
Then, to address the pointer overhead problem, you could use DataBlockEncoding to encode each memstore region individually. A memstore region could accumulate several blocks that get compacted periodically. Given a region size of ~64-256KB, the compaction could be very aggressive and could even be done by the thread writing the data. Again, very similar to how hbase manages the internals of a single region.
This adds moving pieces and complexity but could be developed as a pluggable module that passes the same unit tests as the current memstore.
> Replace memstore's ConcurrentSkipListMap with our own implementation
> --------------------------------------------------------------------
>
> Key: HBASE-3484
> URL: https://issues.apache.org/jira/browse/HBASE-3484
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Priority: Critical
> Attachments: hierarchical-map.txt, memstore_drag.png
>
>
> By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements to it for our use case in MemStore:
> - add an iterator.replace() method which should allow us to do upsert much more cheaply
> - implement a Set directly without having to do Map<KeyValue,KeyValue> to save one reference per entry
> It turns out CSLM is in public domain from its development as part of JSR 166, so we should be OK with licenses.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3484) Replace memstore's
ConcurrentSkipListMap with our own implementation
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215446#comment-13215446 ]
stack commented on HBASE-3484:
------------------------------
Great stuff Todd.
bq. ...copy-on-write sorted array lists.
Could we do this? We'd allocate a new array everytime we did an insert? An array would be cheaper space wise and more efficient scanning, etc., I'd think.... It'd just be the insert and sort that'd be 'expensive'.
Let me have a go at your suggested microbenchmark.
> Replace memstore's ConcurrentSkipListMap with our own implementation
> --------------------------------------------------------------------
>
> Key: HBASE-3484
> URL: https://issues.apache.org/jira/browse/HBASE-3484
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Priority: Critical
> Attachments: hierarchical-map.txt
>
>
> By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements to it for our use case in MemStore:
> - add an iterator.replace() method which should allow us to do upsert much more cheaply
> - implement a Set directly without having to do Map<KeyValue,KeyValue> to save one reference per entry
> It turns out CSLM is in public domain from its development as part of JSR 166, so we should be OK with licenses.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-3484) Replace memstore's
ConcurrentSkipListMap with our own implementation
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410818#comment-13410818 ]
Jean-Daniel Cryans edited comment on HBASE-3484 at 7/10/12 8:21 PM:
--------------------------------------------------------------------
Something that's not log( n ), so a straight line would be ideal :)
Edit: apparently this is thumbdown (n)
was (Author: jdcryans):
Something that's not log(n), so a straight line would be ideal :)
> Replace memstore's ConcurrentSkipListMap with our own implementation
> --------------------------------------------------------------------
>
> Key: HBASE-3484
> URL: https://issues.apache.org/jira/browse/HBASE-3484
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Priority: Critical
> Attachments: hierarchical-map.txt, memstore_drag.png
>
>
> By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements to it for our use case in MemStore:
> - add an iterator.replace() method which should allow us to do upsert much more cheaply
> - implement a Set directly without having to do Map<KeyValue,KeyValue> to save one reference per entry
> It turns out CSLM is in public domain from its development as part of JSR 166, so we should be OK with licenses.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3484) Replace memstore's
ConcurrentSkipListMap with our own implementation
Posted by "Joe Pallas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047342#comment-13047342 ]
Joe Pallas commented on HBASE-3484:
-----------------------------------
I think the performance issue I mentioned above may actually be HBASE-3855.
> Replace memstore's ConcurrentSkipListMap with our own implementation
> --------------------------------------------------------------------
>
> Key: HBASE-3484
> URL: https://issues.apache.org/jira/browse/HBASE-3484
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.92.0
>
>
> By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements to it for our use case in MemStore:
> - add an iterator.replace() method which should allow us to do upsert much more cheaply
> - implement a Set directly without having to do Map<KeyValue,KeyValue> to save one reference per entry
> It turns out CSLM is in public domain from its development as part of JSR 166, so we should be OK with licenses.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HBASE-3484) Replace memstore's
ConcurrentSkipListMap with our own implementation
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987702#action_12987702 ]
Todd Lipcon commented on HBASE-3484:
------------------------------------
Here's the link to the class in Apache Harmony:
https://svn.apache.org/repos/asf/harmony/enhanced/java/branches/java6/classlib/modules/concurrent/src/main/java/java/util/concurrent/ConcurrentSkipListMap.java
> Replace memstore's ConcurrentSkipListMap with our own implementation
> --------------------------------------------------------------------
>
> Key: HBASE-3484
> URL: https://issues.apache.org/jira/browse/HBASE-3484
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.92.0
>
>
> By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements to it for our use case in MemStore:
> - add an iterator.replace() method which should allow us to do upsert much more cheaply
> - implement a Set directly without having to do Map<KeyValue,KeyValue> to save one reference per entry
> It turns out CSLM is in public domain from its development as part of JSR 166, so we should be OK with licenses.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Updated] (HBASE-3484) Replace memstore's
ConcurrentSkipListMap with our own implementation
Posted by "Jean-Daniel Cryans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-3484:
--------------------------------------
Attachment: memstore_drag.png
This is what the slow down currently looks like when using big MemStores. In this graph I flush at around 6GB and there's only 1 region per RS. It seems that the top speed is 100k/s which happens at the very beginning and it can go down to 40k/s.
For those wondering, the dips are caused because we max out our links when the flushes happen all at the same time.
> Replace memstore's ConcurrentSkipListMap with our own implementation
> --------------------------------------------------------------------
>
> Key: HBASE-3484
> URL: https://issues.apache.org/jira/browse/HBASE-3484
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Priority: Critical
> Attachments: hierarchical-map.txt, memstore_drag.png
>
>
> By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements to it for our use case in MemStore:
> - add an iterator.replace() method which should allow us to do upsert much more cheaply
> - implement a Set directly without having to do Map<KeyValue,KeyValue> to save one reference per entry
> It turns out CSLM is in public domain from its development as part of JSR 166, so we should be OK with licenses.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3484) Replace memstore's
ConcurrentSkipListMap with our own implementation
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3484:
-------------------------
Fix Version/s: (was: 0.92.0)
Moving out of 0.92. Don't see it happening in time.
> Replace memstore's ConcurrentSkipListMap with our own implementation
> --------------------------------------------------------------------
>
> Key: HBASE-3484
> URL: https://issues.apache.org/jira/browse/HBASE-3484
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
>
> By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements to it for our use case in MemStore:
> - add an iterator.replace() method which should allow us to do upsert much more cheaply
> - implement a Set directly without having to do Map<KeyValue,KeyValue> to save one reference per entry
> It turns out CSLM is in public domain from its development as part of JSR 166, so we should be OK with licenses.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3484) Replace memstore's
ConcurrentSkipListMap with our own implementation
Posted by "Joe Pallas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025463#comment-13025463 ]
Joe Pallas commented on HBASE-3484:
-----------------------------------
This issue was cited by jdcryans as related to unfortunate performance seen in the following case:
A test program fills a single row of a family with tens of thousands of sequentially increasing qualifiers. Then it performs random gets (or exists) of those qualifiers. The response time seen is (on average) proportional to the ordinal position of the qualifier. If the table is flushed before the random tests begin, then the average response time is basically constant, independent of the qualifier's ordinal position.
I'm not sure that either of the two points in the description actually covers this case, but I don't know enough to say.
> Replace memstore's ConcurrentSkipListMap with our own implementation
> --------------------------------------------------------------------
>
> Key: HBASE-3484
> URL: https://issues.apache.org/jira/browse/HBASE-3484
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.92.0
>
>
> By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements to it for our use case in MemStore:
> - add an iterator.replace() method which should allow us to do upsert much more cheaply
> - implement a Set directly without having to do Map<KeyValue,KeyValue> to save one reference per entry
> It turns out CSLM is in public domain from its development as part of JSR 166, so we should be OK with licenses.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira