[jira] [Comment Edited] (HBASE-11351) Experiments with OffHeap memstores

["Lars Hofhansl (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-11351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14031775#comment-14031775 ] 

Lars Hofhansl edited comment on HBASE-11351 at 6/15/14 3:52 AM:
----------------------------------------------------------------

Next:
* Make it work in 0.96+ (protobufs). Although could keep the current serialization as well
* Better serialization (that makes use of the the sorted nature of the keys and does prefix compression)
* Only store key portion in the key of the map
* Add code to MapDB to allow serialization from an array with offset
* Fork the set implementation so that we do not need to store KV in a map (which in offheap means two copies of the KV)
* Do comparisons off heap! Needs more access into the BTree details (i.e. to the data before it is deserialized onto the heap again)

Not sure how much time I'll get on those. If somebody wants to experiment I'm happy to collaborate.



was (Author: lhofhansl):
Next:
* Make it work in 0.96+ (protobufs). Although could keep the current serialization as well
* Better serialization (that makes use of the the sorted nature of the keys and does prefix compression)
* Only store key portion in the key of the map
* Fork the set implementation so that we do not need to store KV in a map (which in offheap means two copies of the KV)
* Do comparisons off heap! Needs more access into the BTree details (i.e. to the data before it is deserialized onto the heap again)

Not sure how much time I'll get on those. If somebody wants to experiment I'm happy to collaborate.


> Experiments with OffHeap memstores
> ----------------------------------
>
>                 Key: HBASE-11351
>                 URL: https://issues.apache.org/jira/browse/HBASE-11351
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>         Attachments: 11351-0.94-v2.txt, 11351-0.94.txt
>
>
> I have been experimenting with offheap memstores using an offheap BTree implementation of ConcurrentMap provided by MapDB (www.mapdb.org)
> So far I hooked this up in an experimental way with inefficient serialization (in the sense that due to our special usage of the set api we need to back the set by a map, which in the off heap case means we need to serialize and store the value twice. It is used as key and value). But it works.
> It's quite a bit slower and due to need to deserialize upon scan it produces *more* short term garbage.
> Let's use this issue to discuss.



--
This message was sent by Atlassian JIRA
(v6.2#6252)