HBase User Group Meeting - Off Heap Caching Slides:

[Li Pi <li...@cloudera.com>]

Not sure which email list this goes to, but somebody requested these.

https://docs.google.com/present/view?id=d23xkzr_55hgnvngf6

Re: HBase User Group Meeting - Off Heap Caching Slides:

[Li Pi <li...@cloudera.com>]

Yeah, I stumbled onto that as well.

Both our implementations are very similar in context, we both use
DirectByteBuffers, and copy the stored data out of the heap. I haven't
looked at his code too much, but it seems that he optimizes for a more
general case, while my allocator exploits the fact that the cached objects
are very similar in size. He creates his own slice method and pointer
classes, for instance, in order to save a few bytes. I don't bother, as the
average cached object in HBase is far larger.

Additionally, I used the memcached allocation model, where we divide a slab
into equally sized blocks, and simply waste a bit of memory in order to deal
with fragmentation. This works extremely well, as HBase block sizes are
generally very close to eachother.

>From glancing through his code, it seems he allows for fragmented writes,
and has a move function, quite possibly for compaction. I haven't reviewed
the code thoroughly though.

My code is available in hbase trunk right now - so if you'd like to do a
review/comparison, thats available to you. Also, I'd guess the final
difference is that mine has already been integrated into hbase ;).

If theres any improvement that can be made to 4027, feel free to let me know
- it would be greatly appreciated.

~Li

On Mon, Sep 5, 2011 at 1:34 PM, Otis Gospodnetic <otis_gospodnetic@yahoo.com
> wrote:

> Hi Li,
>
> I stumbled upon https://github.com/raffaeleguidi/DirectMemory today and
> then noticed your slides in this email below.
> Could you please compare the 2 approaches/implementations?
>
> Thanks,
> Otis
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Hadoop - HBase
> Hadoop ecosystem search :: http://search-hadoop.com/
>
>
> ----- Original Message -----
> > From: Li Pi <li...@cloudera.com>
> > To: dev@hbase.apache.org; user@hbase.apache.org
> > Cc:
> > Sent: Tuesday, August 23, 2011 5:03 PM
> > Subject: HBase User Group Meeting - Off Heap Caching Slides:
> >
> > Not sure which email list this goes to, but somebody requested these.
> >
> > https://docs.google.com/present/view?id=d23xkzr_55hgnvngf6
> >
>

Re: HBase User Group Meeting - Off Heap Caching Slides:

[Otis Gospodnetic <ot...@yahoo.com>]

Hi Li,

I stumbled upon https://github.com/raffaeleguidi/DirectMemory today and then noticed your slides in this email below.
Could you please compare the 2 approaches/implementations?

Thanks,
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Hadoop - HBase
Hadoop ecosystem search :: http://search-hadoop.com/


----- Original Message -----
> From: Li Pi <li...@cloudera.com>
> To: dev@hbase.apache.org; user@hbase.apache.org
> Cc: 
> Sent: Tuesday, August 23, 2011 5:03 PM
> Subject: HBase User Group Meeting - Off Heap Caching Slides:
> 
> Not sure which email list this goes to, but somebody requested these.
> 
> https://docs.google.com/present/view?id=d23xkzr_55hgnvngf6
>

Re: HBase User Group Meeting - Off Heap Caching Slides:

[Otis Gospodnetic <ot...@yahoo.com>]

Hi Li,

I stumbled upon https://github.com/raffaeleguidi/DirectMemory today and then noticed your slides in this email below.
Could you please compare the 2 approaches/implementations?

Thanks,
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Hadoop - HBase
Hadoop ecosystem search :: http://search-hadoop.com/


----- Original Message -----
> From: Li Pi <li...@cloudera.com>
> To: dev@hbase.apache.org; user@hbase.apache.org
> Cc: 
> Sent: Tuesday, August 23, 2011 5:03 PM
> Subject: HBase User Group Meeting - Off Heap Caching Slides:
> 
> Not sure which email list this goes to, but somebody requested these.
> 
> https://docs.google.com/present/view?id=d23xkzr_55hgnvngf6
>