LruBlockCache alternative

[Benjamin Manes <be...@gmail.com>]

Hi,

I noticed that Accumulo's LruBlockCache [1] appears to be based on HBase's.
I currently have a patch being reviewed in HBASE-15560 [2] that replaces
the pseudo Segmented LRU with the TinyLFU eviction policy. That should
allow the cache to make better predictions based on frequency and recency,
such as improved scan resistance. Full details are in the JIRA ticket. I
think it should be easy to port if there is interest.

Cheers,
Ben

[1]
https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/file/blockfile/cache/LruBlockCache.java
[2] https://issues.apache.org/jira/browse/HBASE-15560

Re: LruBlockCache alternative

[Josh Elser <jo...@gmail.com>]

Sean Busbey wrote:
> On Mon, Apr 4, 2016 at 9:45 AM, Keith Turner<ke...@deenlo.com>  wrote:
>> On Mon, Apr 4, 2016 at 10:20 AM, Josh Elser<jo...@gmail.com>  wrote:
>>
>>> Cool, thanks for the poke, Ben!
>>>
>>> Last I checked, our version of the LRUBlockCache was nearly identical to
>>> what was in HBase 1.x. I would imagine this would be easy to bring over.
>>>
>>> Maybe we can also try to swipe BucketCache while we're at it and get some
>>> off-heap support for blocks.
>>>
>>> Aside: it would be nice if we could somehow find a way to share code like
>>> this across the projects. HBase was interested in FATE for some time, but
>>> eventually created their own new solution. I'm not sure what else exists
>>> that we might want to share between projects.
>>
>>
>> One way to share code is to spin off new projects, like the following :
>>
>>   https://github.com/snazy/ohc
>>
>
> I'd much rather share code under ASF governance. Some code that
> started in Hadoop recently jumped over to Apache Commons, maybe that
> could serve as an example?
>
>
> https://s.apache.org/commons-crypto-vote
>
> https://github.com/intel-hadoop/chimera/blob/master/PROPOSAL.html
>
> https://s.apache.org/commons-crypto-vote-result
>
> Maybe we should start a new thread to go over things we'd like to start sharing?

Ooo, Apache Commons would be a nice home, IMO. Keeping it under the ASF 
governance guidelines would be a plus. I feel like I had a talk about 
this with Billie (and Stack, maybe?) a long time ago. Figuring out the 
proper scope for it will be tricky though. Maybe we should already have 
a list of initial things before taking it to HBase and others? (maybe 
Cassandra would have benefit, too?)

Re: LruBlockCache alternative

[Sean Busbey <bu...@cloudera.com>]

On Mon, Apr 4, 2016 at 9:45 AM, Keith Turner <ke...@deenlo.com> wrote:
> On Mon, Apr 4, 2016 at 10:20 AM, Josh Elser <jo...@gmail.com> wrote:
>
>> Cool, thanks for the poke, Ben!
>>
>> Last I checked, our version of the LRUBlockCache was nearly identical to
>> what was in HBase 1.x. I would imagine this would be easy to bring over.
>>
>> Maybe we can also try to swipe BucketCache while we're at it and get some
>> off-heap support for blocks.
>>
>> Aside: it would be nice if we could somehow find a way to share code like
>> this across the projects. HBase was interested in FATE for some time, but
>> eventually created their own new solution. I'm not sure what else exists
>> that we might want to share between projects.
>
>
>
> One way to share code is to spin off new projects, like the following :
>
>  https://github.com/snazy/ohc
>

I'd much rather share code under ASF governance. Some code that
started in Hadoop recently jumped over to Apache Commons, maybe that
could serve as an example?


https://s.apache.org/commons-crypto-vote

https://github.com/intel-hadoop/chimera/blob/master/PROPOSAL.html

https://s.apache.org/commons-crypto-vote-result

Maybe we should start a new thread to go over things we'd like to start sharing?

Re: LruBlockCache alternative

[Keith Turner <ke...@deenlo.com>]

On Mon, Apr 4, 2016 at 10:20 AM, Josh Elser <jo...@gmail.com> wrote:

> Cool, thanks for the poke, Ben!
>
> Last I checked, our version of the LRUBlockCache was nearly identical to
> what was in HBase 1.x. I would imagine this would be easy to bring over.
>
> Maybe we can also try to swipe BucketCache while we're at it and get some
> off-heap support for blocks.
>
> Aside: it would be nice if we could somehow find a way to share code like
> this across the projects. HBase was interested in FATE for some time, but
> eventually created their own new solution. I'm not sure what else exists
> that we might want to share between projects.



One way to share code is to spin off new projects, like the following :

 https://github.com/snazy/ohc



>
> dlmarion@comcast.net wrote:
>
>> Associated issue: https://issues.apache.org/jira/browse/ACCUMULO-4177
>>
>>
>> -----Original Message-----
>>> From: Christopher [mailto:ctubbsii@apache.org]
>>> Sent: Sunday, April 03, 2016 1:31 PM
>>> To: dev@accumulo.apache.org
>>> Subject: Re: LruBlockCache alternative
>>>
>>> Thanks for the pointer!
>>>
>>> On Sun, Apr 3, 2016, 12:08 Benjamin Manes<be...@gmail.com>  wrote:
>>>
>>> Hi,
>>>>
>>>> I noticed that Accumulo's LruBlockCache [1] appears to be based on
>>>>
>>> HBase's.
>>>
>>>> I currently have a patch being reviewed in HBASE-15560 [2] that
>>>> replaces the pseudo Segmented LRU with the TinyLFU eviction policy.
>>>> That should allow the cache to make better predictions based on
>>>> frequency and recency, such as improved scan resistance. Full details
>>>> are in the JIRA ticket. I think it should be easy to port if there is
>>>> interest.
>>>>
>>>> Cheers,
>>>> Ben
>>>>
>>>> [1]
>>>>
>>>>
>>>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/
>>>
>>>> apache/accumulo/core/file/blockfile/cache/LruBlockCache.java
>>>> [2] https://issues.apache.org/jira/browse/HBASE-15560
>>>>
>>>>
>>

Re: LruBlockCache alternative

[Josh Elser <jo...@gmail.com>]

Cool, thanks for the poke, Ben!

Last I checked, our version of the LRUBlockCache was nearly identical to 
what was in HBase 1.x. I would imagine this would be easy to bring over.

Maybe we can also try to swipe BucketCache while we're at it and get 
some off-heap support for blocks.

Aside: it would be nice if we could somehow find a way to share code 
like this across the projects. HBase was interested in FATE for some 
time, but eventually created their own new solution. I'm not sure what 
else exists that we might want to share between projects.

dlmarion@comcast.net wrote:
> Associated issue: https://issues.apache.org/jira/browse/ACCUMULO-4177
>
>
>> -----Original Message-----
>> From: Christopher [mailto:ctubbsii@apache.org]
>> Sent: Sunday, April 03, 2016 1:31 PM
>> To: dev@accumulo.apache.org
>> Subject: Re: LruBlockCache alternative
>>
>> Thanks for the pointer!
>>
>> On Sun, Apr 3, 2016, 12:08 Benjamin Manes<be...@gmail.com>  wrote:
>>
>>> Hi,
>>>
>>> I noticed that Accumulo's LruBlockCache [1] appears to be based on
>> HBase's.
>>> I currently have a patch being reviewed in HBASE-15560 [2] that
>>> replaces the pseudo Segmented LRU with the TinyLFU eviction policy.
>>> That should allow the cache to make better predictions based on
>>> frequency and recency, such as improved scan resistance. Full details
>>> are in the JIRA ticket. I think it should be easy to port if there is interest.
>>>
>>> Cheers,
>>> Ben
>>>
>>> [1]
>>>
>>>
>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/
>>> apache/accumulo/core/file/blockfile/cache/LruBlockCache.java
>>> [2] https://issues.apache.org/jira/browse/HBASE-15560
>>>
>

RE: LruBlockCache alternative

[dl...@comcast.net]

Associated issue: https://issues.apache.org/jira/browse/ACCUMULO-4177


> -----Original Message-----
> From: Christopher [mailto:ctubbsii@apache.org]
> Sent: Sunday, April 03, 2016 1:31 PM
> To: dev@accumulo.apache.org
> Subject: Re: LruBlockCache alternative
> 
> Thanks for the pointer!
> 
> On Sun, Apr 3, 2016, 12:08 Benjamin Manes <be...@gmail.com> wrote:
> 
> > Hi,
> >
> > I noticed that Accumulo's LruBlockCache [1] appears to be based on
> HBase's.
> > I currently have a patch being reviewed in HBASE-15560 [2] that
> > replaces the pseudo Segmented LRU with the TinyLFU eviction policy.
> > That should allow the cache to make better predictions based on
> > frequency and recency, such as improved scan resistance. Full details
> > are in the JIRA ticket. I think it should be easy to port if there is interest.
> >
> > Cheers,
> > Ben
> >
> > [1]
> >
> >
> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/
> > apache/accumulo/core/file/blockfile/cache/LruBlockCache.java
> > [2] https://issues.apache.org/jira/browse/HBASE-15560
> >

Re: LruBlockCache alternative

[Christopher <ct...@apache.org>]

Thanks for the pointer!

On Sun, Apr 3, 2016, 12:08 Benjamin Manes <be...@gmail.com> wrote:

> Hi,
>
> I noticed that Accumulo's LruBlockCache [1] appears to be based on HBase's.
> I currently have a patch being reviewed in HBASE-15560 [2] that replaces
> the pseudo Segmented LRU with the TinyLFU eviction policy. That should
> allow the cache to make better predictions based on frequency and recency,
> such as improved scan resistance. Full details are in the JIRA ticket. I
> think it should be easy to port if there is interest.
>
> Cheers,
> Ben
>
> [1]
>
> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/file/blockfile/cache/LruBlockCache.java
> [2] https://issues.apache.org/jira/browse/HBASE-15560
>