[jira] [Commented] (HBASE-27587) L1 cache leaks index blocks over time when under subscribed

["Bryan Beaudreault (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-27587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17680401#comment-17680401 ] 

Bryan Beaudreault commented on HBASE-27587:
-------------------------------------------

I was wrong about suspecting compactions above. Actually, this growth is due to regions moving. We do not have hbase.rs.evictblocksonclose enabled, so blocks are not evicted when regions move off. When other regions move on, their INDEX blocks pretty immediately get cached.

I have to think about whether we'd want to enable hbase.rs.evictblocksonclose. At first glance, I think that'd be painful for temporary region displacements. But I actually don't think that happens too much.

I sort of wonder if it'd make sense to have a separate value for evictblocksonclose for L1 vs L2.

> L1 cache leaks index blocks over time when under subscribed
> -----------------------------------------------------------
>
>                 Key: HBASE-27587
>                 URL: https://issues.apache.org/jira/browse/HBASE-27587
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Bryan Beaudreault
>            Priority: Major
>
> Let's say you have CombinedBlockCache enabled. DATA goes to L2, INDEX/BLOOM go to L1.  Your regionserver has index size of 2gb and bloom size of 1gb. So you really only need around 3gb of L1 to fully hold all of the "L1 candidates".
> When data set does not fit into cache, LRU will handle evictions to stay under max. But in the above scenario, if you configure 6gb for L1 (3 more than needed) over time you will end up filling that entire 6gb with old INDEX blocks. Once you reach max, LRU will handle evicting out the oldest ones.
> Since the leak is contained to the configured max L1 size, this isn't a huge issue but it results in heap waste. Under high heap allocations, if you haven't left enough buffer outside memstore, L1, etc, you will start seeing GC pressure. This L1 leak then becomes a little more problematic, because you end up in a circumstance where longer lived regionservers (who've leaked closer to the max L1 size) have less extra buffer available than more newly restarted regionservers.
> The best fix is to appropriately set your L1 size so there is not a lot of excess, but this can be painful to maintain over time as clusters shrink, grow, or data shape changes. It'd be a lot better if the L1 did not leak so you don't have to so finely tune the L1.
> I haven't fully figured out where the leak comes from, but I think it's related to compactions. Perhaps the INDEX blocks are not being evicted as hfiles are compacted away. The leak is very linear over time in our experience.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)