[jira] [Comment Edited] (HBASE-27795) Define RPC API for cache cleaning

["Wellington Chevreuil (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-27795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713627#comment-17713627 ] 

Wellington Chevreuil edited comment on HBASE-27795 at 4/18/23 3:27 PM:
-----------------------------------------------------------------------

Hello [~bbeaudreault], yes sir, contributions/new ideas are very welcome! So let me explain a bit where we are with this. 

Context: We try to cache as much of our data from object store into local file based bucket cache. Once we have the cache warmed up, we don't want to unnecessary have to cache it again. Even if a RS restarts (gracefully or crash), we want it to be able to recover its cache from before, and avoid having to go back to read those blocks from the object store. That's why we did HBASE-27750/HBASE-27551 (we also have HBASE-27389 ongoing).

Problems: 
1) CACHE RECOVERY: We already had a map of the files->blocks persisted into a local file, however we were always prefetching all the files again on region opening. HBASE-27313/HBASE-27686 implemented the functionality to save a list of files whose blocks have been cached, so that we don't need to cache those again. Then the next problem was to make sure we don't evict blocks for the hosted regions.

2) EVICT WISELY: We don't want to evict blocks for files that are still relevant (not compacted away and from regions the given RS is hosting). For that, we needed to turn hbase.rs.evictblocksonclose from default of false, to true. Now the problem is what to do with regions moves?

3) AVOID MOVING REGIONS: RSes restarts take longer than AM does to triggers SCPs and reassign regions from the crash/restarted RSes on the remaining online RSes. This nullifies the feature implemented in #1. So we thought to compromise on region availability by introducing a delay on SCPs to wait for the original server to come online (HBASE-27551). Delaying reassignment is not only sub-optimal, but there's still the problem of regions moving due to region balancer. We have to disable the balancer, until we can get it also taking the cache into consideration (HBASE-27389 ongoing). But regions may still move, say if the delay threshold for reassignment is expired, or if new RSes are added and regions are moved manually, or with the balancer after HBASE-27389 , now the cache will have blocks for regions the RS no longer hosts.

4) EVICT IF REGION MOVE, KEEP IF RS RESTART: That's what we want to do in HBASE-27752. We know when region is closed because of a RS restart. We could change CloseRegionHandler so that when the prefetch persistence is enabled, we never evict. Then we could leave hbase.rs.evictblocksonclose set to true globally, so that blocks would get evicted on regions moves triggered by the balancer or manually. But crashes may still be a problem, if the crashed server comes back after the assignment delay. That's what motivated the cache cleaning feature described in this jira. And yeah, I guess the cleaning logic might well be invoked by some scheduled chore as well, not only manually via API.


was (Author: wchevreuil):
Hello [~bbeaudreault], yes sir, contributions/new ideas are very welcome! So let me explain a bit where we are with this. 

Context: We try to cache as much of our data from object store into local file based bucket cache. Once we have the cache warmed up, we don't want to unnecessary have to cache it again. Even if a RS restarts (gracefully or crash), we want it to be able to recover its cache from before, and avoid having to go back to read those blocks from the object store. That's why we did HBASE-27750/HBASE-27551 (we also have HBASE-27389 ongoing).

Problems: 
1) CACHE RECOVERY: We already had a map of the files->blocks persisted into a local file, however we were always prefetching all the files again on region opening. HBASE-27313/HBASE-27686 implemented the functionality to save a list of files whose blocks have been cached, so that we don't need to cache those again. Then the next problem was to make sure we don't evict blocks for the hosted regions.
2) EVICT WISELY: We don't want to evict blocks for files that are still relevant (not compacted away and from regions the given RS is hosting). For that, we needed to turn hbase.rs.evictblocksonclose from default of false, to true. Now the problem is what to do with regions moves?
3) AVOID MOVING REGIONS: RSes restarts take longer than AM does to triggers SCPs and reassign regions from the crash/restarted RSes on the remaining online RSes. This nullifies the feature implemented in #1. So we thought to compromise on region availability by introducing a delay on SCPs to wait for the original server to come online (HBASE-27551). Delaying reassignment is not only sub-optimal, but there's still the problem of regions moving due to region balancer. We have to disable the balancer, until we can get it also taking the cache into consideration (HBASE-27389 ongoing). But regions may still move, say if the delay threshold for reassignment is expired, or if new RSes are added and regions are moved manually, or with the balancer after HBASE-27389 , now the cache will have blocks for regions the RS no longer hosts.
4) EVICT IF REGION MOVE, KEEP IF RS RESTART: That's what we want to do in HBASE-27752. We know when region is closed because of a RS restart. We could change CloseRegionHandler so that when the prefetch persistence is enabled, we never evict. Then we could leave hbase.rs.evictblocksonclose set to true globally, so that blocks would get evicted on regions moves triggered by the balancer or manually. But crashes may still be a problem, if the crashed server comes back after the assignment delay. That's what motivated the cache cleaning feature described in this jira. And yeah, I guess the cleaning logic might well be invoked by some scheduled chore as well, not only manually via API.

> Define RPC API for cache cleaning
> ---------------------------------
>
>                 Key: HBASE-27795
>                 URL: https://issues.apache.org/jira/browse/HBASE-27795
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Wellington Chevreuil
>            Priority: Major
>
> We should add an RPC API to allow for a "limited manual" cache cleaning. If hbase.rs.evictblocksonclose is set to false, blocks may hang in the cache upon regions move between RSes. 
> The method at the RS level, should compare the files from its online regions against the files in the prefetch list file, evicting blocks from the files in the prefetch list file that are not in any of the online regions for the given RS.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)