[jira] [Commented] (KUDU-3344) Master could do some garbage collection work in CatalogManagerBgTasks thread

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17495706#comment-17495706 ] 

ASF subversion and git services commented on KUDU-3344:
-------------------------------------------------------

Commit 2de290d1b342696cd22c0a800aee5b1bf684a0fa in kudu's branch refs/heads/master from Zhang Yifan
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=2de290d ]

KUDU-3344: catalog manager clean up metadata for deleted tables/tablets

Kudu masters now retain metadata for deleted tables and tablets forever, and
the leader master loads all of them into memory when starts. If we have a lot
of tables and tablets in a cluster, memory usage of the leader master will be
large and it will take a long time to start the leader master. Consider that
in many cases users drop tables and partitions, useless metadata should be
cleaned up in backgroud tasks.

But it's hard to decide when we should clean them up, because the deletion of
tablet replicas is asynchronous. If metadata is deleted before the tablet data
is deleted, the unknown tablet reported by a tablet server will not be processed
by catalog manager and we must delete it manually. So this patch add a new flag
'deleted_table_and_tablet_reserved_secs', its default value is the same as
'unresponsive_ts_rpc_timeout_ms', we can roughly assume that after this amount
of time the tablet data will be actually deleted and it's safe to delete its
metadata entry from sys.catalog table and in-memory map.

Change-Id: Idefa2ee2f5108ba913fe0057a4061c3c28351547
Reviewed-on: http://gerrit.cloudera.org:8080/18152
Tested-by: Alexey Serbin <as...@cloudera.com>
Reviewed-by: Alexey Serbin <as...@cloudera.com>


> Master could do some garbage collection work in CatalogManagerBgTasks thread
> ----------------------------------------------------------------------------
>
>                 Key: KUDU-3344
>                 URL: https://issues.apache.org/jira/browse/KUDU-3344
>             Project: Kudu
>          Issue Type: Improvement
>          Components: master
>            Reporter: YifanZhang
>            Priority: Minor
>
> Kudu master now reserve all tables/tablets' metadata in memory and disks, deleted tables and tablets were marked REMOVED/DELETED/REPLACED state but not really deleted. This could lead to huge memory usage as described in KUDU-3097.  
> I think it's a good idea to cleanup them in the CatalogManagerBgTasks thread. But because the data deletion tasks are done asynchronously by tablet servers, it is uncertain when metadata can be safely deleted.
> Besides, we could also cleanup dead tablet servers from master's in-memory map in this thread, as I mentioned in KUDU-2915.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)