[kudu-CR] KUDU-3344: catalog manager clean up metadata for deleted tables/tablets

["Yifan Zhang (Code Review)" <ge...@cloudera.org>]

Hello Kudu Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18152

to look at the new patch set (#4).

Change subject: KUDU-3344: catalog manager clean up metadata for deleted tables/tablets
......................................................................

KUDU-3344: catalog manager clean up metadata for deleted tables/tablets

Kudu masters now retain metadata for deleted tables and tablets forever, and
the leader master loads all of them into memory when starts. If we have a lot
of tables and tablets in a cluster, memory usage of the leader master will be
large and it will take a long time to start the leader master. Consider that
in many cases users drop tables and partitions, useless metadata should be
cleaned up in backgroud tasks.

But it's hard to decide when we should clean them up, because the deletion of
tablet replicas is asynchronous. If metadata is deleted before the tablet data
is deleted, the unknown tablet reported by a tablet server will not be processed
by catalog manager and we must delete it manually. So this patch add a new flag
'deleted_table_and_tablet_reserved_secs', its default value is the same as
'unresponsive_ts_rpc_timeout_ms', we can roughly assume that after this amount
of time the tablet data will be actually deleted and it's safe to delete its
metadata entry from sys.catalog table and in-memory map.

Change-Id: Idefa2ee2f5108ba913fe0057a4061c3c28351547
---
M src/kudu/client/client.cc
M src/kudu/integration-tests/master-stress-test.cc
M src/kudu/integration-tests/ts_tablet_manager-itest.cc
M src/kudu/master/auto_rebalancer.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master-test.cc
M src/kudu/master/master.proto
8 files changed, 260 insertions(+), 24 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/52/18152/4
-- 
To view, visit http://gerrit.cloudera.org:8080/18152
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idefa2ee2f5108ba913fe0057a4061c3c28351547
Gerrit-Change-Number: 18152
Gerrit-PatchSet: 4
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Kudu Jenkins (120)