[jira] [Commented] (KUDU-2963) Catalog manager never gives up on CreateTablet RPCs

["Bankim Bhavsar (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961550#comment-16961550 ] 

Bankim Bhavsar commented on KUDU-2963:
--------------------------------------

{{AsyncCreateReplica}} is the class that sends CreateTablet request and retries in case of failure.

Constructor of {{AsyncCreateReplica}} sets a default deadline of 30 secs which is less than the default deadline of 1hr set by the base class constructor {{RetryingTSRpcTask}}.
https://github.com/apache/kudu/blob/master/src/kudu/master/catalog_manager.cc#L3390

{{FLAGS_tablet_creation_timeout_ms}} is used in 2 places:
 - while setting deadline for the {{AsyncCreateReplica}}
 - issuing DeleteTablet RPCs in case of failure to CreateTablet after the timeout of FLAGS_tablet_creation_timeout_ms

The deadline is honoured and no additional RPC requests are scheduled on crossing the deadline.
https://github.com/apache/kudu/blob/master/src/kudu/master/catalog_manager.cc#L3276
https://github.com/apache/kudu/blob/master/src/kudu/master/catalog_manager.cc#L3287

I was able to verify this by tweaking {{CreateTableITest::TestCreateWhenMajorityOfReplicasFailCreation}}.
Verified that CreateTablet RCPs time out and no additional ones are issued to the tablet servers.

Next I can work on a unit test that proves that CreateTable RPCs are not retried indefinitely.

----

On other note, I noticed a bug in the {{CreateTableITest::TestCreateWhenMajorityOfReplicasFailCreation}} test that would have been the cause of flakiness fixed by this [commit|https://github.com/apache/kudu/commit/d119d529beb691a84134d02e33ecdce6102a7a35]

https://github.com/apache/kudu/blob/master/src/kudu/integration-tests/create-table-itest.cc#L133-L134

> Catalog manager never gives up on CreateTablet RPCs
> ---------------------------------------------------
>
>                 Key: KUDU-2963
>                 URL: https://issues.apache.org/jira/browse/KUDU-2963
>             Project: Kudu
>          Issue Type: Improvement
>          Components: master
>    Affects Versions: 1.11.0
>            Reporter: Adar Dembo
>            Assignee: Bankim Bhavsar
>            Priority: Major
>              Labels: newbie
>
> This is a problem when there aren't enough live tservers upon which to place a tablet's replicas, or when a chosen tserver doesn't create the replica quickly enough. If the catalog manager decides to replace the tablet, the replaced tablet's CreateTablet RPCs continue to retry ad infinitum. If the previously dead tservers then come back to life, they must needlessly process the CreateTablet RPCs.
> The tablets are eventually deleted, either through explicit DeleteTablet RPCs (triggered by the catalog manager replacement process), or by heartbeating, but it's an unnecessary drain on cluster resources.
> We should probably abort CreateTablet RPCs for tablets that have been removed from their table.
> CreateTableITest_TestCreateWhenMajorityOfReplicasFailCreation demonstrates this acutely.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)