[jira] [Commented] (KUDU-2718) master_failover-itest when HMS is enabled is flaky

["Hao Hao (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16810218#comment-16810218 ] 

Hao Hao commented on KUDU-2718:
-------------------------------

DropTable in HMS is a synchronous call, so I think it should be reflected immediately if we succeeded in dropping the table. It may be DropTable hasn't taken place before CreateTable being retried? But I don't know how DropTable in HMS can take up to ~2 mins.

I also looped the [test 2000 times|http://dist-test.cloudera.org/job?job_id=hao.hao.1554350086.94333], but failed to reproduce the reported error here. Instead I encountered error as {noformat}/data/1/hao/kudu/src/kudu/integration-tests/master_failover-itest.cc:460: Failure
Failed
Bad status: Invalid argument: Error creating table default.table_0 on the master: not enough live tablet servers to create a table with the requested replication factor 3; 2 tablet servers are alive{noformat}
which seems to be the issue described in KUDU-1358. Without the fix for KUDU-1358, to deflake we can retry upon such error.

> master_failover-itest when HMS is enabled is flaky
> --------------------------------------------------
>
>                 Key: KUDU-2718
>                 URL: https://issues.apache.org/jira/browse/KUDU-2718
>             Project: Kudu
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 1.9.0
>            Reporter: Adar Dembo
>            Assignee: Hao Hao
>            Priority: Major
>         Attachments: master_failover-itest.1.txt
>
>
> This was a failure in HmsConfigurations/MasterFailoverTest.TestDeleteTableSync/1, where GetParam() = 2, but it's likely possible in every multi-master test with HMS integration enabled.
> It looks like there was a leader master election at the time that the client tried to create the table being tested. The master managed to create the table in HMS, but then there was a failure replicating in Raft because another master was elected leader. So the client retried the request on a different master, but the HMS piece of CreateTable failed because the HMS already knew about the table.
> Thing is, there's code to roll back the HMS table creation if this happens, so I don't see why the retried CreateTable failed at the HMS with "table already exists". Perhaps this is a case where even though we succeeded in dropping the table from HMS, it doesn't reflect that immediately?
> I'm attaching the full log.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)