[kudu] 02/04: [java] Deflake TestAuthTokenReacquire.testBasicMasterOperations

[mp...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

mpercy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git

commit 5533478b0b50bddd648334e69bad35cda321e9da
Author: Will Berkeley <wd...@gmail.com>
AuthorDate: Wed May 29 16:20:39 2019 -0700

    [java] Deflake TestAuthTokenReacquire.testBasicMasterOperations
    
    This test was flaky because DDL operations aren't exactly-once
    (KUDU-1527). The following could happen with table creation or deletion:
    1. Thread A issues a CreateTable request for Table A.
    2. The master receives the CreateTable request and gets to work creating
       Table A but has not responded to Thread A yet.
    3. Thread B calls `dropConnectionsAndExpireTokens`, dropping connections
       to the master.
    4. Thread A retries the CreateTable RPC, thinking it has failed because
       the connection was dropped.
    5. The master responds with an error because the table exists.
    
    This works around the problem by catching and ignoring exceptions
    indicating that the table already exists in the case of table creation
    and catching and ignoring exceptions that the table is deleted or does
    not exist in the case of table deletion.
    
    Before this patch, I saw 16/100 runs fail in dist-test against TSAN
    binaries. With this patch, I saw 0/100 failures.
    
    Change-Id: Ifef0a254c6eb1cadeb54e9cef52f78c7ccc7b9c5
    Reviewed-on: http://gerrit.cloudera.org:8080/13461
    Tested-by: Kudu Jenkins
    Reviewed-by: Alexey Serbin <as...@cloudera.com>
---
 .../apache/kudu/client/TestAuthTokenReacquire.java | 24 ++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthTokenReacquire.java b/java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthTokenReacquire.java
index c888bef..f58bfd7 100644
--- a/java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthTokenReacquire.java
+++ b/java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthTokenReacquire.java
@@ -125,14 +125,34 @@ public class TestAuthTokenReacquire {
             assertTrue(tableList.getTablesList().isEmpty());
             dropConnectionsAndExpireTokens();
 
-            client.createTable(tableName, basicSchema, getBasicCreateTableOptions());
+            try {
+              client.createTable(tableName, basicSchema, getBasicCreateTableOptions());
+            } catch (KuduException ex) {
+              // Swallow "table already exists" exceptions. This might happen if the thread sends
+              // a CreateTable request, the master receives it, but the connection is dropped by
+              // another thread before the client receives a success response, and then the client
+              // retries.
+              // TODO(KUDU-1537): Remove this workaround when table creation is exactly-once.
+              Status exStatus = ex.getStatus();
+              if (!exStatus.isAlreadyPresent() && !exStatus.isServiceUnavailable()) {
+                throw ex;
+              }
+            }
             dropConnectionsAndExpireTokens();
 
             KuduTable table = client.openTable(tableName);
             assertEquals(basicSchema.getColumnCount(), table.getSchema().getColumnCount());
             dropConnectionsAndExpireTokens();
 
-            client.deleteTable(tableName);
+            try {
+              client.deleteTable(tableName);
+            } catch (KuduException ex) {
+              // See the above comment about table creation. The same idea applies to table deletion.
+              // TODO(KUDU-1537): Remove this workaround when table deletion is exactly-once.
+              if (!ex.getStatus().isNotFound()) {
+                throw ex;
+              }
+            }
             assertFalse(client.tableExists(tableName));
           } catch (Throwable e) {
             //noinspection ThrowableResultOfMethodCallIgnored