[jira] [Created] (HUDI-3067) "Table already exists" error with multiple writers

["Nikita Sheremet (Jira)" <ji...@apache.org>]

Nikita Sheremet created HUDI-3067:
-------------------------------------

             Summary: "Table already exists" error with multiple writers
                 Key: HUDI-3067
                 URL: https://issues.apache.org/jira/browse/HUDI-3067
             Project: Apache Hudi
          Issue Type: Bug
            Reporter: Nikita Sheremet


How reproduce:
 # Set up multiple writing [https://hudi.apache.org/docs/concurrency_control/] for dynamodb (do not forget to set _hoodie.write.lock.dynamodb.region_ and 
{_}hoodie.write.lock.dynamodb.billing_mode{_}). Do not create anty dynamodb table.
 # Run multiple writers to the table

(Tested on aws EMR, so multiple writers is EMR steps)

Expected result - all steps completed.

Actual result: some steps failed with exception 
{code:java}
Caused by: com.amazonaws.services.dynamodbv2.model.ResourceInUseException: Table already exists: truedata_detections (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ResourceInUseException; Request ID:; Proxy: null)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:6214)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:6181)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeCreateTable(AmazonDynamoDBClient.java:1160)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.createTable(AmazonDynamoDBClient.java:1124)
	at org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.createLockTableInDynamoDB(DynamoDBBasedLockProvider.java:188)
	at org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:99)
	at org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:77)
	... 54 more
21/12/19 13:42:06 INFO Yar {code}
This happens because all steps tried to create table at the same time.

 

Suggested solution:

A catch statment for _Table already exists_ exception should be added into dynamodb table creation code. May be with delay and additional check that table is present.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)