[jira] [Updated] (HUDI-2648) Retry FileSystem action when caught runtime exception

["sivabalan narayanan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

sivabalan narayanan updated HUDI-2648:
--------------------------------------
    Fix Version/s: 0.11.0

> Retry FileSystem action when caught runtime exception 
> ------------------------------------------------------
>
>                 Key: HUDI-2648
>                 URL: https://issues.apache.org/jira/browse/HUDI-2648
>             Project: Apache Hudi
>          Issue Type: Task
>            Reporter: Yue Zhang
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 0.11.0
>
>
> Hoodie will do lots of list/get/put/delete etc actions on filesystem.
>  Sometimes will meet the fileSystem performance issue or short service suspension, for example
> Using hoodie on S3, and will throw lots of exception like this 
> {code:java}
> com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 503, AWS Service: Amazon S3, AWS Request ID: J58R9W4E324DWJ1Y, AWS Error Code: SlowDown, AWS Error Message: Please reduce your request rate., S3 Extended Request ID: Pkd6VCJKTuq3NJ6vUQJB1TUznKGvAfjIZkmJOxt+j6oAD5roz+ojKf9rtcBOgUBw4sWeDwxdTv4=com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 503, AWS Service: Amazon S3, AWS Request ID: J58R9W4E324DWJ1Y, AWS Error Code: SlowDown, AWS Error Message: Please reduce your request rate., S3 Extended Request ID: Pkd6VCJKTuq3NJ6vUQJB1TUznKGvAfjIZkmJOxt+j6oAD5roz+ojKf9rtcBOgUBw4sWeDwxdTv4= at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393) at org.apache.hadoop.fs.s3a.S3AFileSystem.createEmptyObject(S3AFileSystem.java:1191) at org.apache.hadoop.fs.s3a.S3AFileSystem.createFakeDirectory(S3AFileSystem.java:1168) at org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:871) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1881) at org.apache.hudi.common.fs.HoodieWrapperFileSystem.lambda$mkdirs$25(HoodieWrapperFileSystem.java:632) at org.apache.hudi.common.fs.HoodieWrapperFileSystem.executeFuncWithTimeMetrics(HoodieWrapperFileSystem.java:100) at org.apache.hudi.common.fs.HoodieWrapperFileSystem.mkdirs(HoodieWrapperFileSystem.java:631) at org.apache.hudi.table.MarkerFiles.create(MarkerFiles.java:195) at org.apache.hudi.io.HoodieWriteHandle.createMarkerFile(HoodieWriteHandle.java:129) at org.apache.hudi.io.HoodieCreateHandle.lambda$new$1(HoodieCreateHandle.java:97) at org.apache.hudi.common.util.RetryHelper.start(RetryHelper.java:61) at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:100) at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:67) at org.apache.hudi.io.CreateHandleFactory.create(CreateHandleFactory.java:34) at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:83) at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:40) at org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37) at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
> {code}
> It's because of meet the throttle of S3 list/get/put/delete.
> This exception will cause app crashed, which can be solved by retry wrapper.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)