[jira] [Updated] (HUDI-2387) Too many HEAD requests from Hudi to S3

["Sourav T (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sourav T updated HUDI-2387:
---------------------------
    Description: 
We are using Apache Hudi from AWS Glue (with PySpark runtime) to store data on S3 bucket. We are observing a very high number of S3 HEAD requests originating from what we believe from Hudi. 

Many a time due to this high number of requests, S3 throws "Status Code: 503; Error Code: SlowDown" causing data losses. 

Is there any any out-of-box feature to debug this further to confirm which Hudi feature causing this? 

  was:
We are using Apache Hudi from AWS Glue (with PySpark runtime) to store data on S3 bucket. We are observing a very high number of S3 HEAD requests originating from what we believe from Hudi.

 

Is there any any out-of-box feature to debug this further to confirm which Hudi feature causing this? 


> Too many HEAD requests from Hudi to S3 
> ---------------------------------------
>
>                 Key: HUDI-2387
>                 URL: https://issues.apache.org/jira/browse/HUDI-2387
>             Project: Apache Hudi
>          Issue Type: Bug
>          Components: Common Core, Spark Integration
>    Affects Versions: 0.8.0
>         Environment: AWS Glue with PySpark
>            Reporter: Sourav T
>            Priority: Major
>
> We are using Apache Hudi from AWS Glue (with PySpark runtime) to store data on S3 bucket. We are observing a very high number of S3 HEAD requests originating from what we believe from Hudi. 
> Many a time due to this high number of requests, S3 throws "Status Code: 503; Error Code: SlowDown" causing data losses. 
> Is there any any out-of-box feature to debug this further to confirm which Hudi feature causing this? 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)