[jira] [Updated] (NIFI-10112) Error to slowdown for ListS3 and FetchS3

["Navneet Kaur (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-10112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Navneet Kaur updated NIFI-10112:
--------------------------------
    Issue Type: Improvement  (was: Wish)

> Error to slowdown for ListS3 and FetchS3
> ----------------------------------------
>
>                 Key: NIFI-10112
>                 URL: https://issues.apache.org/jira/browse/NIFI-10112
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>    Affects Versions: 1.16.1
>         Environment: Linux
>            Reporter: Navneet Kaur
>            Priority: Minor
>
> While listing and fetching objects from S3, and have received this error frequently:
> {color:#ff0000}{{ListS3 Failed to list contents of bucket due to Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: Slowdown;.....)}}{color}
> {color:#ff0000}{{FetchS3 Failed to retrieve S3 object for FlowFile ...; routing to failure: com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: Slowdown;.....)}}{color}
>  
> {color:#172b4d}Scenario: We were listing and fetching many files (1000+) on the buckets which are unique on the path, and was getting this error as transactions per second increased by 5.5k (I couldn't test if it actually increased by this number). Solution to this was setting "Run Schedule" time for ListS3 (5 mins) and FetchS3 (15sec).{color}
> {color:#172b4d}[https://aws.amazon.com/premiumsupport/knowledge-center/s3-resolve-503-slowdown-throttling/]{color}
> {color:#172b4d}
> This solution didn't solve the problem throughly, as I still get this error but sparsely.{color}
>  
> {color:#172b4d}Looking forward to get this handled within the processors so that the transactions per second don't increase beyond the threshold.{color}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)