You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Martijn Visser (Jira)" <ji...@apache.org> on 2023/12/01 15:01:00 UTC

[jira] [Reopened] (FLINK-32878) Add entropy to gcs path for better scalability

     [ https://issues.apache.org/jira/browse/FLINK-32878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martijn Visser reopened FLINK-32878:
------------------------------------

> Add entropy to gcs path for better scalability
> ----------------------------------------------
>
>                 Key: FLINK-32878
>                 URL: https://issues.apache.org/jira/browse/FLINK-32878
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / FileSystem
>    Affects Versions: 1.15.4, 1.16.2, 1.17.1
>            Reporter: Jayadeep Jayaraman
>            Assignee: Cheena
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.19.0
>
>
> Currently GCS is used as a backend for both checkpointing and sink. In both these cases the file names are sequential which causes hotspotting in GCS and results in HTTP 5XX status code.
>  
> As per [GCS best practices]([https://cloud.google.com/storage/docs/request-rate)] it is advisable to spread out the object creation with random names close to the beginning of the file name. 
>  
> There is similar work done already for S3 as part of FLINK-9061 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)