[jira] [Created] (BEAM-12740) Reduce and backoff GCS metadata operations when writing to GCS files

["Sam Whittle (Jira)" <ji...@apache.org>]

Sam Whittle created BEAM-12740:
----------------------------------

             Summary: Reduce and backoff GCS metadata operations when writing to GCS files
                 Key: BEAM-12740
                 URL: https://issues.apache.org/jira/browse/BEAM-12740
             Project: Beam
          Issue Type: Bug
          Components: io-java-gcp
            Reporter: Sam Whittle
            Assignee: Sam Whittle


When issuing GCS operations affecting metadata (ie file-level operations not read/write operations), GCS may return errors indicating backoff. See
https://cloud.google.com/storage/docs/request-rate#ramp-up

If such errors are encountered, currently the exception is not handled by GcsUtil.java and is propagated, causing retries and backoff of all operations at a higher level.  Instead we should backoff and retry only such files that require it.

Additionally FileBasedSink issues deletes for files that have been renamed.  The rename itself should take care of removing the original file and thus we can reduce some metadata operations.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)