You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by "Chamikara Jayalath (JIRA)" <ji...@apache.org> on 2018/01/05 01:38:00 UTC

[jira] [Created] (BEAM-3413) Support copying GCs objects using rewrite operation

Chamikara Jayalath created BEAM-3413:
----------------------------------------

             Summary: Support copying GCs objects using rewrite operation
                 Key: BEAM-3413
                 URL: https://issues.apache.org/jira/browse/BEAM-3413
             Project: Beam
          Issue Type: Improvement
          Components: sdk-java-gcp
            Reporter: Chamikara Jayalath


Currently Java GCS FileSystem users GCS copy() operation to copy objects.
https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java#L744

According to GCS website this can result in HTTP 413 errors if copying takes more than 30 seconds: https://cloud.google.com/storage/docs/json_api/v1/status-codes

The solution suggested there is to use the GCS rewrite() method.

We should add support for this.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)