[jira] [Work logged] (BEAM-8889) Make GcsUtil use GoogleCloudStorage

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-8889?focusedWorklogId=617503&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617503 ]

ASF GitHub Bot logged work on BEAM-8889:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 01/Jul/21 10:59
            Start Date: 01/Jul/21 10:59
    Worklog Time Spent: 10m 
      Work Description: mprashanthsagar commented on a change in pull request #14817:
URL: https://github.com/apache/beam/pull/14817#discussion_r662169853



##########
File path: sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilTest.java
##########
@@ -772,7 +773,11 @@ public void testGCSChannelCloseIdempotent() throws IOException {
         GoogleCloudStorageReadOptions.builder().setFastFailOnNotFound(false).build();
     SeekableByteChannel channel =
         new GoogleCloudStorageReadChannel(
-            null, "dummybucket", "dummyobject", null, new ClientRequestHelper<>(), readOptions);
+            null,
+            new StorageResourceId("dummybucket", "dummyobject"),

Review comment:
       can we maintain `testBucket` and `testObject` for consistency ?

##########
File path: sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilTest.java
##########
@@ -772,7 +773,11 @@ public void testGCSChannelCloseIdempotent() throws IOException {
         GoogleCloudStorageReadOptions.builder().setFastFailOnNotFound(false).build();
     SeekableByteChannel channel =
         new GoogleCloudStorageReadChannel(

Review comment:
       nit : We could remove the dependency on `GoogleCloudStorageReadChannel` and use the higher level of abstraction of `GCSImpl`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

            Worklog Id:     (was: 617503)
    Remaining Estimate: 127h  (was: 127h 10m)
            Time Spent: 41h  (was: 40h 50m)

> Make GcsUtil use GoogleCloudStorage
> -----------------------------------
>
>                 Key: BEAM-8889
>                 URL: https://issues.apache.org/jira/browse/BEAM-8889
>             Project: Beam
>          Issue Type: Improvement
>          Components: io-java-gcp
>    Affects Versions: 2.16.0
>            Reporter: Esun Kim
>            Assignee: VASU NORI
>            Priority: P2
>              Labels: gcs
>             Fix For: 2.22.0
>
>   Original Estimate: 168h
>          Time Spent: 41h
>  Remaining Estimate: 127h
>
> [GcsUtil|https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java] is a primary class to access Google Cloud Storage on Apache Beam. Current implementation directly creates GoogleCloudStorageReadChannel and GoogleCloudStorageWriteChannel by itself to read and write GCS data rather than using [GoogleCloudStorage|https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorage.java] which is an abstract class providing basic IO capability which eventually creates channel objects. This request is about updating GcsUtil to use GoogleCloudStorage to create read and write channel, which is expected flexible because it can easily pick up the new change; e.g. new channel implementation using new protocol without code change.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)