[GitHub] [airflow] ephraimbuddy commented on a change in pull request #7728: [AIRFLOW-5610] Add ability to specify multiple objects to copy in GCSToGCSOperator

[GitBox <gi...@apache.org>]

ephraimbuddy commented on a change in pull request #7728: [AIRFLOW-5610] Add ability to specify multiple objects to copy in GCSToGCSOperator
URL: https://github.com/apache/airflow/pull/7728#discussion_r392667594
 
 

 ##########
 File path: airflow/providers/google/cloud/operators/gcs_to_gcs.py
 ##########
 @@ -40,32 +37,27 @@ class GCSToGCSOperator(BaseOperator):
     :param source_bucket: The source Google Cloud Storage bucket where the
          object is. (templated)
     :type source_bucket: str
-    :param source_object: The source name of the object to copy in the Google cloud
+    :param source_objects: A list of prefix of the objects to copy in the Google cloud
         storage bucket. (templated)
-        You can use only one wildcard for objects (filenames) within your
-        bucket. The wildcard can appear inside the object name or at the
-        end of the object name. Appending a wildcard to the bucket name is
-        unsupported.
-    :type source_object: str
+    :type source_objects: List[str]
     :param destination_bucket: The destination Google Cloud Storage bucket
         where the object should be. If the destination_bucket is None, it defaults
         to source_bucket. (templated)
     :type destination_bucket: str
     :param destination_object: The destination name of the object in the
         destination Google Cloud Storage bucket. (templated)
-        If a wildcard is supplied in the source_object argument, this is the
-        prefix that will be prepended to the final destination objects' paths.
-        Note that the source path's part before the wildcard will be removed;
-        if it needs to be retained it should be appended to destination_object.
-        For example, with prefix ``foo/*`` and destination_object ``blah/``, the
-        file ``foo/baz`` will be copied to ``blah/baz``; to retain the prefix write
-        the destination_object as e.g. ``blah/foo``, in which case the copied file
-        will be named ``blah/foo/baz``.
+        If destination object is not specified, then it defaults to each of the source objects.
+        For example, if source_objects = ['foo/sales','bah/inventory'], then destination will be
+        'foo/sales' and 'bah/inventory' if destination_object is not specified.
     :type destination_object: str
     :param move_object: When move object is True, the object is moved instead
         of copied to the new location. This is the equivalent of a mv command
         as opposed to a cp command.
     :type move_object: bool
+    :type delimiter: str
 
 Review comment:
   Ok. Thanks very much for the review. 
   I'm thinking of removing it entirely and add back `google_cloud_storage_conn_id` which I removed. This is for backward compatibility. The arguments are already 10 so leaving the both arguments will make tests not to pass.
   If `delimiter` is removed, then search will now be based on source object wildcard.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services