[jira] [Commented] (AIRFLOW-1855) Add an Operator to copy files with a specific delimiter in a directory from one GCS bucket to another

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16275208#comment-16275208 ] 

ASF subversion and git services commented on AIRFLOW-1855:
----------------------------------------------------------

Commit 3e321790d537696b8fd1a97dcac2ab7c469fecea in incubator-airflow's branch refs/heads/master from [~kaxilnaik]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=3e32179 ]

[AIRFLOW-1855][AIRFLOW-1866] Add GCS Copy Operator to copy multiple files

Closes #2819 from kaxil/master


> Add an Operator to copy files with a specific delimiter in a directory from one GCS bucket to another
> -----------------------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-1855
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1855
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: contrib, gcp
>    Affects Versions: 1.10.0
>            Reporter: Kaxil Naik
>            Assignee: Kaxil Naik
>            Priority: Minor
>             Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Use case: Copy all the CSV/JSON files from a particular directory in a Bucket to another bucket and in a specific directory (or the same).
> Proposed Approach:
> - Add 'delimiter' argument in GCP hook to filter files with a particular delimiter.
> - Get the list of files to copy and filter it with delimiter using 'list' method in GCP hook
> - Use loop and 'copy' method in GCP hook. 
> Note: Under the hood GCS has no directories. Files are just objects.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)