You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Chris Riccomini (JIRA)" <ji...@apache.org> on 2017/12/02 00:02:23 UTC
[jira] [Resolved] (AIRFLOW-1855) Add an Operator to copy files with
a specific delimiter in a directory from one GCS bucket to another
[ https://issues.apache.org/jira/browse/AIRFLOW-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Riccomini resolved AIRFLOW-1855.
--------------------------------------
Resolution: Fixed
> Add an Operator to copy files with a specific delimiter in a directory from one GCS bucket to another
> -----------------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-1855
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1855
> Project: Apache Airflow
> Issue Type: New Feature
> Components: contrib, gcp
> Affects Versions: 1.10.0
> Reporter: Kaxil Naik
> Assignee: Kaxil Naik
> Priority: Minor
> Fix For: 1.10.0
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Use case: Copy all the CSV/JSON files from a particular directory in a Bucket to another bucket and in a specific directory (or the same).
> Proposed Approach:
> - Add 'delimiter' argument in GCP hook to filter files with a particular delimiter.
> - Get the list of files to copy and filter it with delimiter using 'list' method in GCP hook
> - Use loop and 'copy' method in GCP hook.
> Note: Under the hood GCS has no directories. Files are just objects.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)