You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Kaxil Naik (JIRA)" <ji...@apache.org> on 2017/11/28 10:36:00 UTC
[jira] [Created] (AIRFLOW-1855) Add an Operator to copy files with
a specific delimiter in a directory from one GCS bucket to another
Kaxil Naik created AIRFLOW-1855:
-----------------------------------
Summary: Add an Operator to copy files with a specific delimiter in a directory from one GCS bucket to another
Key: AIRFLOW-1855
URL: https://issues.apache.org/jira/browse/AIRFLOW-1855
Project: Apache Airflow
Issue Type: New Feature
Components: contrib, gcp
Affects Versions: 1.10.0
Reporter: Kaxil Naik
Assignee: Kaxil Naik
Priority: Minor
Fix For: 1.10.0
Use case: Copy all the CSV/JSON files from a particular directory in a Bucket to another bucket and in a specific directory (or the same).
Proposed Approach:
- Add 'delimiter' argument in GCP hook to filter files with a particular delimiter.
- Get the list of files to copy and filter it with delimiter using 'list' method in GCP hook
- Use loop and 'copy' method in GCP hook.
Note: Under the hood GCS has no directories. Files are just objects.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)