[jira] [Resolved] (AIRFLOW-2862) S3ToRedshiftTransfer Copy Command Flexibility

["Ash Berlin-Taylor (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ash Berlin-Taylor resolved AIRFLOW-2862.
----------------------------------------
    Resolution: Duplicate

> S3ToRedshiftTransfer Copy Command Flexibility
> ---------------------------------------------
>
>                 Key: AIRFLOW-2862
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2862
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: operators
>            Reporter: Micheal Ascah
>            Assignee: Micheal Ascah
>            Priority: Minor
>
> Currently, the S3ToRedshiftTransfer class requires that the target table to be loaded is suffixed to the end of the S3 key provided.
> It doesn't seem justifiable that the operator should require the file be named by any convention. The S3 bucket + S3 key should be all that is needed. This makes it possible to load any S3 Key into a Redshift table, rather than only files that have the table name at the end of the S3 key.
> The S3 key parameter should also be template-able so that files created in S3 using timestamps from macros in other tasks in the current DAG run can be used to identify files when loading from S3 to Redshift.
> The command template should change from 
> {code:java}
> COPY {schema}.{table}
>  FROM 's3://{s3_bucket}/{s3_key}/{table}'
>  with credentials
>  'aws_access_key_id={access_key};aws_secret_access_key={secret_key}'
>  {copy_options};{code}
>  To
>  
> {code:java}
> COPY {schema}.{table}
>  FROM 's3://{s3_bucket}/{s3_key}'
>  with credentials
>  'aws_access_key_id={access_key};aws_secret_access_key={secret_key}'
>  {copy_options};
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)