You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/12 07:07:58 UTC
[GitHub] [airflow] dewaldabrie opened a new issue #10298: GCSToS3Operator (and S3Hook) does not allow passing in of custom metadata
dewaldabrie opened a new issue #10298:
URL: https://github.com/apache/airflow/issues/10298
This is related to #8304 which as closed due to becoming stale.
[GCSToS3Operator](https://github.com/apache/airflow/blob/f618cdd19a5cdf09262436978817934d27668997/airflow/providers/amazon/aws/transfers/gcs_to_s3.py#L30) uses the S3Hook to upload files.
The S3Hook uses boto3, which supports attaching metadata (headers) to files via the [`ExtraArgs` parameter](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-uploading-files.html#the-extraargs-parameter). The `ExtraArgs` parameter is already being used in the `S3Hook`'s `upload_fileobj` method, but it does not allow the user to pass in custom metadata.
My proposition is to add the optional params to `GCSToS3Operator` and `S3Hook` to allow the use to pass in custom metadata.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #10298: GCSToS3Operator (and S3Hook) does not allow passing in of custom metadata
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #10298:
URL: https://github.com/apache/airflow/issues/10298
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rootcss commented on issue #10298: GCSToS3Operator (and S3Hook) does not allow passing in of custom metadata
Posted by GitBox <gi...@apache.org>.
rootcss commented on issue #10298:
URL: https://github.com/apache/airflow/issues/10298#issuecomment-694396501
@kaxil please assign this to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org