You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/19 16:25:17 UTC
[GitHub] [airflow] moreaupascal56 opened a new issue, #25833: Airflow Amazon provider S3Hook().download_file() fail when needs encryption arguments (SSECustomerKey etc..)
moreaupascal56 opened a new issue, #25833:
URL: https://github.com/apache/airflow/issues/25833
### Apache Airflow version
2.3.3
### What happened
Bug when trying to use the S3Hook to download a file from S3 with extra parameters for security like an SSECustomerKey.
The function [download_file](https://github.com/apache/airflow/blob/dd72e67524c99e34ba4c62bfb554e4caf877d5ec/airflow/providers/amazon/aws/hooks/s3.py#L854) fetches the `extra_args` from `self` where we can specify the security parameters about encryption as a `dict`.
But [download_file](https://github.com/apache/airflow/blob/dd72e67524c99e34ba4c62bfb554e4caf877d5ec/airflow/providers/amazon/aws/hooks/s3.py#L854) is calling [get_key()](https://github.com/apache/airflow/blob/dd72e67524c99e34ba4c62bfb554e4caf877d5ec/airflow/providers/amazon/aws/hooks/s3.py#L870) which does not use these `extra_args` when calling the [load() method here](https://github.com/apache/airflow/blob/dd72e67524c99e34ba4c62bfb554e4caf877d5ec/airflow/providers/amazon/aws/hooks/s3.py#L472), this results in a `botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request.` error.
This could be fixed like this:
load as says [boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Object.load) is calling [S3.Client.head_object()](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.head_object) which can handle **kwargs and can have all the arguments below:
```
response = client.head_object(
Bucket='string',
IfMatch='string',
IfModifiedSince=datetime(2015, 1, 1),
IfNoneMatch='string',
IfUnmodifiedSince=datetime(2015, 1, 1),
Key='string',
Range='string',
VersionId='string',
SSECustomerAlgorithm='string',
SSECustomerKey='string',
RequestPayer='requester',
PartNumber=123,
ExpectedBucketOwner='string',
ChecksumMode='ENABLED'
)
```
An easy fix would be to give the `extra_args` to `get_key` then to `load(**self.extra_args)
### What you think should happen instead
the extra_args should be used in get_key() and therefore obj.load()
### How to reproduce
Try to use the S3Hook as below to download an encrypted file:
```
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
extra_args={
'SSECustomerAlgorithm': 'YOUR_ALGO',
'SSECustomerKey': YOUR_SSE_C_KEY
}
hook = S3Hook(aws_conn_id=YOUR_S3_CONNECTION, extra_args=extra_args)
hook.download_file(
key=key, bucket_name=bucket_name, local_path=local_path
)
```
### Operating System
any
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #25833: Airflow Amazon provider S3Hook().download_file() fail when needs encryption arguments (SSECustomerKey etc..)
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #25833:
URL: https://github.com/apache/airflow/issues/25833#issuecomment-1265594625
> is this issue still open? @eladkal If so, I can take it
There is open PR https://github.com/apache/airflow/pull/25835
Maybe you can share thoughts on it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow Amazon provider S3Hook().download_file() fail when needs encryption arguments (SSECustomerKey etc..) [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis closed issue #25833: Airflow Amazon provider S3Hook().download_file() fail when needs encryption arguments (SSECustomerKey etc..)
URL: https://github.com/apache/airflow/issues/25833
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow Amazon provider S3Hook().download_file() fail when needs encryption arguments (SSECustomerKey etc..) [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #25833:
URL: https://github.com/apache/airflow/issues/25833#issuecomment-1746256594
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mrmammadov commented on issue #25833: Airflow Amazon provider S3Hook().download_file() fail when needs encryption arguments (SSECustomerKey etc..)
Posted by GitBox <gi...@apache.org>.
mrmammadov commented on issue #25833:
URL: https://github.com/apache/airflow/issues/25833#issuecomment-1264330807
is this issue still open? @eladkal
If so, I can take it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org