You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2023/01/05 12:20:16 UTC
[GitHub] [airflow] set92 opened a new issue, #28743: s3_hook.copy_object() doesn't allow copying files > 5Gb
set92 opened a new issue, #28743:
URL: https://github.com/apache/airflow/issues/28743
### Apache Airflow version
2.5.0
### What happened
I have a DAG in which I copy files from one bucket to another. Today, a call to s3Hook broke because `s3_hook.copy_object()` can't handle more than 5 Gb, and I have a parquet file of 6Gb. My temporal fix was moving it manually, but I thought a permanent fix would be using `s3_hook.copy` instead of `s3_hook.copy_object`, the problem is that it doesn't exist.
In my mind they do the same thing, so if we change in https://github.com/apache/airflow/blob/6cbf9b62e374666693b93f79a36dcb46cc8245c4/airflow/providers/amazon/aws/hooks/s3.py#L814 copy_object to copy wouldn't do the same and also move files bigger than 5 Gb? or there is some other method for this case?
The documentation for copy() is in [here](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.copy).
### What you think should happen instead
I think it shouldn't matter the size of the file while copying files. In my case, the log it showed was:
```
Traceback (most recent call last):
File "/opt/c2/airflow/.venv/lib/python3.9/site-packages/airflow/decorators/base.py", line 188, in execute
return_value = super().execute(context)
File "/opt/c2/airflow/.venv/lib/python3.9/site-packages/airflow/operators/python.py", line 175, in execute
return_value = self.execute_callable()
File "/opt/c2/airflow/.venv/lib/python3.9/site-packages/airflow/operators/python.py", line 193, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/opt/c2/airflow/c1/dags/dataengineers/c3/create_dev_env.py", line 32, in copy_input_data
_copy_input_data(
File "/opt/c2/airflow/c1/dags/dataengineers/c3/actions_create.py", line 155, in _copy_input_data
_copy_s3_data(source_bucket, entity["storage"], dest_bucket, folder_prefix)
File "/opt/c2/airflow/c1/dags/dataengineers/c3/actions_create.py", line 87, in _copy_s3_data
s3_hook.copy_object(
File "/opt/c2/airflow/.venv/lib/python3.9/site-packages/airflow/providers/amazon/aws/hooks/s3.py", line 830, in copy_object
response = self.get_conn().copy_object(
File "/opt/c2/airflow/.venv/lib/python3.9/site-packages/botocore/client.py", line 508, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/opt/c2/airflow/.venv/lib/python3.9/site-packages/botocore/client.py", line 915, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidRequest) when calling the CopyObject operation: The specified copy source is larger than the maximum allowable size for a copy source: 5368709120
```
### How to reproduce
_No response_
### Operating System
ubuntu 20.04
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==6.0.0
apache-airflow-providers-cncf-kubernetes==4.4.0
apache-airflow-providers-common-sql==1.2.0
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-google==6.8.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-postgres==5.2.2
apache-airflow-providers-sendgrid==3.0.0
apache-airflow-providers-slack==6.0.0
apache-airflow-providers-sqlite==3.2.1
### Deployment
Other 3rd-party Helm chart
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #28743: s3_hook.copy_object() doesn't allow copying files > 5Gb
Posted by GitBox <gi...@apache.org>.
Taragolis commented on issue #28743:
URL: https://github.com/apache/airflow/issues/28743#issuecomment-1372552399
@set92 some info in addition to what @eladkal wrote
* `copy_object` method of S3 Client it is wrapper for AWS API and limited by API (same info as mentioned in boto3: https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html)
* `copy` method of S3 Client it is one of helpers which use [s3transfer](https://github.com/boto/s3transfer) capabilities and work by different way and this is not replacement of `copy_object`
Base on your exception you use PythonOperator (task flow decorator) so you could use all capabilities of AWS Hooks, include (but not limited) create boto3 S3 client by use Airflow Connection as resault you could use all client methods described in boto3 documentation
```python
@task
def awesome_s3_copy(**kwargs):
hook = S3Hook(aws_conn_id="awesome-conn-id", region_name="us-east-1", ...)
s3_client = hook.conn
s3_client.copy(...)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #28743: s3_hook.copy_object() doesn't allow copying files > 5Gb
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #28743:
URL: https://github.com/apache/airflow/issues/28743#issuecomment-1372147590
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #28743: s3_hook.copy_object() doesn't allow copying files > 5Gb
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #28743:
URL: https://github.com/apache/airflow/issues/28743#issuecomment-1372209403
You linked to `copy` rather than to `copy_object`
boto explain the limitation when you check the right function :)
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.copy_object
```
Note
You can store individual objects of up to 5 TB in Amazon S3.
You create a copy of your object up to 5 GB in size in a single atomic action using this API.
However, to copy an object greater than 5 GB, you must use the multipart upload Upload Part - Copy (UploadPartCopy) API.
For more information, see [Copy Object Using the REST Multipart Upload API](https://docs.aws.amazon.com/AmazonS3/latest/dev/CopyingObjctsUsingRESTMPUapi.html).
```
So this is not a bug.. this is simply the functionality of boto3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis closed issue #28743: s3_hook.copy_object() doesn't allow copying files > 5Gb
Posted by GitBox <gi...@apache.org>.
Taragolis closed issue #28743: s3_hook.copy_object() doesn't allow copying files > 5Gb
URL: https://github.com/apache/airflow/issues/28743
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org