You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Logical-sh (via GitHub)" <gi...@apache.org> on 2024/08/21 19:11:50 UTC
[I] S3ToGCSOperator broken in deferable mode [airflow]
Logical-sh opened a new issue, #41652:
URL: https://github.com/apache/airflow/issues/41652
### Apache Airflow Provider(s)
google
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon = "==8.27.0"
apache-airflow-providers-google = "==10.21.1"
### Apache Airflow version
2.9.1
### Operating System
GCP Cloud Composer
### Deployment
Google Cloud Composer
### Deployment details
Manually upgraded providers to see if it was still a bug in latest version.
### What happened
Received the following error when trying to use S3ToGCSOperator in deferrable mode.
```
Traceback (most recent call last):
File "/opt/python3.11/lib/python3.11/site-packages/airflow/jobs/triggerer_job_runner.py", line 529, in cleanup_finished_triggers
result = details["task"].result()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python3.11/lib/python3.11/site-packages/airflow/jobs/triggerer_job_runner.py", line 602, in run_trigger
async for event in trigger.run():
File "/opt/python3.11/lib/python3.11/site-packages/airflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py", line 77, in run
jobs_pager = await async_hook.get_jobs(job_names=self.job_names)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python3.11/lib/python3.11/site-packages/airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py", line 581, in get_jobs
client = await self.get_conn()
^^^^^^^^^^^^^^^^^^^^^
File "/opt/python3.11/lib/python3.11/site-packages/airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py", line 567, in get_conn
credentials = (await self.get_sync_hook()).get_credentials()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python3.11/lib/python3.11/site-packages/airflow/providers/google/common/hooks/base_google.py", line 776, in get_sync_hook
self._sync_hook = await sync_to_async(self.sync_hook_class)(**self._hook_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: sync_to_async.<locals>.<lambda>() got an unexpected keyword argument 'gcp_conn_id'
```
### What you think should happen instead
IT should not error when checking the job status on wakeup.
### How to reproduce
Create a S3ToGCSOperator job with deferable enabled.
### Anything else
On every run of the operator, happens after the job is queued and the code trys to check the status I think.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3ToGCSOperator broken in deferable mode [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #41652:
URL: https://github.com/apache/airflow/issues/41652#issuecomment-2302815085
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3ToGCSOperator broken in deferable mode [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #41652:
URL: https://github.com/apache/airflow/issues/41652#issuecomment-2303857444
cc @moiseenkov I think you developed the deferrable mode for this operator?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3ToGCSOperator broken in deferable mode [airflow]
Posted by "moiseenkov (via GitHub)" <gi...@apache.org>.
moiseenkov commented on issue #41652:
URL: https://github.com/apache/airflow/issues/41652#issuecomment-2304766946
The new version was released a couple of hours ago:
`apache-airflow-providers-google==10.22.0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3ToGCSOperator broken in deferable mode [airflow]
Posted by "moiseenkov (via GitHub)" <gi...@apache.org>.
moiseenkov commented on issue #41652:
URL: https://github.com/apache/airflow/issues/41652#issuecomment-2304067202
This bug is fixed by #41417 already. Upgrading the apache-airflow-providers-google==10.22.0 will help. I tested it locally (on the current main branch) and in Composer with apache-airflow-providers-google==10.22.0rc1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3ToGCSOperator broken in deferable mode [airflow]
Posted by "Logical-sh (via GitHub)" <gi...@apache.org>.
Logical-sh commented on issue #41652:
URL: https://github.com/apache/airflow/issues/41652#issuecomment-2304740709
Thanks! Ill give that a try, I did try upgrading to most recent release but didn't think about trying RCs! Will give that a shot today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3ToGCSOperator broken in deferable mode [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #41652:
URL: https://github.com/apache/airflow/issues/41652#issuecomment-2305142614
Cool then closing as solved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3ToGCSOperator broken in deferable mode [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #41652: S3ToGCSOperator broken in deferable mode
URL: https://github.com/apache/airflow/issues/41652
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3ToGCSOperator broken in deferable mode [airflow]
Posted by "moiseenkov (via GitHub)" <gi...@apache.org>.
moiseenkov commented on issue #41652:
URL: https://github.com/apache/airflow/issues/41652#issuecomment-2303932795
> cc @moiseenkov I think you developed the deferrable mode for this operator?
Sure, I am on it already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org