You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "piotrdziuba (via GitHub)" <gi...@apache.org> on 2023/02/14 13:24:09 UTC
[GitHub] [airflow] piotrdziuba opened a new issue, #29535: GCSToBigQueryOperator in defferable mode do not respect impersonation chain
piotrdziuba opened a new issue, #29535:
URL: https://github.com/apache/airflow/issues/29535
### Apache Airflow Provider(s)
google
### Versions of Apache Airflow Providers
apache-airflow-providers-google==8.8.0
### Apache Airflow version
airflow-2.4.3
### Operating System
N/A
### Deployment
Composer
### Deployment details
_No response_
### What happened
When using operator `GCSToBigQueryOperator` with used `impersonation_chain` it works correctly as long as I do not turn on `defferable=True`
```
Traceback (most recent call last):
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py", line 452, in execute_complete
raise AirflowException(event["message"])
airflow.exceptions.AirflowException: 403, message='Forbidden: {\n "error": {\n "code": 403,\n "message": "Access Denied: Job my-gcp-project:EU.airflow_sc_15793_sync_airflow_summary_table_to_bq_import_to_bigquery_2023_02_13T02_00_00_00_00_b7726c7b935bb5c777eb129a5ec3a1e9: Permission bigquery.jobs.get denied on job my-gcp-project:EU.airflow_sc_15793_sync_airflow_summary_table_to_bq_import_to_bigquery_2023_02_13T02_00_00_00_00_b7726c7b935bb5c777eb129a5ec3a1e9 (or it may not exist)
```
### What you think should happen instead
Seems the only provided details for the deferred operation are `project_id` and `gcp_conn_id`, but not the `impersonation_chain`.
https://github.com/apache/airflow/blob/providers-google/8.8.0/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py#L429
Most likely it would be beneficial to rework the `BigQueryInsertJobTrigger` so that it considers use case with the impersonation.
### How to reproduce
_No response_
### Anything else
Possibly related to https://github.com/apache/airflow/issues/29307
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eskarimov commented on issue #29535: GCSToBigQueryOperator in deferrable mode do not respect impersonation chain
Posted by "eskarimov (via GitHub)" <gi...@apache.org>.
eskarimov commented on issue #29535:
URL: https://github.com/apache/airflow/issues/29535#issuecomment-1445354361
Currently `gcloud-aio` package used for the `Trigger` doesn't support impersonated credentials, we should link the current issue to [the upstream one in `gloud-aio`](https://github.com/talkiq/gcloud-aio/issues/421).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29535: GCSToBigQueryOperator in deferrable mode do not respect impersonation chain
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29535:
URL: https://github.com/apache/airflow/issues/29535#issuecomment-1518789498
Agree. W econ re-open if needs be,. PR to implement it the feature is added will be welcome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29535: GCSToBigQueryOperator in deferrable mode do not respect impersonation chain
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29535:
URL: https://github.com/apache/airflow/issues/29535#issuecomment-1445387320
> Currently `gcloud-aio` package used for the `Trigger` doesn't support impersonated credentials, we should link the current issue to [the upstream one in `gloud-aio`](https://github.com/talkiq/gcloud-aio/issues/421).
You just did.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #29535: GCSToBigQueryOperator in deferrable mode do not respect impersonation chain
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #29535: GCSToBigQueryOperator in deferrable mode do not respect impersonation chain
URL: https://github.com/apache/airflow/issues/29535
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #29535: GCSToBigQueryOperator in defferable mode do not respect impersonation chain
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #29535:
URL: https://github.com/apache/airflow/issues/29535#issuecomment-1429744342
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #29535: GCSToBigQueryOperator in deferrable mode do not respect impersonation chain
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #29535:
URL: https://github.com/apache/airflow/issues/29535#issuecomment-1517577421
Do we have a task here?
This seems to be upstream package issue so if we don't actually have a task I think we should close this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org