You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/25 23:12:51 UTC
[GitHub] [airflow] bernyag opened a new issue #21827: Airflow UI exposing AWS S3 secret credentials in XCOM screen when using BigQuery Data Transfer Service
bernyag opened a new issue #21827:
URL: https://github.com/apache/airflow/issues/21827
### Apache Airflow version
2.2.3
### What happened
When using BigQueryDataTransferServiceStartTransferRunsOperator in Airflow to initiate BigQuery Transfer jobs reading data from an AWS S3 bucket, the S3 access key ID as well as the S3 secret access key are being exposed in plain text via the XCOM screen on the Airflow UI when the DAG runs.
### What you expected to happen
Do not share sensitive information on the XCOMs screen.
### How to reproduce
_No response_
### Operating System
-
### Versions of Apache Airflow Providers
composer-1.18.0-airflow-2.2.3
### Deployment
Composer
### Deployment details
Cloud Composer integrated with BigQuery Data Transfer Service and AWS S3 Bucket
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] thelastmessiha commented on issue #21827: Airflow UI exposing AWS S3 secret credentials in XCOM screen when using BigQuery Data Transfer Service
Posted by GitBox <gi...@apache.org>.
thelastmessiha commented on issue #21827:
URL: https://github.com/apache/airflow/issues/21827#issuecomment-1053747254
Yes, I agree this is an issue with the Google operator. @bernyag from Google opened this issue to track it as a result of a Google Support case I opened. My understanding is that it is being internally tracked at Google as well, and that it will be addressed through that channel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #21827: Airflow UI exposing AWS S3 secret credentials in XCOM screen when using BigQuery Data Transfer Service
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #21827:
URL: https://github.com/apache/airflow/issues/21827#issuecomment-1053771851
> Thanks for that information. If that's the case, then it sounds like a good reason to let Google internally decide if it should be addressed at all, and if so, then should that be in the operator or should that be in the API to the BigQuery Transfer Service which makes the secret key available to the operator in the first place.
Very much so - still it's not an issue. It can be addressed by a PR or change in the API. We do not need issues in Airlfow - we are perfectly fine if a change is addressed direcly via PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] thelastmessiha commented on issue #21827: Airflow UI exposing AWS S3 secret credentials in XCOM screen when using BigQuery Data Transfer Service
Posted by GitBox <gi...@apache.org>.
thelastmessiha commented on issue #21827:
URL: https://github.com/apache/airflow/issues/21827#issuecomment-1053742008
@potiuk I think the root cause issue here is not access control to XCOM but that secret keys should not be stored in the metadata database of Airflow in plain text to begin with (which then propagates to the UI's XCOMs screen). Shutting off user access to XCOM does not solve that underlying security issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] thelastmessiha commented on issue #21827: Airflow UI exposing AWS S3 secret credentials in XCOM screen when using BigQuery Data Transfer Service
Posted by GitBox <gi...@apache.org>.
thelastmessiha commented on issue #21827:
URL: https://github.com/apache/airflow/issues/21827#issuecomment-1053752964
Thanks for that information. If that's the case, then it sounds like a good reason to let Google internally decide if it should be addressed at all, and if so, then should that be in the operator or should that be in the API to the BigQuery Transfer Service which makes the secret key available to the operator in the first place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #21827: Airflow UI exposing AWS S3 secret credentials in XCOM screen when using BigQuery Data Transfer Service
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #21827:
URL: https://github.com/apache/airflow/issues/21827
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21827: Airflow UI exposing AWS S3 secret credentials in XCOM screen when using BigQuery Data Transfer Service
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21827:
URL: https://github.com/apache/airflow/issues/21827#issuecomment-1053750640
Still feel free to provide a PR yourself if you want.
BTW. This is not really a "Security" issue, really as anyone who have access to write DAGs can also dump the value as needed to log or anywhere else. And there are plenty other operators where potentially sensitive data is stored in XCom - this is because XCom is used often to pulll/push data from external systems - and this is more decision of Dag Authors what to put there. If you really want to prevent the users in UI from seeing it, disabling access to XCom is the only "real" way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21827: Airflow UI exposing AWS S3 secret credentials in XCOM screen when using BigQuery Data Transfer Service
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21827:
URL: https://github.com/apache/airflow/issues/21827#issuecomment-1053743740
This is a choice of the operator to store it in Airflow. If you think it can be improved - feel free to provide a PR to that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21827: Airflow UI exposing AWS S3 secret credentials in XCOM screen when using BigQuery Data Transfer Service
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21827:
URL: https://github.com/apache/airflow/issues/21827#issuecomment-1053731934
You can configure your users and revoke them resource.XCOM permissions if you do not want them to be able to access xcom
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21827: Airflow UI exposing AWS S3 secret credentials in XCOM screen when using BigQuery Data Transfer Service
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21827:
URL: https://github.com/apache/airflow/issues/21827#issuecomment-1053771851
> Thanks for that information. If that's the case, then it sounds like a good reason to let Google internally decide if it should be addressed at all, and if so, then should that be in the operator or should that be in the API to the BigQuery Transfer Service which makes the secret key available to the operator in the first place.
Very much so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #21827: Airflow UI exposing AWS S3 secret credentials in XCOM screen when using BigQuery Data Transfer Service
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #21827:
URL: https://github.com/apache/airflow/issues/21827#issuecomment-1051347875
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21827: Airflow UI exposing AWS S3 secret credentials in XCOM screen when using BigQuery Data Transfer Service
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21827:
URL: https://github.com/apache/airflow/issues/21827#issuecomment-1053744178
Airflow has ~ 2000 contributors and you can simply become one of those, this is a great way to contribute back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org