You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2025/08/04 11:51:46 UTC
[I] Export import for connections for airflowctl is broken in main [airflow]
potiuk opened a new issue, #54090:
URL: https://github.com/apache/airflow/issues/54090
### Apache Airflow version
main (development)
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
After #53973 where change in our security model has been implemented, to not allow anyone (including connection editing users) to be able to see sensitive data from the connection - effectively making connection sensitive data "write-only". This has an impact on export/import capability for `airflowctl` because airflowctl exposed export/import capability and retrieved connection data via API - and the current model (draft described in https://github.com/apache/airflow/pull/54088) assumes that there is no access to sensitive connection credential via "public API". Only Tasks have access to those via "task-sdk" API.
However this means that exported files will contain "***" (unicode-d) mask instead of real sensitive data - which make them unusable for imports.
There are several ways it can be solved:
1) drop export/import functionality from airflowctl (technically speaking we could only drop export and assume that someone could export using airflow CLI to export data and airflowctl to import it, but it could be confusing).
2) encrypt the exported data (either with FERNET key or with a passphrase entered by the user) - the passphrase by the user is however violating the assumption that no "API" user shoud be able to see the password - we should rather use some security passphrase that is not available to the API user
3) relax expectation of the `airflowctl` user to be able to use "task-sdk" API - but this is also violating the assumptions that no API user should see the credentials.
Possibly there are other optiosns.
@ashb @pierrejeambrun (and of course @bugraoz93 should likely be aware of that) I wonder what your opinions on that are. We had no chance to discuss it before, because decision on changing the model has been taken very hastily without discussing the consequences, but maybe that is a good opportunity to discuss it here.
### What you think should happen instead?
_No response_
### How to reproduce
1) Install airflow
2) add connection
3) run `airflowctl connection export`
4) observe export file containing masked sensitive data that is not "importable" correctly
### Operating System
Any
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Export import for connections for airflowctl is broken in main [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #54090:
URL: https://github.com/apache/airflow/issues/54090#issuecomment-3168966612
But also a "quick" version of it is to remove export (and likely import - it would make little sense to only have one I guess) and schedule the "technical user" for later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Export import for connections for airflowctl is broken in main [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on issue #54090:
URL: https://github.com/apache/airflow/issues/54090#issuecomment-3175399850
Thanks, sounds like a plan!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Export import for connections for airflowctl is broken in main [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on issue #54090:
URL: https://github.com/apache/airflow/issues/54090#issuecomment-3168823161
Yes, the UI and airflowctl are using the same endpoints on those entities. There are a few for UI for specific tasks. Since they are using the same endpoints, I would expect them to behave similarly.
If we reduct in UI, it makes sense to keep the same for it since the audience for GUI and CTL in general is the same.
We can add one more layer, like a new endpoint and role. New role could be better because then we can control for multiple endpoints while we need to consider `GET` and `LIST` endpoints of the reducted entities. I am not sure if these things are more secure since the Read Connection role should also be given carefully.
I also agree that export may be more dangerous, but if you have read access without `reduction`, you can easily automate the export. I think we should evaluate the reduction in general because it also covers getting a single connection and getting all connections. Those are also endpoints and behave similarly. This is basically an export if you all `airflowctl connections list -o yaml > a.yaml` because, as in the airflow CLI, we are returning all, and the list can return yaml, json, etc...
This case also applies to `Variables` if it has some kind of password and needs a reduction, right?
I am also okay to drop this from airflowctl and keep it in airflow as an administrative operation to read database directly since the values are sensitive, but I would like to have similar functionality with the UI if we can, because if you like and get used to commands where you can do the same with things with the UI, for example, I generally use CLI in that case where you feel more control over what you call (of course GUI can be easier), my point is it would be great if we would be consistent in between UI and CTL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Export import for connections for airflowctl is broken in main [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 closed issue #54090: Export import for connections for airflowctl is broken in main
URL: https://github.com/apache/airflow/issues/54090
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Export import for connections for airflowctl is broken in main [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #54090:
URL: https://github.com/apache/airflow/issues/54090#issuecomment-3167142876
> We don't currently, correct, but we could probably very easily detect this via User-Agent header. Or the other option is to create a different UI-only endpoint (we already have other such endpoints)
This can be spoofed by an attacker potentially (via injecting javascript etc.). I think the only way to prevent it, is to have completely separate authentication mechanism dedicated to the endpoint that returns the data.
How about just dropping export via "airflowctl" ? If we consider it really dangerous, we might onle leave it for direct access via "airflow" command. And that even makes perfect sense because exporting/importing connection is almost like you export/import database and for that you need to have access to the database.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Export import for connections for airflowctl is broken in main [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #54090:
URL: https://github.com/apache/airflow/issues/54090#issuecomment-3168963464
Yeah, just endpoint is not enough - it would have to be completely different way (and "technical user" sounds like a good idea - but it would have to define it not as a role, but a flag for the user that should **not allow** it to login to the UI (should not be configurable - you either are "technical" user or "ui" user - but never both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Export import for connections for airflowctl is broken in main [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #54090:
URL: https://github.com/apache/airflow/issues/54090#issuecomment-3167041262
> Import is unaffected right? It's just a "bulk create" operation, so I don't see what needs to change about that.
Yes. Import can be done without problems because we have "write-only" permission.
Most problems is with export - because our model assumes currently that we do not expose such credentials over API to the user (we only expose them via task-sdk to workloads and triggerer)
So my question is do we want to create another role in our model - someone who is able to read the passwords via public API. So you say we need to add such role? Do you think it's secure enough ? Previously the user who could edit connecton could do that - but we considered it not "Securte enough". And I think if we decide on adding it back, we need to be very specific about that role.
I'd say it's a bit not consistent as that user who logs in via the UI will also be able to see those passwords in the UI - so we are just shifting the problem we tried to solve to another role- and we need to add back to our security model that there are users who can read the credentials.
Or maybe we want to introduce a different class of the API and authentication - so far as I understand airflowctl uses the same API as the UI and we have no way to distinguish whether we are calling it from UI or airflowctl?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Export import for connections for airflowctl is broken in main [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on issue #54090:
URL: https://github.com/apache/airflow/issues/54090#issuecomment-3168799094
I like the double endpoint approach. I would love to have a notion of `technical user` which isn't a real person maybe a new class of tokens or something like that, and some sensitive endpoint such as the `retrieve connection without redacting` could be exclusive to `technical user`. (basically there is no real 'user' in the token, that returns None, and the token might be a little bit different)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Export import for connections for airflowctl is broken in main [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on issue #54090:
URL: https://github.com/apache/airflow/issues/54090#issuecomment-3168974548
I agree for short term we can remove them. The moving over CTL from CLI will also take time. We can add the flag to differentiate user later on. I will create a PR later today for removing them. We can always go back in Git and add them back with little changes after ensuring the communication is mature enough according to security
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Export import for connections for airflowctl is broken in main [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on issue #54090:
URL: https://github.com/apache/airflow/issues/54090#issuecomment-3166999361
Import is unaffected right? It's just a "bulk create" operation, so I don't see what needs to change about that.
I'd say this should probably be an extra permission over-and-above "read connections".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Export import for connections for airflowctl is broken in main [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #54090:
URL: https://github.com/apache/airflow/issues/54090#issuecomment-3166800683
I am do not know - I was waiting mostly for comments of @ashb @pierrejeambrun @bugraoz93 as I think it's the consequence of the model we chose, but I am not sure if the good solution is to use different API user, specific permission of a user via public API, or encryption. I'd love to hear their opinion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Export import for connections for airflowctl is broken in main [airflow]
Posted by "Prasanth345 (via GitHub)" <gi...@apache.org>.
Prasanth345 commented on issue #54090:
URL: https://github.com/apache/airflow/issues/54090#issuecomment-3166711635
Hi @potiuk I’d like to contribute towards resolving this issue.
From what I understand, the recent changes in the security model prevent exporting plaintext credentials, which breaks `airflowctl connection export/import`. I’d be happy to explore and propose a solution — whether that’s encrypted export, removing the feature, or adjusting the API access.
Would you prefer one direction over the others, or should I open a draft PR with options?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Export import for connections for airflowctl is broken in main [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on issue #54090:
URL: https://github.com/apache/airflow/issues/54090#issuecomment-3167068888
> we have no way to distinguish whether we are calling it from UI or airflowctl?
We don't currently, correct, but we could probably very easily detect this via User-Agent header. Or the other option is to create a different UI-only endpoint (we already have other such endpoints)
I still think it should be a different permission though -- exporting all connections is a highly sensitive and risky approach -- so I'd feel much happier with export being a separate endpoint, even if it's not a separate permission. My main driver for this is auditability -- if it's a separate endpoint that exposes the password, you can then tell when people get those from acces logs alone.
So I'd probably say both:
1. UI only endpoint that always redacts (and possibly remove the `hide_sensitive_var_conn_fields` setting and make it mandatory?)
2. Add a new permission, ("Connections", "can_read_unredacted") or similar, that is required to either export, or to read a connection without redaction.
3. We could extend the UI views to add a "show senstive" fields button which would use the public API to populate things.
If you call the export endpoint and you don't have the export permission it could either return connections with redaction, or more likely return a 403. And if you call the normal connection GET endpoint without having that new perm it would return it redacted exactly as the UI?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org