You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "andrii-korotkov-verkada (via GitHub)" <gi...@apache.org> on 2025/01/03 22:02:12 UTC
[D] Support IAM auth for a database when connecting via CLI [airflow]
GitHub user andrii-korotkov-verkada created a discussion: Support IAM auth for a database when connecting via CLI
### Description
Right now it looks like a username and password is needed to connect, e.g. https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#connection-cli. It would be great to support IAM auth with auto-refreshing credentials.
### Use case/motivation
Modern security practices include moving away from username/password to IAM auth, e.g. to avoid stale unrotated credentials and risks of leaking them. In some companies, access to the database via username/password might not even be supported anymore. To have CLI usable, IAM auth needs to be supported. Technically, people can get a temporary password for 15 min, but refreshing it manually would be pretty frustrating.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
GitHub link: https://github.com/apache/airflow/discussions/45385
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Support IAM auth for a database when connecting via CLI [airflow]
Posted by "andrii-korotkov-verkada (via GitHub)" <gi...@apache.org>.
GitHub user andrii-korotkov-verkada added a comment to the discussion: Support IAM auth for a database when connecting via CLI
I want to integrate IAM auth for Postgres DB. In my specific case it's about AWS one https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.html, but could be more generic.
I've managed to find docs for that https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html#session-factory, custom factory session. However, earlier section of the doc https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html#configuring-the-connection makes it look like you are responsible for refreshing the credentials, which is not as convenient. Postgres doc https://airflow.apache.org/docs/apache-airflow-providers-postgres/stable/connections/postgres.html also suggests using username and password. Overall, this looks like a case where a documentation can be improved. I'll submit a PR later.
GitHub link: https://github.com/apache/airflow/discussions/45385#discussioncomment-11741603
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Support IAM auth for a database when connecting via CLI [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
GitHub user raphaelauv edited a comment on the discussion: Support IAM auth for a database when connecting via CLI
If you want to use aws rds iam authent for the airflow metdata_db then you must use `airflow_local_settings.py`
https://docs.sqlalchemy.org/en/21/core/engines.html#generating-dynamic-authentication-tokens
this is a working ( not production tested ) example : https://github.com/apache/airflow/issues/71664
GitHub link: https://github.com/apache/airflow/discussions/45385#discussioncomment-18032054
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Support IAM auth for a database when connecting via CLI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Support IAM auth for a database when connecting via CLI
Indeed. There is nothing to prevent using those mechanisms you mention @andrii-korotkov-verkada - for example Google And AWS provider already support what you advocate for. Please double check all the authentication mechanism. What exactly you think is not possible or problematic in the current setup ?
GitHub link: https://github.com/apache/airflow/discussions/45385#discussioncomment-11729645
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Support IAM auth for a database when connecting via CLI [airflow]
Posted by "olfway (via GitHub)" <gi...@apache.org>.
GitHub user olfway added a comment to the discussion: Support IAM auth for a database when connecting via CLI
> Note: For AWS IAM authentication, use iam in the extra connection parameters
and set it to true. Leave the password field empty. This will use the
"aws_default" connection to get the temporary token unless you override
in extras.
> https://airflow.apache.org/docs/apache-airflow-providers-postgres/5.6.1/_modules/airflow/providers/postgres/hooks/postgres.html
It seems it's supported, but I didn't try it myself
GitHub link: https://github.com/apache/airflow/discussions/45385#discussioncomment-17048117
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Support IAM auth for a database when connecting via CLI [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
GitHub user raphaelauv added a comment to the discussion: Support IAM auth for a database when connecting via CLI
If you want to use aws rds iam authent for the airflow metdata_db then you must use `airflow_local_settings.py`
https://docs.sqlalchemy.org/en/21/core/engines.html#generating-dynamic-authentication-tokens
GitHub link: https://github.com/apache/airflow/discussions/45385#discussioncomment-18032054
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Support IAM auth for a database when connecting via CLI [airflow]
Posted by "nathadfield (via GitHub)" <gi...@apache.org>.
GitHub user nathadfield added a comment to the discussion: Support IAM auth for a database when connecting via CLI
@andrii-korotkov-verkada Airflow supports connections for many databases and services with a wide variety of authentication methods. Are you referring to something specific?
GitHub link: https://github.com/apache/airflow/discussions/45385#discussioncomment-11729644
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Support IAM auth for a database when connecting via CLI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Support IAM auth for a database when connecting via CLI
Converting it in a discussion as custom schemes are entirely possible now - more discussion is needed to understand what it is about.
GitHub link: https://github.com/apache/airflow/discussions/45385#discussioncomment-11729650
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org