You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/04 07:51:19 UTC
[GitHub] [airflow] ecerulm opened a new issue #14597: Provide jinja template syntax to access connections
ecerulm opened a new issue #14597:
URL: https://github.com/apache/airflow/issues/14597
**Description**
Expose the connection into the jinja template context via `conn.<connectionname>.{host,port,login,password,extra_config,etc}`
Today is possible to conveniently access [airflow's variables](https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#variables) in jinja templates using `{{ var.value.<variable_name> }}`.
There is no equivalent (to my knowledge for [connections](https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#connections)), I understand that most of the time connection are used programmatically in Operators and Hooks source code, but there are use cases where the connection info has to be pass as parameters to the operators and then it becomes cumbersome to do it without jinja template syntax.
I seen workarounds like using [user defined macros to provide get_login(my_conn_id)](https://stackoverflow.com/questions/65826404/use-airflow-connection-from-a-jinja-template/65873023#65873023
), but I'm after a consistent interface for accessing both variables and connections in the same way
**Use case / motivation**
For example, passing credentials to a [KubernetesPodOperator](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html#howto-operator-kubernetespodoperator) via [env_vars](https://cloud.google.com/composer/docs/how-to/using/using-kubernetes-pod-operator) today has to be done like this:
```
connection = Connection.get_connection_from_secrets('somecredentials')
k = KubernetesPodOperator(
task_id='task1',
env_vars={'MY_VALUE': '{{ var.value.my_value }}', 'PWD': conn.password,},
)
```
where I would prefer to use consistent syntax for both variables and connections like this:
```
# not needed anymore: connection = Connection.get_connection_from_secrets('somecredentials')
k = KubernetesPodOperator(
task_id='task1',
env_vars={'MY_VALUE': '{{ var.value.my_value }}', 'PWD': '{{ conn.somecredentials.password }}',},
)
```
The same applies to `BashOperator` where I sometimes feel the need to pass connection information to the templated script.
**Are you willing to submit a PR?**
yes, I can write the PR.
**Related Issues**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #14597: Provide jinja template syntax to access connections
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #14597:
URL: https://github.com/apache/airflow/issues/14597#issuecomment-791031515
At the beginning, I was concerned that it would be compatible with the rest of the operators and the users would have no problems understanding this feature, but as I read the full description, it actually makes a lot of sense, because Airflow can be used to orchestrate tasks in other systems and then we we need to somehow provide passwords for these systems.
We probably should combine the implementation of thisfeature together with https://github.com/apache/airflow/issues/8421
. Otherwise, it is very easy for an unauthorized person to gain access to this data, which we would like to avoid.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ecerulm edited a comment on issue #14597: Provide jinja template syntax to access connections
Posted by GitBox <gi...@apache.org>.
ecerulm edited a comment on issue #14597:
URL: https://github.com/apache/airflow/issues/14597#issuecomment-791259473
> . Otherwise, it is very easy for an unauthorized person to gain access to this data, which we would like to avoid.
Nothing against that, but it's already very easy for any DAG author to expose any connections secret by simply putting in the DAG
```
from airflow.models import Connection
c = Connection.get_connection_from_secrets(name)
....
task = BashOperator(task_id='expose_secrets', bash_command='echo'+c.password, dag=dag)
```
So I don't think introducing this feature modifies the already existing risk or am I misunderstanding something here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #14597: Provide jinja template syntax to access connections
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #14597:
URL: https://github.com/apache/airflow/issues/14597#issuecomment-869205330
I am also for adding this one. I think it's a bit of an omission that connections cannot be accessed via JINJA macros.
Are you going to work on a PR for that @ecerulm ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ecerulm commented on issue #14597: Provide jinja template syntax to access connections
Posted by GitBox <gi...@apache.org>.
ecerulm commented on issue #14597:
URL: https://github.com/apache/airflow/issues/14597#issuecomment-869414582
I asked in the dev mailing list back in 2021-03-21 and the answer was that without the ability to mask passwords in logs and rendered templates it was bad from the security standpoint so I left it there.
It seems that #8421 and #9638 are fixed now. So I will write a PR now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #14597: Provide jinja template syntax to access connections
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #14597:
URL: https://github.com/apache/airflow/issues/14597
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ecerulm commented on issue #14597: Provide jinja template syntax to access connections
Posted by GitBox <gi...@apache.org>.
ecerulm commented on issue #14597:
URL: https://github.com/apache/airflow/issues/14597#issuecomment-791259473
> . Otherwise, it is very easy for an unauthorized person to gain access to this data, which we would like to avoid.
Nothing against that, but it's already very easy for any DAG author to expose any connections secret by simply putting in the DAG
```
from airflow.models import Connection
c = Connection.get_connection_from_secrets(name)
....
task = BashOperator(task_id='expose_secrets', bash_command='echo'+c.password, dag=dag)
```
So I don't think introducing this feature modifies the already existing risk or am I misunderstanding something here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ecerulm commented on issue #14597: Provide jinja template syntax to access connections
Posted by GitBox <gi...@apache.org>.
ecerulm commented on issue #14597:
URL: https://github.com/apache/airflow/issues/14597#issuecomment-869058271
> I for example create V1Secret for them in the namespace then I can reference them from secrets of KubernetesPodOperator:
Yes, I can manually do that , but
* I do really want to handle the secrets in a single place. I would prefer not to have to update both the Airflow connection and corresponding kubernetes secret every time I rotate the credentials for this particular database.
* The users that can create and modify airflow connections are not necessarily the same with access (and skill) to create a corresponding kubernetes secret (I don't want them to call me everytime that they need to update the password)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #14597: Provide jinja template syntax to access connections
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #14597:
URL: https://github.com/apache/airflow/issues/14597#issuecomment-794468432
I agree which @mik-laj that top level query is bad idea.
As you said getting connections already can be done by using custom macro. So I'm in favour of implementing standard interface.
What I can suggest is to bring this issue to dev list so we may discuss potential security issues. Having a proof of concept would be nice as it may speed up understanding how many changes we need.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #14597: Provide jinja template syntax to access connections
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #14597:
URL: https://github.com/apache/airflow/issues/14597#issuecomment-869075856
Also being able to access a connection is pretty useful for other operators as well (some don’t have another way to store secrets. The alternatives are either to hand-roll a `ConnectionAccessor` like `var` (so why not do this for them), or create a custom operator (a bit heavy for most use cases). Another +1 from me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal edited a comment on issue #14597: Provide jinja template syntax to access connections
Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #14597:
URL: https://github.com/apache/airflow/issues/14597#issuecomment-868956454
Just commenting on the use case - I don't think you should use `env_vars` for that. I for example create `V1Secret` for them in the namespace then I can reference them from secrets of KubernetesPodOperator:
https://github.com/apache/airflow/blob/6d3c6f665a7027da9abe6ef15fcf5593d6eb5377/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py#L192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #14597: Provide jinja template syntax to access connections
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #14597:
URL: https://github.com/apache/airflow/issues/14597#issuecomment-868956454
Just commenting on the use case - I don't think you should use `env_vars` for that. I for example create `V1Secret` for them in the namespace then you can reference them from secrets of KubernetesPodOperator:
https://github.com/apache/airflow/blob/6d3c6f665a7027da9abe6ef15fcf5593d6eb5377/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py#L192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #14597: Provide jinja template syntax to access connections
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #14597:
URL: https://github.com/apache/airflow/issues/14597
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #14597: Provide jinja template syntax to access connections
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #14597:
URL: https://github.com/apache/airflow/issues/14597#issuecomment-791788892
> Nothing against that, but it's already very easy for any DAG author to expose any connections secret by simply putting in the DAG
It is not good practice for top-level code to execute SQL queries.In the community, we even have tests that detect such problems.
https://github.com/apache/airflow/blob/4589375d8bab826b7bf28606c8735057ac8b3cf2/tests/always/test_example_dags.py#L46-L61
I also recommend similar tests for my users.
I also planned to add the ability to detect similar situations to the webserver so that every user would see the warnings that recommend changing the DAG.
https://github.com/apache/airflow/pull/11960
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org