Provide jinja template syntax to access connections

[Ruben Laguna <ru...@gmail.com>]

@turbaszek instructed me to bring the discussion from
https://github.com/apache/airflow/issues/14597 into the dev list:

Today is possible to conveniently access airflow's variables
<https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#variables>
in
jinja templates using {{ var.value.<variable_name> }}.
I think it would be nice to have an similar syntax for connections {{
conn.value.myconn }}

Currently if you want to access connections info from the DAG you need to
resort to any of these two workaround

* c = Connection.get_connection_from_secrets('myconn')
* provide a custom macro at the DAG with `user_defined_macros`
* provide a custom macro via a plugin

All of them are less convenient than having a builtin documented jinja
macro for that.

Does  anybody foresee any security issues by having macros for connections?
Or why connections were left out of jinja in the first place (I suspect
that was a conscious decision)?

-- 
/Rubén

Re: Provide jinja template syntax to access connections

[Kaxil Naik <ka...@gmail.com>]

Yeah, it would good to add that feature only after the security issue
(details exposed via Rendered Template) is fixed.

Regards,
Kaxil

On Tue, Mar 23, 2021 at 10:09 AM Ash Berlin-Taylor <as...@apache.org> wrote:

> Without the ability to mask the connection password (plus possible some of
> the extras) from logs and the webserver this would be bad from a security
> PoV.
>
> Connection passwords would the be viewable in the "Rendered" tab of the
> Task Instance detail, in addition to possible in task logs.
>
> For example
>
> https://github.com/apache/airflow/issues/9638
> https://github.com/apache/airflow/issues/8421
>
> We shouldn't add this connections-via-templates feature until addressing
> both of these, as it makes the problem much much worse.
>
> -ash
>
>
>
> On Tue, 23 Mar, 2021 at 09:59, Ruben Laguna <ru...@gmail.com>
> wrote:
>
> @turbaszek instructed me to bring the discussion from
> https://github.com/apache/airflow/issues/14597 into the dev list:
>
> Today is possible to conveniently access airflow's variables
> <https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#variables> in
> jinja templates using {{ var.value.<variable_name> }}.
> I think it would be nice to have an similar syntax for connections {{
> conn.value.myconn }}
>
> Currently if you want to access connections info from the DAG you need to
> resort to any of these two workaround
>
> * c = Connection.get_connection_from_secrets('myconn')
> * provide a custom macro at the DAG with `user_defined_macros`
> * provide a custom macro via a plugin
>
> All of them are less convenient than having a builtin documented jinja
> macro for that.
>
> Does  anybody foresee any security issues by having macros for
> connections? Or why connections were left out of jinja in the first place
> (I suspect that was a conscious decision)?
>
> --
> /Rubén
>
>

Re: Provide jinja template syntax to access connections

[Ash Berlin-Taylor <as...@apache.org>]

Without the ability to mask the connection password (plus possible some 
of the extras) from logs and the webserver this would be bad from a 
security PoV.

Connection passwords would the be viewable in the "Rendered" tab of the 
Task Instance detail, in addition to possible in task logs.

For example

<https://github.com/apache/airflow/issues/9638>
<https://github.com/apache/airflow/issues/8421>

We shouldn't add this connections-via-templates feature until 
addressing both of these, as it makes the problem much much worse.

-ash



On Tue, 23 Mar, 2021 at 09:59, Ruben Laguna <ru...@gmail.com> 
wrote:
> @turbaszek instructed me to bring the discussion from 
> <https://github.com/apache/airflow/issues/14597> into the dev list:
> 
> Today is possible to conveniently access airflow's variables 
> <https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#variables> 
> in jinja templates using {{ var.value.<variable_name> }}.
> 
> I think it would be nice to have an similar syntax for connections {{ 
> conn.value.myconn }}
> 
> Currently if you want to access connections info from the DAG you 
> need to resort to any of these two workaround
> 
> * c = Connection.get_connection_from_secrets('myconn')
> * provide a custom macro at the DAG with `user_defined_macros`
> * provide a custom macro via a plugin
> 
> All of them are less convenient than having a builtin documented 
> jinja macro for that.
> 
> Does  anybody foresee any security issues by having macros for 
> connections? Or why connections were left out of jinja in the first 
> place (I suspect that was a conscious decision)?
> 
> --
> /Rubén