You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Guilherme Marthe <gu...@enjoei.com.br> on 2017/01/18 17:45:36 UTC
Good jinja templating in airflow examples
Hey folks!
I am trying to write a dag that works well with jinja templating, since
through my study of the documentation, is the best way to ensure
compatibility with the back-fill function.
Are you guys aware of any examples online with this functionality working?
The, documentation is still a bit dry on insightful examples, and I am a
noobie developer, so I am trying to make the best use of the tool in the
"correct way".
Thank you in advance for any resources you guys can share!
Gui
Ps: I ve been told I can email you guys with questions like these :) Hope
this is not a hassle ^_^
Re: Good jinja templating in airflow examples
Posted by Boris Tyukin <bo...@boristyukin.com>.
Max, this is awesome! Here is the working example if someone needs it - you
almost typed it right :)
def foo(templates_dict, *args, **kwargs):
return templates_dict['ds']
# All of the values in the `templates_dict` should get templated by the
Airflow engine,
# meaning that in this context, `foo` should print the value of the `ds`.
t4 = PythonOperator(
task_id='t4',
python_callable=foo,
provide_context=True,
templates_dict={
'ds': '{{ ds }}'
},
dag=dag)
On Thu, Jan 19, 2017 at 11:58 AM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:
> I didn't test the code bellow, but hopefully you get the idea:
> ----------------------
>
> def foo(templates_dict, *args, **kwargs):
> print(templates_dict['ds'])
>
> op = PythonOperator('a_task_id', python_callable=foo, templates_dict={'ds':
> '{{ ds }}'})
> # All of the values in the `templates_dict` should get templated by the
> Airflow engine, meaning that in this context, `foo` should print the value
> of the `ds`.
>
> Max
>
> On Thu, Jan 19, 2017 at 6:22 AM, Boris Tyukin <bo...@boristyukin.com>
> wrote:
>
> > Maxime, I have a related question. Can you explain how template_dict work
> > with PythonOperator? Documentation is very vague about it. Example would
> be
> > nice. I could not find a way to use jinja template with Python operator
> > i.e. passing templated parameters to a pythonoperator callable. Someone
> > told me the only way to do it is to create a wrapper function and use
> xcom
> > or macro from there
> >
> > On Wed, Jan 18, 2017 at 10:09 PM, Maxime Beauchemin <
> > maximebeauchemin@gmail.com> wrote:
> >
> > > Here's the list of variables and macros exposed in the jinja context:
> > > https://airflow.apache.org/code.html#macros
> > >
> > > Those are all exposed by the framework, meaning they are usable for any
> > > templated field without doing anything special.
> > >
> > > There are ways to pass your own variables and methods using the
> `params`
> > > attribute of any operator as shown in the tutorial here
> > > <https://airflow.apache.org/tutorial.html#example-pipeline-definition
> >,
> > > and
> > > there's also a `user_defined_macros` parameter as a dict you can pass
> > while
> > > creating the DAG object. Every key in that dictionary is made available
> > in
> > > the global jinja namespace.
> > >
> > > https://airflow.apache.org/code.html#airflow.models.DAG
> > >
> > > Max
> > >
> > > On Wed, Jan 18, 2017 at 11:31 AM, Boris Tyukin <bo...@boristyukin.com>
> > > wrote:
> > >
> > > > Hi Guilherme,
> > > >
> > > > I guess it depends what exactly you want to do as not everything
> works
> > > with
> > > > jinja.
> > > >
> > > > From documentation:
> > > >
> > > > https://pythonhosted.org/airflow/concepts.html#jinja-templating
> > > >
> > > > You can use Jinja templating with every parameter that is marked as
> > > > “templated” in the documentation.
> > > >
> > > > You can open source code for operators and see what parameters are
> > > actually
> > > > templated.
> > > >
> > > > For example, if you open source code for BashOperator
> > > > https://pythonhosted.org/airflow/_modules/bash_
> > > operator.html#BashOperator
> > > > you will see this line of code:
> > > > template_fields = ('bash_command', 'env')
> > > >
> > > > It means only bash_command and env will work with jinja templates.
> > > >
> > > > On Wed, Jan 18, 2017 at 12:45 PM, Guilherme Marthe <
> > > > guilherme.marthe@enjoei.com.br> wrote:
> > > >
> > > > > Hey folks!
> > > > >
> > > > > I am trying to write a dag that works well with jinja templating,
> > since
> > > > > through my study of the documentation, is the best way to ensure
> > > > > compatibility with the back-fill function.
> > > > >
> > > > > Are you guys aware of any examples online with this functionality
> > > > working?
> > > > > The, documentation is still a bit dry on insightful examples, and I
> > am
> > > a
> > > > > noobie developer, so I am trying to make the best use of the tool
> in
> > > the
> > > > > "correct way".
> > > > >
> > > > > Thank you in advance for any resources you guys can share!
> > > > >
> > > > > Gui
> > > > >
> > > > > Ps: I ve been told I can email you guys with questions like these
> :)
> > > Hope
> > > > > this is not a hassle ^_^
> > > > >
> > > >
> > >
> >
>
Re: Good jinja templating in airflow examples
Posted by Maxime Beauchemin <ma...@gmail.com>.
I didn't test the code bellow, but hopefully you get the idea:
----------------------
def foo(templates_dict, *args, **kwargs):
print(templates_dict['ds'])
op = PythonOperator('a_task_id', python_callable=foo, templates_dict={'ds':
'{{ ds }}'})
# All of the values in the `templates_dict` should get templated by the
Airflow engine, meaning that in this context, `foo` should print the value
of the `ds`.
Max
On Thu, Jan 19, 2017 at 6:22 AM, Boris Tyukin <bo...@boristyukin.com> wrote:
> Maxime, I have a related question. Can you explain how template_dict work
> with PythonOperator? Documentation is very vague about it. Example would be
> nice. I could not find a way to use jinja template with Python operator
> i.e. passing templated parameters to a pythonoperator callable. Someone
> told me the only way to do it is to create a wrapper function and use xcom
> or macro from there
>
> On Wed, Jan 18, 2017 at 10:09 PM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > Here's the list of variables and macros exposed in the jinja context:
> > https://airflow.apache.org/code.html#macros
> >
> > Those are all exposed by the framework, meaning they are usable for any
> > templated field without doing anything special.
> >
> > There are ways to pass your own variables and methods using the `params`
> > attribute of any operator as shown in the tutorial here
> > <https://airflow.apache.org/tutorial.html#example-pipeline-definition>,
> > and
> > there's also a `user_defined_macros` parameter as a dict you can pass
> while
> > creating the DAG object. Every key in that dictionary is made available
> in
> > the global jinja namespace.
> >
> > https://airflow.apache.org/code.html#airflow.models.DAG
> >
> > Max
> >
> > On Wed, Jan 18, 2017 at 11:31 AM, Boris Tyukin <bo...@boristyukin.com>
> > wrote:
> >
> > > Hi Guilherme,
> > >
> > > I guess it depends what exactly you want to do as not everything works
> > with
> > > jinja.
> > >
> > > From documentation:
> > >
> > > https://pythonhosted.org/airflow/concepts.html#jinja-templating
> > >
> > > You can use Jinja templating with every parameter that is marked as
> > > “templated” in the documentation.
> > >
> > > You can open source code for operators and see what parameters are
> > actually
> > > templated.
> > >
> > > For example, if you open source code for BashOperator
> > > https://pythonhosted.org/airflow/_modules/bash_
> > operator.html#BashOperator
> > > you will see this line of code:
> > > template_fields = ('bash_command', 'env')
> > >
> > > It means only bash_command and env will work with jinja templates.
> > >
> > > On Wed, Jan 18, 2017 at 12:45 PM, Guilherme Marthe <
> > > guilherme.marthe@enjoei.com.br> wrote:
> > >
> > > > Hey folks!
> > > >
> > > > I am trying to write a dag that works well with jinja templating,
> since
> > > > through my study of the documentation, is the best way to ensure
> > > > compatibility with the back-fill function.
> > > >
> > > > Are you guys aware of any examples online with this functionality
> > > working?
> > > > The, documentation is still a bit dry on insightful examples, and I
> am
> > a
> > > > noobie developer, so I am trying to make the best use of the tool in
> > the
> > > > "correct way".
> > > >
> > > > Thank you in advance for any resources you guys can share!
> > > >
> > > > Gui
> > > >
> > > > Ps: I ve been told I can email you guys with questions like these :)
> > Hope
> > > > this is not a hassle ^_^
> > > >
> > >
> >
>
Re: Good jinja templating in airflow examples
Posted by Boris Tyukin <bo...@boristyukin.com>.
Maxime, I have a related question. Can you explain how template_dict work
with PythonOperator? Documentation is very vague about it. Example would be
nice. I could not find a way to use jinja template with Python operator
i.e. passing templated parameters to a pythonoperator callable. Someone
told me the only way to do it is to create a wrapper function and use xcom
or macro from there
On Wed, Jan 18, 2017 at 10:09 PM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:
> Here's the list of variables and macros exposed in the jinja context:
> https://airflow.apache.org/code.html#macros
>
> Those are all exposed by the framework, meaning they are usable for any
> templated field without doing anything special.
>
> There are ways to pass your own variables and methods using the `params`
> attribute of any operator as shown in the tutorial here
> <https://airflow.apache.org/tutorial.html#example-pipeline-definition>,
> and
> there's also a `user_defined_macros` parameter as a dict you can pass while
> creating the DAG object. Every key in that dictionary is made available in
> the global jinja namespace.
>
> https://airflow.apache.org/code.html#airflow.models.DAG
>
> Max
>
> On Wed, Jan 18, 2017 at 11:31 AM, Boris Tyukin <bo...@boristyukin.com>
> wrote:
>
> > Hi Guilherme,
> >
> > I guess it depends what exactly you want to do as not everything works
> with
> > jinja.
> >
> > From documentation:
> >
> > https://pythonhosted.org/airflow/concepts.html#jinja-templating
> >
> > You can use Jinja templating with every parameter that is marked as
> > “templated” in the documentation.
> >
> > You can open source code for operators and see what parameters are
> actually
> > templated.
> >
> > For example, if you open source code for BashOperator
> > https://pythonhosted.org/airflow/_modules/bash_
> operator.html#BashOperator
> > you will see this line of code:
> > template_fields = ('bash_command', 'env')
> >
> > It means only bash_command and env will work with jinja templates.
> >
> > On Wed, Jan 18, 2017 at 12:45 PM, Guilherme Marthe <
> > guilherme.marthe@enjoei.com.br> wrote:
> >
> > > Hey folks!
> > >
> > > I am trying to write a dag that works well with jinja templating, since
> > > through my study of the documentation, is the best way to ensure
> > > compatibility with the back-fill function.
> > >
> > > Are you guys aware of any examples online with this functionality
> > working?
> > > The, documentation is still a bit dry on insightful examples, and I am
> a
> > > noobie developer, so I am trying to make the best use of the tool in
> the
> > > "correct way".
> > >
> > > Thank you in advance for any resources you guys can share!
> > >
> > > Gui
> > >
> > > Ps: I ve been told I can email you guys with questions like these :)
> Hope
> > > this is not a hassle ^_^
> > >
> >
>
Re: Good jinja templating in airflow examples
Posted by Maxime Beauchemin <ma...@gmail.com>.
Here's the list of variables and macros exposed in the jinja context:
https://airflow.apache.org/code.html#macros
Those are all exposed by the framework, meaning they are usable for any
templated field without doing anything special.
There are ways to pass your own variables and methods using the `params`
attribute of any operator as shown in the tutorial here
<https://airflow.apache.org/tutorial.html#example-pipeline-definition>, and
there's also a `user_defined_macros` parameter as a dict you can pass while
creating the DAG object. Every key in that dictionary is made available in
the global jinja namespace.
https://airflow.apache.org/code.html#airflow.models.DAG
Max
On Wed, Jan 18, 2017 at 11:31 AM, Boris Tyukin <bo...@boristyukin.com>
wrote:
> Hi Guilherme,
>
> I guess it depends what exactly you want to do as not everything works with
> jinja.
>
> From documentation:
>
> https://pythonhosted.org/airflow/concepts.html#jinja-templating
>
> You can use Jinja templating with every parameter that is marked as
> “templated” in the documentation.
>
> You can open source code for operators and see what parameters are actually
> templated.
>
> For example, if you open source code for BashOperator
> https://pythonhosted.org/airflow/_modules/bash_operator.html#BashOperator
> you will see this line of code:
> template_fields = ('bash_command', 'env')
>
> It means only bash_command and env will work with jinja templates.
>
> On Wed, Jan 18, 2017 at 12:45 PM, Guilherme Marthe <
> guilherme.marthe@enjoei.com.br> wrote:
>
> > Hey folks!
> >
> > I am trying to write a dag that works well with jinja templating, since
> > through my study of the documentation, is the best way to ensure
> > compatibility with the back-fill function.
> >
> > Are you guys aware of any examples online with this functionality
> working?
> > The, documentation is still a bit dry on insightful examples, and I am a
> > noobie developer, so I am trying to make the best use of the tool in the
> > "correct way".
> >
> > Thank you in advance for any resources you guys can share!
> >
> > Gui
> >
> > Ps: I ve been told I can email you guys with questions like these :) Hope
> > this is not a hassle ^_^
> >
>
Re: Good jinja templating in airflow examples
Posted by Boris Tyukin <bo...@boristyukin.com>.
Hi Guilherme,
I guess it depends what exactly you want to do as not everything works with
jinja.
From documentation:
https://pythonhosted.org/airflow/concepts.html#jinja-templating
You can use Jinja templating with every parameter that is marked as
“templated” in the documentation.
You can open source code for operators and see what parameters are actually
templated.
For example, if you open source code for BashOperator
https://pythonhosted.org/airflow/_modules/bash_operator.html#BashOperator
you will see this line of code:
template_fields = ('bash_command', 'env')
It means only bash_command and env will work with jinja templates.
On Wed, Jan 18, 2017 at 12:45 PM, Guilherme Marthe <
guilherme.marthe@enjoei.com.br> wrote:
> Hey folks!
>
> I am trying to write a dag that works well with jinja templating, since
> through my study of the documentation, is the best way to ensure
> compatibility with the back-fill function.
>
> Are you guys aware of any examples online with this functionality working?
> The, documentation is still a bit dry on insightful examples, and I am a
> noobie developer, so I am trying to make the best use of the tool in the
> "correct way".
>
> Thank you in advance for any resources you guys can share!
>
> Gui
>
> Ps: I ve been told I can email you guys with questions like these :) Hope
> this is not a hassle ^_^
>