You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Or Sher <or...@gmail.com> on 2016/06/15 13:49:24 UTC
Iterating task xcom keys inside template
Hi all,
For the purpose of data validation tests I have two adjacent operators:
Python - Parsing predefined json of tests and saves the result as an xcom
where each key is a test id.
Email - I'd like to be able to iterate over all keys from the previous task
inside the predefined email template so I won't need to change it every
time I change the tests json.
It seems like it's not supported. Is that right?
What are my choices?
If I won't find anything else, I assume I can always use one key for all of
the tests combined into one json.
Re: Iterating task xcom keys inside template
Posted by Maxime Beauchemin <ma...@gmail.com>.
Try changing:
{% for key, value in task_instance.xcom_pull(task_
ids='data_validation_tests', key='tests_results') %}
to
{% for key, value in task_instance.xcom_pull(task_
ids='data_validation_tests', key='tests_results').items() %}
Maybe?
Max
On Thu, Jun 16, 2016 at 11:51 AM, Jeremiah Lowin <jl...@apache.org> wrote:
> Or, I'm afraid my jinja-fu is sorely lacking and I'm not going to be much
> help to you here... perhaps Maxime or one of the other devs is more
> experienced with templates and can offer some advice?
>
> Apologies!
> Jeremiah
>
> On Thu, Jun 16, 2016 at 2:40 AM Or Sher <or...@gmail.com> wrote:
>
>> Thanks Jeremiah!
>>
>> Looking at the code it seems like get_many can return only many xcom
>> values
>> from different task_ids and not different keys from the same task_id.
>> But it doesn't really matter as I would really like to do the iteration
>> somehow inside the template.
>> I feel like building the template inside the previous python operator is
>> kind of missing the whole point of templates as I can already create the
>> whole "already rendered" email message at the same time.
>>
>> I tried pushing one big json xcom to reflect all of the tests, but now
>> I'm struggling
>> with iterating string json inside the template.
>> The following template fails on the "for" line:
>> VALIDATION_EMAIL_CONTENT = """
>> <b>Date:</b> {{ ds }}<br>
>> <p>
>> <table>
>> {% for key, value in
>> task_instance.xcom_pull(task_ids='data_validation_tests',
>> key='tests_results') %}
>> {{ key|string() }}
>> {{ value|string() }}
>> {% endfor %}
>> ..
>> ..
>>
>> File "<template>", line 5, in top-level template code
>> ValueError: too many values to unpack
>>
>> I'm guessing it's because the xcom_pull returns a string but I didn't
>> managed to cast it to a json yet..
>>
>> On Wed, Jun 15, 2016 at 5:33 PM Jeremiah Lowin <jl...@apache.org> wrote:
>>
>> > Hi Or,
>> >
>> > There is support for pulling multiple XComs at once (see
>> XCom.get_many())
>> > but it won't be directly available inside a jinja template, where you
>> you
>> > are typically calling ti.xcom_pull() and returning a single XCom at a
>> time,
>> > using a key you know in advance.
>> >
>> > I would suggest that you do this processing either inside your existing
>> > PythonOperator or in a subsequent PythonOperator that calls
>> XCom.get_many()
>> > and produces the final template for your email operator. It is
>> > theoretically possible for you to do this in your email template using
>> > template macros, but I only have a basic knowledge of that (and it's a
>> > convoluted process) so I don't want to give you bad advice. If you can
>> do
>> > your preprocessing in your PythonOperator I think you will be happier,
>> > especially because the result will be visible in the Airflow UI and
>> logs.
>> >
>> > Best,
>> > Jeremiah
>> >
>> > On Wed, Jun 15, 2016 at 9:49 AM Or Sher <or...@gmail.com> wrote:
>> >
>> > > Hi all,
>> > >
>> > > For the purpose of data validation tests I have two adjacent
>> operators:
>> > > Python - Parsing predefined json of tests and saves the result as an
>> xcom
>> > > where each key is a test id.
>> > >
>> > > Email - I'd like to be able to iterate over all keys from the previous
>> > task
>> > > inside the predefined email template so I won't need to change it
>> every
>> > > time I change the tests json.
>> > >
>> > > It seems like it's not supported. Is that right?
>> > > What are my choices?
>> > > If I won't find anything else, I assume I can always use one key for
>> all
>> > of
>> > > the tests combined into one json.
>> > >
>> >
>>
>
Re: Iterating task xcom keys inside template
Posted by Chris Riccomini <cr...@apache.org>.
I *think* it might be because xcom_pull only returns one element per task:
https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/example_xcom.py#L35
You're calling with just a single task_id, so I think you should be doing
`for element in ...` rather than `for key, value in ...`.
On Thu, Jun 16, 2016 at 11:51 AM, Jeremiah Lowin <jl...@apache.org> wrote:
> Or, I'm afraid my jinja-fu is sorely lacking and I'm not going to be much
> help to you here... perhaps Maxime or one of the other devs is more
> experienced with templates and can offer some advice?
>
> Apologies!
> Jeremiah
>
> On Thu, Jun 16, 2016 at 2:40 AM Or Sher <or...@gmail.com> wrote:
>
> > Thanks Jeremiah!
> >
> > Looking at the code it seems like get_many can return only many xcom
> values
> > from different task_ids and not different keys from the same task_id.
> > But it doesn't really matter as I would really like to do the iteration
> > somehow inside the template.
> > I feel like building the template inside the previous python operator is
> > kind of missing the whole point of templates as I can already create the
> > whole "already rendered" email message at the same time.
> >
> > I tried pushing one big json xcom to reflect all of the tests, but now
> > I'm struggling
> > with iterating string json inside the template.
> > The following template fails on the "for" line:
> > VALIDATION_EMAIL_CONTENT = """
> > <b>Date:</b> {{ ds }}<br>
> > <p>
> > <table>
> > {% for key, value in
> > task_instance.xcom_pull(task_ids='data_validation_tests',
> > key='tests_results') %}
> > {{ key|string() }}
> > {{ value|string() }}
> > {% endfor %}
> > ..
> > ..
> >
> > File "<template>", line 5, in top-level template code
> > ValueError: too many values to unpack
> >
> > I'm guessing it's because the xcom_pull returns a string but I didn't
> > managed to cast it to a json yet..
> >
> > On Wed, Jun 15, 2016 at 5:33 PM Jeremiah Lowin <jl...@apache.org>
> wrote:
> >
> > > Hi Or,
> > >
> > > There is support for pulling multiple XComs at once (see
> XCom.get_many())
> > > but it won't be directly available inside a jinja template, where you
> you
> > > are typically calling ti.xcom_pull() and returning a single XCom at a
> > time,
> > > using a key you know in advance.
> > >
> > > I would suggest that you do this processing either inside your existing
> > > PythonOperator or in a subsequent PythonOperator that calls
> > XCom.get_many()
> > > and produces the final template for your email operator. It is
> > > theoretically possible for you to do this in your email template using
> > > template macros, but I only have a basic knowledge of that (and it's a
> > > convoluted process) so I don't want to give you bad advice. If you can
> do
> > > your preprocessing in your PythonOperator I think you will be happier,
> > > especially because the result will be visible in the Airflow UI and
> logs.
> > >
> > > Best,
> > > Jeremiah
> > >
> > > On Wed, Jun 15, 2016 at 9:49 AM Or Sher <or...@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > For the purpose of data validation tests I have two adjacent
> operators:
> > > > Python - Parsing predefined json of tests and saves the result as an
> > xcom
> > > > where each key is a test id.
> > > >
> > > > Email - I'd like to be able to iterate over all keys from the
> previous
> > > task
> > > > inside the predefined email template so I won't need to change it
> every
> > > > time I change the tests json.
> > > >
> > > > It seems like it's not supported. Is that right?
> > > > What are my choices?
> > > > If I won't find anything else, I assume I can always use one key for
> > all
> > > of
> > > > the tests combined into one json.
> > > >
> > >
> >
>
Re: Iterating task xcom keys inside template
Posted by Jeremiah Lowin <jl...@apache.org>.
Or, I'm afraid my jinja-fu is sorely lacking and I'm not going to be much
help to you here... perhaps Maxime or one of the other devs is more
experienced with templates and can offer some advice?
Apologies!
Jeremiah
On Thu, Jun 16, 2016 at 2:40 AM Or Sher <or...@gmail.com> wrote:
> Thanks Jeremiah!
>
> Looking at the code it seems like get_many can return only many xcom values
> from different task_ids and not different keys from the same task_id.
> But it doesn't really matter as I would really like to do the iteration
> somehow inside the template.
> I feel like building the template inside the previous python operator is
> kind of missing the whole point of templates as I can already create the
> whole "already rendered" email message at the same time.
>
> I tried pushing one big json xcom to reflect all of the tests, but now
> I'm struggling
> with iterating string json inside the template.
> The following template fails on the "for" line:
> VALIDATION_EMAIL_CONTENT = """
> <b>Date:</b> {{ ds }}<br>
> <p>
> <table>
> {% for key, value in
> task_instance.xcom_pull(task_ids='data_validation_tests',
> key='tests_results') %}
> {{ key|string() }}
> {{ value|string() }}
> {% endfor %}
> ..
> ..
>
> File "<template>", line 5, in top-level template code
> ValueError: too many values to unpack
>
> I'm guessing it's because the xcom_pull returns a string but I didn't
> managed to cast it to a json yet..
>
> On Wed, Jun 15, 2016 at 5:33 PM Jeremiah Lowin <jl...@apache.org> wrote:
>
> > Hi Or,
> >
> > There is support for pulling multiple XComs at once (see XCom.get_many())
> > but it won't be directly available inside a jinja template, where you you
> > are typically calling ti.xcom_pull() and returning a single XCom at a
> time,
> > using a key you know in advance.
> >
> > I would suggest that you do this processing either inside your existing
> > PythonOperator or in a subsequent PythonOperator that calls
> XCom.get_many()
> > and produces the final template for your email operator. It is
> > theoretically possible for you to do this in your email template using
> > template macros, but I only have a basic knowledge of that (and it's a
> > convoluted process) so I don't want to give you bad advice. If you can do
> > your preprocessing in your PythonOperator I think you will be happier,
> > especially because the result will be visible in the Airflow UI and logs.
> >
> > Best,
> > Jeremiah
> >
> > On Wed, Jun 15, 2016 at 9:49 AM Or Sher <or...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > For the purpose of data validation tests I have two adjacent operators:
> > > Python - Parsing predefined json of tests and saves the result as an
> xcom
> > > where each key is a test id.
> > >
> > > Email - I'd like to be able to iterate over all keys from the previous
> > task
> > > inside the predefined email template so I won't need to change it every
> > > time I change the tests json.
> > >
> > > It seems like it's not supported. Is that right?
> > > What are my choices?
> > > If I won't find anything else, I assume I can always use one key for
> all
> > of
> > > the tests combined into one json.
> > >
> >
>
Re: Iterating task xcom keys inside template
Posted by Or Sher <or...@gmail.com>.
Thanks Jeremiah!
Looking at the code it seems like get_many can return only many xcom values
from different task_ids and not different keys from the same task_id.
But it doesn't really matter as I would really like to do the iteration
somehow inside the template.
I feel like building the template inside the previous python operator is
kind of missing the whole point of templates as I can already create the
whole "already rendered" email message at the same time.
I tried pushing one big json xcom to reflect all of the tests, but now
I'm struggling
with iterating string json inside the template.
The following template fails on the "for" line:
VALIDATION_EMAIL_CONTENT = """
<b>Date:</b> {{ ds }}<br>
<p>
<table>
{% for key, value in
task_instance.xcom_pull(task_ids='data_validation_tests',
key='tests_results') %}
{{ key|string() }}
{{ value|string() }}
{% endfor %}
..
..
File "<template>", line 5, in top-level template code
ValueError: too many values to unpack
I'm guessing it's because the xcom_pull returns a string but I didn't
managed to cast it to a json yet..
On Wed, Jun 15, 2016 at 5:33 PM Jeremiah Lowin <jl...@apache.org> wrote:
> Hi Or,
>
> There is support for pulling multiple XComs at once (see XCom.get_many())
> but it won't be directly available inside a jinja template, where you you
> are typically calling ti.xcom_pull() and returning a single XCom at a time,
> using a key you know in advance.
>
> I would suggest that you do this processing either inside your existing
> PythonOperator or in a subsequent PythonOperator that calls XCom.get_many()
> and produces the final template for your email operator. It is
> theoretically possible for you to do this in your email template using
> template macros, but I only have a basic knowledge of that (and it's a
> convoluted process) so I don't want to give you bad advice. If you can do
> your preprocessing in your PythonOperator I think you will be happier,
> especially because the result will be visible in the Airflow UI and logs.
>
> Best,
> Jeremiah
>
> On Wed, Jun 15, 2016 at 9:49 AM Or Sher <or...@gmail.com> wrote:
>
> > Hi all,
> >
> > For the purpose of data validation tests I have two adjacent operators:
> > Python - Parsing predefined json of tests and saves the result as an xcom
> > where each key is a test id.
> >
> > Email - I'd like to be able to iterate over all keys from the previous
> task
> > inside the predefined email template so I won't need to change it every
> > time I change the tests json.
> >
> > It seems like it's not supported. Is that right?
> > What are my choices?
> > If I won't find anything else, I assume I can always use one key for all
> of
> > the tests combined into one json.
> >
>
Re: Iterating task xcom keys inside template
Posted by Jeremiah Lowin <jl...@apache.org>.
Hi Or,
There is support for pulling multiple XComs at once (see XCom.get_many())
but it won't be directly available inside a jinja template, where you you
are typically calling ti.xcom_pull() and returning a single XCom at a time,
using a key you know in advance.
I would suggest that you do this processing either inside your existing
PythonOperator or in a subsequent PythonOperator that calls XCom.get_many()
and produces the final template for your email operator. It is
theoretically possible for you to do this in your email template using
template macros, but I only have a basic knowledge of that (and it's a
convoluted process) so I don't want to give you bad advice. If you can do
your preprocessing in your PythonOperator I think you will be happier,
especially because the result will be visible in the Airflow UI and logs.
Best,
Jeremiah
On Wed, Jun 15, 2016 at 9:49 AM Or Sher <or...@gmail.com> wrote:
> Hi all,
>
> For the purpose of data validation tests I have two adjacent operators:
> Python - Parsing predefined json of tests and saves the result as an xcom
> where each key is a test id.
>
> Email - I'd like to be able to iterate over all keys from the previous task
> inside the predefined email template so I won't need to change it every
> time I change the tests json.
>
> It seems like it's not supported. Is that right?
> What are my choices?
> If I won't find anything else, I assume I can always use one key for all of
> the tests combined into one json.
>