You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Edward Tomazic <ET...@Scentregroup.com> on 2018/01/31 08:05:15 UTC
XCom and Sub DAGs
Hi,
Have a general question re the use of XCom between Sub DAGs.
I have the following DAG structure:
dag.task-1 (SubDagOperator)
subdag1.task-1 (PythonOperator) >> subdag1.task-2 (SubDagOperator)
subdag2.task-3 (BigQueryOperator) >> subdag2.task-4 (BigQueryOperator)
subdag1.task-1 returns a value which is automatically pushed to XCom with a key of "return_value". I then try to pull the value from subdag2.task-3 with the following template:
{{ task_instance.xcom_pull(task_ids='task-1', key='return_value') }}
The template returns the value "None".
Can XCom be used to communicate between tasks of different DAGs? And if so, what am I doing wrong?
Kind regards,
Edward Tomazic
Data W/H Solution Architect
Scentre Group IT Data Management
________________________________
SCENTRE GROUP
T
+61 2 9358 7887
F
+61 2 9028 8500
M
+61 437 272 696
E
etomazic@scentregroup.com<ma...@scentregroup.com>
FB
Scentre Group<https://www.facebook.com/#!/Westfield?fref=ts>
W
scentregroup.com<http://www.scentregroup.com>
[https://res.cloudinary.com/scentre-group-au/image/upload/v1510274486/email/logo.jpg]<http://www.scentregroup.com/>
This e-mail may contain information which is confidential, subject to legal privilege and /or protected by copyright. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy all copies of this email.
RE: XCom and Sub DAGs
Posted by Edward Tomazic <ET...@Scentregroup.com>.
Hello Wilson,
I have now included the dag_id and it works like a charm - thankyou.
Very powerful, as I can also pass a dictionary of values!
BTW, when do you suppose the doc_md markup documentation will be rendered in sub DAGs and their tasks?
Regards,
Edward.
-----Original Message-----
From: Wilson Lian [mailto:wwlian@google.com.INVALID]
Sent: Thursday, 1 February 2018 8:45 AM
To: dev@airflow.incubator.apache.org
Subject: Re: XCom and Sub DAGs
By default xcom_pull assumes you're pulling from the same DAG ID. You can override this with the dag_id kwarg in xcom_pull.
https://github.com/apache/incubator-airflow/blob/6ee4bbd4b1bc4b3f275f7946e2bcdd123970e2dd/airflow/models.py#L1829
best,
Wilson
On Wed, Jan 31, 2018 at 12:05 AM, Edward Tomazic <ET...@scentregroup.com>
wrote:
> Hi,
>
> Have a general question re the use of XCom between Sub
> DAGs.
>
>
>
> I have the following DAG structure:
>
>
>
> dag.task-1 (SubDagOperator)
>
> subdag1.task-1 (PythonOperator) >> subdag1.task-2 (SubDagOperator)
>
> subdag2.task-3
> (BigQueryOperator) >> subdag2.task-4 (BigQueryOperator)
>
>
>
> subdag1.task-1 returns a value which is automatically pushed to XCom
> with a key of “return_value”. I then try to pull the value from
> subdag2.task-3 with the following template:
>
>
>
> {{ task_instance*.*xcom_pull(task_ids*=*'task-1',
> key*=*'return_value') }}
>
>
>
> The template returns the value “None”.
>
>
>
> Can XCom be used to communicate between tasks of different DAGs? And
> if so, what am I doing wrong?
>
>
>
> Kind regards,
>
>
>
> *Edward Tomazic*
>
> Data W/H Solution Architect
>
> Scentre Group IT Data Management
> ------------------------------
>
> SCENTRE GROUP
>
> T
>
> +61 2 9358 7887 <+61%202%209358%207887>
>
> F
>
> +61 2 9028 8500 <+61%202%209028%208500>
>
> M
>
> +61 437 272 696 <+61%20437%20272%20696>
>
> E
>
> etomazic@scentregroup.com
>
> FB
>
> Scentre Group <https://www.facebook.com/#!/Westfield?fref=ts>
>
> W
>
> scentregroup.com <http://www.scentregroup.com>
>
>
>
> [image:
> https://res.cloudinary.com/scentre-group-au/image/upload/v1510274486/e
> mail/logo.jpg]
> <http://www.scentregroup.com/>
>
>
>
> This e-mail may contain information which is confidential, subject to
> legal privilege and /or protected by copyright. If you are not the
> intended recipient (or have received this e-mail in error) please
> notify the sender immediately and destroy all copies of this email.
>
This e-mail may contain information which is confidential, subject to legal privilege and /or protected by copyright. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy all copies of this email.
Re: XCom and Sub DAGs
Posted by Wilson Lian <ww...@google.com.INVALID>.
By default xcom_pull assumes you're pulling from the same DAG ID. You can
override this with the dag_id kwarg in xcom_pull.
https://github.com/apache/incubator-airflow/blob/6ee4bbd4b1bc4b3f275f7946e2bcdd123970e2dd/airflow/models.py#L1829
best,
Wilson
On Wed, Jan 31, 2018 at 12:05 AM, Edward Tomazic <ET...@scentregroup.com>
wrote:
> Hi,
>
> Have a general question re the use of XCom between Sub
> DAGs.
>
>
>
> I have the following DAG structure:
>
>
>
> dag.task-1 (SubDagOperator)
>
> subdag1.task-1 (PythonOperator) >> subdag1.task-2 (SubDagOperator)
>
> subdag2.task-3
> (BigQueryOperator) >> subdag2.task-4 (BigQueryOperator)
>
>
>
> subdag1.task-1 returns a value which is automatically pushed to XCom with
> a key of “return_value”. I then try to pull the value from subdag2.task-3
> with the following template:
>
>
>
> {{ task_instance*.*xcom_pull(task_ids*=*'task-1', key*=*'return_value') }}
>
>
>
> The template returns the value “None”.
>
>
>
> Can XCom be used to communicate between tasks of different DAGs? And if
> so, what am I doing wrong?
>
>
>
> Kind regards,
>
>
>
> *Edward Tomazic*
>
> Data W/H Solution Architect
>
> Scentre Group IT Data Management
> ------------------------------
>
> SCENTRE GROUP
>
> T
>
> +61 2 9358 7887 <+61%202%209358%207887>
>
> F
>
> +61 2 9028 8500 <+61%202%209028%208500>
>
> M
>
> +61 437 272 696 <+61%20437%20272%20696>
>
> E
>
> etomazic@scentregroup.com
>
> FB
>
> Scentre Group <https://www.facebook.com/#!/Westfield?fref=ts>
>
> W
>
> scentregroup.com <http://www.scentregroup.com>
>
>
>
> [image:
> https://res.cloudinary.com/scentre-group-au/image/upload/v1510274486/email/logo.jpg]
> <http://www.scentregroup.com/>
>
>
>
> This e-mail may contain information which is confidential, subject to
> legal privilege and /or protected by copyright. If you are not the intended
> recipient (or have received this e-mail in error) please notify the sender
> immediately and destroy all copies of this email.
>