Pulling XCom from parent DAG in a sub DAG template

[Daniel Huang <dx...@gmail.com>]

Hey everyone,
I currently have a parent DAG that generates a filename based on the
current timestamp. That same filename needs to be used down the line in
multiple operators, including some subdag operators. So I was hoping to use
XComs to pass this filename along. It works fine for operators within the
parent DAG, but not for any operators in the sub DAGs because the dag_id
arg on xcom_pull() defaults to self.dag_id. I'm pulling this XCom value
from templates, so I don't want to have to hardcode a parent's dag id in it
because it should work for different parent dags. The sub dag template
would still have to make assumptions about what is available in xcom under
a task id, but I think that's alright?

So my question is if there is a way to programmatically specify the parent
dag id in an xcom pull from within a template or if there's a better way of
doing what I'm trying to achieve. Note, I also have sub dags within sub
dags that require the same filename.

If there is no way to this already, I did come across
https://issues.apache.org/jira/browse/AIRFLOW-54 and wondered if we need
something similar for the dag_id arg, like ti.xcom_pull(dag_id='@parent',
task_ids='foo').

Thanks,
Daniel

Re: Pulling XCom from parent DAG in a sub DAG template

[Daniel Huang <da...@upsight.com>]

Ah that did it, thanks!

On Wed, Feb 1, 2017 at 11:58 PM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:

> I think `dag.parent_dag.dag_id` should work, either in a template of
> through the context in a PythonOperator.
>
> Max
>
> On Wed, Feb 1, 2017 at 4:23 PM, Daniel Huang <dx...@gmail.com> wrote:
>
> > Hey everyone,
> > I currently have a parent DAG that generates a filename based on the
> > current timestamp. That same filename needs to be used down the line in
> > multiple operators, including some subdag operators. So I was hoping to
> use
> > XComs to pass this filename along. It works fine for operators within the
> > parent DAG, but not for any operators in the sub DAGs because the dag_id
> > arg on xcom_pull() defaults to self.dag_id. I'm pulling this XCom value
> > from templates, so I don't want to have to hardcode a parent's dag id in
> it
> > because it should work for different parent dags. The sub dag template
> > would still have to make assumptions about what is available in xcom
> under
> > a task id, but I think that's alright?
> >
> > So my question is if there is a way to programmatically specify the
> parent
> > dag id in an xcom pull from within a template or if there's a better way
> of
> > doing what I'm trying to achieve. Note, I also have sub dags within sub
> > dags that require the same filename.
> >
> > If there is no way to this already, I did come across
> > https://issues.apache.org/jira/browse/AIRFLOW-54 and wondered if we need
> > something similar for the dag_id arg, like ti.xcom_pull(dag_id='@parent',
> > task_ids='foo').
> >
> > Thanks,
> > Daniel
> >
>

Re: Pulling XCom from parent DAG in a sub DAG template

[Maxime Beauchemin <ma...@gmail.com>]

I think `dag.parent_dag.dag_id` should work, either in a template of
through the context in a PythonOperator.

Max

On Wed, Feb 1, 2017 at 4:23 PM, Daniel Huang <dx...@gmail.com> wrote:

> Hey everyone,
> I currently have a parent DAG that generates a filename based on the
> current timestamp. That same filename needs to be used down the line in
> multiple operators, including some subdag operators. So I was hoping to use
> XComs to pass this filename along. It works fine for operators within the
> parent DAG, but not for any operators in the sub DAGs because the dag_id
> arg on xcom_pull() defaults to self.dag_id. I'm pulling this XCom value
> from templates, so I don't want to have to hardcode a parent's dag id in it
> because it should work for different parent dags. The sub dag template
> would still have to make assumptions about what is available in xcom under
> a task id, but I think that's alright?
>
> So my question is if there is a way to programmatically specify the parent
> dag id in an xcom pull from within a template or if there's a better way of
> doing what I'm trying to achieve. Note, I also have sub dags within sub
> dags that require the same filename.
>
> If there is no way to this already, I did come across
> https://issues.apache.org/jira/browse/AIRFLOW-54 and wondered if we need
> something similar for the dag_id arg, like ti.xcom_pull(dag_id='@parent',
> task_ids='foo').
>
> Thanks,
> Daniel
>