You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/09/24 08:05:29 UTC
[GitHub] [airflow] bhavaniravi opened a new issue #18491: Parent DAG conf missing when using SubDagOperator
bhavaniravi opened a new issue #18491:
URL: https://github.com/apache/airflow/issues/18491
### Apache Airflow version
2.1.0
### Operating System
Debian GNU/Linux 10 (buster)
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other
### Deployment details
Astronomer
Breeze
### What happened
When running a DAG with a SubDag with `trigger_conf` as of 1.x, the conf is passed to the SubDAG. But since 2.0, the behavior is changed.
When sending `trigger_conf` from UI, it's not passed along to SubDags, whereas when the DAG is triggered from the terminal CLI, I was able to access the `conf` using `dag_run.conf`
### What you expected to happen
The conf of parent DAG should be passed along to all subdags
### How to reproduce
1. Use the following DAG
<details>
<summary>Click to expand!</summary>
```
# [START example_subdag_operator]
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.subdag import SubDagOperator
from airflow.utils.dates import days_ago
from datetime import datetime
DAG_NAME = 'example_subdag_operator'
args = {
'owner': 'airflow',
}
def subddag_python(dag_run, **kwargs):
print (kwargs)
print (f"{dag_run.dag_id} {dag_run.conf}")
def subdag(parent_dag_name, child_dag_name, args):
"""
Generate a DAG to be used as a subdag.
:param str parent_dag_name: Id of the parent DAG
:param str child_dag_name: Id of the child DAG
:param dict args: Default arguments to provide to the subdag
:return: DAG to use as a subdag
:rtype: airflow.models.DAG
"""
dag_subdag = DAG(
dag_id=f'{parent_dag_name}.{child_dag_name}',
default_args=args,
start_date=datetime(2021, 1, 1),
catchup=False,
schedule_interval="@daily",
)
tn = PythonOperator(
task_id=f'subdag_python',
python_callable=subddag_python, # make sure you don't include the () of the function
provide_context=True,
dag=dag_subdag
)
return dag_subdag
def simple_python(dag_run):
print (f"{dag_run.dag_id} {dag_run.conf}")
with DAG(
dag_id=DAG_NAME, default_args=args, start_date=days_ago(2), schedule_interval="@once", tags=['example']
) as dag:
tn = PythonOperator(
task_id=f'simple_python',
python_callable=simple_python, # make sure you don't include the () of the function
provide_context=True,
)
section_1 = SubDagOperator(
task_id='section-1',
subdag=subdag(DAG_NAME, 'section-1', args),
# conf="{{ dag_run.conf }}"
)
tn >> section_1
```
</details>
2. Trigger the dag with `conf` from Airflow UI `{"a": 1}`
You will see that the SubDag python operator doesn't print any conf
3. Trigger the same dag with conf from Airflow CLI, you will the conf dict printed
```
airflow dags trigger example_subdag_operator --conf="{\"1\":1}"
```
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] khalidmammadov commented on issue #18491: Parent DAG trigger_conf missing in SubDagOperator (only when triggered from UI)
Posted by GitBox <gi...@apache.org>.
khalidmammadov commented on issue #18491:
URL: https://github.com/apache/airflow/issues/18491#issuecomment-930232935
In that case is there "deprecated feature change request" (or deprecated or something like that) label to put on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bhavaniravi commented on issue #18491: Parent DAG trigger_conf missing in SubDagOperator (only when triggered from UI)
Posted by GitBox <gi...@apache.org>.
bhavaniravi commented on issue #18491:
URL: https://github.com/apache/airflow/issues/18491#issuecomment-926568356
Can someone help me what would be a good starting point to fix this issue?
I am going back and forth between `subdag.py` and `trigger_dag.py` but not sure where the actual conf gets set for the subdag's dagrun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #18491: Parent DAG trigger_conf missing in SubDagOperator (only when triggered from UI)
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #18491:
URL: https://github.com/apache/airflow/issues/18491#issuecomment-930801560
If this is an easy simple fix I think it's fine to fix it if @bhavaniravi wish to work on it.
We also have https://github.com/apache/airflow/issues/18559 which is related
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18491: Parent DAG trigger_conf missing in SubDagOperator (only when triggered from UI)
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18491:
URL: https://github.com/apache/airflow/issues/18491#issuecomment-930562977
Yeah. If we have workaround, I'd say we should not invest in SubDag . Shall we close the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bhavaniravi commented on issue #18491: Parent DAG trigger_conf missing in SubDagOperator (only when triggered from UI)
Posted by GitBox <gi...@apache.org>.
bhavaniravi commented on issue #18491:
URL: https://github.com/apache/airflow/issues/18491#issuecomment-930235392
A workaround for this issue is to use XCom push from parent dag and use XCom pull in subdag
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #18491: Parent DAG trigger_conf missing in SubDagOperator (only when triggered from UI)
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #18491:
URL: https://github.com/apache/airflow/issues/18491#issuecomment-926665179
`SubDagOperator` is deprecated. Not sure if we still work on deprecated items
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18491: Parent DAG trigger_conf missing in SubDagOperator (only when triggered from UI)
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18491:
URL: https://github.com/apache/airflow/issues/18491#issuecomment-930562977
Yeah. If we have workaround, I'd say we should not invest in SubDag . Shall we close the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] khalidmammadov commented on issue #18491: Parent DAG trigger_conf missing in SubDagOperator (only when triggered from UI)
Posted by GitBox <gi...@apache.org>.
khalidmammadov commented on issue #18491:
URL: https://github.com/apache/airflow/issues/18491#issuecomment-930232935
In that case is there "deprecated feature change request" (or deprecated or something like that) label to put on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bhavaniravi commented on issue #18491: Parent DAG trigger_conf missing in SubDagOperator (only when triggered from UI)
Posted by GitBox <gi...@apache.org>.
bhavaniravi commented on issue #18491:
URL: https://github.com/apache/airflow/issues/18491#issuecomment-930235392
A workaround for this issue is to use XCom push from parent dag and use XCom pull in subdag
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org