You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/14 21:01:20 UTC
[GitHub] [airflow] josh-fell commented on a diff in pull request #24452: Update dag-run.rst to ref dag_run.conf
josh-fell commented on code in PR #24452:
URL: https://github.com/apache/airflow/pull/24452#discussion_r897311128
##########
docs/apache-airflow/dag-run.rst:
##########
@@ -265,8 +265,32 @@ Example of a parameterized DAG:
dag=dag,
)
+Example of a parameterized DAG that reads the configuration from the context:
+
+.. code-block:: python
+
+ import pendulum
+
+ from airflow import DAG
+ from airflow.operators.python import PythonOperator
+
+ dag = DAG(
+ "example_parameterized_dag",
+ schedule_interval=None,
+ start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
+ catchup=False,
+ )
+
+ def print_conf(**context):
+ conf1 = context['dag_run'].conf['conf1']
+ print(conf1)
+
+ parameterized_task = PythonOperator(
+ task_id="parameterized_task",
+ python_callable=print_conf,
+ dag=dag,
+ )
-**Note**: The parameters from ``dag_run.conf`` can only be used in a template field of an operator.
Review Comment:
I think this note is still worthwhile but could use an addition of more context rather than deleting entirely. If you choose to access `dag_run.conf` in a Jinja expression then it can only be done in a template field of an operator. WDYT?
##########
docs/apache-airflow/dag-run.rst:
##########
@@ -265,8 +265,32 @@ Example of a parameterized DAG:
dag=dag,
)
+Example of a parameterized DAG that reads the configuration from the context:
+
+.. code-block:: python
+
+ import pendulum
+
+ from airflow import DAG
+ from airflow.operators.python import PythonOperator
+
+ dag = DAG(
+ "example_parameterized_dag",
+ schedule_interval=None,
+ start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
+ catchup=False,
+ )
+
+ def print_conf(**context):
+ conf1 = context['dag_run'].conf['conf1']
+ print(conf1)
+
+ parameterized_task = PythonOperator(
+ task_id="parameterized_task",
+ python_callable=print_conf,
+ dag=dag,
+ )
Review Comment:
```suggestion
import pendulum
from airflow import DAG
dag = DAG("example_parameterized_dag",
schedule_interval=None,
start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
catchup=False,
)
@dag.task(task_id="parameterized_task")
def print_conf(dag_run=None):
conf1 = dag_run.conf["conf1"]
print(conf1)
print_conf()
```
We've moved away from using `PythonOperator` with preference for the TaskFlow API. I don't think the docs overall reflect this (...yet) but the example DAGs were all updated with this in mind. Also with the TaskFlow API you can [access context variables directly](https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html#accessing-context-variables-in-decorated-tasks) without needing `kwargs`.
##########
docs/apache-airflow/dag-run.rst:
##########
@@ -265,8 +265,32 @@ Example of a parameterized DAG:
dag=dag,
)
+Example of a parameterized DAG that reads the configuration from the context:
+
+.. code-block:: python
+
+ import pendulum
+
+ from airflow import DAG
+ from airflow.operators.python import PythonOperator
+
+ dag = DAG(
+ "example_parameterized_dag",
+ schedule_interval=None,
+ start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
+ catchup=False,
+ )
+
+ def print_conf(**context):
+ conf1 = context['dag_run'].conf['conf1']
+ print(conf1)
+
+ parameterized_task = PythonOperator(
+ task_id="parameterized_task",
+ python_callable=print_conf,
+ dag=dag,
+ )
Review Comment:
```suggestion
import pendulum
from airflow import DAG
dag = DAG("example_parameterized_dag",
schedule_interval=None,
start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
catchup=False,
)
@dag.task(task_id="parameterized_task")
def print_conf(dag_run=None):
conf1 = dag_run.conf["conf1"]
print(conf1)
print_conf()
```
We've moved away from using `PythonOperator` with preference for the TaskFlow API. I don't think the docs overall reflect this (...yet) but the example DAGs were all updated with this in mind. Also with the TaskFlow API you can [access context variables directly](https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html#accessing-context-variables-in-decorated-tasks) without needing `kwargs`. I suppose this is could be argued as a style preference but helpful to have another example in the docs IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org