You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/14 19:50:03 UTC
[GitHub] [airflow] patricker opened a new pull request, #24452: Update dag-run.rst to ref dag_run.conf
patricker opened a new pull request, #24452:
URL: https://github.com/apache/airflow/pull/24452
Docs made it sound like there was no way to access dag_run.conf outside of a templated field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #24452: Update dag-run.rst to ref dag_run.conf
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #24452:
URL: https://github.com/apache/airflow/pull/24452#discussion_r903696338
##########
docs/apache-airflow/dag-run.rst:
##########
@@ -264,9 +264,33 @@ Example of a parameterized DAG:
bash_command="echo value: {{ dag_run.conf['conf1'] }}",
dag=dag,
)
+
+**Note**: The parameters from ``dag_run.conf`` can only be used in a template field of an operator.
Review Comment:
Let’s make this a normal paragraph; I feel it reads better combined with the newly added content. Something like this:
> Note that the parameters from […]. Example of a parametrized DAG […]:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24452: Update dag-run.rst to ref dag_run.conf
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24452:
URL: https://github.com/apache/airflow/pull/24452#issuecomment-1159814737
Static checks failing. Please fix them @patricker and rebase. Installing pre-commits helps as a lot of problems are fixed for you automatically when you commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on a diff in pull request #24452: Update dag-run.rst to ref dag_run.conf
Posted by GitBox <gi...@apache.org>.
josh-fell commented on code in PR #24452:
URL: https://github.com/apache/airflow/pull/24452#discussion_r897311128
##########
docs/apache-airflow/dag-run.rst:
##########
@@ -265,8 +265,32 @@ Example of a parameterized DAG:
dag=dag,
)
+Example of a parameterized DAG that reads the configuration from the context:
+
+.. code-block:: python
+
+ import pendulum
+
+ from airflow import DAG
+ from airflow.operators.python import PythonOperator
+
+ dag = DAG(
+ "example_parameterized_dag",
+ schedule_interval=None,
+ start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
+ catchup=False,
+ )
+
+ def print_conf(**context):
+ conf1 = context['dag_run'].conf['conf1']
+ print(conf1)
+
+ parameterized_task = PythonOperator(
+ task_id="parameterized_task",
+ python_callable=print_conf,
+ dag=dag,
+ )
-**Note**: The parameters from ``dag_run.conf`` can only be used in a template field of an operator.
Review Comment:
I think this note is still worthwhile but could use an addition of more context rather than deleting entirely. If you choose to access `dag_run.conf` in a Jinja expression then it can only be done in a template field of an operator. WDYT?
##########
docs/apache-airflow/dag-run.rst:
##########
@@ -265,8 +265,32 @@ Example of a parameterized DAG:
dag=dag,
)
+Example of a parameterized DAG that reads the configuration from the context:
+
+.. code-block:: python
+
+ import pendulum
+
+ from airflow import DAG
+ from airflow.operators.python import PythonOperator
+
+ dag = DAG(
+ "example_parameterized_dag",
+ schedule_interval=None,
+ start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
+ catchup=False,
+ )
+
+ def print_conf(**context):
+ conf1 = context['dag_run'].conf['conf1']
+ print(conf1)
+
+ parameterized_task = PythonOperator(
+ task_id="parameterized_task",
+ python_callable=print_conf,
+ dag=dag,
+ )
Review Comment:
```suggestion
import pendulum
from airflow import DAG
dag = DAG("example_parameterized_dag",
schedule_interval=None,
start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
catchup=False,
)
@dag.task(task_id="parameterized_task")
def print_conf(dag_run=None):
conf1 = dag_run.conf["conf1"]
print(conf1)
print_conf()
```
We've moved away from using `PythonOperator` with preference for the TaskFlow API. I don't think the docs overall reflect this (...yet) but the example DAGs were all updated with this in mind. Also with the TaskFlow API you can [access context variables directly](https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html#accessing-context-variables-in-decorated-tasks) without needing `kwargs`.
##########
docs/apache-airflow/dag-run.rst:
##########
@@ -265,8 +265,32 @@ Example of a parameterized DAG:
dag=dag,
)
+Example of a parameterized DAG that reads the configuration from the context:
+
+.. code-block:: python
+
+ import pendulum
+
+ from airflow import DAG
+ from airflow.operators.python import PythonOperator
+
+ dag = DAG(
+ "example_parameterized_dag",
+ schedule_interval=None,
+ start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
+ catchup=False,
+ )
+
+ def print_conf(**context):
+ conf1 = context['dag_run'].conf['conf1']
+ print(conf1)
+
+ parameterized_task = PythonOperator(
+ task_id="parameterized_task",
+ python_callable=print_conf,
+ dag=dag,
+ )
Review Comment:
```suggestion
import pendulum
from airflow import DAG
dag = DAG("example_parameterized_dag",
schedule_interval=None,
start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
catchup=False,
)
@dag.task(task_id="parameterized_task")
def print_conf(dag_run=None):
conf1 = dag_run.conf["conf1"]
print(conf1)
print_conf()
```
We've moved away from using `PythonOperator` with preference for the TaskFlow API. I don't think the docs overall reflect this (...yet) but the example DAGs were all updated with this in mind. Also with the TaskFlow API you can [access context variables directly](https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html#accessing-context-variables-in-decorated-tasks) without needing `kwargs`. I suppose this is could be argued as a style preference but helpful to have another example in the docs IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed pull request #24452: Update dag-run.rst to ref dag_run.conf
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #24452: Update dag-run.rst to ref dag_run.conf
URL: https://github.com/apache/airflow/pull/24452
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] patricker commented on pull request #24452: Update dag-run.rst to ref dag_run.conf
Posted by GitBox <gi...@apache.org>.
patricker commented on PR #24452:
URL: https://github.com/apache/airflow/pull/24452#issuecomment-1163282294
I'll try and wrap this up in the next couple days, thanks for the feedback!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #24452: Update dag-run.rst to ref dag_run.conf
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #24452:
URL: https://github.com/apache/airflow/pull/24452#issuecomment-1207302323
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] patricker commented on a diff in pull request #24452: Update dag-run.rst to ref dag_run.conf
Posted by GitBox <gi...@apache.org>.
patricker commented on code in PR #24452:
URL: https://github.com/apache/airflow/pull/24452#discussion_r898213762
##########
docs/apache-airflow/dag-run.rst:
##########
@@ -265,8 +265,32 @@ Example of a parameterized DAG:
dag=dag,
)
+Example of a parameterized DAG that reads the configuration from the context:
+
+.. code-block:: python
+
+ import pendulum
+
+ from airflow import DAG
+ from airflow.operators.python import PythonOperator
+
+ dag = DAG(
+ "example_parameterized_dag",
+ schedule_interval=None,
+ start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
+ catchup=False,
+ )
+
+ def print_conf(**context):
+ conf1 = context['dag_run'].conf['conf1']
+ print(conf1)
+
+ parameterized_task = PythonOperator(
+ task_id="parameterized_task",
+ python_callable=print_conf,
+ dag=dag,
+ )
-**Note**: The parameters from ``dag_run.conf`` can only be used in a template field of an operator.
Review Comment:
I put it back in as-is. Now that there is more than one example, it has the context of being with a parameterized DAG example, so i think it makes more sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] patricker commented on a diff in pull request #24452: Update dag-run.rst to ref dag_run.conf
Posted by GitBox <gi...@apache.org>.
patricker commented on code in PR #24452:
URL: https://github.com/apache/airflow/pull/24452#discussion_r898214744
##########
docs/apache-airflow/dag-run.rst:
##########
@@ -265,8 +265,32 @@ Example of a parameterized DAG:
dag=dag,
)
+Example of a parameterized DAG that reads the configuration from the context:
+
+.. code-block:: python
+
+ import pendulum
+
+ from airflow import DAG
+ from airflow.operators.python import PythonOperator
+
+ dag = DAG(
+ "example_parameterized_dag",
+ schedule_interval=None,
+ start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
+ catchup=False,
+ )
+
+ def print_conf(**context):
+ conf1 = context['dag_run'].conf['conf1']
+ print(conf1)
+
+ parameterized_task = PythonOperator(
+ task_id="parameterized_task",
+ python_callable=print_conf,
+ dag=dag,
+ )
Review Comment:
I wrote it up slightly differently. Are you OK with my use of `context = get_current_context()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on a diff in pull request #24452: Update dag-run.rst to ref dag_run.conf
Posted by GitBox <gi...@apache.org>.
josh-fell commented on code in PR #24452:
URL: https://github.com/apache/airflow/pull/24452#discussion_r898346143
##########
docs/apache-airflow/dag-run.rst:
##########
@@ -265,8 +265,32 @@ Example of a parameterized DAG:
dag=dag,
)
+Example of a parameterized DAG that reads the configuration from the context:
+
+.. code-block:: python
+
+ import pendulum
+
+ from airflow import DAG
+ from airflow.operators.python import PythonOperator
+
+ dag = DAG(
+ "example_parameterized_dag",
+ schedule_interval=None,
+ start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
+ catchup=False,
+ )
+
+ def print_conf(**context):
+ conf1 = context['dag_run'].conf['conf1']
+ print(conf1)
+
+ parameterized_task = PythonOperator(
+ task_id="parameterized_task",
+ python_callable=print_conf,
+ dag=dag,
+ )
Review Comment:
`get_current_context()` was really purpose-built to [access context variables deep in the stack](https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html#accessing-context-variables-in-decorated-tasks) without having to pass them around from the task callable to other callables.
But, maybe another example could be added to access `dag_run.conf` in a callable that isn't a task using `get_current_context()`? I don't think more examples could hurt 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org