You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Chais (via GitHub)" <gi...@apache.org> on 2025/11/13 15:31:46 UTC
[D] Using parameter to select from dict prevents parsing [airflow]
GitHub user Chais created a discussion: Using parameter to select from dict prevents parsing
I have a workflow where I want to run a backup process from an active machine to a hot standby. Depending on the environment (development or production) these are different machines. Here's a minimal working example of what I had in mind:
```py
from datetime import datetime
from airflow.decorators import dag, task
from airflow.models.param import Param
@dag(
"playground",
"Try things",
schedule=None,
start_date=datetime.now(),
params={"selector": Param(type="string", title="Selector", enum=["a", "b"], default="a", description="Pick one")},
)
def playground():
values = {"a": "something", "b": "something else"}
@task.python
def task_print(value: str):
print(value)
task_print(values["{{ params.selector }}"])
playground()
if __name__ == "__main__":
playground().test()
```
However, with this approach I get the following error during serialisation:
```
File "/path/to/dag/dir/dag_playground.py", line 28, in playground
task_print(values["{{ params.selector }}"])
~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '{{ params.selector }}'
```
This indicates that the template isn't rendered during serialisation.
Any ideas how to circumvent this issue?
GitHub link: https://github.com/apache/airflow/discussions/58275
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Using parameter to select from dict prevents parsing [airflow]
Posted by "aayostem (via GitHub)" <gi...@apache.org>.
GitHub user aayostem edited a comment on the discussion: Using parameter to select from dict prevents parsing
Helo @Chais, template strings cannot be used inside Python expressions during DAG serialization. The templating happens at runtime, but the DAG parsing/serialization happens before that.
what you can do it to use ti - task Instance- in a python function
`
from datetime import datetime
from airflow.decorators import dag, task
from airflow.models.param import Param
@dag(
"playground",
"Try things",
schedule=None,
start_date=datetime.now(),
params={
"selector": Param(
type="string",
title="Selector",
enum=["a", "b"],
default="a",
description="Pick one"
)
},
)
def playground():
values = {"a": "something", "b": "something else"}
@task.python
def task_print(**kwargs):
ti = kwargs['ti']
dag_run = kwargs['dag_run']
selector = dag_run.conf.get('selector', 'a')
value = values[selector]
print(value)
task_print()
playground_dag = playground()
`
i hope this help
GitHub link: https://github.com/apache/airflow/discussions/58275#discussioncomment-14966932
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Using parameter to select from dict prevents parsing [airflow]
Posted by "aayostem (via GitHub)" <gi...@apache.org>.
GitHub user aayostem edited a comment on the discussion: Using parameter to select from dict prevents parsing
Helo @Chais, template strings cannot be used inside Python expressions during DAG serialization. The templating happens at runtime, but the DAG parsing/serialization happens before that.
what you can do it to use ti - task Instance- in a python function
```python
from datetime import datetime
from airflow.decorators import dag, task
from airflow.models.param import Param
@dag(
"playground",
"Try things",
schedule=None,
start_date=datetime.now(),
params={
"selector": Param(
type="string",
title="Selector",
enum=["a", "b"],
default="a",
description="Pick one"
)
},
)
def playground():
values = {"a": "something", "b": "something else"}
@task.python
def task_print(**kwargs):
ti = kwargs['ti']
dag_run = kwargs['dag_run']
selector = dag_run.conf.get('selector', 'a')
value = values[selector]
print(value)
task_print()
playground_dag = playground()
````
i hope this help
GitHub link: https://github.com/apache/airflow/discussions/58275#discussioncomment-14966932
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Using parameter to select from dict prevents parsing [airflow]
Posted by "Chais (via GitHub)" <gi...@apache.org>.
GitHub user Chais closed a discussion: Using parameter to select from dict prevents parsing
I have a workflow where I want to run a backup process from an active machine to a hot standby. Depending on the environment (development or production) these are different machines. Here's a minimal working example of what I had in mind:
```py
from datetime import datetime
from airflow.decorators import dag, task
from airflow.models.param import Param
@dag(
"playground",
"Try things",
schedule=None,
start_date=datetime.now(),
params={"selector": Param(type="string", title="Selector", enum=["a", "b"], default="a", description="Pick one")},
)
def playground():
values = {"a": "something", "b": "something else"}
@task.python
def task_print(value: str):
print(value)
task_print(values["{{ params.selector }}"])
playground()
if __name__ == "__main__":
playground().test()
```
However, with this approach I get the following error during serialisation:
```
File "/path/to/dag/dir/dag_playground.py", line 28, in playground
task_print(values["{{ params.selector }}"])
~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '{{ params.selector }}'
```
This indicates that the template isn't rendered during serialisation.
Any ideas how to circumvent this issue?
GitHub link: https://github.com/apache/airflow/discussions/58275
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Using parameter to select from dict prevents parsing [airflow]
Posted by "aayostem (via GitHub)" <gi...@apache.org>.
GitHub user aayostem added a comment to the discussion: Using parameter to select from dict prevents parsing
Helo @Chais, template strings cannot be used inside Python expressions during DAG serialization. The templating happens at runtime, but the DAG parsing/serialization happens before that.
what you can do it to use ti - task Instance- in a python function
from datetime import datetime
from airflow.decorators import dag, task
from airflow.models.param import Param
@dag(
"playground",
"Try things",
schedule=None,
start_date=datetime.now(),
params={
"selector": Param(
type="string",
title="Selector",
enum=["a", "b"],
default="a",
description="Pick one"
)
},
)
def playground():
values = {"a": "something", "b": "something else"}
@task.python
def task_print(**kwargs):
ti = kwargs['ti']
dag_run = kwargs['dag_run']
selector = dag_run.conf.get('selector', 'a')
value = values[selector]
print(value)
task_print()
playground_dag = playground()`
`
i hope this help
GitHub link: https://github.com/apache/airflow/discussions/58275#discussioncomment-14966932
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org