You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "AbigailColwell (via GitHub)" <gi...@apache.org> on 2025/03/13 16:15:21 UTC
[I] `KubernetesPodOperator` does not properly handle templated volumes and volume_mounts [airflow]
AbigailColwell opened a new issue, #47738:
URL: https://github.com/apache/airflow/issues/47738
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
`apache-airflow-providers-cncf-kubernetes==10.3.0`
### Apache Airflow version
`apache-airflow==2.10.5`
### Operating System
macOs Sequioa 15.3.1
### Deployment
Virtualenv installation
### Deployment details
1. Create virtual environment with uv
- `uv venv --python 3.12`
- `source .venv/bin/activate`
- `uv pip install apache-airflow-providers-cncf-kubernetes`
### What happened
```log
Traceback (most recent call last):
File "/Users/acolwell/Code/airflow-dags/dynamic_volume_dag.py", line 51, in <module>
example_k8s_volume_template_error_dag = example_k8s_volume_template_error()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/acolwell/Code/airflow-dags/.venv/lib/python3.12/site-packages/airflow/models/dag.py", line 4307, in factory
f(**f_kwargs)
File "/Users/acolwell/Code/airflow-dags/dynamic_volume_dag.py", line 42, in example_k8s_volume_template_error
run_pod = KubernetesPodOperator(
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/acolwell/Code/airflow-dags/.venv/lib/python3.12/site-packages/airflow/models/baseoperator.py", line 506, in apply_defaults
result = func(self, **kwargs, default_args=default_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/acolwell/Code/airflow-dags/.venv/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 347, in __init__
volume_mounts = [convert_volume_mount(v) for v in volume_mounts] if volume_mounts else []
^^^^^^^^^^^^^
File "/Users/acolwell/Code/airflow-dags/.venv/lib/python3.12/site-packages/airflow/models/xcom_arg.py", line 329, in __iter__
raise TypeError("'XComArg' object is not iterable")
TypeError: 'XComArg' object is not iterable
```
### What you think should happen instead
The DAG should parse without issue and should properly set up the volumes and mounts when run
### How to reproduce
1. Set up virtual environment as described above
2. Create a dag `dynanic_volume_dag.py`
```python
"""
Example DAG showing error when using KubernetesPodOperator trying to template a volume_mount or volume
"""
from datetime import datetime
from airflow.decorators import dag, task
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
@dag(
schedule=None,
start_date=datetime(2023, 8, 28),
catchup=False,
)
def example_k8s_volume_template_error():
"""
Example DAG showing error when using KubernetesPodOperator trying to template a volume_mount or volume
"""
@task(multiple_outputs=True)
def generate_mount_config():
"""Generate the volume_mounts and volumes configuration. Assume this is a complex function that generates the configuration"""
return {
"volume_mounts": [
{
"name": "example-volume",
"mountPath": "/example",
}
],
"volumes": [
{
"name": "example-volume",
"persistent_volume_claim": {
"claim_name": "example-claim",
},
}
],
}
mount_config = generate_mount_config()
run_pod = KubernetesPodOperator(
task_id="run_pod",
image="alpine",
cmds=["sh", "-c", "ls /example"],
volume_mounts=mount_config["volume_mounts"],
volumes=mount_config["volumes"],
)
example_k8s_volume_template_error_dag = example_k8s_volume_template_error()
```
3. Run the dag
```shell
python dynamic_volume_dag.py
```
### Anything else
I believe we should just be able to move the calls to `convert_volume_mount` and `convert_volumes` into the `build_pod_request_obj` method, similar to how `env_vars` is handled
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `KubernetesPodOperator` does not properly handle templated volumes and volume_mounts [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #47738:
URL: https://github.com/apache/airflow/issues/47738#issuecomment-5000228001
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `KubernetesPodOperator` does not properly handle templated volumes and volume_mounts [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 closed issue #47738: `KubernetesPodOperator` does not properly handle templated volumes and volume_mounts
URL: https://github.com/apache/airflow/issues/47738
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `KubernetesPodOperator` does not properly handle templated volumes and volume_mounts [airflow]
Posted by "insomnes (via GitHub)" <gi...@apache.org>.
insomnes commented on issue #47738:
URL: https://github.com/apache/airflow/issues/47738#issuecomment-2841881100
I was able to reproduce it, the problem looks the same as described in
https://github.com/apache/airflow/discussions/42638
The problem is: we have xcom object before "templating" so iteration is failed in `__init__`
> I believe we should just be able to move the calls to convert_volume_mount and convert_volumes into the build_pod_request_obj method, similar to how env_vars is handled
I need to dig a bit more into how volumes are configured, that could be a solution, but I could not say for now, how would it affect other places, will check it a bit later this week.
The volume also requires k8s (or legacy airflow) models anyway, if I remember correctly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `KubernetesPodOperator` does not properly handle templated volumes and volume_mounts [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #47738:
URL: https://github.com/apache/airflow/issues/47738#issuecomment-2721844768
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `KubernetesPodOperator` does not properly handle templated volumes and volume_mounts [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #47738:
URL: https://github.com/apache/airflow/issues/47738#issuecomment-2833385978
cc @insomnes will you have time to look into this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `KubernetesPodOperator` does not properly handle templated volumes and volume_mounts [airflow]
Posted by "insomnes (via GitHub)" <gi...@apache.org>.
insomnes commented on issue #47738:
URL: https://github.com/apache/airflow/issues/47738#issuecomment-2834103817
Sure, I will try to take a look today or tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `KubernetesPodOperator` does not properly handle templated volumes and volume_mounts [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on issue #47738:
URL: https://github.com/apache/airflow/issues/47738#issuecomment-5069862141
Should be solved as part of #70296.
The problem seems to be logic in the constructor that should run in the `execute()` method.
```python
volume_mounts = [convert_volume_mount(v) for v in volume_mounts] if volume_mounts else []
self.volume_mounts = volume_mounts
volumes = [convert_volume(volume) for volume in volumes] if volumes else []
self.volumes = volumes
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org