You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "javi-cortes (via GitHub)" <gi...@apache.org> on 2025/11/03 12:49:04 UTC
[I] PydanticSerializationError when using DAG-level pod_override (V1Pod) with KubernetesExecutor [airflow]
javi-cortes opened a new issue, #57743:
URL: https://github.com/apache/airflow/issues/57743
### Apache Airflow version
3.1.1
### If "Other Airflow 2/3 version" selected, which one?
_No response_
### What happened?
Hi!
I’m using Airflow 3.1.0 / 3.1.1 with the KubernetesExecutor, and I’m overriding the pod via executor_config in default_args.
This works perfectly when applied at task level, but when I set it at DAG level (via default_args), the UI (FastAPI) returns a 500 Internal Server Error because the response serializer can’t handle the V1Pod object.
Error from api-server logs:
`
INFO: 10.101.11.64:44642 - "GET /api/v2/dags/pip_freeze_dag_override/details HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
[...]
File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 188, in serialize_response
return field.serialize(
File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/_compat.py", line 152, in serialize
return self._type_adapter.dump_python(
File "/home/airflow/.local/lib/python3.12/site-packages/pydantic/type_adapter.py", line 572, in dump_python
return self.serializer.to_python(
pydantic_core._pydantic_core.PydanticSerializationError: Unable to serialize unknown type: <class 'kubernetes.client.models.v1_pod.V1Pod'>
`
Task runs fine when pod_override is passed directly to the operator.
UI / API call fails (/api/v2/dags/{dag_id}/details) because V1Pod is not JSON-serializable by Pydantic.
Scheduler logs show a proper V1Pod object being used if the task is triggered manually.
The serialization failure happens when the UI (or API) calls the /api/v2/dags/{dag_id}/details endpoint, which is implemented as:
`
@dags_router.get("/{dag_id}/details")
def get_dag_details(...):
dag = get_latest_version_of_dag(dag_bag, dag_id, session)
dag_model = session.get(DagModel, dag_id)
for key, value in dag.__dict__.items():
if not key.startswith("_") and not hasattr(dag_model, key):
setattr(dag_model, key, value)
return dag_model
`
The returned dag_model is then serialized into a DAGDetailsResponse Pydantic schema.
When the DAG contains executor_config={"pod_override": <V1Pod>}, the serializer hits a PydanticSerializationError because V1Pod isn’t JSON-serializable.
If you sanitize it into a dict, the scheduler later fails, since PodGenerator.from_obj() rejects non-V1Pod objects.
PodGenerator only deserializes YAML templates (via deserialize_model_file()), so there’s currently no code path that handles this correctly for dicts returned by the API.
### What you think should happen instead?
The API response should serialize cleanly even when executor_config["pod_override"] contains a V1Pod.
Alternatively, Airflow could auto-sanitize or exclude V1Pod fields when building the DAG details response.
### How to reproduce
Minimal Reproducible Example:
`from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime
from kubernetes import client as k8s
default_args = {
"executor_config": {
"pod_override": k8s.V1Pod(
spec=k8s.V1PodSpec(
containers=[
k8s.V1Container(
name="base",
image="busybox",
image_pull_policy="IfNotPresent",
)
],
restart_policy="Never",
),
)
}
}
with DAG(
dag_id="example_pod_override_dag",
start_date=datetime(2024, 1, 1),
schedule=None,
catchup=False,
default_args=default_args, # ← DAG-level
) as dag:
BashOperator(
task_id="pip_freeze",
bash_command="python -m pip --version && python -m pip freeze",
# inherits executor_config from default_args
)
`
### Operating System
Linux, Ubuntu 24.04
### Versions of Apache Airflow Providers
3.1.0
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
k8s official helm chart
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PydanticSerializationError when using DAG-level pod_override (V1Pod) with KubernetesExecutor [airflow]
Posted by "xunyang-tay (via GitHub)" <gi...@apache.org>.
xunyang-tay commented on issue #57743:
URL: https://github.com/apache/airflow/issues/57743#issuecomment-3881796071
@sdg9670f Which version of airflow is this fixed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PydanticSerializationError when using DAG-level pod_override (V1Pod) with KubernetesExecutor [airflow]
Posted by "sdg9670f (via GitHub)" <gi...@apache.org>.
sdg9670f commented on issue #57743:
URL: https://github.com/apache/airflow/issues/57743#issuecomment-3734884021
I’m experiencing the same issue. The same error occurs via the API, though the DAG does run when I use the trigger button. However, I think it needs a fix.
Airflow 3.1.5, apache-airflow-providers-cncf-kubernetes 10.12.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PydanticSerializationError when using DAG-level pod_override (V1Pod) with KubernetesExecutor [airflow]
Posted by "sdg9670f (via GitHub)" <gi...@apache.org>.
sdg9670f commented on issue #57743:
URL: https://github.com/apache/airflow/issues/57743#issuecomment-3881831740
> [@sdg9670f](https://github.com/sdg9670f) Which version of airflow is this fixed?
@xunyang-tay airflow 3.1.7 !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PydanticSerializationError when using DAG-level pod_override (V1Pod) with KubernetesExecutor [airflow]
Posted by "MadhuPolu (via GitHub)" <gi...@apache.org>.
MadhuPolu commented on issue #57743:
URL: https://github.com/apache/airflow/issues/57743#issuecomment-3643115000
## If anyone looking whether the bug exists in the latest Airflow version (3.1.4)
I can confirm this issue **still exists in Airflow 3.1.4** (tested on 2025-12-11).
### Environment
- **Airflow Version**: 3.1.4
- **Executor**: KubernetesExecutor
- **Deployment**: Official Apache Airflow Helm Chart
- **Python**: 3.12
- **Kubernetes Provider**: Latest
### Reproduction
When using `executor_config` with `V1Pod` in `default_args` at DAG level:
from kubernetes.client import models as k8s
```
default_args = {
'executor_config': {
'pod_override': k8s.V1Pod(
spec=k8s.V1PodSpec(
node_selector={'cloud.google.com/gke-nodepool': 'arm-worker-pool'},
tolerations=[
k8s.V1Toleration(
key='cloud.google.com/gke-nodepool',
operator='Equal',
value='arm-worker-pool',
effect='NoSchedule'
)
],
containers=[k8s.V1Container(name="base")]
)
)
}
}
@dag(
dag_id='kickoff__table_onboarding',
default_args=default_args, # ← DAG-level executor_config
...
)### Error Details
**Full Stack Trace from API Server Logs:**
pydantic_core._pydantic_core.PydanticSerializationError: Unable to serialize unknown type: <class 'kubernetes.client.models.v1_pod.V1Pod'>
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 334, in app
content = await serialize_response(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 188, in serialize_response
return field.serialize(
^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/_compat.py", line 152, in serialize
return self._type_adapter.dump_python(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/pydantic/type_adapter.py", line 605, in dump_python
return self.serializer.to_python(
^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.PydanticSerializationError: Unable to serialize unknown type: <class 'kubernetes.client.models.v1_pod.V1Pod'>### Impact
- **Tasks execute successfully** (scheduler handles V1Pod objects correctly)
- **API endpoints fail** with 500 Internal Server Error when accessing:
- `/api/v2/dags/{dag_id}/details`
- Any endpoint that serializes DAG metadata containing executor_config
- **UI becomes partially unusable** for affected DAGs
```
### Current Workaround
Apply `executor_config` at **task level** instead of DAG level:
```
from kubernetes.client import models as k8s
K8S_EXECUTOR_ARM64_CONFIG = {
'pod_override': k8s.V1Pod(
spec=k8s.V1PodSpec(
node_selector={'cloud.google.com/gke-nodepool': 'arm-worker-pool'},
tolerations=[...],
containers=[k8s.V1Container(name="base")]
)
)
}
@task(executor_config=K8S_EXECUTOR_ARM64_CONFIG, pool=POOL_NAME)
def my_task():
passThis is verbose and error-prone, as every single task must explicitly include `executor_config`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PydanticSerializationError when using DAG-level pod_override (V1Pod) with KubernetesExecutor [airflow]
Posted by "sdg9670f (via GitHub)" <gi...@apache.org>.
sdg9670f commented on issue #57743:
URL: https://github.com/apache/airflow/issues/57743#issuecomment-4189805366
I fixed it.
Convert the V1Pod object into a standard Python dictionary using the .to_dict() method. This allows the API server to serialize the configuration properly while maintaining full functionality for the executor.
```python
# Before (Causes UI Error)
executor_config = {
"pod_override": k8s.V1Pod(
spec=k8s.V1PodSpec(
# ... internal configurations ...
)
)
}
# After (Fixed)
executor_config = {
"pod_override": k8s.V1Pod(
spec=k8s.V1PodSpec(
# ... internal configurations ...
)
).to_dict() # Added .to_dict() for JSON serialization
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PydanticSerializationError when using DAG-level pod_override (V1Pod) with KubernetesExecutor [airflow]
Posted by "sdg9670f (via GitHub)" <gi...@apache.org>.
sdg9670f commented on issue #57743:
URL: https://github.com/apache/airflow/issues/57743#issuecomment-3881494788
It's fixed! working well 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PydanticSerializationError when using DAG-level pod_override (V1Pod) with KubernetesExecutor [airflow]
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on issue #57743:
URL: https://github.com/apache/airflow/issues/57743#issuecomment-4259885681
Unassigning you @sunank200 so that this is available for other contributors to pick up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PydanticSerializationError when using DAG-level pod_override (V1Pod) with KubernetesExecutor [airflow]
Posted by "jvstein (via GitHub)" <gi...@apache.org>.
jvstein commented on issue #57743:
URL: https://github.com/apache/airflow/issues/57743#issuecomment-3881486043
Confirming the problem still exists on Airflow 3.1.7.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PydanticSerializationError when using DAG-level pod_override (V1Pod) with KubernetesExecutor [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on issue #57743:
URL: https://github.com/apache/airflow/issues/57743#issuecomment-3663974171
@sunank200 assigning to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PydanticSerializationError when using DAG-level pod_override (V1Pod) with KubernetesExecutor [airflow]
Posted by "xunyang-tay (via GitHub)" <gi...@apache.org>.
xunyang-tay commented on issue #57743:
URL: https://github.com/apache/airflow/issues/57743#issuecomment-3630442987
Hi team, is there any update on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PydanticSerializationError when using DAG-level pod_override (V1Pod) with KubernetesExecutor [airflow]
Posted by "RenGeng (via GitHub)" <gi...@apache.org>.
RenGeng commented on issue #57743:
URL: https://github.com/apache/airflow/issues/57743#issuecomment-3745511867
I can confirm the same issue on Airflow 3.1.6, apache-airflow-providers-cncf-kubernetes==10.12.0, python 3.11
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PydanticSerializationError when using DAG-level pod_override (V1Pod) with KubernetesExecutor [airflow]
Posted by "sdg9670f (via GitHub)" <gi...@apache.org>.
sdg9670f commented on issue #57743:
URL: https://github.com/apache/airflow/issues/57743#issuecomment-4027918955
It's not fixed. Some dags still have this error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PydanticSerializationError when using DAG-level pod_override (V1Pod) with KubernetesExecutor [airflow]
Posted by "sunank200 (via GitHub)" <gi...@apache.org>.
sunank200 commented on issue #57743:
URL: https://github.com/apache/airflow/issues/57743#issuecomment-3630921868
I can take a look at this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PydanticSerializationError when using DAG-level pod_override (V1Pod) with KubernetesExecutor [airflow]
Posted by "jvstein (via GitHub)" <gi...@apache.org>.
jvstein commented on issue #57743:
URL: https://github.com/apache/airflow/issues/57743#issuecomment-3887425457
I put together a quick branch (on top of the 3.1.7 tag) which at least gets past the . I'm not sure the format of the serialization output is what is desired here, though.
https://github.com/apache/airflow/compare/3.1.7...jvstein:airflow:fix_default_args_serialization
<img width="937" height="1028" alt="Image" src="https://github.com/user-attachments/assets/ec9d8a7a-ed4a-4c2d-9445-72618e6b00db" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PydanticSerializationError when using DAG-level pod_override (V1Pod) with KubernetesExecutor [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #57743:
URL: https://github.com/apache/airflow/issues/57743#issuecomment-3480364976
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PydanticSerializationError when using DAG-level pod_override (V1Pod) with KubernetesExecutor [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on issue #57743:
URL: https://github.com/apache/airflow/issues/57743#issuecomment-3480394036
Self assigning to investigate this, likely tomorrow / next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PydanticSerializationError when using DAG-level pod_override (V1Pod) with KubernetesExecutor [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy closed issue #57743: PydanticSerializationError when using DAG-level pod_override (V1Pod) with KubernetesExecutor
URL: https://github.com/apache/airflow/issues/57743
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PydanticSerializationError when using DAG-level pod_override (V1Pod) with KubernetesExecutor [airflow]
Posted by "jvstein (via GitHub)" <gi...@apache.org>.
jvstein commented on issue #57743:
URL: https://github.com/apache/airflow/issues/57743#issuecomment-3884748127
> > [@sdg9670f](https://github.com/sdg9670f) Which version of airflow is this fixed?
>
> @xunyang-tay airflow 3.1.7 !
On 3.1.7, I'm still getting a 500 error on the API request (and no content in the UI) when visiting the Details tab for a similarly configured DAG. That works for you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org