You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vatsrahul1001 (via GitHub)" <gi...@apache.org> on 2026/06/22 04:26:36 UTC
[PR] Fix KubernetesExecutor scheduler crash when pod_override is queued in-cluster [airflow]
vatsrahul1001 opened a new pull request, #68819:
URL: https://github.com/apache/airflow/pull/68819
### Problem
When the scheduler runs **in-cluster**, the kubernetes Python client attaches an in-cluster `Configuration` to every `V1Pod`, and that `Configuration.refresh_api_key_hook` is a **local closure** (`InClusterConfigLoader._set_config.<locals>._refresh_api_key`). `pickle` cannot serialize a local closure.
`KubernetesExecutor` puts each task's `KubernetesJob` (which embeds the task's `pod_override` `V1Pod`) onto a `multiprocessing.Manager` queue, whose `.put()` pickles synchronously in the scheduler. So any in-cluster deployment where a task sets a `V1Pod` `pod_override` crashes the scheduler in a loop:
```
_pickle.PicklingError: Can't pickle local object
<function InClusterConfigLoader._set_config.<locals>._refresh_api_key at ...>
when serializing kubernetes.client.configuration.Configuration ...
when serializing dict item 'pod_override'
when serializing ... ExecuteTask ...
File ".../executors/kubernetes_executor.py", line 230, in execute_async
self.task_queue.put(KubernetesJob(key, command, kube_executor_config, pod_template_file))
```
This is **independent of the Airflow version** — it is gated on having a `V1Pod` `pod_override` and an in-cluster scheduler. It surfaces with kubernetes client **36.x** (whose `Configuration` carries the unpicklable hook), and pinning the client is not a viable fix for everyone (35.x has its own [`no_proxy` bug #2520](https://github.com/kubernetes-client/python/issues/2520), so deployments that need 36.x for that fix cannot downgrade).
### Fix
Serialize the `pod_override` to a plain dict (dropping the `Configuration`) **before** it is queued, using the existing `PodGenerator.serialize_pod`, and rebuild the `V1Pod` worker-side in `run_next` via `PodGenerator.deserialize_model_dict`. The worker already reconstructs its own kube client, so nothing is lost. The worker uses `kube_executor_config` for the pod override; the same `V1Pod` is also referenced by the workload's `executor_config`, so that copy is sanitized too (it is otherwise pickled as part of the workload).
This keeps `pod_override` working regardless of the kubernetes client version, rather than depending on a client version whose `Configuration` happens to be picklable. The worker side already sanitizes pods with `sanitize_for_serialization` (kubernetes_executor_utils.py:492) — this closes the same gap on the scheduler→queue boundary.
### Tests
- New regression test `test_execute_async_queues_picklable_pod_override`: builds a `pod_override` whose `Configuration` carries an unpicklable local-closure hook, asserts the raw pod is unpicklable (reproduces the crash), then asserts the queued `KubernetesJob` pickles cleanly and the serialized dict round-trips back to an equivalent `V1Pod`.
- Updated `test_pod_template_file_override_in_executor_config` to reflect the queued `kube_executor_config` now being a dict; its existing `run_next` round-trip assertion (the reconstructed pod) is unchanged and still passes.
- Full `test_kubernetes_executor.py` suite passes (111 passed, 1 skipped AF<3.2-only).
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 4.8)
Generated-by: Claude Code (Opus 4.8) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KubernetesExecutor scheduler crash when pod_override is queued in-cluster [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #68819:
URL: https://github.com/apache/airflow/pull/68819#discussion_r3451109259
##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py:
##########
@@ -73,6 +74,30 @@
)
+def _reset_local_vars_configuration(obj: Any) -> None:
Review Comment:
```suggestion
def _reset_local_vars_configuration(node: Any) -> None:
```
IMO, this is more readable. I don't quite understand what `obj` is in the original context.
Also, should we type it? looks like we know the potential type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KubernetesExecutor scheduler crash when pod_override is queued in-cluster [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #68819:
URL: https://github.com/apache/airflow/pull/68819#discussion_r3451109259
##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py:
##########
@@ -73,6 +74,30 @@
)
+def _reset_local_vars_configuration(obj: Any) -> None:
Review Comment:
```suggestion
def _reset_local_vars_configuration(config: Any) -> None:
```
IMO, this is more readable. I don't quite understand what `obj` is in the original context.
Also, should we type it? looks like we know the potential type.
##########
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/executors/test_kubernetes_executor.py:
##########
@@ -915,6 +917,123 @@ def test_pod_template_file_override_in_executor_config(
finally:
executor.end()
+ @staticmethod
+ def _install_unpicklable_incluster_hook(pod):
+ """Simulate the in-cluster kubernetes v36 default config: an unpicklable local-closure hook
+ on every nested model object's ``Configuration`` (``InClusterConfigLoader._set_config.<locals>.
+ _refresh_api_key``)."""
+
+ def _make_hook():
+ def _refresh_api_key(config):
+ return None
+
+ return _refresh_api_key
+
+ for obj in (
+ pod,
+ pod.metadata,
+ pod.spec,
+ pod.spec.containers[0],
+ pod.spec.containers[0].resources,
+ ):
+ cfg = Configuration()
+ cfg.refresh_api_key_hook = _make_hook()
+ obj.local_vars_configuration = cfg
+
+ def test_reset_local_vars_configuration_strips_hook_recursively(self):
+ """``_reset_local_vars_configuration`` must replace every nested config with a hook-free one."""
+ # Build a pod with the unpicklable in-cluster hook on every nested config; confirm it is
+ # unpicklable first, then that the reset makes it picklable.
+ pod = k8s.V1Pod(
+ metadata=k8s.V1ObjectMeta(),
+ spec=k8s.V1PodSpec(
+ containers=[k8s.V1Container(name="base", resources=k8s.V1ResourceRequirements())]
+ ),
+ )
+ self._install_unpicklable_incluster_hook(pod)
+ with pytest.raises((pickle.PicklingError, AttributeError, TypeError)):
Review Comment:
Would like to check when will `AttributeError`, `TypeError` happen in different versions?
##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py:
##########
@@ -73,6 +74,30 @@
)
+def _reset_local_vars_configuration(obj: Any) -> None:
+ """
+ Reset ``local_vars_configuration`` to a fresh ``Configuration()`` on every kubernetes model object.
+
+ kubernetes-python-client v36.0.0 changed model constructors to call ``Configuration.get_default_copy()``
+ instead of ``Configuration()``, so objects created after incluster config was loaded capture the global
+ config whose ``refresh_api_key_hook`` is an unpicklable local closure function. We fall back to the
+ pre-v36 behaviour which is picklable. A fresh ``Configuration()`` keeps ``client_side_validation`` so
+ the worker-side ``reconcile_pods`` setters still work; ``None`` would break them.
+ """
+ if obj is None or isinstance(obj, (str, int, float, bool)):
+ return
+ if hasattr(obj, "openapi_types"):
+ obj.local_vars_configuration = Configuration()
+ for attr in obj.openapi_types:
+ _reset_local_vars_configuration(getattr(obj, attr, None))
+ elif isinstance(obj, dict):
+ for v in obj.values():
+ _reset_local_vars_configuration(v)
+ elif isinstance(obj, (list, tuple)):
+ for item in obj:
+ _reset_local_vars_configuration(item)
Review Comment:
What should happen if we go into the `else` block? Is it expected to do nothing? or not expcted to happen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KubernetesExecutor scheduler crash when pod_override is queued in-cluster [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on PR #68819:
URL: https://github.com/apache/airflow/pull/68819#issuecomment-4767622865
Ah, thanks for closing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KubernetesExecutor scheduler crash when pod_override is queued in-cluster [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on PR #68819:
URL: https://github.com/apache/airflow/pull/68819#issuecomment-4766511784
Thanks for the review @amoghrajesh! Addressed the nits:
- Trimmed the docstring to your suggested wording (kept one line on why fresh `Configuration()` over `None` — it preserves `client_side_validation` so worker-side reconcile doesn't break).
- Removed the redundant call-site comment.
- Moved the test imports (`pickle`, `Configuration`, `_reset_local_vars_configuration`) to top level.
Full executor suite green (112 passed, 1 AF<3.2 skip).
---
Drafted-by: Claude Code (Opus 4.8); reviewed by @vatsrahul1001 before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KubernetesExecutor scheduler crash when pod_override is queued in-cluster [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #68819:
URL: https://github.com/apache/airflow/pull/68819#issuecomment-4808187561
> I've opened [#68831](https://github.com/astronomer/airflow/issues/68831), which fixes the same crash one layer up — at the airflow-core deserialize path where that Configuration actually gets attached. The `pod_override` the scheduler pickles onto the queue is produced by `BaseSerialization.deserialize / ensure_pod_is_valid_after_unpickling` (via `ExecutorConfigType`), not by the executor itself; deserializing there through a fresh `Configuration()` keeps the pod (and every nested model) picklable for every consumer of a deserialized V1Pod, not just the `KubernetesExecutor` queue. It also gives `PodGenerator.deserialize_model_dict` the same treatment, so the provider's worker-side reconstruction is covered too.
>
> Since that resolves the crash at the source and makes the executor-level reset redundant, I think we can consolidate on [#68831](https://github.com/astronomer/airflow/issues/68831) and close this one. Let me know if that's reasonable?
@ephraimbuddy The fix is currently in Astronomer's private repo, can you please make create a PR in the upstream repo?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KubernetesExecutor scheduler crash when pod_override is queued in-cluster [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 closed pull request #68819: Fix KubernetesExecutor scheduler crash when pod_override is queued in-cluster
URL: https://github.com/apache/airflow/pull/68819
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KubernetesExecutor scheduler crash when pod_override is queued in-cluster [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #68819:
URL: https://github.com/apache/airflow/pull/68819#discussion_r3450479016
##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py:
##########
@@ -226,6 +226,21 @@ def execute_async(
pod_template_file = executor_config.get("pod_template_file", None)
else:
pod_template_file = None
+
+ # Serialize any ``pod_override`` ``V1Pod`` to a plain dict before putting it on the
+ # multiprocessing queue. When the scheduler runs in-cluster, the kubernetes client attaches an
+ # in-cluster ``Configuration`` to every ``V1Pod`` whose ``refresh_api_key_hook`` is a local
+ # closure (``InClusterConfigLoader._set_config.<locals>._refresh_api_key``). ``pickle`` cannot
+ # serialize a local closure, so queuing a live ``V1Pod`` raises ``PicklingError`` and crashes the
+ # scheduler. ``run_next`` deserializes the dict back into a ``V1Pod`` worker-side. The same
+ # ``V1Pod`` object is also referenced by the workload's ``executor_config``, so sanitize that copy
+ # too (it is otherwise pickled as part of the workload, even though the worker rebuilds the pod
+ # override from ``kube_executor_config``).
+ if kube_executor_config is not None:
+ kube_executor_config = PodGenerator.serialize_pod(kube_executor_config)
+ if executor_config and executor_config.get("pod_override") is not None:
+ executor_config["pod_override"] = PodGenerator.serialize_pod(executor_config["pod_override"])
Review Comment:
Maybe something like this would be a better fix:
```diff
diff --git a/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py b/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py
--- a/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py (revision 491c167075187a48dc010862cb2ee9a708e822ef)
+++ b/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py (date 1782112916157)
@@ -73,6 +73,32 @@
)
+def _strip_local_vars_configuration(obj: Any) -> None:
+ """
+ Null out local_vars_configuration on all kubernetes model objects recursively.
+
+ kubernetes-python-client v36 changed model constructors to call
+ Configuration.get_default_copy() instead of Configuration(), so every
+ model object created after load_incluster_config() runs captures the global
+ Configuration, which holds an unpicklable refresh_api_key_hook closure.
+ The field is only used for API calls within the same process; workers
+ reinitialize their own kube config, so nulling it before the multiprocessing
+ queue put loses nothing functional.
+ """
+ if obj is None or isinstance(obj, (str, int, float, bool)):
+ return
+ if hasattr(obj, "openapi_types"):
+ obj.local_vars_configuration = None
+ for attr in obj.openapi_types:
+ _strip_local_vars_configuration(getattr(obj, attr, None))
+ elif isinstance(obj, dict):
+ for v in obj.values():
+ _strip_local_vars_configuration(v)
+ elif isinstance(obj, (list, tuple)):
+ for item in obj:
+ _strip_local_vars_configuration(item)
+
+
class KubernetesExecutor(BaseExecutor):
"""Executor for Kubernetes."""
@@ -226,6 +252,7 @@
pod_template_file = executor_config.get("pod_template_file", None)
else:
pod_template_file = None
+ _strip_local_vars_configuration(kube_executor_config)
self.event_buffer[key] = (TaskInstanceState.QUEUED, self.scheduler_job_id)
self.task_queue.put(KubernetesJob(key, command, kube_executor_config, pod_template_file))
# We keep a temporary local record that we've handled this so we don't
```
This keeps `V1Pod` as the type through the queue, does not mutate `executor_config` in-place, and works identically on v35 where `local_vars_configuration` was already a plain `Configuration()` with no hook
##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py:
##########
@@ -226,6 +226,21 @@ def execute_async(
pod_template_file = executor_config.get("pod_template_file", None)
else:
pod_template_file = None
+
+ # Serialize any ``pod_override`` ``V1Pod`` to a plain dict before putting it on the
+ # multiprocessing queue. When the scheduler runs in-cluster, the kubernetes client attaches an
+ # in-cluster ``Configuration`` to every ``V1Pod`` whose ``refresh_api_key_hook`` is a local
+ # closure (``InClusterConfigLoader._set_config.<locals>._refresh_api_key``). ``pickle`` cannot
+ # serialize a local closure, so queuing a live ``V1Pod`` raises ``PicklingError`` and crashes the
+ # scheduler. ``run_next`` deserializes the dict back into a ``V1Pod`` worker-side. The same
+ # ``V1Pod`` object is also referenced by the workload's ``executor_config``, so sanitize that copy
+ # too (it is otherwise pickled as part of the workload, even though the worker rebuilds the pod
+ # override from ``kube_executor_config``).
+ if kube_executor_config is not None:
+ kube_executor_config = PodGenerator.serialize_pod(kube_executor_config)
+ if executor_config and executor_config.get("pod_override") is not None:
+ executor_config["pod_override"] = PodGenerator.serialize_pod(executor_config["pod_override"])
Review Comment:
The root cause here is that `kubernetes-python-client` v36 changed model constructors from `Configuration()` to `Configuration.get_default_copy()`, which causes every `V1Pod` created after `in-cluster`config loads to silently capture the global config, including its unpicklable `refresh_api_key_hook` closure defined in https://github.com/kubernetes-client/python/blob/master/kubernetes/aio/config/incluster_config.py#L100
The serialize/deserialize approach works but treats the symptom at the wrong layer. `local_vars_configuration` is API client state that has no business being on a pod template override object, it ended up there as a side effect of the v36 constructor change, not by intent. Rather than round-tripping through a dict, nulling it out before the queue put is more direct and correct imo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KubernetesExecutor scheduler crash when pod_override is queued in-cluster [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on PR #68819:
URL: https://github.com/apache/airflow/pull/68819#issuecomment-4766372857
Thanks @amoghrajesh — agreed on the "wrong layer" point, switched away from serialize/deserialize. I went with your in-place approach but with one correction.
**Nulling `local_vars_configuration` breaks worker-side reconcile.** When I tested `local_vars_configuration = None` on a real in-cluster deployment, `run_next` → `PodGenerator.construct_pod` → `reconcile_specs` failed:
```
File ".../kubernetes/client/models/v1_pod_spec.py", line 349, in containers
if self.local_vars_configuration.client_side_validation and containers is None:
AttributeError: 'NoneType' object has no attribute 'client_side_validation'
```
The model setters hit during `reconcile_pods` dereference `self.local_vars_configuration.client_side_validation`, so `None` raises and the task fails with `PodReconciliationError`.
**Fix: reset it to a fresh `Configuration()` instead of `None`** — exactly what v35 model constructors produced. That's picklable (no in-cluster `refresh_api_key_hook`) **and** keeps `client_side_validation`, so reconcile works. Implemented as `_reset_local_vars_configuration` (recursive), called on `kube_executor_config` before `task_queue.put`. As you noted, `from_obj` returns the same object referenced by the workload's `executor_config`, so the single call covers both; the pod keeps its `V1Pod` type through the queue, so `run_next` is unchanged.
Validated on the deployment (no crash, task runs/finishes) and added a regression test that asserts both **picklability** and that **`run_next`/`reconcile_pods` succeeds** (the assertion the `None` approach fails). Full executor suite green.
Root cause is the v36 model-constructor change (`Configuration()` → `Configuration.get_default_copy()`, kubernetes-client/python#2532 / OpenAPITools/openapi-generator#8500); tracked in #68827.
---
Drafted-by: Claude Code (Opus 4.8); reviewed by @vatsrahul1001 before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KubernetesExecutor scheduler crash when pod_override is queued in-cluster [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #68819:
URL: https://github.com/apache/airflow/pull/68819#discussion_r3450813871
##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py:
##########
@@ -226,6 +258,15 @@ def execute_async(
pod_template_file = executor_config.get("pod_template_file", None)
else:
pod_template_file = None
+
+ # Reset ``local_vars_configuration`` on the ``pod_override`` to a fresh ``Configuration()`` before
+ # putting it on the multiprocessing queue. kubernetes-python-client v36 makes model constructors
+ # capture the global in-cluster ``Configuration`` (with an unpicklable ``refresh_api_key_hook``
+ # closure), so queuing the pod raises ``PicklingError`` and crashes the scheduler.
+ # ``kube_executor_config`` is the same ``V1Pod`` object referenced by the workload's
+ # ``executor_config``, so this also makes the workload picklable. No-op on v35.
+ _reset_local_vars_configuration(kube_executor_config)
Review Comment:
Restates what doc-string states, adds no value
```suggestion
```
##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py:
##########
@@ -73,6 +74,37 @@
)
+def _reset_local_vars_configuration(obj: Any) -> None:
+ """
+ Reset ``local_vars_configuration`` to a fresh ``Configuration`` on every kubernetes model object.
+
+ kubernetes-python-client v36 changed model constructors to call
+ ``Configuration.get_default_copy()`` instead of ``Configuration()``, so every model object
+ created after ``load_incluster_config()`` runs captures the global ``Configuration``, which holds
+ an unpicklable ``refresh_api_key_hook`` closure -- pickling such a pod onto the multiprocessing
+ queue raises ``PicklingError`` and crashes the scheduler.
+
+ Replace it with a fresh ``Configuration()`` (exactly what v35 model constructors used): it carries
+ no in-cluster auth hook so it is picklable, while still providing ``client_side_validation`` so the
+ worker-side ``PodGenerator.reconcile_pods`` setters keep working. Setting it to ``None`` instead
+ would break reconcile -- model setters dereference
+ ``self.local_vars_configuration.client_side_validation``. The field is only API-client state used
+ within the same process; workers reinitialize their own kube config, so nothing is lost. No-op on v35.
Review Comment:
```suggestion
Reset local_vars_configuration to a fresh Configuration() on every kubernetes model object.
kubernetes-python-client v36.0.0 changed model constructors to call ``Configuration.get_default_copy()``
instead of ``Configuration()``, so objects created after incluster config was loaded capture the global
config whose `refresh_api_key_hook` is an unpicklable local closure function. We choose to fallback to the pre-v36 behaviour which is picklable.
```
##########
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/executors/test_kubernetes_executor.py:
##########
@@ -915,6 +915,132 @@ def test_pod_template_file_override_in_executor_config(
finally:
executor.end()
+ @staticmethod
+ def _install_unpicklable_incluster_hook(pod):
+ """Simulate the in-cluster kubernetes v36 default config: an unpicklable local-closure hook
+ on every nested model object's ``Configuration`` (``InClusterConfigLoader._set_config.<locals>.
+ _refresh_api_key``)."""
+ from kubernetes.client import Configuration
Review Comment:
Top level import pls.
##########
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/executors/test_kubernetes_executor.py:
##########
@@ -915,6 +915,132 @@ def test_pod_template_file_override_in_executor_config(
finally:
executor.end()
+ @staticmethod
+ def _install_unpicklable_incluster_hook(pod):
+ """Simulate the in-cluster kubernetes v36 default config: an unpicklable local-closure hook
+ on every nested model object's ``Configuration`` (``InClusterConfigLoader._set_config.<locals>.
+ _refresh_api_key``)."""
+ from kubernetes.client import Configuration
+
+ def _make_hook():
+ def _refresh_api_key(config):
+ return None
+
+ return _refresh_api_key
+
+ for obj in (
+ pod,
+ pod.metadata,
+ pod.spec,
+ pod.spec.containers[0],
+ pod.spec.containers[0].resources,
+ ):
+ cfg = Configuration()
+ cfg.refresh_api_key_hook = _make_hook()
+ obj.local_vars_configuration = cfg
+
+ def test_reset_local_vars_configuration_strips_hook_recursively(self):
+ """``_reset_local_vars_configuration`` must replace every nested config with a hook-free one."""
+ import pickle
Review Comment:
Top level import pls.
##########
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/executors/test_kubernetes_executor.py:
##########
@@ -915,6 +915,132 @@ def test_pod_template_file_override_in_executor_config(
finally:
executor.end()
+ @staticmethod
+ def _install_unpicklable_incluster_hook(pod):
+ """Simulate the in-cluster kubernetes v36 default config: an unpicklable local-closure hook
+ on every nested model object's ``Configuration`` (``InClusterConfigLoader._set_config.<locals>.
+ _refresh_api_key``)."""
+ from kubernetes.client import Configuration
+
+ def _make_hook():
+ def _refresh_api_key(config):
+ return None
+
+ return _refresh_api_key
+
+ for obj in (
+ pod,
+ pod.metadata,
+ pod.spec,
+ pod.spec.containers[0],
+ pod.spec.containers[0].resources,
+ ):
+ cfg = Configuration()
+ cfg.refresh_api_key_hook = _make_hook()
+ obj.local_vars_configuration = cfg
+
+ def test_reset_local_vars_configuration_strips_hook_recursively(self):
+ """``_reset_local_vars_configuration`` must replace every nested config with a hook-free one."""
+ import pickle
+
+ from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import (
+ _reset_local_vars_configuration,
+ )
Review Comment:
Same here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KubernetesExecutor scheduler crash when pod_override is queued in-cluster [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on PR #68819:
URL: https://github.com/apache/airflow/pull/68819#issuecomment-4767601856
Closing in favour of https://github.com/apache/airflow/pull/68831
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org