You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Vamsi-klu (via GitHub)" <gi...@apache.org> on 2026/02/15 04:39:37 UTC
[PR] KubernetesHook: add AWS exec-auth botocore guardrails for EKS token flow (#60943) [airflow]
Vamsi-klu opened a new pull request, #61936:
URL: https://github.com/apache/airflow/pull/61936
## Why this change
Issue #60943 reports intermittent KubernetesPodOperator task failures on Celery workers when multiple tasks start together and kubeconfig uses `aws eks get-token` exec auth.
The failure mode is subtle:
- the auth subprocess (`aws eks get-token`) can fail due to older botocore race behavior around `~/.aws/cli/cache`
- Kubernetes client then proceeds with invalid/empty auth and surfaces a generic `403 Forbidden`
- this looks identical to real RBAC failures, so operators often lose time debugging the wrong problem
This PR adds **explicit runtime guardrails** for that path so operators get a clear signal before task execution fails in a misleading way.
## Impact of the change
This adds a policy-driven runtime check only when kubeconfig exec auth actually uses `aws eks get-token`:
- `warn` (default): emits an actionable warning if botocore is vulnerable (`< 1.40.2`) or version cannot be detected
- `fail`: hard-fails early with a clear error to enforce platform policy
- `ignore`: bypasses the check when users intentionally manage this externally
Operational impact:
- **Improves diagnosability** of a production issue that often appears as ambiguous `403`
- **Reduces MTTR** by surfacing root-cause guidance at connection/auth setup time
- **Adds governance controls** for teams that need strict enforcement (`fail`) without forcing everyone into that mode
- **Keeps backwards compatibility** with default `warn`
## Scope and non-goals
- Scope is intentionally limited to the AWS EKS exec-auth path (`aws eks get-token`) because this is the concrete failing path in #60943.
- This PR does **not** change Kubernetes retry semantics for `403` responses, and does not change auth flow for non-AWS exec plugins.
## Configuration
New Kubernetes connection extra:
- `exec_auth_aws_cli_version_check_mode`: `warn` (default) | `fail` | `ignore`
## Validation
- Added unit coverage for:
- kubeconfig exec-auth detection (`aws eks get-token`)
- botocore version parsing from `aws --version`
- mode behavior (`warn`, `fail`, `ignore`, invalid fallback)
- integration points in `get_conn` and default kubeconfig client path
- Test command used:
- `AIRFLOW_HOME=/tmp/airflow-60943 uv run --python 3.12 -m pytest providers/cncf/kubernetes/tests/unit/cncf/kubernetes/hooks/test_kubernetes.py -q`
closes #60943
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KubernetesHook: add AWS exec-auth botocore guardrails for EKS token flow (#60943) [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on PR #61936:
URL: https://github.com/apache/airflow/pull/61936#issuecomment-3940356499
Collaborators for this PR: @codingrealitylabs and @girlcoder-gaming. They helped me raise this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KubernetesHook: add AWS exec-auth botocore guardrails for EKS token flow (#60943) [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #61936:
URL: https://github.com/apache/airflow/pull/61936#discussion_r2819490402
##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/hooks/kubernetes.py:
##########
@@ -56,6 +60,11 @@
from kubernetes.client.models import CoreV1Event, CoreV1EventList, V1Job, V1Pod
LOADING_KUBE_CONFIG_FILE_RESOURCE = "Loading Kubernetes configuration file kube_config from {}..."
+_AWS_EXEC_AUTH_VERSION_CHECK_MODE_FIELD = "exec_auth_aws_cli_version_check_mode"
+_AWS_EXEC_AUTH_VERSION_CHECK_MODES = {"warn", "fail", "ignore"}
+_AWS_EXEC_AUTH_AWS_BINARY_NAMES = {"aws", "aws.exe", "aws2", "aws2.exe"}
+_AWS_EXEC_AUTH_FIXED_BOTOCORE_VERSION = (1, 40, 2)
+_BOTOCORE_VERSION_PATTERN = re.compile(r"botocore/(?P<version>\d+(?:\.\d+){1,2})")
Review Comment:
Off the top of my head I can't think of previous case quite like this. Can you @eladkal?
I see both sides, I don't like AWS stuff in the K8s hook, but I also see the point @Vamsi-klu mentions. And the workaround sounds quite a bit more complicated than the change already is, which I don't love :grimacing:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KubernetesHook: add AWS exec-auth botocore guardrails for EKS token flow (#60943) [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on PR #61936:
URL: https://github.com/apache/airflow/pull/61936#issuecomment-3903542611
@Srabasti i have updated the PR with the relevant checks. Can you please review this and let me know your feedback? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KubernetesHook: add AWS exec-auth botocore guardrails for EKS token flow (#60943) [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #61936:
URL: https://github.com/apache/airflow/pull/61936#discussion_r2849034496
##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/hooks/kubernetes.py:
##########
@@ -56,6 +60,11 @@
from kubernetes.client.models import CoreV1Event, CoreV1EventList, V1Job, V1Pod
LOADING_KUBE_CONFIG_FILE_RESOURCE = "Loading Kubernetes configuration file kube_config from {}..."
+_AWS_EXEC_AUTH_VERSION_CHECK_MODE_FIELD = "exec_auth_aws_cli_version_check_mode"
+_AWS_EXEC_AUTH_VERSION_CHECK_MODES = {"warn", "fail", "ignore"}
+_AWS_EXEC_AUTH_AWS_BINARY_NAMES = {"aws", "aws.exe", "aws2", "aws2.exe"}
+_AWS_EXEC_AUTH_FIXED_BOTOCORE_VERSION = (1, 40, 2)
+_BOTOCORE_VERSION_PATTERN = re.compile(r"botocore/(?P<version>\d+(?:\.\d+){1,2})")
Review Comment:
I really don't understand what we are trying to solve here.
but if k8s provider needs optional stuff from amazon provider it should have set it as optional dependency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KubernetesHook: add AWS exec-auth botocore guardrails for EKS token flow (#60943) [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #61936:
URL: https://github.com/apache/airflow/pull/61936#discussion_r2810482924
##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/hooks/kubernetes.py:
##########
@@ -56,6 +60,11 @@
from kubernetes.client.models import CoreV1Event, CoreV1EventList, V1Job, V1Pod
LOADING_KUBE_CONFIG_FILE_RESOURCE = "Loading Kubernetes configuration file kube_config from {}..."
+_AWS_EXEC_AUTH_VERSION_CHECK_MODE_FIELD = "exec_auth_aws_cli_version_check_mode"
+_AWS_EXEC_AUTH_VERSION_CHECK_MODES = {"warn", "fail", "ignore"}
+_AWS_EXEC_AUTH_AWS_BINARY_NAMES = {"aws", "aws.exe", "aws2", "aws2.exe"}
+_AWS_EXEC_AUTH_FIXED_BOTOCORE_VERSION = (1, 40, 2)
+_BOTOCORE_VERSION_PATTERN = re.compile(r"botocore/(?P<version>\d+(?:\.\d+){1,2})")
Review Comment:
Thanks for the feedback @jscheffl, that's a fair point about keeping the K8s provider cloud agnostic.
The reason this was placed in KubernetesHook is that the vulnerability affects any kubeconfig using aws eks get-token exec auth, not just users going through EksHook or EksPodOperator. Many users configure a generic Kubernetes connection with a kubeconfig file that happens to use AWS exec auth and they never interact with the AWS provider at all.
That said, I agree AWS specific constants and detection logic shouldn't live here. How about this approach:
In the K8s provider, add a minimal, generic exec auth validation hook point in KubernetesHook.get_conn() (for example a discoverable entry point or registry pattern like _validate_exec_auth(kubeconfig, context)). No AWS/GCP/Azure specific code, just a generic extension mechanism.
In the AWS provider, register a validator that handles the AWS specific detection (aws eks get-token binary matching, botocore version parsing, subprocess call) and the connection extra field (exec_auth_aws_cli_version_check_mode). All AWS constants, helpers, tests, and docs move there.
This way the K8s hook stays cloud agnostic while still protecting users who use generic KubernetesHook with an EKS kubeconfig. It also makes the pattern extensible so any other provider could register their own exec auth validators in the future.
Would this approach work for you, or would you prefer everything moved entirely into the AWS provider package?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KubernetesHook: add AWS exec-auth botocore guardrails for EKS token flow (#60943) [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #61936:
URL: https://github.com/apache/airflow/pull/61936#discussion_r2813954322
##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/hooks/kubernetes.py:
##########
@@ -56,6 +60,11 @@
from kubernetes.client.models import CoreV1Event, CoreV1EventList, V1Job, V1Pod
LOADING_KUBE_CONFIG_FILE_RESOURCE = "Loading Kubernetes configuration file kube_config from {}..."
+_AWS_EXEC_AUTH_VERSION_CHECK_MODE_FIELD = "exec_auth_aws_cli_version_check_mode"
+_AWS_EXEC_AUTH_VERSION_CHECK_MODES = {"warn", "fail", "ignore"}
+_AWS_EXEC_AUTH_AWS_BINARY_NAMES = {"aws", "aws.exe", "aws2", "aws2.exe"}
+_AWS_EXEC_AUTH_FIXED_BOTOCORE_VERSION = (1, 40, 2)
+_BOTOCORE_VERSION_PATTERN = re.compile(r"botocore/(?P<version>\d+(?:\.\d+){1,2})")
Review Comment:
Not an expert here... @o-nikolas / @eladkal how were such things handled in the past if cloud-provider specific stuff needed to be integrated into a base provider package?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KubernetesHook: add AWS exec-auth botocore guardrails for EKS token flow (#60943) [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #61936:
URL: https://github.com/apache/airflow/pull/61936#discussion_r2810065854
##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/hooks/kubernetes.py:
##########
@@ -56,6 +60,11 @@
from kubernetes.client.models import CoreV1Event, CoreV1EventList, V1Job, V1Pod
LOADING_KUBE_CONFIG_FILE_RESOURCE = "Loading Kubernetes configuration file kube_config from {}..."
+_AWS_EXEC_AUTH_VERSION_CHECK_MODE_FIELD = "exec_auth_aws_cli_version_check_mode"
+_AWS_EXEC_AUTH_VERSION_CHECK_MODES = {"warn", "fail", "ignore"}
+_AWS_EXEC_AUTH_AWS_BINARY_NAMES = {"aws", "aws.exe", "aws2", "aws2.exe"}
+_AWS_EXEC_AUTH_FIXED_BOTOCORE_VERSION = (1, 40, 2)
+_BOTOCORE_VERSION_PATTERN = re.compile(r"botocore/(?P<version>\d+(?:\.\d+){1,2})")
Review Comment:
I do not understand why AKS specific settings are introduced in K8s provider package (as well as RST docs above and tests below). Why is this not makde in the AWS specific package?
I my view K8s standard package should not be tainted by AWS, Google or Azure specific handling if no K8s standard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KubernetesHook: add AWS exec-auth botocore guardrails for EKS token flow (#60943) [airflow]
Posted by "Srabasti (via GitHub)" <gi...@apache.org>.
Srabasti commented on PR #61936:
URL: https://github.com/apache/airflow/pull/61936#issuecomment-3903453537
Static tests are failing @Vamsi-klu
Please run prek locally in your branch. Prek will fix any formatting errors, and then you can push the commit from your branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KubernetesHook: add AWS exec-auth botocore guardrails for EKS token flow (#60943) [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #61936:
URL: https://github.com/apache/airflow/pull/61936#discussion_r2855678618
##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/hooks/kubernetes.py:
##########
@@ -56,6 +60,11 @@
from kubernetes.client.models import CoreV1Event, CoreV1EventList, V1Job, V1Pod
LOADING_KUBE_CONFIG_FILE_RESOURCE = "Loading Kubernetes configuration file kube_config from {}..."
+_AWS_EXEC_AUTH_VERSION_CHECK_MODE_FIELD = "exec_auth_aws_cli_version_check_mode"
+_AWS_EXEC_AUTH_VERSION_CHECK_MODES = {"warn", "fail", "ignore"}
+_AWS_EXEC_AUTH_AWS_BINARY_NAMES = {"aws", "aws.exe", "aws2", "aws2.exe"}
+_AWS_EXEC_AUTH_FIXED_BOTOCORE_VERSION = (1, 40, 2)
+_BOTOCORE_VERSION_PATTERN = re.compile(r"botocore/(?P<version>\d+(?:\.\d+){1,2})")
Review Comment:
I'm pretty hesitant to add this much code to add a min version check like this, especially one that isn't really directly related to this provider.
The [lower bound in the amazon provider](https://github.com/apache/airflow/blob/6d6f8bf3d04fc3ee90a3dfd6f5c1e980dabb549a/providers/amazon/pyproject.toml#L69) is already higher than what we'd defend against here. And, our constraint files are already putting in a higher version. (2.11.1 has the new version, 2.11.0 did not, fwiw)
If we went with the proposed generic validator + validate botocore version in the aws provider, we'd not have a problem because of the min version over there (ignoring _when_ that min was bumped over there).
I'm just wondering if this is now not really a likely issue with the latest versions of stuff already...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org