[PR] Let fetch_access_token_for_cluster reuse a caller's boto3 session [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli opened a new pull request, #74399:
URL: https://github.com/apache/airflow/pull/74399

   `fetch_access_token_for_cluster` now takes an optional `session` argument. Callers that already hold a boto3 session can pass it in, and the token gets signed with that session's credentials and region instead of a fresh session built from the environment. It defaults to `None`, which keeps today's behavior, so existing callers are unaffected.
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (please specify the tool below)
   
   Generated-by: Claude Code (claude-fable-5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["eladkal (via GitHub)" <gi...@apache.org>]

eladkal commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4218475819


##########
providers/amazon/docs/executors/eks-executor.rst:
##########
@@ -0,0 +1,232 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. |executorName| replace:: EKS
+
+.. _eks_executor:
+
+================
+AWS EKS Executor
+================
+
+The EKS executor runs each Airflow task in its own pod on an Amazon EKS cluster.
+
+This executor extends the Kubernetes executor that ships in the ``cncf.kubernetes``
+provider. Pod scheduling, pod templates, per-task pod overrides, and log handling all
+behave the same way they do under
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor`. The part this executor
+adds is authentication. It builds the Kubernetes client for your EKS cluster from an
+Airflow AWS connection, so the scheduler does not need a kubeconfig file on disk and
+you do not have to refresh cluster credentials yourself.
+
+Because the executor inherits its pod behavior, everything written about the
+Kubernetes executor still applies, including the requirement for a database backend
+other than SQLite.
+
+For a quick start guide please see :ref:`here <eks_setup_guide>`.
+
+Requirements
+------------
+
+The executor needs ``apache-airflow-providers-cncf-kubernetes`` version 10.24.0 or
+newer, the release that added the ``client_factory`` setting the executor is built on.
+Install the ``cncf.kubernetes`` extra of the Amazon provider together with that
+version:
+
+.. code-block:: bash
+
+    pip install 'apache-airflow-providers-amazon[cncf.kubernetes]' \
+        'apache-airflow-providers-cncf-kubernetes>=10.24.0'
+
+The extra by itself declares a much older floor, because the Amazon provider's other
+Kubernetes integrations still work with it. A fresh install resolves to the newest
+``cncf.kubernetes`` regardless, so naming the version matters when an older one is
+already pinned in your environment. If the installed version is too old, the executor
+raises an error when it starts up rather than falling back to a client it cannot
+authenticate with.
+
+The AWS credentials the executor uses must be allowed to call ``eks:DescribeCluster``
+on the cluster, and the IAM principal behind those credentials must be granted
+access inside the cluster itself. On modern clusters this means an EKS access entry;
+on older clusters it means an entry in the ``aws-auth`` config map. Without that
+in-cluster grant the scheduler can read the cluster description but every Kubernetes
+API call is rejected.

Review Comment:
   Can this case be avoided by code checks? 



##########
providers/amazon/docs/executors/eks-executor.rst:
##########
@@ -0,0 +1,232 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. |executorName| replace:: EKS
+
+.. _eks_executor:
+
+================
+AWS EKS Executor
+================
+
+The EKS executor runs each Airflow task in its own pod on an Amazon EKS cluster.
+
+This executor extends the Kubernetes executor that ships in the ``cncf.kubernetes``
+provider. Pod scheduling, pod templates, per-task pod overrides, and log handling all
+behave the same way they do under
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor`. The part this executor
+adds is authentication. It builds the Kubernetes client for your EKS cluster from an
+Airflow AWS connection, so the scheduler does not need a kubeconfig file on disk and
+you do not have to refresh cluster credentials yourself.
+
+Because the executor inherits its pod behavior, everything written about the
+Kubernetes executor still applies, including the requirement for a database backend
+other than SQLite.
+
+For a quick start guide please see :ref:`here <eks_setup_guide>`.
+
+Requirements
+------------
+
+The executor needs ``apache-airflow-providers-cncf-kubernetes`` version 10.24.0 or
+newer, the release that added the ``client_factory`` setting the executor is built on.
+Install the ``cncf.kubernetes`` extra of the Amazon provider together with that
+version:
+
+.. code-block:: bash
+
+    pip install 'apache-airflow-providers-amazon[cncf.kubernetes]' \
+        'apache-airflow-providers-cncf-kubernetes>=10.24.0'
+
+The extra by itself declares a much older floor, because the Amazon provider's other
+Kubernetes integrations still work with it. A fresh install resolves to the newest
+``cncf.kubernetes`` regardless, so naming the version matters when an older one is
+already pinned in your environment. If the installed version is too old, the executor
+raises an error when it starts up rather than falling back to a client it cannot
+authenticate with.
+

Review Comment:
   Lets drop this paragraph. this should be checked with the code.
   If someone imports the executor without a compatible k8s provider version then just raise exception and explain this in the exception.



##########
providers/amazon/docs/executors/eks-executor.rst:
##########
@@ -0,0 +1,232 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. |executorName| replace:: EKS
+
+.. _eks_executor:
+
+================
+AWS EKS Executor
+================
+
+The EKS executor runs each Airflow task in its own pod on an Amazon EKS cluster.
+
+This executor extends the Kubernetes executor that ships in the ``cncf.kubernetes``
+provider. Pod scheduling, pod templates, per-task pod overrides, and log handling all
+behave the same way they do under
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor`. The part this executor
+adds is authentication. It builds the Kubernetes client for your EKS cluster from an
+Airflow AWS connection, so the scheduler does not need a kubeconfig file on disk and
+you do not have to refresh cluster credentials yourself.
+
+Because the executor inherits its pod behavior, everything written about the
+Kubernetes executor still applies, including the requirement for a database backend
+other than SQLite.
+
+For a quick start guide please see :ref:`here <eks_setup_guide>`.
+
+Requirements
+------------
+
+The executor needs ``apache-airflow-providers-cncf-kubernetes`` version 10.24.0 or
+newer, the release that added the ``client_factory`` setting the executor is built on.
+Install the ``cncf.kubernetes`` extra of the Amazon provider together with that
+version:
+
+.. code-block:: bash
+
+    pip install 'apache-airflow-providers-amazon[cncf.kubernetes]' \
+        'apache-airflow-providers-cncf-kubernetes>=10.24.0'
+
+The extra by itself declares a much older floor, because the Amazon provider's other
+Kubernetes integrations still work with it. A fresh install resolves to the newest
+``cncf.kubernetes`` regardless, so naming the version matters when an older one is
+already pinned in your environment. If the installed version is too old, the executor
+raises an error when it starts up rather than falling back to a client it cannot
+authenticate with.
+
+The AWS credentials the executor uses must be allowed to call ``eks:DescribeCluster``
+on the cluster, and the IAM principal behind those credentials must be granted
+access inside the cluster itself. On modern clusters this means an EKS access entry;
+on older clusters it means an entry in the ``aws-auth`` config map. Without that
+in-cluster grant the scheduler can read the cluster description but every Kubernetes
+API call is rejected.
+
+How authentication works
+------------------------
+
+When the executor starts, it plugs its own Kubernetes client into the Kubernetes
+executor. Every process that needs a client, including the pod watcher that Airflow runs
+as a separate process, builds its own from the ``[aws_eks_executor]`` settings.
+
+To build a client, the executor describes the cluster to find its API endpoint and
+certificate authority, then mints an authentication token for it. An EKS token is a presigned
+STS URL that stays valid for roughly fifteen minutes, while the scheduler holds a
+single client for as long as it runs. To keep the token current, the executor
+registers a refresh hook that the Kubernetes client calls on every authenticated
+request. Minting a token is a local signing operation with no network call, so
+refreshing that often costs very little. Rotation of the underlying AWS credentials
+is handled by botocore in the usual way.

Review Comment:
   what is usual way? This is ambiguous. lets avoid describing it and just replace this with a link to the relevant botocore docs where they explain how they do it?



##########
providers/amazon/docs/executors/eks-executor.rst:
##########
@@ -0,0 +1,232 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. |executorName| replace:: EKS
+
+.. _eks_executor:
+
+================
+AWS EKS Executor
+================
+
+The EKS executor runs each Airflow task in its own pod on an Amazon EKS cluster.
+
+This executor extends the Kubernetes executor that ships in the ``cncf.kubernetes``
+provider. Pod scheduling, pod templates, per-task pod overrides, and log handling all
+behave the same way they do under
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor`. The part this executor
+adds is authentication. It builds the Kubernetes client for your EKS cluster from an
+Airflow AWS connection, so the scheduler does not need a kubeconfig file on disk and
+you do not have to refresh cluster credentials yourself.
+
+Because the executor inherits its pod behavior, everything written about the
+Kubernetes executor still applies, including the requirement for a database backend
+other than SQLite.
+
+For a quick start guide please see :ref:`here <eks_setup_guide>`.
+
+Requirements
+------------
+
+The executor needs ``apache-airflow-providers-cncf-kubernetes`` version 10.24.0 or
+newer, the release that added the ``client_factory`` setting the executor is built on.
+Install the ``cncf.kubernetes`` extra of the Amazon provider together with that
+version:
+
+.. code-block:: bash
+
+    pip install 'apache-airflow-providers-amazon[cncf.kubernetes]' \
+        'apache-airflow-providers-cncf-kubernetes>=10.24.0'
+
+The extra by itself declares a much older floor, because the Amazon provider's other
+Kubernetes integrations still work with it. A fresh install resolves to the newest
+``cncf.kubernetes`` regardless, so naming the version matters when an older one is
+already pinned in your environment. If the installed version is too old, the executor
+raises an error when it starts up rather than falling back to a client it cannot
+authenticate with.
+
+The AWS credentials the executor uses must be allowed to call ``eks:DescribeCluster``
+on the cluster, and the IAM principal behind those credentials must be granted
+access inside the cluster itself. On modern clusters this means an EKS access entry;
+on older clusters it means an entry in the ``aws-auth`` config map. Without that
+in-cluster grant the scheduler can read the cluster description but every Kubernetes
+API call is rejected.
+
+How authentication works
+------------------------
+
+When the executor starts, it plugs its own Kubernetes client into the Kubernetes
+executor. Every process that needs a client, including the pod watcher that Airflow runs
+as a separate process, builds its own from the ``[aws_eks_executor]`` settings.
+
+To build a client, the executor describes the cluster to find its API endpoint and
+certificate authority, then mints an authentication token for it. An EKS token is a presigned
+STS URL that stays valid for roughly fifteen minutes, while the scheduler holds a
+single client for as long as it runs. To keep the token current, the executor
+registers a refresh hook that the Kubernetes client calls on every authenticated
+request. Minting a token is a local signing operation with no network call, so
+refreshing that often costs very little. Rotation of the underlying AWS credentials
+is handled by botocore in the usual way.
+
+Before it accepts any tasks, the executor checks that the cluster is ``ACTIVE`` (or
+``UPDATING``) and, unless ``check_health_on_startup`` is turned off, that it is allowed
+to list pods in its namespace. It refuses to start if either check fails.

Review Comment:
   why updating is valid? wha is check_health_on_startup? lets avoid referencing a setting as hard string. can you link to the setting section?



##########
providers/amazon/docs/executors/eks-executor.rst:
##########
@@ -0,0 +1,232 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. |executorName| replace:: EKS
+
+.. _eks_executor:
+
+================
+AWS EKS Executor
+================
+
+The EKS executor runs each Airflow task in its own pod on an Amazon EKS cluster.
+
+This executor extends the Kubernetes executor that ships in the ``cncf.kubernetes``
+provider. Pod scheduling, pod templates, per-task pod overrides, and log handling all
+behave the same way they do under
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor`. The part this executor
+adds is authentication. It builds the Kubernetes client for your EKS cluster from an
+Airflow AWS connection, so the scheduler does not need a kubeconfig file on disk and
+you do not have to refresh cluster credentials yourself.
+
+Because the executor inherits its pod behavior, everything written about the
+Kubernetes executor still applies, including the requirement for a database backend
+other than SQLite.
+
+For a quick start guide please see :ref:`here <eks_setup_guide>`.
+
+Requirements
+------------
+
+The executor needs ``apache-airflow-providers-cncf-kubernetes`` version 10.24.0 or
+newer, the release that added the ``client_factory`` setting the executor is built on.
+Install the ``cncf.kubernetes`` extra of the Amazon provider together with that
+version:
+
+.. code-block:: bash
+
+    pip install 'apache-airflow-providers-amazon[cncf.kubernetes]' \
+        'apache-airflow-providers-cncf-kubernetes>=10.24.0'
+
+The extra by itself declares a much older floor, because the Amazon provider's other
+Kubernetes integrations still work with it. A fresh install resolves to the newest
+``cncf.kubernetes`` regardless, so naming the version matters when an older one is
+already pinned in your environment. If the installed version is too old, the executor
+raises an error when it starts up rather than falling back to a client it cannot
+authenticate with.
+
+The AWS credentials the executor uses must be allowed to call ``eks:DescribeCluster``
+on the cluster, and the IAM principal behind those credentials must be granted
+access inside the cluster itself. On modern clusters this means an EKS access entry;
+on older clusters it means an entry in the ``aws-auth`` config map. Without that
+in-cluster grant the scheduler can read the cluster description but every Kubernetes
+API call is rejected.
+
+How authentication works
+------------------------
+
+When the executor starts, it plugs its own Kubernetes client into the Kubernetes
+executor. Every process that needs a client, including the pod watcher that Airflow runs
+as a separate process, builds its own from the ``[aws_eks_executor]`` settings.
+
+To build a client, the executor describes the cluster to find its API endpoint and
+certificate authority, then mints an authentication token for it. An EKS token is a presigned
+STS URL that stays valid for roughly fifteen minutes, while the scheduler holds a
+single client for as long as it runs. To keep the token current, the executor
+registers a refresh hook that the Kubernetes client calls on every authenticated
+request. Minting a token is a local signing operation with no network call, so
+refreshing that often costs very little. Rotation of the underlying AWS credentials
+is handled by botocore in the usual way.
+
+Before it accepts any tasks, the executor checks that the cluster is ``ACTIVE`` (or
+``UPDATING``) and, unless ``check_health_on_startup`` is turned off, that it is allowed
+to list pods in its namespace. It refuses to start if either check fails.
+
+.. _eks_config_options:
+
+Config Options
+--------------
+
+The executor reads its own settings from an ``aws_eks_executor`` section in
+``airflow.cfg``. You can also set any of them with an environment variable using the
+``AIRFLOW__AWS_EKS_EXECUTOR__<OPTION_NAME>`` form, for example
+``AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME=airflow-eks-cluster``. For more information on
+how to set these options, see `Setting Configuration Options
+<https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html>`__.

Review Comment:
   internal links should not be with http. if doc changes location we won't know it.



##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py:
##########
@@ -0,0 +1,145 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""AwsEksExecutor: run Airflow tasks as pods on an Amazon EKS cluster."""
+
+from __future__ import annotations
+
+import os
+from typing import TYPE_CHECKING
+
+from airflow.providers.common.compat.sdk import conf
+
+try:
+    from kubernetes.client.rest import ApiException
+
+    from airflow.providers.cncf.kubernetes import __version__ as cncf_kubernetes_version
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+    from airflow.providers.cncf.kubernetes.get_provider_info import (
+        get_provider_info as get_cncf_kubernetes_provider_info,
+    )
+except ImportError as e:
+    raise ImportError(
+        "AwsEksExecutor requires the cncf.kubernetes provider; install it with "
+        "pip install 'apache-airflow-providers-amazon[cncf.kubernetes]'"

Review Comment:
   probably best also to mention min version of supported cncf in the note?



##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py:
##########
@@ -0,0 +1,130 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kubernetes client factories for the AwsEksExecutor."""
+
+# Internal to AwsEksExecutor, not a public API. The executor writes the import paths of
+# _get_eks_kube_client and _get_eks_async_kube_client into [kubernetes_executor] client_factory
+# and async_client_factory, and cncf.kubernetes re-resolves them by that path in every process
+# that builds a client, including the spawned pod watcher. Keep eks_executor's paths in step.
+
+from __future__ import annotations
+
+import os
+import tempfile
+from base64 import b64decode
+from typing import TYPE_CHECKING
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.aws.hooks.eks import EksHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.providers.amazon.aws.utils.eks_get_token import fetch_access_token_for_cluster
+from airflow.providers.common.compat.sdk import conf
+
+if TYPE_CHECKING:
+    from kubernetes import client
+    from kubernetes_asyncio import client as async_client
+
+# UPDATING still serves the Kubernetes API, so only creating, deleting and failed clusters are refused.
+_USABLE_CLUSTER_STATUSES = ("ACTIVE", "UPDATING")
+
+
+def _get_eks_kube_client() -> client.CoreV1Api:
+    """Build a Kubernetes client for the configured EKS cluster, in memory and without a kubeconfig."""
+    from kubernetes import client
+
+    configuration = client.Configuration()
+    _configure_eks_auth(configuration)
+    return client.CoreV1Api(client.ApiClient(configuration=configuration))
+
+
+def _get_eks_async_kube_client() -> async_client.CoreV1Api:
+    """Build the asynchronous Kubernetes client used when ``async_pod_creation`` is enabled."""
+    from kubernetes_asyncio import client as async_client
+
+    configuration = async_client.Configuration()
+    _configure_eks_auth(configuration)
+    return async_client.CoreV1Api(async_client.ApiClient(configuration=configuration))
+
+
+def _configure_eks_auth(configuration: client.Configuration | async_client.Configuration) -> None:
+    cluster_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None)
+    if not cluster_name:
+        raise ValueError(f"[{CONFIG_GROUP_NAME}] cluster_name is required to build an EKS client")
+    region_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.REGION_NAME, fallback=None)
+    conn_id = conf.get(
+        CONFIG_GROUP_NAME,
+        AllEksConfigKeys.AWS_CONN_ID,
+        fallback=CONFIG_DEFAULTS[AllEksConfigKeys.AWS_CONN_ID],
+    )
+
+    eks_hook = EksHook(aws_conn_id=conn_id, region_name=region_name)
+    cluster = eks_hook.conn.describe_cluster(name=cluster_name)["cluster"]
+    if cluster["status"] not in _USABLE_CLUSTER_STATUSES:
+        raise ValueError(
+            f"EKS cluster {cluster_name} is {cluster['status']}; the executor needs it to be ACTIVE"
+        )
+    session = eks_hook.get_session()
+
+    # EKS only accepts tokens presigned against the regional STS endpoint; some regions
+    # otherwise default to the global one. Same dance as EksHook.generate_config_file.
+    os.environ["AWS_STS_REGIONAL_ENDPOINTS"] = "regional"
+    try:
+        sts_endpoint = StsHook(
+            aws_conn_id=conn_id, region_name=session.region_name
+        ).conn_client_meta.endpoint_url
+    finally:
+        del os.environ["AWS_STS_REGIONAL_ENDPOINTS"]
+    sts_url = f"{sts_endpoint}/?Action=GetCallerIdentity&Version=2011-06-15"
+
+    configuration.host = cluster["endpoint"]
+    configuration.ssl_ca_cert = _write_cluster_ca_file(cluster["certificateAuthority"]["data"])
+    # Key the bearer auth under both identifiers. kubernetes-client >= 36 looks the token up under
+    # "BearerToken" and only aliases the api_key (not api_key_prefix) back to the legacy
+    # "authorization" slot (see kubernetes-client/python#2595); older clients use "authorization".
+    # Setting the prefix only under "authorization" makes the newer client emit the raw token with
+    # no "Bearer " prefix, which the API server rejects with 401, so set both slots.
+    for identifier in ("BearerToken", "authorization"):
+        configuration.api_key_prefix[identifier] = "Bearer"

Review Comment:
   This is too much wording. We can avoid this all together by checking the underlying library version.
   If >=36 do X
   else do Y
   that way once we bump min library version would also be easier to adjust the code.



##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py:
##########
@@ -0,0 +1,145 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""AwsEksExecutor: run Airflow tasks as pods on an Amazon EKS cluster."""
+
+from __future__ import annotations
+
+import os
+from typing import TYPE_CHECKING
+
+from airflow.providers.common.compat.sdk import conf
+
+try:
+    from kubernetes.client.rest import ApiException
+
+    from airflow.providers.cncf.kubernetes import __version__ as cncf_kubernetes_version
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+    from airflow.providers.cncf.kubernetes.get_provider_info import (
+        get_provider_info as get_cncf_kubernetes_provider_info,
+    )
+except ImportError as e:
+    raise ImportError(
+        "AwsEksExecutor requires the cncf.kubernetes provider; install it with "
+        "pip install 'apache-airflow-providers-amazon[cncf.kubernetes]'"
+    ) from e
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+
+# Import paths, because cncf.kubernetes re-resolves the factories in each process (see _client_factory).
+_FACTORY_MODULE = "airflow.providers.amazon.aws.executors.eks._client_factory"
+_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_kube_client"
+_ASYNC_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_async_kube_client"
+# Only used in the error message below. The check looks for the client_factory option instead of
+# comparing versions, because an unreleased source tree still reports the previous release.
+MIN_CNCF_KUBERNETES_VERSION = "10.24.0"
+
+
+class AwsEksExecutor(KubernetesExecutor):
+    """
+    A KubernetesExecutor that authenticates against an Amazon EKS cluster.
+
+    Builds the Kubernetes client from ``[aws_eks_executor]`` configuration and keeps the
+    short-lived EKS token fresh in-process. All pod-level behaviour comes unchanged from the
+    KubernetesExecutor and its ``[kubernetes_executor]`` configuration.
+    """
+
+    # The client factories read the un-prefixed [aws_eks_executor] section, so every team
+    # would land on the same cluster.
+    supports_multi_team: bool = False
+
+    def __init__(self, *args, **kwargs):
+        self._validate_eks_config()
+        self._require_client_factory_support()
+        self._ensure_client_factory()
+        super().__init__(*args, **kwargs)
+
+    def start(self) -> None:
+        """Call this when the Executor is run for the first time by the scheduler."""
+        super().start()
+        check_health = conf.getboolean(
+            CONFIG_GROUP_NAME,
+            AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP,
+            fallback=CONFIG_DEFAULTS[AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP],
+        )
+
+        if not check_health:
+            return
+
+        self.log.info("Starting EKS Executor and determining health...")
+        try:
+            self.check_health()
+        except RuntimeError:
+            self.log.error("Stopping the Airflow Scheduler from starting until the issue is resolved.")
+            raise
+
+    def check_health(self) -> None:
+        """
+        Make a test Kubernetes API call to check the health of the EKS Executor.
+
+        Building the client only proves the cluster exists; without this, a missing EKS access
+        entry or RBAC binding only shows up later as a watcher error loop.
+        """
+        if TYPE_CHECKING:
+            assert self.kube_client
+        namespace = self.kube_config.kube_namespace
+        try:
+            self.kube_client.list_namespaced_pod(namespace, limit=1)
+        except ApiException as e:
+            raise RuntimeError(
+                f"EKS Executor health check has failed because: cannot list pods in namespace {namespace} "
+                f"({e.status} {e.reason}). Check the EKS access entry and RBAC for the executor's IAM role."
+            ) from e
+        self.log.info("EKS Executor health check has succeeded.")
+
+    @staticmethod
+    def _require_client_factory_support() -> None:
+        # A cncf.kubernetes without the seam imports fine but never reads client_factory, so it
+        # would build a default client from kubeconfig and fail later as an authentication error.
+        provider_config = get_cncf_kubernetes_provider_info().get("config", {})
+        if "client_factory" not in provider_config.get("kubernetes_executor", {}).get("options", {}):
+            raise ImportError(
+                "AwsEksExecutor requires apache-airflow-providers-cncf-kubernetes>="
+                f"{MIN_CNCF_KUBERNETES_VERSION}, which honors the [kubernetes_executor] "
+                f"client_factory setting. The installed {cncf_kubernetes_version} ignores it."
+            )

Review Comment:
   I prefer that all checks against supported versions be done at the import level
   If we don't have a good version we should never reach to this function.
   
   That way functions should not worry about supported versions checks.



##########
providers/amazon/docs/executors/eks-executor.rst:
##########


Review Comment:
   I think we need also an edit to the AWS executors docs to answer the question:
   if I am AWS user who want to understand my options how do I choose between Eks vs Lambda vs ECS?



##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py:
##########
@@ -0,0 +1,130 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kubernetes client factories for the AwsEksExecutor."""
+
+# Internal to AwsEksExecutor, not a public API. The executor writes the import paths of
+# _get_eks_kube_client and _get_eks_async_kube_client into [kubernetes_executor] client_factory
+# and async_client_factory, and cncf.kubernetes re-resolves them by that path in every process
+# that builds a client, including the spawned pod watcher. Keep eks_executor's paths in step.
+
+from __future__ import annotations
+
+import os
+import tempfile
+from base64 import b64decode
+from typing import TYPE_CHECKING
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.aws.hooks.eks import EksHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.providers.amazon.aws.utils.eks_get_token import fetch_access_token_for_cluster
+from airflow.providers.common.compat.sdk import conf
+
+if TYPE_CHECKING:
+    from kubernetes import client
+    from kubernetes_asyncio import client as async_client
+
+# UPDATING still serves the Kubernetes API, so only creating, deleting and failed clusters are refused.
+_USABLE_CLUSTER_STATUSES = ("ACTIVE", "UPDATING")
+
+
+def _get_eks_kube_client() -> client.CoreV1Api:
+    """Build a Kubernetes client for the configured EKS cluster, in memory and without a kubeconfig."""
+    from kubernetes import client
+
+    configuration = client.Configuration()
+    _configure_eks_auth(configuration)
+    return client.CoreV1Api(client.ApiClient(configuration=configuration))
+
+
+def _get_eks_async_kube_client() -> async_client.CoreV1Api:
+    """Build the asynchronous Kubernetes client used when ``async_pod_creation`` is enabled."""
+    from kubernetes_asyncio import client as async_client
+
+    configuration = async_client.Configuration()
+    _configure_eks_auth(configuration)
+    return async_client.CoreV1Api(async_client.ApiClient(configuration=configuration))
+
+
+def _configure_eks_auth(configuration: client.Configuration | async_client.Configuration) -> None:
+    cluster_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None)
+    if not cluster_name:
+        raise ValueError(f"[{CONFIG_GROUP_NAME}] cluster_name is required to build an EKS client")
+    region_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.REGION_NAME, fallback=None)
+    conn_id = conf.get(
+        CONFIG_GROUP_NAME,
+        AllEksConfigKeys.AWS_CONN_ID,
+        fallback=CONFIG_DEFAULTS[AllEksConfigKeys.AWS_CONN_ID],
+    )
+
+    eks_hook = EksHook(aws_conn_id=conn_id, region_name=region_name)
+    cluster = eks_hook.conn.describe_cluster(name=cluster_name)["cluster"]
+    if cluster["status"] not in _USABLE_CLUSTER_STATUSES:
+        raise ValueError(
+            f"EKS cluster {cluster_name} is {cluster['status']}; the executor needs it to be ACTIVE"
+        )

Review Comment:
   Docs mention that cluster also accept tasks in `UPDATING` state but here we check for active.
   I assume doc entry is wrong?



##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py:
##########
@@ -0,0 +1,130 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kubernetes client factories for the AwsEksExecutor."""
+
+# Internal to AwsEksExecutor, not a public API. The executor writes the import paths of
+# _get_eks_kube_client and _get_eks_async_kube_client into [kubernetes_executor] client_factory
+# and async_client_factory, and cncf.kubernetes re-resolves them by that path in every process
+# that builds a client, including the spawned pod watcher. Keep eks_executor's paths in step.
+
+from __future__ import annotations
+
+import os
+import tempfile
+from base64 import b64decode
+from typing import TYPE_CHECKING
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.aws.hooks.eks import EksHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.providers.amazon.aws.utils.eks_get_token import fetch_access_token_for_cluster
+from airflow.providers.common.compat.sdk import conf
+
+if TYPE_CHECKING:
+    from kubernetes import client
+    from kubernetes_asyncio import client as async_client
+
+# UPDATING still serves the Kubernetes API, so only creating, deleting and failed clusters are refused.
+_USABLE_CLUSTER_STATUSES = ("ACTIVE", "UPDATING")
+
+
+def _get_eks_kube_client() -> client.CoreV1Api:
+    """Build a Kubernetes client for the configured EKS cluster, in memory and without a kubeconfig."""
+    from kubernetes import client
+
+    configuration = client.Configuration()
+    _configure_eks_auth(configuration)
+    return client.CoreV1Api(client.ApiClient(configuration=configuration))
+
+
+def _get_eks_async_kube_client() -> async_client.CoreV1Api:
+    """Build the asynchronous Kubernetes client used when ``async_pod_creation`` is enabled."""
+    from kubernetes_asyncio import client as async_client
+
+    configuration = async_client.Configuration()
+    _configure_eks_auth(configuration)
+    return async_client.CoreV1Api(async_client.ApiClient(configuration=configuration))
+
+
+def _configure_eks_auth(configuration: client.Configuration | async_client.Configuration) -> None:
+    cluster_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None)
+    if not cluster_name:
+        raise ValueError(f"[{CONFIG_GROUP_NAME}] cluster_name is required to build an EKS client")
+    region_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.REGION_NAME, fallback=None)
+    conn_id = conf.get(
+        CONFIG_GROUP_NAME,
+        AllEksConfigKeys.AWS_CONN_ID,
+        fallback=CONFIG_DEFAULTS[AllEksConfigKeys.AWS_CONN_ID],
+    )
+
+    eks_hook = EksHook(aws_conn_id=conn_id, region_name=region_name)
+    cluster = eks_hook.conn.describe_cluster(name=cluster_name)["cluster"]
+    if cluster["status"] not in _USABLE_CLUSTER_STATUSES:
+        raise ValueError(
+            f"EKS cluster {cluster_name} is {cluster['status']}; the executor needs it to be ACTIVE"
+        )
+    session = eks_hook.get_session()
+
+    # EKS only accepts tokens presigned against the regional STS endpoint; some regions
+    # otherwise default to the global one. Same dance as EksHook.generate_config_file.
+    os.environ["AWS_STS_REGIONAL_ENDPOINTS"] = "regional"
+    try:
+        sts_endpoint = StsHook(
+            aws_conn_id=conn_id, region_name=session.region_name
+        ).conn_client_meta.endpoint_url
+    finally:
+        del os.environ["AWS_STS_REGIONAL_ENDPOINTS"]
+    sts_url = f"{sts_endpoint}/?Action=GetCallerIdentity&Version=2011-06-15"
+
+    configuration.host = cluster["endpoint"]
+    configuration.ssl_ca_cert = _write_cluster_ca_file(cluster["certificateAuthority"]["data"])
+    # Key the bearer auth under both identifiers. kubernetes-client >= 36 looks the token up under
+    # "BearerToken" and only aliases the api_key (not api_key_prefix) back to the legacy
+    # "authorization" slot (see kubernetes-client/python#2595); older clients use "authorization".
+    # Setting the prefix only under "authorization" makes the newer client emit the raw token with
+    # no "Bearer " prefix, which the API server rejects with 401, so set both slots.
+    for identifier in ("BearerToken", "authorization"):
+        configuration.api_key_prefix[identifier] = "Bearer"
+
+    # The EKS token is a presigned STS URL valid for ~15 minutes, but the scheduler holds
+    # one client for its whole lifetime. refresh_api_key_hook runs on every authenticated
+    # request (Configuration.get_api_key_with_prefix), and minting is a local SigV4 signing
+    # with no network round-trip, so re-minting per request keeps the token fresh at
+    # negligible cost. Botocore refreshes the session's own credentials when they rotate.
+    def refresh_api_key(config: client.Configuration | async_client.Configuration) -> None:
+        token = fetch_access_token_for_cluster(
+            cluster_name, sts_url, region_name=session.region_name, session=session
+        )

Review Comment:
   I am not sure I understand this one.
   Shouldn't the refresh request be in a side car container?



##########
providers/amazon/docs/executors/eks-executor.rst:
##########
@@ -0,0 +1,232 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. |executorName| replace:: EKS
+
+.. _eks_executor:
+
+================
+AWS EKS Executor
+================
+
+The EKS executor runs each Airflow task in its own pod on an Amazon EKS cluster.
+
+This executor extends the Kubernetes executor that ships in the ``cncf.kubernetes``
+provider. Pod scheduling, pod templates, per-task pod overrides, and log handling all
+behave the same way they do under
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor`. The part this executor
+adds is authentication. It builds the Kubernetes client for your EKS cluster from an
+Airflow AWS connection, so the scheduler does not need a kubeconfig file on disk and
+you do not have to refresh cluster credentials yourself.
+
+Because the executor inherits its pod behavior, everything written about the
+Kubernetes executor still applies, including the requirement for a database backend
+other than SQLite.
+
+For a quick start guide please see :ref:`here <eks_setup_guide>`.
+
+Requirements
+------------
+
+The executor needs ``apache-airflow-providers-cncf-kubernetes`` version 10.24.0 or
+newer, the release that added the ``client_factory`` setting the executor is built on.
+Install the ``cncf.kubernetes`` extra of the Amazon provider together with that
+version:
+
+.. code-block:: bash
+
+    pip install 'apache-airflow-providers-amazon[cncf.kubernetes]' \
+        'apache-airflow-providers-cncf-kubernetes>=10.24.0'
+
+The extra by itself declares a much older floor, because the Amazon provider's other
+Kubernetes integrations still work with it. A fresh install resolves to the newest
+``cncf.kubernetes`` regardless, so naming the version matters when an older one is
+already pinned in your environment. If the installed version is too old, the executor
+raises an error when it starts up rather than falling back to a client it cannot
+authenticate with.
+
+The AWS credentials the executor uses must be allowed to call ``eks:DescribeCluster``
+on the cluster, and the IAM principal behind those credentials must be granted
+access inside the cluster itself. On modern clusters this means an EKS access entry;
+on older clusters it means an entry in the ``aws-auth`` config map. Without that
+in-cluster grant the scheduler can read the cluster description but every Kubernetes
+API call is rejected.
+
+How authentication works
+------------------------
+
+When the executor starts, it plugs its own Kubernetes client into the Kubernetes
+executor. Every process that needs a client, including the pod watcher that Airflow runs
+as a separate process, builds its own from the ``[aws_eks_executor]`` settings.
+
+To build a client, the executor describes the cluster to find its API endpoint and
+certificate authority, then mints an authentication token for it. An EKS token is a presigned
+STS URL that stays valid for roughly fifteen minutes, while the scheduler holds a
+single client for as long as it runs. To keep the token current, the executor
+registers a refresh hook that the Kubernetes client calls on every authenticated
+request. Minting a token is a local signing operation with no network call, so
+refreshing that often costs very little. Rotation of the underlying AWS credentials
+is handled by botocore in the usual way.
+
+Before it accepts any tasks, the executor checks that the cluster is ``ACTIVE`` (or
+``UPDATING``) and, unless ``check_health_on_startup`` is turned off, that it is allowed
+to list pods in its namespace. It refuses to start if either check fails.
+
+.. _eks_config_options:
+
+Config Options
+--------------
+
+The executor reads its own settings from an ``aws_eks_executor`` section in
+``airflow.cfg``. You can also set any of them with an environment variable using the
+``AIRFLOW__AWS_EKS_EXECUTOR__<OPTION_NAME>`` form, for example
+``AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME=airflow-eks-cluster``. For more information on
+how to set these options, see `Setting Configuration Options
+<https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html>`__.
+
+Required config options:
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  CLUSTER_NAME - The name of the Amazon EKS cluster that tasks run on. The
+   executor refuses to start if this is unset. Required.
+
+Optional config options:
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  CONN_ID - The Airflow connection (i.e. credentials) used by the EKS
+   executor to make API calls to Amazon EKS. Defaults to ``aws_default``.
+-  REGION_NAME - The AWS Region the cluster is in. When this is left empty, the
+   region comes from the standard boto3 resolution order.
+-  CHECK_HEALTH_ON_STARTUP - Whether to check on startup that the executor can
+   list pods in its namespace. Defaults to ``True``.
+
+Pod-level configuration
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Settings that describe the worker pods themselves stay in the
+``[kubernetes_executor]`` section, exactly as they are for the Kubernetes executor.
+This includes ``namespace``, ``pod_template_file``, ``worker_container_repository``,
+``worker_container_tag``, and ``delete_worker_pods``. See
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor` for the full list
+and for how pod templates and ``pod_override`` work.
+
+Leave ``client_factory`` and ``async_client_factory`` in that section unset. The
+executor sets both itself, and raises an error at startup if either is set to something
+else, so that a stale or conflicting setting cannot silently send tasks to the wrong
+cluster.
+
+The worker pod template
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Set ``[kubernetes_executor] pod_template_file`` to the path of a YAML file describing
+the worker pod. Treat it as required. There is no usable default: when the setting is
+empty the Kubernetes executor only logs a warning that the model file does not exist
+and then builds a worker pod from an empty template. That pod is missing the container
+the executor needs, so the failure arrives later and somewhere else, usually as a
+rejected pod or a task that never reports back.
+
+The file must define a container named ``base`` as the first entry in
+``spec.containers``, and that container has to run your Airflow worker image.
+The ``pod_template_file`` section of
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor` covers the full set
+of requirements and includes templates for Dags baked into the image, Dags on a volume,
+and git-sync. Any of those works here unchanged, since this executor only replaces how
+the Kubernetes client is authenticated.
+
+.. _eks_logging:
+
+.. include:: general.rst
+  :start-after: .. BEGIN LOGGING
+  :end-before: .. END LOGGING
+
+-  Worker pods are deleted once their task finishes (``[kubernetes_executor]
+   delete_worker_pods``), and their logs go with them, so configure remote
+   logging to CloudWatch Logs or S3 to keep task logs viewable in the Airflow UI.
+-  The remote logging configuration must be set on the worker pods as well as on
+   the scheduler and API server, and the worker pods need an IAM role, for
+   example through EKS Pod Identity, that can write to the log destination.
+
+.. _eks_setup_guide:
+
+Setting up an EKS Executor for Apache Airflow
+---------------------------------------------
+
+Grant access to the cluster
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Create an EKS access entry for the IAM role or user that the scheduler runs as, and
+associate a policy that allows it to create and watch pods in the namespace you plan
+to use. If you manage cluster access through the ``aws-auth`` config map instead, add
+the principal there and map it to a Kubernetes group with the same permissions.
+
+Configure Airflow
+~~~~~~~~~~~~~~~~~
+
+Select the executor and name your cluster:
+
+.. code-block:: ini
+
+    [core]
+    executor = airflow.providers.amazon.aws.executors.eks.eks_executor.AwsEksExecutor
+
+    [aws_eks_executor]
+    cluster_name = airflow-eks-cluster
+    region_name = us-east-1
+    conn_id = aws_default
+
+    [kubernetes_executor]
+    namespace = airflow
+    pod_template_file = /opt/airflow/pod_templates/worker_template.yaml
+    worker_container_repository = my-account.dkr.ecr.us-east-1.amazonaws.com/airflow
+    worker_container_tag = latest
+
+The worker image must contain Airflow and the Amazon provider, and it needs access
+to your Dag files and a network path to the Airflow API server, in the same way any
+Kubernetes executor worker does.
+
+Task logging
+~~~~~~~~~~~~
+
+Configure remote logging as described in the :ref:`logging <eks_logging>` section, so
+that task logs stay viewable in the Airflow UI after the worker pods are deleted.
+
+Verify the setup
+~~~~~~~~~~~~~~~~
+
+Start the scheduler and trigger a small Dag. A successful run shows a worker pod
+appearing in your chosen namespace and the task finishing in the Airflow UI. If the
+scheduler logs an authentication or forbidden error from the Kubernetes API, the
+in-cluster access grant for your IAM principal is the first thing to check.
+
+Multi-team deployments
+----------------------
+
+The executor cannot be used as a team executor yet, because its settings are read from
+the un-prefixed ``[aws_eks_executor]`` section and every team would share one cluster.

Review Comment:
   what is team executor? are you referencing multi team feature? if so please link to the relevant doc section.
   
   By `yet` you mean it's planned soon or is it long term plan? if so lets add github issue for users to be able to interact with us on that feature request



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on PR #74399:
URL: https://github.com/apache/airflow/pull/74399#issuecomment-6177952725

   @eladkal Any thoughts? Now that it's [working e2e](https://aws-mwaa.github.io/#/open-source/system-tests/executor/eks/dashboard.html), I'm happy to merge this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4291755499


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py:
##########
@@ -0,0 +1,137 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""AwsEksExecutor: run Airflow tasks as pods on an Amazon EKS cluster."""
+
+from __future__ import annotations
+
+import os
+from typing import TYPE_CHECKING
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException, conf
+
+_CNCF_KUBERNETES_REQUIRED = "AwsEksExecutor requires apache-airflow-providers-cncf-kubernetes>=10.24.0"
+
+try:
+    from kubernetes.client.rest import ApiException
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+    from airflow.providers.cncf.kubernetes.get_provider_info import (
+        get_provider_info as get_cncf_kubernetes_provider_info,
+    )
+except ImportError as e:
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED) from e
+
+# Check for the option, not the version, since unreleased trees report the previous version.
+if "client_factory" not in (
+    get_cncf_kubernetes_provider_info().get("config", {}).get("kubernetes_executor", {}).get("options", {})
+):
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED)
+
+_FACTORY_MODULE = "airflow.providers.amazon.aws.executors.eks._client_factory"
+_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_kube_client"
+_ASYNC_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_async_kube_client"
+
+
+class AwsEksExecutor(KubernetesExecutor):
+    """
+    A KubernetesExecutor that authenticates against an Amazon EKS cluster.
+
+    Builds the Kubernetes client from ``[aws_eks_executor]`` configuration and keeps the
+    short-lived EKS token fresh in-process. All pod-level behaviour comes unchanged from the
+    KubernetesExecutor and its ``[kubernetes_executor]`` configuration.
+    """
+
+    supports_multi_team: bool = True
+
+    def __init__(self, *args, **kwargs):
+        self._validate_eks_config()
+        super().__init__(*args, **kwargs)
+        # Needs team_name, which super().__init__ sets.
+        self._ensure_client_factory()
+
+    def start(self) -> None:
+        """Call this when the Executor is run for the first time by the scheduler."""
+        super().start()
+        check_health = conf.getboolean(
+            CONFIG_GROUP_NAME,
+            AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP,
+            fallback=CONFIG_DEFAULTS[AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP],
+        )
+
+        if not check_health:
+            return
+
+        self.log.info("Starting EKS Executor and determining health...")
+        try:
+            self.check_health()
+        except RuntimeError:
+            self.log.error("Stopping the Airflow Scheduler from starting until the issue is resolved.")
+            raise
+
+    def check_health(self) -> None:
+        """
+        Make a test Kubernetes API call to check the health of the EKS Executor.
+
+        Building the client only proves the cluster exists; without this, a missing EKS access
+        entry or RBAC binding only shows up later as a watcher error loop.
+        """
+        if TYPE_CHECKING:
+            assert self.kube_client
+        namespace = self.kube_config.kube_namespace
+        try:
+            self.kube_client.list_namespaced_pod(namespace, limit=1)
+        except ApiException as e:
+            raise RuntimeError(
+                f"EKS Executor health check has failed because: cannot list pods in namespace {namespace} "
+                f"({e.status} {e.reason}). The IAM principal behind the AWS connection needs an EKS access "
+                "entry (or aws-auth mapping) on the cluster that allows create, get, list, watch, patch and "
+                f"delete on pods and get on pods/log in namespace {namespace}. See 'Grant access to the "
+                "cluster' in the AwsEksExecutor docs."
+            ) from e
+        self.log.info("EKS Executor health check has succeeded.")
+
+    @staticmethod
+    def _validate_eks_config() -> None:
+        if not conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None):
+            raise ValueError(
+                f"AwsEksExecutor requires [{CONFIG_GROUP_NAME}] cluster_name to be set. "
+                "Set it in airflow.cfg or with AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME."
+            )
+
+    def _ensure_client_factory(self) -> None:
+        # A team's factory is looked up in team config only, with no global fallback.
+        team_name = self.team_name
+        team_kwargs = {"team_name": team_name} if team_name else {}
+        env_var_prefix = f"AIRFLOW__{team_name.upper()}___" if team_name else "AIRFLOW__"
+        for key, path in (
+            ("client_factory", _CLIENT_FACTORY_PATH),
+            ("async_client_factory", _ASYNC_CLIENT_FACTORY_PATH),
+        ):
+            configured = conf.get("kubernetes_executor", key, fallback=None, **team_kwargs)
+            if configured and configured != path:
+                raise ValueError(
+                    f"AwsEksExecutor sets [kubernetes_executor] {key} itself, so leave it unset; "
+                    f"it is set to {configured}"
+                )
+            if not configured:
+                # An env var, not conf.set, so the spawned pod watcher process sees it too.
+                os.environ[f"{env_var_prefix}KUBERNETES_EXECUTOR__{key.upper()}"] = path

Review Comment:
   This is an interesting point, and I'd expect this to be quite rare that a single airflow runtime is simultaneously running `KubernetesExecutor` and `AwsEksExecutor`, but in theory it is possible, and this would fail silently. To properly deal with it, this issue https://github.com/apache/airflow/issues/74844 needs to be handled and the requisite changes in cncf need to be present. In the meantime, I've added a startup guard to check for this specific circumstance and fail fast.
   
   Resolving this comment for now as it your first comment captures the same sentiment, so let's direct reviewer attention to that comment to agree on an approach.



##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py:
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kubernetes client factories for the AwsEksExecutor."""
+
+# Internal to AwsEksExecutor, which writes these functions' import paths into
+# [kubernetes_executor] client_factory and async_client_factory.
+
+from __future__ import annotations
+
+import os
+import tempfile
+import time
+from base64 import b64decode
+from typing import TYPE_CHECKING
+
+import kubernetes
+from packaging.version import Version
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.aws.hooks.eks import EksHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.providers.amazon.aws.utils.eks_get_token import (
+    TOKEN_EXPIRATION_MINUTES,
+    fetch_access_token_for_cluster,
+)
+from airflow.providers.common.compat.sdk import conf
+
+if TYPE_CHECKING:
+    from kubernetes import client
+    from kubernetes_asyncio import client as async_client
+
+# UPDATING still serves the Kubernetes API, so every other status is refused.
+_USABLE_CLUSTER_STATUSES = ("ACTIVE", "UPDATING")
+
+
+def _get_sync_token_key(kubernetes_version: str) -> str:
+    # kubernetes 36 moved the bearer token from the "authorization" api_key to "BearerToken".
+    return "BearerToken" if Version(kubernetes_version).major >= 36 else "authorization"
+
+
+_SYNC_TOKEN_KEY = _get_sync_token_key(kubernetes.__version__)
+
+
+def _get_eks_kube_client() -> client.CoreV1Api:
+    """Build a Kubernetes client for the configured EKS cluster, in memory and without a kubeconfig."""
+    from kubernetes import client
+
+    configuration = client.Configuration()
+    _configure_eks_auth(configuration, _SYNC_TOKEN_KEY)
+    return client.CoreV1Api(client.ApiClient(configuration=configuration))
+
+
+def _get_eks_async_kube_client() -> async_client.CoreV1Api:
+    """Build the asynchronous Kubernetes client used when ``async_pod_creation`` is enabled."""
+    from kubernetes_asyncio import client as async_client
+
+    configuration = async_client.Configuration()
+    # kubernetes_asyncio has always used "BearerToken".
+    _configure_eks_auth(configuration, "BearerToken")
+    return async_client.CoreV1Api(async_client.ApiClient(configuration=configuration))
+
+
+def _configure_eks_auth(
+    configuration: client.Configuration | async_client.Configuration, token_key: str
+) -> None:
+    cluster_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None)
+    if not cluster_name:
+        raise ValueError(
+            f"[{CONFIG_GROUP_NAME}] cluster_name is required to build an EKS client. "
+            "Set it in airflow.cfg or with AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME."
+        )
+    region_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.REGION_NAME, fallback=None)
+    conn_id = conf.get(
+        CONFIG_GROUP_NAME,
+        AllEksConfigKeys.AWS_CONN_ID,
+        fallback=CONFIG_DEFAULTS[AllEksConfigKeys.AWS_CONN_ID],
+    )
+
+    eks_hook = EksHook(aws_conn_id=conn_id, region_name=region_name)
+    cluster = eks_hook.conn.describe_cluster(name=cluster_name)["cluster"]
+    if cluster["status"] not in _USABLE_CLUSTER_STATUSES:
+        raise ValueError(
+            f"EKS cluster {cluster_name} is {cluster['status']}; the executor needs it to be ACTIVE or "
+            f"UPDATING. Wait for it to become ACTIVE, or point [{CONFIG_GROUP_NAME}] cluster_name at a "
+            "usable cluster."
+        )
+    session = eks_hook.get_session()
+
+    # EKS only accepts tokens presigned against the regional STS endpoint.
+    os.environ["AWS_STS_REGIONAL_ENDPOINTS"] = "regional"
+    try:
+        sts_endpoint = StsHook(
+            aws_conn_id=conn_id, region_name=session.region_name
+        ).conn_client_meta.endpoint_url
+    finally:
+        del os.environ["AWS_STS_REGIONAL_ENDPOINTS"]

Review Comment:
   Good catch, fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["o-nikolas (via GitHub)" <gi...@apache.org>]

o-nikolas commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4296461943


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py:
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kubernetes client factories for the AwsEksExecutor."""
+
+# Internal to AwsEksExecutor, which writes these functions' import paths into
+# [kubernetes_executor] client_factory and async_client_factory.
+
+from __future__ import annotations
+
+import os
+import tempfile
+import time
+from base64 import b64decode
+from typing import TYPE_CHECKING
+
+import kubernetes
+from packaging.version import Version
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.aws.hooks.eks import EksHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.providers.amazon.aws.utils.eks_get_token import (
+    TOKEN_EXPIRATION_MINUTES,
+    fetch_access_token_for_cluster,
+)
+from airflow.providers.common.compat.sdk import conf
+
+if TYPE_CHECKING:
+    from kubernetes import client
+    from kubernetes_asyncio import client as async_client
+
+# UPDATING still serves the Kubernetes API, so every other status is refused.
+_USABLE_CLUSTER_STATUSES = ("ACTIVE", "UPDATING")
+
+
+def _get_sync_token_key(kubernetes_version: str) -> str:
+    # kubernetes 36 moved the bearer token from the "authorization" api_key to "BearerToken".
+    return "BearerToken" if Version(kubernetes_version).major >= 36 else "authorization"
+
+
+_SYNC_TOKEN_KEY = _get_sync_token_key(kubernetes.__version__)
+
+
+def _get_eks_kube_client() -> client.CoreV1Api:
+    """Build a Kubernetes client for the configured EKS cluster, in memory and without a kubeconfig."""
+    from kubernetes import client
+
+    configuration = client.Configuration()
+    _configure_eks_auth(configuration, _SYNC_TOKEN_KEY)
+    return client.CoreV1Api(client.ApiClient(configuration=configuration))
+
+
+def _get_eks_async_kube_client() -> async_client.CoreV1Api:
+    """Build the asynchronous Kubernetes client used when ``async_pod_creation`` is enabled."""
+    from kubernetes_asyncio import client as async_client
+
+    configuration = async_client.Configuration()
+    # kubernetes_asyncio has always used "BearerToken".
+    _configure_eks_auth(configuration, "BearerToken")
+    return async_client.CoreV1Api(async_client.ApiClient(configuration=configuration))
+
+
+def _configure_eks_auth(
+    configuration: client.Configuration | async_client.Configuration, token_key: str
+) -> None:
+    cluster_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None)
+    if not cluster_name:
+        raise ValueError(
+            f"[{CONFIG_GROUP_NAME}] cluster_name is required to build an EKS client. "
+            "Set it in airflow.cfg or with AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME."
+        )
+    region_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.REGION_NAME, fallback=None)
+    conn_id = conf.get(
+        CONFIG_GROUP_NAME,
+        AllEksConfigKeys.AWS_CONN_ID,
+        fallback=CONFIG_DEFAULTS[AllEksConfigKeys.AWS_CONN_ID],
+    )
+
+    eks_hook = EksHook(aws_conn_id=conn_id, region_name=region_name)
+    cluster = eks_hook.conn.describe_cluster(name=cluster_name)["cluster"]
+    if cluster["status"] not in _USABLE_CLUSTER_STATUSES:
+        raise ValueError(
+            f"EKS cluster {cluster_name} is {cluster['status']}; the executor needs it to be ACTIVE or "
+            f"UPDATING. Wait for it to become ACTIVE, or point [{CONFIG_GROUP_NAME}] cluster_name at a "
+            "usable cluster."
+        )
+    session = eks_hook.get_session()
+
+    # EKS only accepts tokens presigned against the regional STS endpoint.
+    os.environ["AWS_STS_REGIONAL_ENDPOINTS"] = "regional"
+    try:
+        sts_endpoint = StsHook(
+            aws_conn_id=conn_id, region_name=session.region_name
+        ).conn_client_meta.endpoint_url
+    finally:
+        del os.environ["AWS_STS_REGIONAL_ENDPOINTS"]
+    sts_url = f"{sts_endpoint}/?Action=GetCallerIdentity&Version=2011-06-15"
+
+    configuration.host = cluster["endpoint"]
+    configuration.ssl_ca_cert = _write_cluster_ca_file(cluster["certificateAuthority"]["data"])
+    configuration.api_key_prefix[token_key] = "Bearer"
+
+    # EKS tokens expire after about 15 minutes, so get a new one shortly before the current one expires.
+    token: str | None = None
+    refresh_at = 0.0
+
+    def refresh_api_key(config: client.Configuration | async_client.Configuration) -> None:
+        nonlocal token, refresh_at
+        if token is None or time.monotonic() >= refresh_at:
+            token = fetch_access_token_for_cluster(
+                cluster_name, sts_url, region_name=session.region_name, session=session
+            )
+            refresh_at = time.monotonic() + TOKEN_EXPIRATION_MINUTES * 60
+        config.api_key[token_key] = token
+
+    configuration.refresh_api_key_hook = refresh_api_key
+    refresh_api_key(configuration)
+
+
+def _write_cluster_ca_file(base64_ca_data: str) -> str:
+    # The kubernetes client reads the CA file lazily, so it must outlive this function.

Review Comment:
   Will these not accumulate over time? Especially on an API server that is serving a lot of AwsEksExecutor logs?



##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py:
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kubernetes client factories for the AwsEksExecutor."""
+
+# Internal to AwsEksExecutor, which writes these functions' import paths into
+# [kubernetes_executor] client_factory and async_client_factory.
+
+from __future__ import annotations
+
+import os
+import tempfile
+import time
+from base64 import b64decode
+from typing import TYPE_CHECKING
+
+import kubernetes
+from packaging.version import Version
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.aws.hooks.eks import EksHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.providers.amazon.aws.utils.eks_get_token import (
+    TOKEN_EXPIRATION_MINUTES,
+    fetch_access_token_for_cluster,
+)
+from airflow.providers.common.compat.sdk import conf
+
+if TYPE_CHECKING:
+    from kubernetes import client
+    from kubernetes_asyncio import client as async_client
+
+# UPDATING still serves the Kubernetes API, so every other status is refused.
+_USABLE_CLUSTER_STATUSES = ("ACTIVE", "UPDATING")
+
+
+def _get_sync_token_key(kubernetes_version: str) -> str:
+    # kubernetes 36 moved the bearer token from the "authorization" api_key to "BearerToken".
+    return "BearerToken" if Version(kubernetes_version).major >= 36 else "authorization"
+
+
+_SYNC_TOKEN_KEY = _get_sync_token_key(kubernetes.__version__)
+
+
+def _get_eks_kube_client() -> client.CoreV1Api:
+    """Build a Kubernetes client for the configured EKS cluster, in memory and without a kubeconfig."""
+    from kubernetes import client
+
+    configuration = client.Configuration()
+    _configure_eks_auth(configuration, _SYNC_TOKEN_KEY)
+    return client.CoreV1Api(client.ApiClient(configuration=configuration))
+
+
+def _get_eks_async_kube_client() -> async_client.CoreV1Api:
+    """Build the asynchronous Kubernetes client used when ``async_pod_creation`` is enabled."""
+    from kubernetes_asyncio import client as async_client
+
+    configuration = async_client.Configuration()
+    # kubernetes_asyncio has always used "BearerToken".
+    _configure_eks_auth(configuration, "BearerToken")
+    return async_client.CoreV1Api(async_client.ApiClient(configuration=configuration))
+
+
+def _configure_eks_auth(
+    configuration: client.Configuration | async_client.Configuration, token_key: str
+) -> None:
+    cluster_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None)

Review Comment:
   What's the story around config reads like this not using team based reads? Is that coming? If so, should we raise an exception if we detect team config until it's supported?



##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py:
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kubernetes client factories for the AwsEksExecutor."""
+
+# Internal to AwsEksExecutor, which writes these functions' import paths into
+# [kubernetes_executor] client_factory and async_client_factory.
+
+from __future__ import annotations
+
+import os
+import tempfile
+import time
+from base64 import b64decode
+from typing import TYPE_CHECKING
+
+import kubernetes
+from packaging.version import Version
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.aws.hooks.eks import EksHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.providers.amazon.aws.utils.eks_get_token import (
+    TOKEN_EXPIRATION_MINUTES,
+    fetch_access_token_for_cluster,
+)
+from airflow.providers.common.compat.sdk import conf
+
+if TYPE_CHECKING:
+    from kubernetes import client
+    from kubernetes_asyncio import client as async_client
+
+# UPDATING still serves the Kubernetes API, so every other status is refused.
+_USABLE_CLUSTER_STATUSES = ("ACTIVE", "UPDATING")
+
+
+def _get_sync_token_key(kubernetes_version: str) -> str:
+    # kubernetes 36 moved the bearer token from the "authorization" api_key to "BearerToken".
+    return "BearerToken" if Version(kubernetes_version).major >= 36 else "authorization"
+
+
+_SYNC_TOKEN_KEY = _get_sync_token_key(kubernetes.__version__)
+
+
+def _get_eks_kube_client() -> client.CoreV1Api:
+    """Build a Kubernetes client for the configured EKS cluster, in memory and without a kubeconfig."""
+    from kubernetes import client
+
+    configuration = client.Configuration()
+    _configure_eks_auth(configuration, _SYNC_TOKEN_KEY)
+    return client.CoreV1Api(client.ApiClient(configuration=configuration))
+
+
+def _get_eks_async_kube_client() -> async_client.CoreV1Api:
+    """Build the asynchronous Kubernetes client used when ``async_pod_creation`` is enabled."""
+    from kubernetes_asyncio import client as async_client
+
+    configuration = async_client.Configuration()
+    # kubernetes_asyncio has always used "BearerToken".
+    _configure_eks_auth(configuration, "BearerToken")
+    return async_client.CoreV1Api(async_client.ApiClient(configuration=configuration))
+
+
+def _configure_eks_auth(
+    configuration: client.Configuration | async_client.Configuration, token_key: str
+) -> None:
+    cluster_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None)
+    if not cluster_name:
+        raise ValueError(
+            f"[{CONFIG_GROUP_NAME}] cluster_name is required to build an EKS client. "
+            "Set it in airflow.cfg or with AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME."
+        )
+    region_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.REGION_NAME, fallback=None)
+    conn_id = conf.get(
+        CONFIG_GROUP_NAME,
+        AllEksConfigKeys.AWS_CONN_ID,
+        fallback=CONFIG_DEFAULTS[AllEksConfigKeys.AWS_CONN_ID],
+    )
+
+    eks_hook = EksHook(aws_conn_id=conn_id, region_name=region_name)
+    cluster = eks_hook.conn.describe_cluster(name=cluster_name)["cluster"]
+    if cluster["status"] not in _USABLE_CLUSTER_STATUSES:
+        raise ValueError(
+            f"EKS cluster {cluster_name} is {cluster['status']}; the executor needs it to be ACTIVE or "
+            f"UPDATING. Wait for it to become ACTIVE, or point [{CONFIG_GROUP_NAME}] cluster_name at a "
+            "usable cluster."
+        )
+    session = eks_hook.get_session()
+
+    # EKS only accepts tokens presigned against the regional STS endpoint.
+    os.environ["AWS_STS_REGIONAL_ENDPOINTS"] = "regional"
+    try:
+        sts_endpoint = StsHook(
+            aws_conn_id=conn_id, region_name=session.region_name
+        ).conn_client_meta.endpoint_url
+    finally:
+        del os.environ["AWS_STS_REGIONAL_ENDPOINTS"]
+    sts_url = f"{sts_endpoint}/?Action=GetCallerIdentity&Version=2011-06-15"
+
+    configuration.host = cluster["endpoint"]
+    configuration.ssl_ca_cert = _write_cluster_ca_file(cluster["certificateAuthority"]["data"])
+    configuration.api_key_prefix[token_key] = "Bearer"
+
+    # EKS tokens expire after about 15 minutes, so get a new one shortly before the current one expires.
+    token: str | None = None
+    refresh_at = 0.0

Review Comment:
   so the first refresh happens immediately?



##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py:
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kubernetes client factories for the AwsEksExecutor."""
+
+# Internal to AwsEksExecutor, which writes these functions' import paths into
+# [kubernetes_executor] client_factory and async_client_factory.
+
+from __future__ import annotations
+
+import os
+import tempfile
+import time
+from base64 import b64decode
+from typing import TYPE_CHECKING
+
+import kubernetes
+from packaging.version import Version
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.aws.hooks.eks import EksHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.providers.amazon.aws.utils.eks_get_token import (
+    TOKEN_EXPIRATION_MINUTES,
+    fetch_access_token_for_cluster,
+)
+from airflow.providers.common.compat.sdk import conf
+
+if TYPE_CHECKING:
+    from kubernetes import client
+    from kubernetes_asyncio import client as async_client
+
+# UPDATING still serves the Kubernetes API, so every other status is refused.
+_USABLE_CLUSTER_STATUSES = ("ACTIVE", "UPDATING")
+
+
+def _get_sync_token_key(kubernetes_version: str) -> str:
+    # kubernetes 36 moved the bearer token from the "authorization" api_key to "BearerToken".
+    return "BearerToken" if Version(kubernetes_version).major >= 36 else "authorization"
+
+
+_SYNC_TOKEN_KEY = _get_sync_token_key(kubernetes.__version__)
+
+
+def _get_eks_kube_client() -> client.CoreV1Api:
+    """Build a Kubernetes client for the configured EKS cluster, in memory and without a kubeconfig."""
+    from kubernetes import client
+
+    configuration = client.Configuration()
+    _configure_eks_auth(configuration, _SYNC_TOKEN_KEY)
+    return client.CoreV1Api(client.ApiClient(configuration=configuration))
+
+
+def _get_eks_async_kube_client() -> async_client.CoreV1Api:
+    """Build the asynchronous Kubernetes client used when ``async_pod_creation`` is enabled."""
+    from kubernetes_asyncio import client as async_client
+
+    configuration = async_client.Configuration()
+    # kubernetes_asyncio has always used "BearerToken".
+    _configure_eks_auth(configuration, "BearerToken")
+    return async_client.CoreV1Api(async_client.ApiClient(configuration=configuration))
+
+
+def _configure_eks_auth(
+    configuration: client.Configuration | async_client.Configuration, token_key: str
+) -> None:
+    cluster_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None)
+    if not cluster_name:
+        raise ValueError(
+            f"[{CONFIG_GROUP_NAME}] cluster_name is required to build an EKS client. "
+            "Set it in airflow.cfg or with AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME."
+        )
+    region_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.REGION_NAME, fallback=None)
+    conn_id = conf.get(
+        CONFIG_GROUP_NAME,
+        AllEksConfigKeys.AWS_CONN_ID,
+        fallback=CONFIG_DEFAULTS[AllEksConfigKeys.AWS_CONN_ID],
+    )
+
+    eks_hook = EksHook(aws_conn_id=conn_id, region_name=region_name)
+    cluster = eks_hook.conn.describe_cluster(name=cluster_name)["cluster"]
+    if cluster["status"] not in _USABLE_CLUSTER_STATUSES:
+        raise ValueError(
+            f"EKS cluster {cluster_name} is {cluster['status']}; the executor needs it to be ACTIVE or "
+            f"UPDATING. Wait for it to become ACTIVE, or point [{CONFIG_GROUP_NAME}] cluster_name at a "
+            "usable cluster."
+        )
+    session = eks_hook.get_session()
+
+    # EKS only accepts tokens presigned against the regional STS endpoint.
+    os.environ["AWS_STS_REGIONAL_ENDPOINTS"] = "regional"
+    try:
+        sts_endpoint = StsHook(
+            aws_conn_id=conn_id, region_name=session.region_name
+        ).conn_client_meta.endpoint_url
+    finally:
+        del os.environ["AWS_STS_REGIONAL_ENDPOINTS"]

Review Comment:
   Is this becoming sketchy now? We run it in some existing code that sets up credentials for the EksPodOperator, but that ran in a worker that was thrown away. I worry that this is now being run in the API server, possibly on multiple threads. Is there a safer way to do this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4299994399


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py:
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kubernetes client factories for the AwsEksExecutor."""
+
+# Internal to AwsEksExecutor, which writes these functions' import paths into
+# [kubernetes_executor] client_factory and async_client_factory.
+
+from __future__ import annotations
+
+import os
+import tempfile
+import time
+from base64 import b64decode
+from typing import TYPE_CHECKING
+
+import kubernetes
+from packaging.version import Version
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.aws.hooks.eks import EksHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.providers.amazon.aws.utils.eks_get_token import (
+    TOKEN_EXPIRATION_MINUTES,
+    fetch_access_token_for_cluster,
+)
+from airflow.providers.common.compat.sdk import conf
+
+if TYPE_CHECKING:
+    from kubernetes import client
+    from kubernetes_asyncio import client as async_client
+
+# UPDATING still serves the Kubernetes API, so every other status is refused.
+_USABLE_CLUSTER_STATUSES = ("ACTIVE", "UPDATING")
+
+
+def _get_sync_token_key(kubernetes_version: str) -> str:
+    # kubernetes 36 moved the bearer token from the "authorization" api_key to "BearerToken".
+    return "BearerToken" if Version(kubernetes_version).major >= 36 else "authorization"
+
+
+_SYNC_TOKEN_KEY = _get_sync_token_key(kubernetes.__version__)
+
+
+def _get_eks_kube_client() -> client.CoreV1Api:
+    """Build a Kubernetes client for the configured EKS cluster, in memory and without a kubeconfig."""
+    from kubernetes import client
+
+    configuration = client.Configuration()
+    _configure_eks_auth(configuration, _SYNC_TOKEN_KEY)
+    return client.CoreV1Api(client.ApiClient(configuration=configuration))
+
+
+def _get_eks_async_kube_client() -> async_client.CoreV1Api:
+    """Build the asynchronous Kubernetes client used when ``async_pod_creation`` is enabled."""
+    from kubernetes_asyncio import client as async_client
+
+    configuration = async_client.Configuration()
+    # kubernetes_asyncio has always used "BearerToken".
+    _configure_eks_auth(configuration, "BearerToken")
+    return async_client.CoreV1Api(async_client.ApiClient(configuration=configuration))
+
+
+def _configure_eks_auth(
+    configuration: client.Configuration | async_client.Configuration, token_key: str
+) -> None:
+    cluster_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None)

Review Comment:
   I've outlined the issue in this comment from @ferruzzi
   
   https://github.com/apache/airflow/pull/74399#discussion_r4288802873
   
   Let me know if you agree with the plan laid out there.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Let fetch_access_token_for_cluster reuse a caller's boto3 session [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4211351183


##########
providers/amazon/src/airflow/providers/amazon/aws/utils/eks_get_token.py:
##########
@@ -49,9 +49,12 @@ def get_parser():
     return parser
 
 
-def fetch_access_token_for_cluster(eks_cluster_name: str, sts_url: str, region_name: str) -> str:
-    # This will use the credentials from the caller set as the standard AWS env variables
-    session = boto3.Session(region_name=region_name)
+def fetch_access_token_for_cluster(
+    eks_cluster_name: str, sts_url: str, region_name: str, session: boto3.Session | None = None

Review Comment:
   Will be consumed by coming PR!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4291715024


##########
providers/amazon/tests/unit/amazon/aws/executors/eks/test_eks_executor.py:
##########
@@ -0,0 +1,154 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import importlib
+import os
+import sys
+from unittest import mock
+
+import pytest
+from kubernetes.client import CoreV1Api
+from kubernetes.client.rest import ApiException
+
+from airflow.providers.amazon.aws.executors.eks import AwsEksExecutor, _client_factory
+from airflow.providers.amazon.aws.executors.eks.eks_executor import (
+    _ASYNC_CLIENT_FACTORY_PATH,
+    _CLIENT_FACTORY_PATH,
+)
+from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException, conf
+
+from tests_common.test_utils.config import conf_vars
+from tests_common.test_utils.version_compat import AIRFLOW_V_3_2_PLUS
+
+EKS_EXECUTOR_MODULE = "airflow.providers.amazon.aws.executors.eks.eks_executor"
+CLIENT_FACTORY_ENV_VAR = "AIRFLOW__KUBERNETES_EXECUTOR__CLIENT_FACTORY"
+TEAMS = [
+    pytest.param(None, "AIRFLOW__", id="global"),
+    pytest.param(
+        "team_a",
+        "AIRFLOW__TEAM_A___",
+        id="team",
+        marks=pytest.mark.skipif(not AIRFLOW_V_3_2_PLUS, reason="Multi-team requires Airflow 3.2+"),
+    ),
+]
+
+
+def _team_kwargs(team_name: str | None) -> dict:
+    return {"team_name": team_name} if team_name else {}
+
+
+@pytest.fixture(autouse=True)
+def eks_config():
+    with mock.patch.dict(os.environ), conf_vars({("aws_eks_executor", "cluster_name"): "test-eks-cluster"}):
+        for prefix in ("AIRFLOW__", "AIRFLOW__TEAM_A___"):
+            for key in ("CLIENT_FACTORY", "ASYNC_CLIENT_FACTORY"):
+                os.environ.pop(f"{prefix}KUBERNETES_EXECUTOR__{key}", None)
+        yield
+
+
+class TestAwsEksExecutor:

Review Comment:
   Similar to the above comment, `KubernetesExecutor` and `AwsEksExecutor` use the same cluster for all teams as of right now. I will deal with that limitation as a follow-on, described by the issue https://github.com/apache/airflow/issues/74844 and then I will add these tests.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4318847430


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py:
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kubernetes client factories for the AwsEksExecutor."""
+
+# Internal to AwsEksExecutor, which writes these functions' import paths into
+# [kubernetes_executor] client_factory and async_client_factory.
+
+from __future__ import annotations
+
+import os
+import tempfile
+import time
+from base64 import b64decode
+from typing import TYPE_CHECKING
+
+import kubernetes
+from packaging.version import Version
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.aws.hooks.eks import EksHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.providers.amazon.aws.utils.eks_get_token import (
+    TOKEN_EXPIRATION_MINUTES,
+    fetch_access_token_for_cluster,
+)
+from airflow.providers.common.compat.sdk import conf
+
+if TYPE_CHECKING:
+    from kubernetes import client
+    from kubernetes_asyncio import client as async_client
+
+# UPDATING still serves the Kubernetes API, so every other status is refused.
+_USABLE_CLUSTER_STATUSES = ("ACTIVE", "UPDATING")
+
+
+def _get_sync_token_key(kubernetes_version: str) -> str:
+    # kubernetes 36 moved the bearer token from the "authorization" api_key to "BearerToken".
+    return "BearerToken" if Version(kubernetes_version).major >= 36 else "authorization"
+
+
+_SYNC_TOKEN_KEY = _get_sync_token_key(kubernetes.__version__)
+
+
+def _get_eks_kube_client() -> client.CoreV1Api:
+    """Build a Kubernetes client for the configured EKS cluster, in memory and without a kubeconfig."""
+    from kubernetes import client
+
+    configuration = client.Configuration()
+    _configure_eks_auth(configuration, _SYNC_TOKEN_KEY)
+    return client.CoreV1Api(client.ApiClient(configuration=configuration))
+
+
+def _get_eks_async_kube_client() -> async_client.CoreV1Api:
+    """Build the asynchronous Kubernetes client used when ``async_pod_creation`` is enabled."""
+    from kubernetes_asyncio import client as async_client
+
+    configuration = async_client.Configuration()
+    # kubernetes_asyncio has always used "BearerToken".
+    _configure_eks_auth(configuration, "BearerToken")
+    return async_client.CoreV1Api(async_client.ApiClient(configuration=configuration))
+
+
+def _configure_eks_auth(
+    configuration: client.Configuration | async_client.Configuration, token_key: str
+) -> None:
+    cluster_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None)
+    if not cluster_name:
+        raise ValueError(
+            f"[{CONFIG_GROUP_NAME}] cluster_name is required to build an EKS client. "
+            "Set it in airflow.cfg or with AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME."
+        )
+    region_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.REGION_NAME, fallback=None)
+    conn_id = conf.get(
+        CONFIG_GROUP_NAME,
+        AllEksConfigKeys.AWS_CONN_ID,
+        fallback=CONFIG_DEFAULTS[AllEksConfigKeys.AWS_CONN_ID],
+    )
+
+    eks_hook = EksHook(aws_conn_id=conn_id, region_name=region_name)
+    cluster = eks_hook.conn.describe_cluster(name=cluster_name)["cluster"]
+    if cluster["status"] not in _USABLE_CLUSTER_STATUSES:
+        raise ValueError(
+            f"EKS cluster {cluster_name} is {cluster['status']}; the executor needs it to be ACTIVE or "
+            f"UPDATING. Wait for it to become ACTIVE, or point [{CONFIG_GROUP_NAME}] cluster_name at a "
+            "usable cluster."
+        )
+    session = eks_hook.get_session()
+
+    # EKS only accepts tokens presigned against the regional STS endpoint.
+    os.environ["AWS_STS_REGIONAL_ENDPOINTS"] = "regional"
+    try:
+        sts_endpoint = StsHook(
+            aws_conn_id=conn_id, region_name=session.region_name
+        ).conn_client_meta.endpoint_url
+    finally:
+        del os.environ["AWS_STS_REGIONAL_ENDPOINTS"]

Review Comment:
   Sounds good, adding a line in the EksExecutor specific docs that mentions the minimum boto version. This is also already stated in the AMPP docs. Since AMPP enforces the right version of boto at install time, there's no need to additionally have runtime validation. This is the pattern across other dependencies too



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4288488642


##########
providers/amazon/docs/executors/eks-executor.rst:
##########
@@ -0,0 +1,213 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. |executorName| replace:: EKS
+
+.. _eks_executor:
+
+================
+AWS EKS Executor
+================
+
+The EKS executor runs each Airflow task in its own pod on an Amazon EKS cluster.
+
+It extends the Kubernetes executor from the ``cncf.kubernetes`` provider, so pod scheduling,
+pod templates, per-task pod overrides, log handling and requirements such as a database
+backend other than SQLite are all as described in
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor`. What it adds is
+authentication: it builds the Kubernetes client for your EKS cluster from an Airflow AWS
+connection, so the scheduler needs no kubeconfig file and you do not have to refresh cluster
+credentials yourself.
+
+For a quick start guide please see :ref:`here <eks_setup_guide>`.
+
+Requirements
+------------
+
+The executor needs ``apache-airflow-providers-cncf-kubernetes`` version 10.24.0 or
+newer, the release that added the
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__client_factory`
+setting the executor is built on. Install the ``cncf.kubernetes`` extra of the Amazon
+provider together with that version:
+
+.. code-block:: bash
+
+    pip install 'apache-airflow-providers-amazon[cncf.kubernetes]' \
+        'apache-airflow-providers-cncf-kubernetes>=10.24.0'
+
+The AWS credentials the executor uses must be allowed to call ``eks:DescribeCluster``
+on the cluster, and their IAM principal must also be granted access inside the cluster, as
+described in :ref:`eks_grant_access`. Without that in-cluster grant every Kubernetes API call
+is rejected.
+
+How authentication works
+------------------------
+
+When the executor starts, it plugs its own Kubernetes client into the Kubernetes
+executor. Every process that needs a client, including the separate pod watcher process,
+builds its own from the :ref:`[aws_eks_executor] <eks_config_options>` settings: it
+describes the cluster to find its API endpoint and certificate authority, then gets an
+authentication token for it. An EKS token is a presigned STS URL that stays valid for
+roughly fifteen minutes, so the executor registers a refresh hook that the Kubernetes client
+calls on every authenticated request. The hook reuses the current token and gets a new one
+a minute before it expires.
+
+Before it accepts any tasks, the executor checks that the cluster is ``ACTIVE`` or
+``UPDATING``, since EKS keeps the Kubernetes API available during an update (see `Update
+existing cluster to new Kubernetes version
+<https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html>`__). Unless
+:ref:`check_health_on_startup <eks_config_options>` is turned off, it also checks that it
+is allowed to list pods in its namespace. It refuses to start if either check fails.
+
+.. _eks_config_options:
+
+Config Options
+--------------
+
+The executor reads its own settings from an ``aws_eks_executor`` section in
+``airflow.cfg``, or from environment variables of the form
+``AIRFLOW__AWS_EKS_EXECUTOR__<OPTION_NAME>``, for example
+``AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME=airflow-eks-cluster``. For more information on
+how to set these options, see :doc:`apache-airflow:howto/set-config`.
+
+Required config options:
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  CLUSTER_NAME - The name of the Amazon EKS cluster that tasks run on. The
+   executor refuses to start if this is unset. Required.
+
+Optional config options:
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  CONN_ID - The Airflow connection (i.e. credentials) used by the EKS
+   executor to make API calls to Amazon EKS. Defaults to ``aws_default``.
+-  REGION_NAME - The AWS Region the cluster is in. When this is left empty, it
+   falls back to the :ref:`AWS connection's <howto/connection:aws>` region, then
+   boto3's.
+-  CHECK_HEALTH_ON_STARTUP - Whether to check on startup that the executor can
+   list pods in its namespace. Defaults to ``True``.
+
+Pod-level configuration
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Settings for the worker pods stay in the
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor`
+section, exactly as for the Kubernetes executor. This includes
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__namespace`,
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__pod_template_file`,
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__worker_container_repository`,
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__worker_container_tag`, and
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__delete_worker_pods`.
+See :doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor` for the full list, and
+for how :ref:`pod templates <apache-airflow-providers-cncf-kubernetes:concepts:pod_template_file>` and
+:ref:`apache-airflow-providers-cncf-kubernetes:concepts:pod_override` work.
+
+Leave :ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__client_factory` and
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__async_client_factory`
+unset. The executor sets both itself, and raises an error at startup if either is set to
+something else, so that a conflicting setting cannot silently send tasks to the wrong cluster.
+
+Set :ref:`[kubernetes_executor] pod_template_file
+<apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__pod_template_file>`
+to a YAML file describing the worker pod, and treat it as required. When it is empty, the
+Kubernetes executor only logs a warning and builds the worker pod from an empty template, so
+the failure shows up later, usually as a rejected pod or a task that never reports back. The
+template's first container must be named ``base`` and run your Airflow worker image. The
+:ref:`apache-airflow-providers-cncf-kubernetes:concepts:pod_template_file` section of the
+Kubernetes executor docs covers the full requirements, and its example templates work here
+unchanged.
+
+.. _eks_logging:
+
+.. include:: general.rst
+  :start-after: .. BEGIN LOGGING
+  :end-before: .. END LOGGING
+
+-  Worker pods are deleted when their task succeeds (:ref:`[kubernetes_executor]
+   delete_worker_pods <apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__delete_worker_pods>`),
+   and their logs go with them, so configure remote logging to CloudWatch Logs or S3
+   to keep task logs viewable in the Airflow UI.
+-  The remote logging configuration must be set on the worker pods as well as on
+   the scheduler and API server, and the worker pods need an IAM role, for
+   example through EKS Pod Identity, that can write to the log destination.
+
+.. _eks_setup_guide:
+
+Setting up an EKS Executor for Apache Airflow
+---------------------------------------------
+
+.. _eks_grant_access:
+
+Grant access to the cluster
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Create an `EKS access entry
+<https://docs.aws.amazon.com/eks/latest/userguide/access-entries.html>`__ for the IAM
+role or user that the scheduler runs as, and associate a policy that allows create, get,

Review Comment:
   Agreed, reworded to:
   
   > Create an EKS access entry for the IAM role or user behind the executor's AWS connection (`[aws_eks_executor] conn_id`, `aws_default` by default), and associate a policy that allows create, get, list, watch, patch and delete on pods, and get on pods/log, in the namespace you plan to use.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4288459595


##########
providers/amazon/docs/executors/eks-executor.rst:
##########
@@ -0,0 +1,213 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. |executorName| replace:: EKS
+
+.. _eks_executor:
+
+================
+AWS EKS Executor
+================
+
+The EKS executor runs each Airflow task in its own pod on an Amazon EKS cluster.
+
+It extends the Kubernetes executor from the ``cncf.kubernetes`` provider, so pod scheduling,
+pod templates, per-task pod overrides, log handling and requirements such as a database
+backend other than SQLite are all as described in
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor`. What it adds is
+authentication: it builds the Kubernetes client for your EKS cluster from an Airflow AWS
+connection, so the scheduler needs no kubeconfig file and you do not have to refresh cluster
+credentials yourself.
+
+For a quick start guide please see :ref:`here <eks_setup_guide>`.
+
+Requirements
+------------
+
+The executor needs ``apache-airflow-providers-cncf-kubernetes`` version 10.24.0 or
+newer, the release that added the
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__client_factory`
+setting the executor is built on. Install the ``cncf.kubernetes`` extra of the Amazon
+provider together with that version:
+
+.. code-block:: bash
+
+    pip install 'apache-airflow-providers-amazon[cncf.kubernetes]' \
+        'apache-airflow-providers-cncf-kubernetes>=10.24.0'
+
+The AWS credentials the executor uses must be allowed to call ``eks:DescribeCluster``
+on the cluster, and their IAM principal must also be granted access inside the cluster, as
+described in :ref:`eks_grant_access`. Without that in-cluster grant every Kubernetes API call
+is rejected.
+
+How authentication works
+------------------------
+
+When the executor starts, it plugs its own Kubernetes client into the Kubernetes
+executor. Every process that needs a client, including the separate pod watcher process,
+builds its own from the :ref:`[aws_eks_executor] <eks_config_options>` settings: it
+describes the cluster to find its API endpoint and certificate authority, then gets an
+authentication token for it. An EKS token is a presigned STS URL that stays valid for
+roughly fifteen minutes, so the executor registers a refresh hook that the Kubernetes client
+calls on every authenticated request. The hook reuses the current token and gets a new one
+a minute before it expires.
+
+Before it accepts any tasks, the executor checks that the cluster is ``ACTIVE`` or
+``UPDATING``, since EKS keeps the Kubernetes API available during an update (see `Update
+existing cluster to new Kubernetes version
+<https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html>`__). Unless
+:ref:`check_health_on_startup <eks_config_options>` is turned off, it also checks that it
+is allowed to list pods in its namespace. It refuses to start if either check fails.
+
+.. _eks_config_options:
+
+Config Options
+--------------
+
+The executor reads its own settings from an ``aws_eks_executor`` section in
+``airflow.cfg``, or from environment variables of the form
+``AIRFLOW__AWS_EKS_EXECUTOR__<OPTION_NAME>``, for example
+``AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME=airflow-eks-cluster``. For more information on
+how to set these options, see :doc:`apache-airflow:howto/set-config`.
+
+Required config options:
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  CLUSTER_NAME - The name of the Amazon EKS cluster that tasks run on. The
+   executor refuses to start if this is unset. Required.
+
+Optional config options:
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  CONN_ID - The Airflow connection (i.e. credentials) used by the EKS
+   executor to make API calls to Amazon EKS. Defaults to ``aws_default``.
+-  REGION_NAME - The AWS Region the cluster is in. When this is left empty, it
+   falls back to the :ref:`AWS connection's <howto/connection:aws>` region, then
+   boto3's.
+-  CHECK_HEALTH_ON_STARTUP - Whether to check on startup that the executor can
+   list pods in its namespace. Defaults to ``True``.
+
+Pod-level configuration
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Settings for the worker pods stay in the
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor`
+section, exactly as for the Kubernetes executor. This includes
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__namespace`,
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__pod_template_file`,
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__worker_container_repository`,
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__worker_container_tag`, and
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__delete_worker_pods`.
+See :doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor` for the full list, and
+for how :ref:`pod templates <apache-airflow-providers-cncf-kubernetes:concepts:pod_template_file>` and
+:ref:`apache-airflow-providers-cncf-kubernetes:concepts:pod_override` work.
+
+Leave :ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__client_factory` and
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__async_client_factory`
+unset. The executor sets both itself, and raises an error at startup if either is set to
+something else, so that a conflicting setting cannot silently send tasks to the wrong cluster.
+
+Set :ref:`[kubernetes_executor] pod_template_file
+<apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__pod_template_file>`
+to a YAML file describing the worker pod, and treat it as required. When it is empty, the
+Kubernetes executor only logs a warning and builds the worker pod from an empty template, so
+the failure shows up later, usually as a rejected pod or a task that never reports back. The

Review Comment:
   No, it isn't guaranteed to fail, so I'll soften the language to reflect that. Building worker pods from an empty template works if the parameters `worker_container_repository` and `worker_container_tag` point to an image with everything the task needs. The way in which we can fail faster is that, at the task-level, if the pod ends up with no image at all (no template, no `pod_override` image, and no repository/tag), then the task fails right away with a legible error message. I've added that now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4287637290


##########
providers/amazon/docs/executors/eks-executor.rst:
##########
@@ -0,0 +1,213 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. |executorName| replace:: EKS
+
+.. _eks_executor:
+
+================
+AWS EKS Executor
+================
+
+The EKS executor runs each Airflow task in its own pod on an Amazon EKS cluster.
+
+It extends the Kubernetes executor from the ``cncf.kubernetes`` provider, so pod scheduling,
+pod templates, per-task pod overrides, log handling and requirements such as a database
+backend other than SQLite are all as described in
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor`. What it adds is
+authentication: it builds the Kubernetes client for your EKS cluster from an Airflow AWS
+connection, so the scheduler needs no kubeconfig file and you do not have to refresh cluster
+credentials yourself.
+
+For a quick start guide please see :ref:`here <eks_setup_guide>`.
+
+Requirements
+------------
+
+The executor needs ``apache-airflow-providers-cncf-kubernetes`` version 10.24.0 or

Review Comment:
   Yes, it's checked at import time. Added a line in the doc that states this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on PR #74399:
URL: https://github.com/apache/airflow/pull/74399#issuecomment-6163338307

   Using the Eks Executor depends on cncf.kubernetes >= 10.24.0 but I haven't bumped the version floor in this PR, and instead just throw an error if the user is on an older version of cncf. @vincbeck do you agree with this, or prefer that I bump the version floor?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on PR #74399:
URL: https://github.com/apache/airflow/pull/74399#issuecomment-6163145668

   As of today 60/60 system tests are successfully running on the AWS [system test dashboard](https://aws-mwaa.github.io/#/open-source/system-tests/executor/eks/dashboard.html) using the EksExecutor


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4291655987


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py:
##########
@@ -0,0 +1,137 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""AwsEksExecutor: run Airflow tasks as pods on an Amazon EKS cluster."""
+
+from __future__ import annotations
+
+import os
+from typing import TYPE_CHECKING
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException, conf
+
+_CNCF_KUBERNETES_REQUIRED = "AwsEksExecutor requires apache-airflow-providers-cncf-kubernetes>=10.24.0"
+
+try:
+    from kubernetes.client.rest import ApiException
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+    from airflow.providers.cncf.kubernetes.get_provider_info import (
+        get_provider_info as get_cncf_kubernetes_provider_info,
+    )
+except ImportError as e:
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED) from e
+
+# Check for the option, not the version, since unreleased trees report the previous version.
+if "client_factory" not in (
+    get_cncf_kubernetes_provider_info().get("config", {}).get("kubernetes_executor", {}).get("options", {})
+):
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED)
+
+_FACTORY_MODULE = "airflow.providers.amazon.aws.executors.eks._client_factory"
+_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_kube_client"
+_ASYNC_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_async_kube_client"
+
+
+class AwsEksExecutor(KubernetesExecutor):
+    """
+    A KubernetesExecutor that authenticates against an Amazon EKS cluster.
+
+    Builds the Kubernetes client from ``[aws_eks_executor]`` configuration and keeps the
+    short-lived EKS token fresh in-process. All pod-level behaviour comes unchanged from the
+    KubernetesExecutor and its ``[kubernetes_executor]`` configuration.
+    """
+
+    supports_multi_team: bool = True
+
+    def __init__(self, *args, **kwargs):
+        self._validate_eks_config()
+        super().__init__(*args, **kwargs)
+        # Needs team_name, which super().__init__ sets.
+        self._ensure_client_factory()
+
+    def start(self) -> None:
+        """Call this when the Executor is run for the first time by the scheduler."""
+        super().start()
+        check_health = conf.getboolean(
+            CONFIG_GROUP_NAME,
+            AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP,
+            fallback=CONFIG_DEFAULTS[AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP],
+        )
+
+        if not check_health:
+            return
+
+        self.log.info("Starting EKS Executor and determining health...")
+        try:
+            self.check_health()
+        except RuntimeError:
+            self.log.error("Stopping the Airflow Scheduler from starting until the issue is resolved.")
+            raise
+
+    def check_health(self) -> None:
+        """
+        Make a test Kubernetes API call to check the health of the EKS Executor.
+
+        Building the client only proves the cluster exists; without this, a missing EKS access
+        entry or RBAC binding only shows up later as a watcher error loop.
+        """
+        if TYPE_CHECKING:
+            assert self.kube_client
+        namespace = self.kube_config.kube_namespace
+        try:
+            self.kube_client.list_namespaced_pod(namespace, limit=1)
+        except ApiException as e:
+            raise RuntimeError(
+                f"EKS Executor health check has failed because: cannot list pods in namespace {namespace} "
+                f"({e.status} {e.reason}). The IAM principal behind the AWS connection needs an EKS access "
+                "entry (or aws-auth mapping) on the cluster that allows create, get, list, watch, patch and "
+                f"delete on pods and get on pods/log in namespace {namespace}. See 'Grant access to the "
+                "cluster' in the AwsEksExecutor docs."
+            ) from e
+        self.log.info("EKS Executor health check has succeeded.")
+
+    @staticmethod
+    def _validate_eks_config() -> None:
+        if not conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None):

Review Comment:
   It's a good point to bring up, and it's something I did consider explicitly when writing #73014. To keep the changes to `cncf` minimal, I decided to bring `AwsEksExecutor` to multi-team feature parity with `KubernetesExecutor` ([docs](https://github.com/apache/airflow/blob/313255f8db6543f8482bd1c391d97a46567da89f/providers/amazon/docs/executors/eks-executor.rst?plain=1#L203-L207)), which reads its cluster settings globally, so teams share one cluster ([kube_client.py](https://github.com/apache/airflow/blob/313255f8db6543f8482bd1c391d97a46567da89f/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/kube_client.py#L180-L181)):
   
   ~~~python
       if in_cluster is None:
           in_cluster = conf.getboolean("kubernetes_executor", "in_cluster")
   ~~~
   
   On the other hand, the AWS executors read their whole section per team, so each team can have its own cluster ([ecs_executor.py](https://github.com/apache/airflow/blob/313255f8db6543f8482bd1c391d97a46567da89f/providers/amazon/src/airflow/providers/amazon/aws/executors/ecs/ecs_executor.py#L142)):
   
   ~~~python
           self.cluster = self.conf.get(CONFIG_GROUP_NAME, AllEcsConfigKeys.CLUSTER)
   ~~~
   
   Getting EKS (and KubernetesExecutor) to that level needs cncf to pass the team name to the client factory, which is a small change I'll do as a follow-up. I'll file an issue for it and target for it to be included in the next cncf release.
   
   If you, @o-nikolas, and @vincbeck are happy with that plan, then we'll treat this as a follow-on.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4299895514


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py:
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kubernetes client factories for the AwsEksExecutor."""
+
+# Internal to AwsEksExecutor, which writes these functions' import paths into
+# [kubernetes_executor] client_factory and async_client_factory.
+
+from __future__ import annotations
+
+import os
+import tempfile
+import time
+from base64 import b64decode
+from typing import TYPE_CHECKING
+
+import kubernetes
+from packaging.version import Version
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.aws.hooks.eks import EksHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.providers.amazon.aws.utils.eks_get_token import (
+    TOKEN_EXPIRATION_MINUTES,
+    fetch_access_token_for_cluster,
+)
+from airflow.providers.common.compat.sdk import conf
+
+if TYPE_CHECKING:
+    from kubernetes import client
+    from kubernetes_asyncio import client as async_client
+
+# UPDATING still serves the Kubernetes API, so every other status is refused.
+_USABLE_CLUSTER_STATUSES = ("ACTIVE", "UPDATING")
+
+
+def _get_sync_token_key(kubernetes_version: str) -> str:
+    # kubernetes 36 moved the bearer token from the "authorization" api_key to "BearerToken".
+    return "BearerToken" if Version(kubernetes_version).major >= 36 else "authorization"
+
+
+_SYNC_TOKEN_KEY = _get_sync_token_key(kubernetes.__version__)
+
+
+def _get_eks_kube_client() -> client.CoreV1Api:
+    """Build a Kubernetes client for the configured EKS cluster, in memory and without a kubeconfig."""
+    from kubernetes import client
+
+    configuration = client.Configuration()
+    _configure_eks_auth(configuration, _SYNC_TOKEN_KEY)
+    return client.CoreV1Api(client.ApiClient(configuration=configuration))
+
+
+def _get_eks_async_kube_client() -> async_client.CoreV1Api:
+    """Build the asynchronous Kubernetes client used when ``async_pod_creation`` is enabled."""
+    from kubernetes_asyncio import client as async_client
+
+    configuration = async_client.Configuration()
+    # kubernetes_asyncio has always used "BearerToken".
+    _configure_eks_auth(configuration, "BearerToken")
+    return async_client.CoreV1Api(async_client.ApiClient(configuration=configuration))
+
+
+def _configure_eks_auth(
+    configuration: client.Configuration | async_client.Configuration, token_key: str
+) -> None:
+    cluster_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None)
+    if not cluster_name:
+        raise ValueError(
+            f"[{CONFIG_GROUP_NAME}] cluster_name is required to build an EKS client. "
+            "Set it in airflow.cfg or with AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME."
+        )
+    region_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.REGION_NAME, fallback=None)
+    conn_id = conf.get(
+        CONFIG_GROUP_NAME,
+        AllEksConfigKeys.AWS_CONN_ID,
+        fallback=CONFIG_DEFAULTS[AllEksConfigKeys.AWS_CONN_ID],
+    )
+
+    eks_hook = EksHook(aws_conn_id=conn_id, region_name=region_name)
+    cluster = eks_hook.conn.describe_cluster(name=cluster_name)["cluster"]
+    if cluster["status"] not in _USABLE_CLUSTER_STATUSES:
+        raise ValueError(
+            f"EKS cluster {cluster_name} is {cluster['status']}; the executor needs it to be ACTIVE or "
+            f"UPDATING. Wait for it to become ACTIVE, or point [{CONFIG_GROUP_NAME}] cluster_name at a "
+            "usable cluster."
+        )
+    session = eks_hook.get_session()
+
+    # EKS only accepts tokens presigned against the regional STS endpoint.
+    os.environ["AWS_STS_REGIONAL_ENDPOINTS"] = "regional"
+    try:
+        sts_endpoint = StsHook(
+            aws_conn_id=conn_id, region_name=session.region_name
+        ).conn_client_meta.endpoint_url
+    finally:
+        del os.environ["AWS_STS_REGIONAL_ENDPOINTS"]
+    sts_url = f"{sts_endpoint}/?Action=GetCallerIdentity&Version=2011-06-15"
+
+    configuration.host = cluster["endpoint"]
+    configuration.ssl_ca_cert = _write_cluster_ca_file(cluster["certificateAuthority"]["data"])
+    configuration.api_key_prefix[token_key] = "Bearer"
+
+    # EKS tokens expire after about 15 minutes, so get a new one shortly before the current one expires.
+    token: str | None = None
+    refresh_at = 0.0
+
+    def refresh_api_key(config: client.Configuration | async_client.Configuration) -> None:
+        nonlocal token, refresh_at
+        if token is None or time.monotonic() >= refresh_at:
+            token = fetch_access_token_for_cluster(
+                cluster_name, sts_url, region_name=session.region_name, session=session
+            )
+            refresh_at = time.monotonic() + TOKEN_EXPIRATION_MINUTES * 60
+        config.api_key[token_key] = token
+
+    configuration.refresh_api_key_hook = refresh_api_key
+    refresh_api_key(configuration)
+
+
+def _write_cluster_ca_file(base64_ca_data: str) -> str:
+    # The kubernetes client reads the CA file lazily, so it must outlive this function.

Review Comment:
   Good catch, added two things to manage the file storage needs:
   1. file deletion step when a python process exits, so when the api server or scheduler or any other process exits, it will sweep through the created CA files and delete them
   2. Each process now writes one CA file per cluster, and re-uses it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["o-nikolas (via GitHub)" <gi...@apache.org>]

o-nikolas commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4259437593


##########
providers/amazon/docs/executors/eks-executor.rst:
##########
@@ -0,0 +1,213 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. |executorName| replace:: EKS
+
+.. _eks_executor:
+
+================
+AWS EKS Executor
+================
+
+The EKS executor runs each Airflow task in its own pod on an Amazon EKS cluster.
+
+It extends the Kubernetes executor from the ``cncf.kubernetes`` provider, so pod scheduling,
+pod templates, per-task pod overrides, log handling and requirements such as a database
+backend other than SQLite are all as described in
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor`. What it adds is
+authentication: it builds the Kubernetes client for your EKS cluster from an Airflow AWS
+connection, so the scheduler needs no kubeconfig file and you do not have to refresh cluster
+credentials yourself.
+
+For a quick start guide please see :ref:`here <eks_setup_guide>`.
+
+Requirements
+------------
+
+The executor needs ``apache-airflow-providers-cncf-kubernetes`` version 10.24.0 or
+newer, the release that added the
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__client_factory`
+setting the executor is built on. Install the ``cncf.kubernetes`` extra of the Amazon
+provider together with that version:
+
+.. code-block:: bash
+
+    pip install 'apache-airflow-providers-amazon[cncf.kubernetes]' \
+        'apache-airflow-providers-cncf-kubernetes>=10.24.0'
+
+The AWS credentials the executor uses must be allowed to call ``eks:DescribeCluster``
+on the cluster, and their IAM principal must also be granted access inside the cluster, as
+described in :ref:`eks_grant_access`. Without that in-cluster grant every Kubernetes API call
+is rejected.
+
+How authentication works
+------------------------
+
+When the executor starts, it plugs its own Kubernetes client into the Kubernetes
+executor. Every process that needs a client, including the separate pod watcher process,
+builds its own from the :ref:`[aws_eks_executor] <eks_config_options>` settings: it
+describes the cluster to find its API endpoint and certificate authority, then gets an
+authentication token for it. An EKS token is a presigned STS URL that stays valid for
+roughly fifteen minutes, so the executor registers a refresh hook that the Kubernetes client
+calls on every authenticated request. The hook reuses the current token and gets a new one
+a minute before it expires.
+
+Before it accepts any tasks, the executor checks that the cluster is ``ACTIVE`` or
+``UPDATING``, since EKS keeps the Kubernetes API available during an update (see `Update
+existing cluster to new Kubernetes version
+<https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html>`__). Unless
+:ref:`check_health_on_startup <eks_config_options>` is turned off, it also checks that it
+is allowed to list pods in its namespace. It refuses to start if either check fails.
+
+.. _eks_config_options:
+
+Config Options
+--------------
+
+The executor reads its own settings from an ``aws_eks_executor`` section in
+``airflow.cfg``, or from environment variables of the form
+``AIRFLOW__AWS_EKS_EXECUTOR__<OPTION_NAME>``, for example
+``AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME=airflow-eks-cluster``. For more information on
+how to set these options, see :doc:`apache-airflow:howto/set-config`.
+
+Required config options:
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  CLUSTER_NAME - The name of the Amazon EKS cluster that tasks run on. The
+   executor refuses to start if this is unset. Required.

Review Comment:
   Very minor style nit: Between the heading and the text itself, it tells the user it's not required three separate ways. We can probably drop the last `Required.` IMHO



##########
providers/amazon/docs/executors/index.rst:
##########
@@ -20,10 +20,42 @@
 Amazon Executors
 ================
 
+Each Amazon executor runs every Airflow task in its own container, pod or function
+invocation. They differ in which AWS service runs it.
+
+Choosing an executor
+--------------------
+
+.. list-table::

Review Comment:
   I'd say we drop this for now, I think it needs a bit more iteration. The "good fir when" column is not very helpful when distilled down to a single sentence (some of which are just kind of true-isms). Not opposed to it, but I think it needs more iteration that shouldn't be in this PR.



##########
providers/amazon/docs/executors/eks-executor.rst:
##########
@@ -0,0 +1,213 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. |executorName| replace:: EKS
+
+.. _eks_executor:
+
+================
+AWS EKS Executor
+================
+
+The EKS executor runs each Airflow task in its own pod on an Amazon EKS cluster.
+
+It extends the Kubernetes executor from the ``cncf.kubernetes`` provider, so pod scheduling,
+pod templates, per-task pod overrides, log handling and requirements such as a database
+backend other than SQLite are all as described in
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor`. What it adds is
+authentication: it builds the Kubernetes client for your EKS cluster from an Airflow AWS
+connection, so the scheduler needs no kubeconfig file and you do not have to refresh cluster
+credentials yourself.
+
+For a quick start guide please see :ref:`here <eks_setup_guide>`.
+
+Requirements
+------------
+
+The executor needs ``apache-airflow-providers-cncf-kubernetes`` version 10.24.0 or
+newer, the release that added the
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__client_factory`
+setting the executor is built on. Install the ``cncf.kubernetes`` extra of the Amazon
+provider together with that version:
+
+.. code-block:: bash
+
+    pip install 'apache-airflow-providers-amazon[cncf.kubernetes]' \
+        'apache-airflow-providers-cncf-kubernetes>=10.24.0'
+
+The AWS credentials the executor uses must be allowed to call ``eks:DescribeCluster``
+on the cluster, and their IAM principal must also be granted access inside the cluster, as
+described in :ref:`eks_grant_access`. Without that in-cluster grant every Kubernetes API call
+is rejected.
+
+How authentication works
+------------------------
+
+When the executor starts, it plugs its own Kubernetes client into the Kubernetes
+executor. Every process that needs a client, including the separate pod watcher process,
+builds its own from the :ref:`[aws_eks_executor] <eks_config_options>` settings: it
+describes the cluster to find its API endpoint and certificate authority, then gets an
+authentication token for it. An EKS token is a presigned STS URL that stays valid for
+roughly fifteen minutes, so the executor registers a refresh hook that the Kubernetes client
+calls on every authenticated request. The hook reuses the current token and gets a new one
+a minute before it expires.
+
+Before it accepts any tasks, the executor checks that the cluster is ``ACTIVE`` or
+``UPDATING``, since EKS keeps the Kubernetes API available during an update (see `Update
+existing cluster to new Kubernetes version
+<https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html>`__). Unless
+:ref:`check_health_on_startup <eks_config_options>` is turned off, it also checks that it
+is allowed to list pods in its namespace. It refuses to start if either check fails.
+
+.. _eks_config_options:
+
+Config Options
+--------------
+
+The executor reads its own settings from an ``aws_eks_executor`` section in
+``airflow.cfg``, or from environment variables of the form
+``AIRFLOW__AWS_EKS_EXECUTOR__<OPTION_NAME>``, for example
+``AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME=airflow-eks-cluster``. For more information on
+how to set these options, see :doc:`apache-airflow:howto/set-config`.
+
+Required config options:
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  CLUSTER_NAME - The name of the Amazon EKS cluster that tasks run on. The
+   executor refuses to start if this is unset. Required.
+
+Optional config options:
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  CONN_ID - The Airflow connection (i.e. credentials) used by the EKS
+   executor to make API calls to Amazon EKS. Defaults to ``aws_default``.
+-  REGION_NAME - The AWS Region the cluster is in. When this is left empty, it
+   falls back to the :ref:`AWS connection's <howto/connection:aws>` region, then
+   boto3's.
+-  CHECK_HEALTH_ON_STARTUP - Whether to check on startup that the executor can
+   list pods in its namespace. Defaults to ``True``.
+
+Pod-level configuration
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Settings for the worker pods stay in the
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor`
+section, exactly as for the Kubernetes executor. This includes
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__namespace`,
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__pod_template_file`,
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__worker_container_repository`,
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__worker_container_tag`, and
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__delete_worker_pods`.
+See :doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor` for the full list, and
+for how :ref:`pod templates <apache-airflow-providers-cncf-kubernetes:concepts:pod_template_file>` and
+:ref:`apache-airflow-providers-cncf-kubernetes:concepts:pod_override` work.
+
+Leave :ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__client_factory` and
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__async_client_factory`
+unset. The executor sets both itself, and raises an error at startup if either is set to
+something else, so that a conflicting setting cannot silently send tasks to the wrong cluster.
+
+Set :ref:`[kubernetes_executor] pod_template_file
+<apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__pod_template_file>`
+to a YAML file describing the worker pod, and treat it as required. When it is empty, the
+Kubernetes executor only logs a warning and builds the worker pod from an empty template, so
+the failure shows up later, usually as a rejected pod or a task that never reports back. The

Review Comment:
   Is it guaranteed to show up as a failure? If so, can we do more than just warn.



##########
providers/amazon/docs/executors/eks-executor.rst:
##########
@@ -0,0 +1,213 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. |executorName| replace:: EKS
+
+.. _eks_executor:
+
+================
+AWS EKS Executor
+================
+
+The EKS executor runs each Airflow task in its own pod on an Amazon EKS cluster.
+
+It extends the Kubernetes executor from the ``cncf.kubernetes`` provider, so pod scheduling,
+pod templates, per-task pod overrides, log handling and requirements such as a database
+backend other than SQLite are all as described in
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor`. What it adds is
+authentication: it builds the Kubernetes client for your EKS cluster from an Airflow AWS
+connection, so the scheduler needs no kubeconfig file and you do not have to refresh cluster
+credentials yourself.
+
+For a quick start guide please see :ref:`here <eks_setup_guide>`.
+
+Requirements
+------------
+
+The executor needs ``apache-airflow-providers-cncf-kubernetes`` version 10.24.0 or
+newer, the release that added the
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__client_factory`
+setting the executor is built on. Install the ``cncf.kubernetes`` extra of the Amazon
+provider together with that version:
+
+.. code-block:: bash
+
+    pip install 'apache-airflow-providers-amazon[cncf.kubernetes]' \
+        'apache-airflow-providers-cncf-kubernetes>=10.24.0'
+
+The AWS credentials the executor uses must be allowed to call ``eks:DescribeCluster``
+on the cluster, and their IAM principal must also be granted access inside the cluster, as
+described in :ref:`eks_grant_access`. Without that in-cluster grant every Kubernetes API call
+is rejected.
+
+How authentication works
+------------------------
+
+When the executor starts, it plugs its own Kubernetes client into the Kubernetes
+executor. Every process that needs a client, including the separate pod watcher process,
+builds its own from the :ref:`[aws_eks_executor] <eks_config_options>` settings: it
+describes the cluster to find its API endpoint and certificate authority, then gets an
+authentication token for it. An EKS token is a presigned STS URL that stays valid for
+roughly fifteen minutes, so the executor registers a refresh hook that the Kubernetes client
+calls on every authenticated request. The hook reuses the current token and gets a new one
+a minute before it expires.
+
+Before it accepts any tasks, the executor checks that the cluster is ``ACTIVE`` or
+``UPDATING``, since EKS keeps the Kubernetes API available during an update (see `Update
+existing cluster to new Kubernetes version
+<https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html>`__). Unless
+:ref:`check_health_on_startup <eks_config_options>` is turned off, it also checks that it
+is allowed to list pods in its namespace. It refuses to start if either check fails.
+
+.. _eks_config_options:
+
+Config Options
+--------------
+
+The executor reads its own settings from an ``aws_eks_executor`` section in
+``airflow.cfg``, or from environment variables of the form
+``AIRFLOW__AWS_EKS_EXECUTOR__<OPTION_NAME>``, for example
+``AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME=airflow-eks-cluster``. For more information on
+how to set these options, see :doc:`apache-airflow:howto/set-config`.
+
+Required config options:
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  CLUSTER_NAME - The name of the Amazon EKS cluster that tasks run on. The
+   executor refuses to start if this is unset. Required.
+
+Optional config options:
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  CONN_ID - The Airflow connection (i.e. credentials) used by the EKS
+   executor to make API calls to Amazon EKS. Defaults to ``aws_default``.
+-  REGION_NAME - The AWS Region the cluster is in. When this is left empty, it
+   falls back to the :ref:`AWS connection's <howto/connection:aws>` region, then
+   boto3's.
+-  CHECK_HEALTH_ON_STARTUP - Whether to check on startup that the executor can
+   list pods in its namespace. Defaults to ``True``.
+
+Pod-level configuration
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Settings for the worker pods stay in the
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor`
+section, exactly as for the Kubernetes executor. This includes
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__namespace`,
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__pod_template_file`,
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__worker_container_repository`,
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__worker_container_tag`, and
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__delete_worker_pods`.
+See :doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor` for the full list, and
+for how :ref:`pod templates <apache-airflow-providers-cncf-kubernetes:concepts:pod_template_file>` and
+:ref:`apache-airflow-providers-cncf-kubernetes:concepts:pod_override` work.
+
+Leave :ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__client_factory` and
+:ref:`apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__async_client_factory`
+unset. The executor sets both itself, and raises an error at startup if either is set to
+something else, so that a conflicting setting cannot silently send tasks to the wrong cluster.
+
+Set :ref:`[kubernetes_executor] pod_template_file
+<apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__pod_template_file>`
+to a YAML file describing the worker pod, and treat it as required. When it is empty, the
+Kubernetes executor only logs a warning and builds the worker pod from an empty template, so
+the failure shows up later, usually as a rejected pod or a task that never reports back. The
+template's first container must be named ``base`` and run your Airflow worker image. The
+:ref:`apache-airflow-providers-cncf-kubernetes:concepts:pod_template_file` section of the
+Kubernetes executor docs covers the full requirements, and its example templates work here
+unchanged.
+
+.. _eks_logging:
+
+.. include:: general.rst
+  :start-after: .. BEGIN LOGGING
+  :end-before: .. END LOGGING
+
+-  Worker pods are deleted when their task succeeds (:ref:`[kubernetes_executor]
+   delete_worker_pods <apache-airflow-providers-cncf-kubernetes:config:kubernetes_executor__delete_worker_pods>`),
+   and their logs go with them, so configure remote logging to CloudWatch Logs or S3
+   to keep task logs viewable in the Airflow UI.
+-  The remote logging configuration must be set on the worker pods as well as on
+   the scheduler and API server, and the worker pods need an IAM role, for
+   example through EKS Pod Identity, that can write to the log destination.
+
+.. _eks_setup_guide:
+
+Setting up an EKS Executor for Apache Airflow
+---------------------------------------------
+
+.. _eks_grant_access:
+
+Grant access to the cluster
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Create an `EKS access entry
+<https://docs.aws.amazon.com/eks/latest/userguide/access-entries.html>`__ for the IAM
+role or user that the scheduler runs as, and associate a policy that allows create, get,

Review Comment:
   nit: The executor runs inside the scheduler, but I think it's a bit misleading to say scheduler here. What people normally think of as the scheduler does not include the executor.



##########
providers/amazon/docs/executors/index.rst:
##########
@@ -20,10 +20,42 @@
 Amazon Executors
 ================
 
+Each Amazon executor runs every Airflow task in its own container, pod or function
+invocation. They differ in which AWS service runs it.

Review Comment:
   I'm not sure we should try generalize all the executors with a one-liner sentence like that. For example Batch can be configured to run multiple tasks on a single EC2 host. 



##########
providers/amazon/docs/executors/eks-executor.rst:
##########
@@ -0,0 +1,213 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. |executorName| replace:: EKS
+
+.. _eks_executor:
+
+================
+AWS EKS Executor
+================
+
+The EKS executor runs each Airflow task in its own pod on an Amazon EKS cluster.
+
+It extends the Kubernetes executor from the ``cncf.kubernetes`` provider, so pod scheduling,
+pod templates, per-task pod overrides, log handling and requirements such as a database
+backend other than SQLite are all as described in
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor`. What it adds is
+authentication: it builds the Kubernetes client for your EKS cluster from an Airflow AWS
+connection, so the scheduler needs no kubeconfig file and you do not have to refresh cluster
+credentials yourself.
+
+For a quick start guide please see :ref:`here <eks_setup_guide>`.
+
+Requirements
+------------
+
+The executor needs ``apache-airflow-providers-cncf-kubernetes`` version 10.24.0 or

Review Comment:
   Do we validate this at all at runtime? Or just let the user run into the exception wherever it breaks naturally?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4299970468


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py:
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kubernetes client factories for the AwsEksExecutor."""
+
+# Internal to AwsEksExecutor, which writes these functions' import paths into
+# [kubernetes_executor] client_factory and async_client_factory.
+
+from __future__ import annotations
+
+import os
+import tempfile
+import time
+from base64 import b64decode
+from typing import TYPE_CHECKING
+
+import kubernetes
+from packaging.version import Version
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.aws.hooks.eks import EksHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.providers.amazon.aws.utils.eks_get_token import (
+    TOKEN_EXPIRATION_MINUTES,
+    fetch_access_token_for_cluster,
+)
+from airflow.providers.common.compat.sdk import conf
+
+if TYPE_CHECKING:
+    from kubernetes import client
+    from kubernetes_asyncio import client as async_client
+
+# UPDATING still serves the Kubernetes API, so every other status is refused.
+_USABLE_CLUSTER_STATUSES = ("ACTIVE", "UPDATING")
+
+
+def _get_sync_token_key(kubernetes_version: str) -> str:
+    # kubernetes 36 moved the bearer token from the "authorization" api_key to "BearerToken".
+    return "BearerToken" if Version(kubernetes_version).major >= 36 else "authorization"
+
+
+_SYNC_TOKEN_KEY = _get_sync_token_key(kubernetes.__version__)
+
+
+def _get_eks_kube_client() -> client.CoreV1Api:
+    """Build a Kubernetes client for the configured EKS cluster, in memory and without a kubeconfig."""
+    from kubernetes import client
+
+    configuration = client.Configuration()
+    _configure_eks_auth(configuration, _SYNC_TOKEN_KEY)
+    return client.CoreV1Api(client.ApiClient(configuration=configuration))
+
+
+def _get_eks_async_kube_client() -> async_client.CoreV1Api:
+    """Build the asynchronous Kubernetes client used when ``async_pod_creation`` is enabled."""
+    from kubernetes_asyncio import client as async_client
+
+    configuration = async_client.Configuration()
+    # kubernetes_asyncio has always used "BearerToken".
+    _configure_eks_auth(configuration, "BearerToken")
+    return async_client.CoreV1Api(async_client.ApiClient(configuration=configuration))
+
+
+def _configure_eks_auth(
+    configuration: client.Configuration | async_client.Configuration, token_key: str
+) -> None:
+    cluster_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None)
+    if not cluster_name:
+        raise ValueError(
+            f"[{CONFIG_GROUP_NAME}] cluster_name is required to build an EKS client. "
+            "Set it in airflow.cfg or with AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME."
+        )
+    region_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.REGION_NAME, fallback=None)
+    conn_id = conf.get(
+        CONFIG_GROUP_NAME,
+        AllEksConfigKeys.AWS_CONN_ID,
+        fallback=CONFIG_DEFAULTS[AllEksConfigKeys.AWS_CONN_ID],
+    )
+
+    eks_hook = EksHook(aws_conn_id=conn_id, region_name=region_name)
+    cluster = eks_hook.conn.describe_cluster(name=cluster_name)["cluster"]
+    if cluster["status"] not in _USABLE_CLUSTER_STATUSES:
+        raise ValueError(
+            f"EKS cluster {cluster_name} is {cluster['status']}; the executor needs it to be ACTIVE or "
+            f"UPDATING. Wait for it to become ACTIVE, or point [{CONFIG_GROUP_NAME}] cluster_name at a "
+            "usable cluster."
+        )
+    session = eks_hook.get_session()
+
+    # EKS only accepts tokens presigned against the regional STS endpoint.
+    os.environ["AWS_STS_REGIONAL_ENDPOINTS"] = "regional"
+    try:
+        sts_endpoint = StsHook(
+            aws_conn_id=conn_id, region_name=session.region_name
+        ).conn_client_meta.endpoint_url
+    finally:
+        del os.environ["AWS_STS_REGIONAL_ENDPOINTS"]

Review Comment:
   Initially I carried this over from older Airflow code, but since version `1.40` of boto (AMPP version floor is `1.42`), this is the default behavior anyways, so there's no point of writing it explicitly anymore. And, as you mention, there is that risk of multiple threads overwriting the global var. So, now that it's deleted, it should be fine.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Let fetch_access_token_for_cluster reuse a caller's boto3 session [airflow]

["vincbeck (via GitHub)" <gi...@apache.org>]

vincbeck commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4211324854


##########
providers/amazon/src/airflow/providers/amazon/aws/utils/eks_get_token.py:
##########
@@ -49,9 +49,12 @@ def get_parser():
     return parser
 
 
-def fetch_access_token_for_cluster(eks_cluster_name: str, sts_url: str, region_name: str) -> str:
-    # This will use the credentials from the caller set as the standard AWS env variables
-    session = boto3.Session(region_name=region_name)
+def fetch_access_token_for_cluster(
+    eks_cluster_name: str, sts_url: str, region_name: str, session: boto3.Session | None = None

Review Comment:
   But you do not seem to use it? Or is it for future feature?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Let fetch_access_token_for_cluster reuse a caller's boto3 session [airflow]

["vincbeck (via GitHub)" <gi...@apache.org>]

vincbeck commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4211407335


##########
providers/amazon/src/airflow/providers/amazon/aws/utils/eks_get_token.py:
##########
@@ -49,9 +49,12 @@ def get_parser():
     return parser
 
 
-def fetch_access_token_for_cluster(eks_cluster_name: str, sts_url: str, region_name: str) -> str:
-    # This will use the credentials from the caller set as the standard AWS env variables
-    session = boto3.Session(region_name=region_name)
+def fetch_access_token_for_cluster(
+    eks_cluster_name: str, sts_url: str, region_name: str, session: boto3.Session | None = None

Review Comment:
   Could you combine them? That's always easier to review a feature when you use it :) It also avoid having useless code in case you forget/drop the incoming PR



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Let fetch_access_token_for_cluster reuse a caller's boto3 session [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4212142924


##########
providers/amazon/src/airflow/providers/amazon/aws/utils/eks_get_token.py:
##########
@@ -49,9 +49,12 @@ def get_parser():
     return parser
 
 
-def fetch_access_token_for_cluster(eks_cluster_name: str, sts_url: str, region_name: str) -> str:
-    # This will use the credentials from the caller set as the standard AWS env variables
-    session = boto3.Session(region_name=region_name)
+def fetch_access_token_for_cluster(
+    eks_cluster_name: str, sts_url: str, region_name: str, session: boto3.Session | None = None

Review Comment:
   Sounds good, I will bring the upcoming PR I was planning into this one!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["eladkal (via GitHub)" <gi...@apache.org>]

eladkal commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4214789164


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py:
##########
@@ -0,0 +1,145 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""AwsEksExecutor: run Airflow tasks as pods on an Amazon EKS cluster."""
+
+from __future__ import annotations
+
+import os
+from typing import TYPE_CHECKING
+
+from airflow.providers.common.compat.sdk import conf
+
+try:
+    from kubernetes.client.rest import ApiException
+
+    from airflow.providers.cncf.kubernetes import __version__ as cncf_kubernetes_version
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+    from airflow.providers.cncf.kubernetes.get_provider_info import (
+        get_provider_info as get_cncf_kubernetes_provider_info,
+    )
+except ImportError as e:
+    raise ImportError(
+        "AwsEksExecutor requires the cncf.kubernetes provider; install it with "
+        "pip install 'apache-airflow-providers-amazon[cncf.kubernetes]'"
+    ) from e
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+
+# Import paths, because cncf.kubernetes re-resolves the factories in each process (see _client_factory).
+_FACTORY_MODULE = "airflow.providers.amazon.aws.executors.eks._client_factory"
+_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_kube_client"
+_ASYNC_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_async_kube_client"
+# Only used in the error message below. The check looks for the client_factory option instead of
+# comparing versions, because an unreleased source tree still reports the previous release.
+MIN_CNCF_KUBERNETES_VERSION = "10.24.0"
+
+
+class AwsEksExecutor(KubernetesExecutor):

Review Comment:
   Is that the naming convention?
   Shouldn't it be just EksExecutor?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on PR #74399:
URL: https://github.com/apache/airflow/pull/74399#issuecomment-6203217514

   Thanks @o-nikolas , addressed your docs comments!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["o-nikolas (via GitHub)" <gi...@apache.org>]

o-nikolas commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4319875528


##########
providers/amazon/src/airflow/providers/amazon/get_provider_info.py:
##########
@@ -1567,6 +1567,39 @@ def get_provider_info():
                     },
                 },
             },
+            "aws_eks_executor": {
+                "description": "This section only applies if you are using the AwsEksExecutor in\nAirflow's ``[core]`` configuration. It requires the ``cncf.kubernetes``\nextra of this provider. The executor supplies the Kubernetes client for\nthe configured Amazon EKS cluster; all pod-level settings are still read\nfrom the ``[kubernetes_executor]`` section.\nFor boto3 credential management, see\nhttps://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html\n",
+                "options": {
+                    "conn_id": {
+                        "description": "The Airflow connection (i.e. credentials) used by the EKS executor to\nmake API calls to Amazon EKS.\n",
+                        "version_added": "9.38.0",

Review Comment:
   All instances of 9.38.0 need to be changed to 9.39.0



##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py:
##########
@@ -0,0 +1,226 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""AwsEksExecutor: run Airflow tasks as pods on an Amazon EKS cluster."""
+
+from __future__ import annotations
+
+import os
+from http import HTTPStatus
+from typing import TYPE_CHECKING
+
+from airflow.executors.executor_loader import ExecutorLoader
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.version_compat import AIRFLOW_V_3_2_PLUS
+from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException, conf
+
+if TYPE_CHECKING:
+    from kubernetes.client import models as k8s
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor_types import KubernetesJob
+
+_CNCF_KUBERNETES_REQUIRED = "AwsEksExecutor requires apache-airflow-providers-cncf-kubernetes>=10.24.0"
+
+try:
+    from kubernetes.client.rest import ApiException
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+    from airflow.providers.cncf.kubernetes.get_provider_info import (
+        get_provider_info as get_cncf_kubernetes_provider_info,
+    )
+except ImportError as e:
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED) from e
+
+# Check for the option, not the version, since unreleased trees report the previous version.
+if "client_factory" not in (
+    get_cncf_kubernetes_provider_info().get("config", {}).get("kubernetes_executor", {}).get("options", {})
+):
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED)
+
+_FACTORY_MODULE = "airflow.providers.amazon.aws.executors.eks._client_factory"
+_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_kube_client"
+_ASYNC_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_async_kube_client"
+# Airflow 2 hybrid executors that wrap a KubernetesExecutor.
+_HYBRID_KUBERNETES_EXECUTORS = ("CeleryKubernetesExecutor", "LocalKubernetesExecutor")
+
+
+class AwsEksExecutor(KubernetesExecutor):
+    """
+    A KubernetesExecutor that authenticates against an Amazon EKS cluster.
+
+    Builds the Kubernetes client from ``[aws_eks_executor]`` configuration and keeps the
+    short-lived EKS token fresh in-process. All pod-level behaviour comes unchanged from the
+    KubernetesExecutor and its ``[kubernetes_executor]`` configuration.
+    """
+
+    supports_multi_team: bool = True
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        # These need team_name, which super().__init__ sets.
+        self._check_no_team_eks_config()
+        self._validate_eks_config()
+        self._check_no_kubernetes_executor_in_scope()
+        self._ensure_client_factory()
+
+    def start(self) -> None:
+        """Call this when the Executor is run for the first time by the scheduler."""
+        super().start()
+        if TYPE_CHECKING:
+            assert self.kube_scheduler
+        # Both pod creation paths build every worker pod here, so check the image before it is sent.
+        build_pod_request = self.kube_scheduler._build_pod_request
+
+        def _build_pod_request_with_image(job: KubernetesJob) -> k8s.V1Pod:
+            pod = build_pod_request(job)
+            self._check_worker_image(pod)
+            return pod
+
+        self.kube_scheduler._build_pod_request = _build_pod_request_with_image  # type: ignore[method-assign,assignment]

Review Comment:
   Messing with this private function is a bit brittle. If that thing ever changes name or the logic is moved elsewhere this is going to regress. Probably not hugely likely, but I think it's not worth the small bit we gain (which I think makes a better fit in the upstream executor anyway).



##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py:
##########
@@ -0,0 +1,226 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""AwsEksExecutor: run Airflow tasks as pods on an Amazon EKS cluster."""
+
+from __future__ import annotations
+
+import os
+from http import HTTPStatus
+from typing import TYPE_CHECKING
+
+from airflow.executors.executor_loader import ExecutorLoader
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.version_compat import AIRFLOW_V_3_2_PLUS
+from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException, conf
+
+if TYPE_CHECKING:
+    from kubernetes.client import models as k8s
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor_types import KubernetesJob
+
+_CNCF_KUBERNETES_REQUIRED = "AwsEksExecutor requires apache-airflow-providers-cncf-kubernetes>=10.24.0"
+
+try:
+    from kubernetes.client.rest import ApiException
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+    from airflow.providers.cncf.kubernetes.get_provider_info import (
+        get_provider_info as get_cncf_kubernetes_provider_info,
+    )
+except ImportError as e:
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED) from e
+
+# Check for the option, not the version, since unreleased trees report the previous version.
+if "client_factory" not in (
+    get_cncf_kubernetes_provider_info().get("config", {}).get("kubernetes_executor", {}).get("options", {})
+):
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED)
+
+_FACTORY_MODULE = "airflow.providers.amazon.aws.executors.eks._client_factory"
+_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_kube_client"
+_ASYNC_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_async_kube_client"
+# Airflow 2 hybrid executors that wrap a KubernetesExecutor.
+_HYBRID_KUBERNETES_EXECUTORS = ("CeleryKubernetesExecutor", "LocalKubernetesExecutor")
+
+
+class AwsEksExecutor(KubernetesExecutor):
+    """
+    A KubernetesExecutor that authenticates against an Amazon EKS cluster.
+
+    Builds the Kubernetes client from ``[aws_eks_executor]`` configuration and keeps the
+    short-lived EKS token fresh in-process. All pod-level behaviour comes unchanged from the
+    KubernetesExecutor and its ``[kubernetes_executor]`` configuration.
+    """
+
+    supports_multi_team: bool = True
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        # These need team_name, which super().__init__ sets.
+        self._check_no_team_eks_config()
+        self._validate_eks_config()
+        self._check_no_kubernetes_executor_in_scope()
+        self._ensure_client_factory()
+
+    def start(self) -> None:
+        """Call this when the Executor is run for the first time by the scheduler."""
+        super().start()
+        if TYPE_CHECKING:
+            assert self.kube_scheduler
+        # Both pod creation paths build every worker pod here, so check the image before it is sent.
+        build_pod_request = self.kube_scheduler._build_pod_request
+
+        def _build_pod_request_with_image(job: KubernetesJob) -> k8s.V1Pod:
+            pod = build_pod_request(job)
+            self._check_worker_image(pod)
+            return pod
+
+        self.kube_scheduler._build_pod_request = _build_pod_request_with_image  # type: ignore[method-assign,assignment]
+
+        check_health = conf.getboolean(
+            CONFIG_GROUP_NAME,
+            AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP,
+            fallback=CONFIG_DEFAULTS[AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP],
+        )
+
+        if not check_health:
+            return
+
+        self.log.info("Starting EKS Executor and determining health...")
+        try:
+            self.check_health()
+        except RuntimeError:
+            self.log.error("Stopping the Airflow Scheduler from starting until the issue is resolved.")
+            raise
+
+    def check_health(self) -> None:
+        """
+        Make a test Kubernetes API call to check the health of the EKS Executor.
+
+        Building the client only proves the cluster exists; without this, a missing EKS access
+        entry or RBAC binding only shows up later as a watcher error loop.
+        """
+        if TYPE_CHECKING:
+            assert self.kube_client
+        namespace = self.kube_config.kube_namespace
+        try:
+            self.kube_client.list_namespaced_pod(namespace, limit=1)
+        except ApiException as e:
+            raise RuntimeError(
+                f"EKS Executor health check has failed because: cannot list pods in namespace {namespace} "
+                f"({e.status} {e.reason}). The IAM principal behind the AWS connection needs an EKS access "
+                "entry (or aws-auth mapping) on the cluster that allows create, get, list, watch, patch and "
+                f"delete on pods and get on pods/log in namespace {namespace}. See 'Grant access to the "
+                "cluster' in the AwsEksExecutor docs."
+            ) from e
+        self.log.info("EKS Executor health check has succeeded.")
+
+    def _check_worker_image(self, pod: k8s.V1Pod) -> None:

Review Comment:
   Is this an EKS thing? Should this be in the regular kubernetes executor?



##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py:
##########
@@ -0,0 +1,226 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""AwsEksExecutor: run Airflow tasks as pods on an Amazon EKS cluster."""
+
+from __future__ import annotations
+
+import os
+from http import HTTPStatus
+from typing import TYPE_CHECKING
+
+from airflow.executors.executor_loader import ExecutorLoader
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.version_compat import AIRFLOW_V_3_2_PLUS
+from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException, conf
+
+if TYPE_CHECKING:
+    from kubernetes.client import models as k8s
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor_types import KubernetesJob
+
+_CNCF_KUBERNETES_REQUIRED = "AwsEksExecutor requires apache-airflow-providers-cncf-kubernetes>=10.24.0"
+
+try:
+    from kubernetes.client.rest import ApiException
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+    from airflow.providers.cncf.kubernetes.get_provider_info import (
+        get_provider_info as get_cncf_kubernetes_provider_info,
+    )
+except ImportError as e:
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED) from e
+
+# Check for the option, not the version, since unreleased trees report the previous version.
+if "client_factory" not in (
+    get_cncf_kubernetes_provider_info().get("config", {}).get("kubernetes_executor", {}).get("options", {})
+):
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED)
+
+_FACTORY_MODULE = "airflow.providers.amazon.aws.executors.eks._client_factory"
+_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_kube_client"
+_ASYNC_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_async_kube_client"
+# Airflow 2 hybrid executors that wrap a KubernetesExecutor.
+_HYBRID_KUBERNETES_EXECUTORS = ("CeleryKubernetesExecutor", "LocalKubernetesExecutor")
+
+
+class AwsEksExecutor(KubernetesExecutor):
+    """
+    A KubernetesExecutor that authenticates against an Amazon EKS cluster.
+
+    Builds the Kubernetes client from ``[aws_eks_executor]`` configuration and keeps the
+    short-lived EKS token fresh in-process. All pod-level behaviour comes unchanged from the
+    KubernetesExecutor and its ``[kubernetes_executor]`` configuration.
+    """
+
+    supports_multi_team: bool = True
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        # These need team_name, which super().__init__ sets.
+        self._check_no_team_eks_config()
+        self._validate_eks_config()
+        self._check_no_kubernetes_executor_in_scope()
+        self._ensure_client_factory()
+
+    def start(self) -> None:
+        """Call this when the Executor is run for the first time by the scheduler."""
+        super().start()
+        if TYPE_CHECKING:
+            assert self.kube_scheduler
+        # Both pod creation paths build every worker pod here, so check the image before it is sent.
+        build_pod_request = self.kube_scheduler._build_pod_request
+
+        def _build_pod_request_with_image(job: KubernetesJob) -> k8s.V1Pod:
+            pod = build_pod_request(job)
+            self._check_worker_image(pod)
+            return pod
+
+        self.kube_scheduler._build_pod_request = _build_pod_request_with_image  # type: ignore[method-assign,assignment]
+
+        check_health = conf.getboolean(
+            CONFIG_GROUP_NAME,
+            AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP,
+            fallback=CONFIG_DEFAULTS[AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP],
+        )
+
+        if not check_health:
+            return
+
+        self.log.info("Starting EKS Executor and determining health...")
+        try:
+            self.check_health()
+        except RuntimeError:
+            self.log.error("Stopping the Airflow Scheduler from starting until the issue is resolved.")
+            raise
+
+    def check_health(self) -> None:
+        """
+        Make a test Kubernetes API call to check the health of the EKS Executor.
+
+        Building the client only proves the cluster exists; without this, a missing EKS access
+        entry or RBAC binding only shows up later as a watcher error loop.
+        """
+        if TYPE_CHECKING:
+            assert self.kube_client
+        namespace = self.kube_config.kube_namespace
+        try:
+            self.kube_client.list_namespaced_pod(namespace, limit=1)
+        except ApiException as e:
+            raise RuntimeError(
+                f"EKS Executor health check has failed because: cannot list pods in namespace {namespace} "
+                f"({e.status} {e.reason}). The IAM principal behind the AWS connection needs an EKS access "
+                "entry (or aws-auth mapping) on the cluster that allows create, get, list, watch, patch and "
+                f"delete on pods and get on pods/log in namespace {namespace}. See 'Grant access to the "
+                "cluster' in the AwsEksExecutor docs."
+            ) from e
+        self.log.info("EKS Executor health check has succeeded.")
+
+    def _check_worker_image(self, pod: k8s.V1Pod) -> None:
+        if pod.spec.containers and pod.spec.containers[0].image:
+            return
+        section = (
+            f"team {self.team_name}'s [kubernetes_executor]" if self.team_name else "[kubernetes_executor]"
+        )
+        annotations = pod.metadata.annotations or {}
+        # Raised as the 422 the API server would return, so KubernetesExecutor fails the task.
+        raise ApiException(
+            status=HTTPStatus.UNPROCESSABLE_ENTITY,
+            reason=(
+                f"Task {annotations.get('dag_id')}.{annotations.get('task_id')} has no worker image, so its "
+                "pod was not created. Set an image in one of three ways: on the base container of the pod "
+                f"template ({section} pod_template_file, or the task's executor_config pod_template_file), "
+                f"in the task's executor_config pod_override, or with {section} "
+                "worker_container_repository and worker_container_tag."
+            ),
+        )
+
+    @staticmethod
+    def _validate_eks_config() -> None:
+        if not conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None):
+            raise ValueError(
+                f"AwsEksExecutor requires [{CONFIG_GROUP_NAME}] cluster_name to be set. "
+                "Set it in airflow.cfg or with AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME."
+            )
+
+    def _check_no_team_eks_config(self) -> None:
+        # [aws_eks_executor] is only read globally for now, so refuse team values rather than ignore them.
+        if not self.team_name:
+            return
+        # self.conf looks options up for this team only, falling back to the built-in defaults.
+        team_keys = sorted(
+            key
+            for key in AllEksConfigKeys()
+            if self.conf.get(CONFIG_GROUP_NAME, key, fallback=None) not in (None, CONFIG_DEFAULTS.get(key))
+        )
+        if team_keys:
+            raise ValueError(
+                f"AwsEksExecutor doesn't support team-specific [{CONFIG_GROUP_NAME}] config yet, but team "
+                f"{self.team_name} sets {', '.join(team_keys)}. Set these in the global "
+                f"[{CONFIG_GROUP_NAME}] section instead. Per-team clusters are tracked at "
+                "https://github.com/apache/airflow/issues/74844"
+            )
+
+    def _check_no_kubernetes_executor_in_scope(self) -> None:
+        executor_names = (
+            ExecutorLoader.get_executor_names(validate_teams=False)
+            if AIRFLOW_V_3_2_PLUS
+            else ExecutorLoader.get_executor_names()
+        )
+        for executor_name in executor_names:
+            # ExecutorName has no team_name before Airflow 3.1.
+            if getattr(executor_name, "team_name", None) != self.team_name:
+                continue
+            executor_cls, _ = ExecutorLoader.import_executor_cls(executor_name)
+            if issubclass(executor_cls, AwsEksExecutor) or not (
+                issubclass(executor_cls, KubernetesExecutor)
+                or executor_cls.__name__ in _HYBRID_KUBERNETES_EXECUTORS
+            ):
+                continue
+            scope = f"team {self.team_name}" if self.team_name else "the global scope"
+            raise ValueError(

Review Comment:
   All this broken team stuff bums me out, but I know we're going to fast follow on it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4319941005


##########
providers/amazon/docs/executors/eks-executor.rst:
##########
@@ -0,0 +1,232 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. |executorName| replace:: EKS
+
+.. _eks_executor:
+
+================
+AWS EKS Executor
+================
+
+The EKS executor runs each Airflow task in its own pod on an Amazon EKS cluster.
+
+This executor extends the Kubernetes executor that ships in the ``cncf.kubernetes``
+provider. Pod scheduling, pod templates, per-task pod overrides, and log handling all
+behave the same way they do under
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor`. The part this executor
+adds is authentication. It builds the Kubernetes client for your EKS cluster from an
+Airflow AWS connection, so the scheduler does not need a kubeconfig file on disk and
+you do not have to refresh cluster credentials yourself.
+
+Because the executor inherits its pod behavior, everything written about the
+Kubernetes executor still applies, including the requirement for a database backend
+other than SQLite.
+
+For a quick start guide please see :ref:`here <eks_setup_guide>`.
+
+Requirements
+------------
+
+The executor needs ``apache-airflow-providers-cncf-kubernetes`` version 10.24.0 or
+newer, the release that added the ``client_factory`` setting the executor is built on.
+Install the ``cncf.kubernetes`` extra of the Amazon provider together with that
+version:
+
+.. code-block:: bash
+
+    pip install 'apache-airflow-providers-amazon[cncf.kubernetes]' \
+        'apache-airflow-providers-cncf-kubernetes>=10.24.0'
+
+The extra by itself declares a much older floor, because the Amazon provider's other
+Kubernetes integrations still work with it. A fresh install resolves to the newest
+``cncf.kubernetes`` regardless, so naming the version matters when an older one is
+already pinned in your environment. If the installed version is too old, the executor
+raises an error when it starts up rather than falling back to a client it cannot
+authenticate with.
+
+The AWS credentials the executor uses must be allowed to call ``eks:DescribeCluster``
+on the cluster, and the IAM principal behind those credentials must be granted
+access inside the cluster itself. On modern clusters this means an EKS access entry;
+on older clusters it means an entry in the ``aws-auth`` config map. Without that
+in-cluster grant the scheduler can read the cluster description but every Kubernetes
+API call is rejected.
+
+How authentication works
+------------------------
+
+When the executor starts, it plugs its own Kubernetes client into the Kubernetes
+executor. Every process that needs a client, including the pod watcher that Airflow runs
+as a separate process, builds its own from the ``[aws_eks_executor]`` settings.
+
+To build a client, the executor describes the cluster to find its API endpoint and
+certificate authority, then mints an authentication token for it. An EKS token is a presigned
+STS URL that stays valid for roughly fifteen minutes, while the scheduler holds a
+single client for as long as it runs. To keep the token current, the executor
+registers a refresh hook that the Kubernetes client calls on every authenticated
+request. Minting a token is a local signing operation with no network call, so
+refreshing that often costs very little. Rotation of the underlying AWS credentials
+is handled by botocore in the usual way.
+
+Before it accepts any tasks, the executor checks that the cluster is ``ACTIVE`` (or
+``UPDATING``) and, unless ``check_health_on_startup`` is turned off, that it is allowed
+to list pods in its namespace. It refuses to start if either check fails.
+
+.. _eks_config_options:
+
+Config Options
+--------------
+
+The executor reads its own settings from an ``aws_eks_executor`` section in
+``airflow.cfg``. You can also set any of them with an environment variable using the
+``AIRFLOW__AWS_EKS_EXECUTOR__<OPTION_NAME>`` form, for example
+``AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME=airflow-eks-cluster``. For more information on
+how to set these options, see `Setting Configuration Options
+<https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html>`__.
+
+Required config options:
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  CLUSTER_NAME - The name of the Amazon EKS cluster that tasks run on. The
+   executor refuses to start if this is unset. Required.
+
+Optional config options:
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  CONN_ID - The Airflow connection (i.e. credentials) used by the EKS
+   executor to make API calls to Amazon EKS. Defaults to ``aws_default``.
+-  REGION_NAME - The AWS Region the cluster is in. When this is left empty, the
+   region comes from the standard boto3 resolution order.
+-  CHECK_HEALTH_ON_STARTUP - Whether to check on startup that the executor can
+   list pods in its namespace. Defaults to ``True``.
+
+Pod-level configuration
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Settings that describe the worker pods themselves stay in the
+``[kubernetes_executor]`` section, exactly as they are for the Kubernetes executor.
+This includes ``namespace``, ``pod_template_file``, ``worker_container_repository``,
+``worker_container_tag``, and ``delete_worker_pods``. See
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor` for the full list
+and for how pod templates and ``pod_override`` work.
+
+Leave ``client_factory`` and ``async_client_factory`` in that section unset. The
+executor sets both itself, and raises an error at startup if either is set to something
+else, so that a stale or conflicting setting cannot silently send tasks to the wrong
+cluster.
+
+The worker pod template
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Set ``[kubernetes_executor] pod_template_file`` to the path of a YAML file describing
+the worker pod. Treat it as required. There is no usable default: when the setting is
+empty the Kubernetes executor only logs a warning that the model file does not exist
+and then builds a worker pod from an empty template. That pod is missing the container
+the executor needs, so the failure arrives later and somewhere else, usually as a
+rejected pod or a task that never reports back.
+
+The file must define a container named ``base`` as the first entry in
+``spec.containers``, and that container has to run your Airflow worker image.
+The ``pod_template_file`` section of
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor` covers the full set
+of requirements and includes templates for Dags baked into the image, Dags on a volume,
+and git-sync. Any of those works here unchanged, since this executor only replaces how
+the Kubernetes client is authenticated.
+
+.. _eks_logging:
+
+.. include:: general.rst
+  :start-after: .. BEGIN LOGGING
+  :end-before: .. END LOGGING
+
+-  Worker pods are deleted once their task finishes (``[kubernetes_executor]
+   delete_worker_pods``), and their logs go with them, so configure remote
+   logging to CloudWatch Logs or S3 to keep task logs viewable in the Airflow UI.
+-  The remote logging configuration must be set on the worker pods as well as on
+   the scheduler and API server, and the worker pods need an IAM role, for
+   example through EKS Pod Identity, that can write to the log destination.
+
+.. _eks_setup_guide:
+
+Setting up an EKS Executor for Apache Airflow
+---------------------------------------------
+
+Grant access to the cluster
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Create an EKS access entry for the IAM role or user that the scheduler runs as, and
+associate a policy that allows it to create and watch pods in the namespace you plan
+to use. If you manage cluster access through the ``aws-auth`` config map instead, add
+the principal there and map it to a Kubernetes group with the same permissions.
+
+Configure Airflow
+~~~~~~~~~~~~~~~~~
+
+Select the executor and name your cluster:
+
+.. code-block:: ini
+
+    [core]
+    executor = airflow.providers.amazon.aws.executors.eks.eks_executor.AwsEksExecutor
+
+    [aws_eks_executor]
+    cluster_name = airflow-eks-cluster
+    region_name = us-east-1
+    conn_id = aws_default
+
+    [kubernetes_executor]
+    namespace = airflow
+    pod_template_file = /opt/airflow/pod_templates/worker_template.yaml
+    worker_container_repository = my-account.dkr.ecr.us-east-1.amazonaws.com/airflow
+    worker_container_tag = latest
+
+The worker image must contain Airflow and the Amazon provider, and it needs access
+to your Dag files and a network path to the Airflow API server, in the same way any
+Kubernetes executor worker does.
+
+Task logging
+~~~~~~~~~~~~
+
+Configure remote logging as described in the :ref:`logging <eks_logging>` section, so
+that task logs stay viewable in the Airflow UI after the worker pods are deleted.
+
+Verify the setup
+~~~~~~~~~~~~~~~~
+
+Start the scheduler and trigger a small Dag. A successful run shows a worker pod
+appearing in your chosen namespace and the task finishing in the Airflow UI. If the
+scheduler logs an authentication or forbidden error from the Kubernetes API, the
+in-cluster access grant for your IAM principal is the first thing to check.
+
+Multi-team deployments
+----------------------
+
+The executor cannot be used as a team executor yet, because its settings are read from
+the un-prefixed ``[aws_eks_executor]`` section and every team would share one cluster.

Review Comment:
   For posterity, the agreed upon multi-team approach is covered in this review comment https://github.com/apache/airflow/pull/74399#discussion_r4317213748



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4225650975


##########
providers/amazon/docs/executors/eks-executor.rst:
##########
@@ -0,0 +1,232 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. |executorName| replace:: EKS
+
+.. _eks_executor:
+
+================
+AWS EKS Executor
+================
+
+The EKS executor runs each Airflow task in its own pod on an Amazon EKS cluster.
+
+This executor extends the Kubernetes executor that ships in the ``cncf.kubernetes``
+provider. Pod scheduling, pod templates, per-task pod overrides, and log handling all
+behave the same way they do under
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor`. The part this executor
+adds is authentication. It builds the Kubernetes client for your EKS cluster from an
+Airflow AWS connection, so the scheduler does not need a kubeconfig file on disk and
+you do not have to refresh cluster credentials yourself.
+
+Because the executor inherits its pod behavior, everything written about the
+Kubernetes executor still applies, including the requirement for a database backend
+other than SQLite.
+
+For a quick start guide please see :ref:`here <eks_setup_guide>`.
+
+Requirements
+------------
+
+The executor needs ``apache-airflow-providers-cncf-kubernetes`` version 10.24.0 or
+newer, the release that added the ``client_factory`` setting the executor is built on.
+Install the ``cncf.kubernetes`` extra of the Amazon provider together with that
+version:
+
+.. code-block:: bash
+
+    pip install 'apache-airflow-providers-amazon[cncf.kubernetes]' \
+        'apache-airflow-providers-cncf-kubernetes>=10.24.0'
+
+The extra by itself declares a much older floor, because the Amazon provider's other
+Kubernetes integrations still work with it. A fresh install resolves to the newest
+``cncf.kubernetes`` regardless, so naming the version matters when an older one is
+already pinned in your environment. If the installed version is too old, the executor
+raises an error when it starts up rather than falling back to a client it cannot
+authenticate with.
+
+The AWS credentials the executor uses must be allowed to call ``eks:DescribeCluster``
+on the cluster, and the IAM principal behind those credentials must be granted
+access inside the cluster itself. On modern clusters this means an EKS access entry;
+on older clusters it means an entry in the ``aws-auth`` config map. Without that
+in-cluster grant the scheduler can read the cluster description but every Kubernetes
+API call is rejected.
+
+How authentication works
+------------------------
+
+When the executor starts, it plugs its own Kubernetes client into the Kubernetes
+executor. Every process that needs a client, including the pod watcher that Airflow runs
+as a separate process, builds its own from the ``[aws_eks_executor]`` settings.
+
+To build a client, the executor describes the cluster to find its API endpoint and
+certificate authority, then mints an authentication token for it. An EKS token is a presigned
+STS URL that stays valid for roughly fifteen minutes, while the scheduler holds a
+single client for as long as it runs. To keep the token current, the executor
+registers a refresh hook that the Kubernetes client calls on every authenticated
+request. Minting a token is a local signing operation with no network call, so
+refreshing that often costs very little. Rotation of the underlying AWS credentials
+is handled by botocore in the usual way.
+
+Before it accepts any tasks, the executor checks that the cluster is ``ACTIVE`` (or
+``UPDATING``) and, unless ``check_health_on_startup`` is turned off, that it is allowed
+to list pods in its namespace. It refuses to start if either check fails.
+
+.. _eks_config_options:
+
+Config Options
+--------------
+
+The executor reads its own settings from an ``aws_eks_executor`` section in
+``airflow.cfg``. You can also set any of them with an environment variable using the
+``AIRFLOW__AWS_EKS_EXECUTOR__<OPTION_NAME>`` form, for example
+``AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME=airflow-eks-cluster``. For more information on
+how to set these options, see `Setting Configuration Options
+<https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html>`__.
+
+Required config options:
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  CLUSTER_NAME - The name of the Amazon EKS cluster that tasks run on. The
+   executor refuses to start if this is unset. Required.
+
+Optional config options:
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  CONN_ID - The Airflow connection (i.e. credentials) used by the EKS
+   executor to make API calls to Amazon EKS. Defaults to ``aws_default``.
+-  REGION_NAME - The AWS Region the cluster is in. When this is left empty, the
+   region comes from the standard boto3 resolution order.
+-  CHECK_HEALTH_ON_STARTUP - Whether to check on startup that the executor can
+   list pods in its namespace. Defaults to ``True``.
+
+Pod-level configuration
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Settings that describe the worker pods themselves stay in the
+``[kubernetes_executor]`` section, exactly as they are for the Kubernetes executor.
+This includes ``namespace``, ``pod_template_file``, ``worker_container_repository``,
+``worker_container_tag``, and ``delete_worker_pods``. See
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor` for the full list
+and for how pod templates and ``pod_override`` work.
+
+Leave ``client_factory`` and ``async_client_factory`` in that section unset. The
+executor sets both itself, and raises an error at startup if either is set to something
+else, so that a stale or conflicting setting cannot silently send tasks to the wrong
+cluster.
+
+The worker pod template
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Set ``[kubernetes_executor] pod_template_file`` to the path of a YAML file describing
+the worker pod. Treat it as required. There is no usable default: when the setting is
+empty the Kubernetes executor only logs a warning that the model file does not exist
+and then builds a worker pod from an empty template. That pod is missing the container
+the executor needs, so the failure arrives later and somewhere else, usually as a
+rejected pod or a task that never reports back.
+
+The file must define a container named ``base`` as the first entry in
+``spec.containers``, and that container has to run your Airflow worker image.
+The ``pod_template_file`` section of
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor` covers the full set
+of requirements and includes templates for Dags baked into the image, Dags on a volume,
+and git-sync. Any of those works here unchanged, since this executor only replaces how
+the Kubernetes client is authenticated.
+
+.. _eks_logging:
+
+.. include:: general.rst
+  :start-after: .. BEGIN LOGGING
+  :end-before: .. END LOGGING
+
+-  Worker pods are deleted once their task finishes (``[kubernetes_executor]
+   delete_worker_pods``), and their logs go with them, so configure remote
+   logging to CloudWatch Logs or S3 to keep task logs viewable in the Airflow UI.
+-  The remote logging configuration must be set on the worker pods as well as on
+   the scheduler and API server, and the worker pods need an IAM role, for
+   example through EKS Pod Identity, that can write to the log destination.
+
+.. _eks_setup_guide:
+
+Setting up an EKS Executor for Apache Airflow
+---------------------------------------------
+
+Grant access to the cluster
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Create an EKS access entry for the IAM role or user that the scheduler runs as, and
+associate a policy that allows it to create and watch pods in the namespace you plan
+to use. If you manage cluster access through the ``aws-auth`` config map instead, add
+the principal there and map it to a Kubernetes group with the same permissions.
+
+Configure Airflow
+~~~~~~~~~~~~~~~~~
+
+Select the executor and name your cluster:
+
+.. code-block:: ini
+
+    [core]
+    executor = airflow.providers.amazon.aws.executors.eks.eks_executor.AwsEksExecutor
+
+    [aws_eks_executor]
+    cluster_name = airflow-eks-cluster
+    region_name = us-east-1
+    conn_id = aws_default
+
+    [kubernetes_executor]
+    namespace = airflow
+    pod_template_file = /opt/airflow/pod_templates/worker_template.yaml
+    worker_container_repository = my-account.dkr.ecr.us-east-1.amazonaws.com/airflow
+    worker_container_tag = latest
+
+The worker image must contain Airflow and the Amazon provider, and it needs access
+to your Dag files and a network path to the Airflow API server, in the same way any
+Kubernetes executor worker does.
+
+Task logging
+~~~~~~~~~~~~
+
+Configure remote logging as described in the :ref:`logging <eks_logging>` section, so
+that task logs stay viewable in the Airflow UI after the worker pods are deleted.
+
+Verify the setup
+~~~~~~~~~~~~~~~~
+
+Start the scheduler and trigger a small Dag. A successful run shows a worker pod
+appearing in your chosen namespace and the task finishing in the Airflow UI. If the
+scheduler logs an authentication or forbidden error from the Kubernetes API, the
+in-cluster access grant for your IAM principal is the first thing to check.
+
+Multi-team deployments
+----------------------
+
+The executor cannot be used as a team executor yet, because its settings are read from
+the un-prefixed ``[aws_eks_executor]`` section and every team would share one cluster.

Review Comment:
   Yes, referencing multi-team. Linking to the doc now. Also, I've updated the docs to reflect that the `AwsEksExecutor` can be used as a team executor, the main limitation is that all teams must share the same cluster (which is the same limitation as `KubernetesExecutor`). This can be addressed in a follow-up if desired. @o-nikolas @vincbeck let me know if you have thoughts on the multi-team capabilities you'd like to support.



##########
providers/amazon/docs/executors/eks-executor.rst:
##########
@@ -0,0 +1,232 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. |executorName| replace:: EKS
+
+.. _eks_executor:
+
+================
+AWS EKS Executor
+================
+
+The EKS executor runs each Airflow task in its own pod on an Amazon EKS cluster.
+
+This executor extends the Kubernetes executor that ships in the ``cncf.kubernetes``
+provider. Pod scheduling, pod templates, per-task pod overrides, and log handling all
+behave the same way they do under
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor`. The part this executor
+adds is authentication. It builds the Kubernetes client for your EKS cluster from an
+Airflow AWS connection, so the scheduler does not need a kubeconfig file on disk and
+you do not have to refresh cluster credentials yourself.
+
+Because the executor inherits its pod behavior, everything written about the
+Kubernetes executor still applies, including the requirement for a database backend
+other than SQLite.
+
+For a quick start guide please see :ref:`here <eks_setup_guide>`.
+
+Requirements
+------------
+
+The executor needs ``apache-airflow-providers-cncf-kubernetes`` version 10.24.0 or
+newer, the release that added the ``client_factory`` setting the executor is built on.
+Install the ``cncf.kubernetes`` extra of the Amazon provider together with that
+version:
+
+.. code-block:: bash
+
+    pip install 'apache-airflow-providers-amazon[cncf.kubernetes]' \
+        'apache-airflow-providers-cncf-kubernetes>=10.24.0'
+
+The extra by itself declares a much older floor, because the Amazon provider's other
+Kubernetes integrations still work with it. A fresh install resolves to the newest
+``cncf.kubernetes`` regardless, so naming the version matters when an older one is
+already pinned in your environment. If the installed version is too old, the executor
+raises an error when it starts up rather than falling back to a client it cannot
+authenticate with.
+
+The AWS credentials the executor uses must be allowed to call ``eks:DescribeCluster``
+on the cluster, and the IAM principal behind those credentials must be granted
+access inside the cluster itself. On modern clusters this means an EKS access entry;
+on older clusters it means an entry in the ``aws-auth`` config map. Without that
+in-cluster grant the scheduler can read the cluster description but every Kubernetes
+API call is rejected.
+
+How authentication works
+------------------------
+
+When the executor starts, it plugs its own Kubernetes client into the Kubernetes
+executor. Every process that needs a client, including the pod watcher that Airflow runs
+as a separate process, builds its own from the ``[aws_eks_executor]`` settings.
+
+To build a client, the executor describes the cluster to find its API endpoint and
+certificate authority, then mints an authentication token for it. An EKS token is a presigned
+STS URL that stays valid for roughly fifteen minutes, while the scheduler holds a
+single client for as long as it runs. To keep the token current, the executor
+registers a refresh hook that the Kubernetes client calls on every authenticated
+request. Minting a token is a local signing operation with no network call, so
+refreshing that often costs very little. Rotation of the underlying AWS credentials
+is handled by botocore in the usual way.
+
+Before it accepts any tasks, the executor checks that the cluster is ``ACTIVE`` (or
+``UPDATING``) and, unless ``check_health_on_startup`` is turned off, that it is allowed
+to list pods in its namespace. It refuses to start if either check fails.

Review Comment:
   An EKS cluster will [continue working](https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html#:~:text=To%20ensure%20that%20the%20API%20server%20endpoint%20for%20your%20cluster%20is%20always%20accessible) while it's being updated, so it's fine to carry on with startup.
   
   As for `check_health_on_startup`, agreed, linking to it now.



##########
providers/amazon/docs/executors/eks-executor.rst:
##########
@@ -0,0 +1,232 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. |executorName| replace:: EKS
+
+.. _eks_executor:
+
+================
+AWS EKS Executor
+================
+
+The EKS executor runs each Airflow task in its own pod on an Amazon EKS cluster.
+
+This executor extends the Kubernetes executor that ships in the ``cncf.kubernetes``
+provider. Pod scheduling, pod templates, per-task pod overrides, and log handling all
+behave the same way they do under
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor`. The part this executor
+adds is authentication. It builds the Kubernetes client for your EKS cluster from an
+Airflow AWS connection, so the scheduler does not need a kubeconfig file on disk and
+you do not have to refresh cluster credentials yourself.
+
+Because the executor inherits its pod behavior, everything written about the
+Kubernetes executor still applies, including the requirement for a database backend
+other than SQLite.
+
+For a quick start guide please see :ref:`here <eks_setup_guide>`.
+
+Requirements
+------------
+
+The executor needs ``apache-airflow-providers-cncf-kubernetes`` version 10.24.0 or
+newer, the release that added the ``client_factory`` setting the executor is built on.
+Install the ``cncf.kubernetes`` extra of the Amazon provider together with that
+version:
+
+.. code-block:: bash
+
+    pip install 'apache-airflow-providers-amazon[cncf.kubernetes]' \
+        'apache-airflow-providers-cncf-kubernetes>=10.24.0'
+
+The extra by itself declares a much older floor, because the Amazon provider's other
+Kubernetes integrations still work with it. A fresh install resolves to the newest
+``cncf.kubernetes`` regardless, so naming the version matters when an older one is
+already pinned in your environment. If the installed version is too old, the executor
+raises an error when it starts up rather than falling back to a client it cannot
+authenticate with.
+
+The AWS credentials the executor uses must be allowed to call ``eks:DescribeCluster``
+on the cluster, and the IAM principal behind those credentials must be granted
+access inside the cluster itself. On modern clusters this means an EKS access entry;
+on older clusters it means an entry in the ``aws-auth`` config map. Without that
+in-cluster grant the scheduler can read the cluster description but every Kubernetes
+API call is rejected.
+
+How authentication works
+------------------------
+
+When the executor starts, it plugs its own Kubernetes client into the Kubernetes
+executor. Every process that needs a client, including the pod watcher that Airflow runs
+as a separate process, builds its own from the ``[aws_eks_executor]`` settings.
+
+To build a client, the executor describes the cluster to find its API endpoint and
+certificate authority, then mints an authentication token for it. An EKS token is a presigned
+STS URL that stays valid for roughly fifteen minutes, while the scheduler holds a
+single client for as long as it runs. To keep the token current, the executor
+registers a refresh hook that the Kubernetes client calls on every authenticated
+request. Minting a token is a local signing operation with no network call, so
+refreshing that often costs very little. Rotation of the underlying AWS credentials
+is handled by botocore in the usual way.

Review Comment:
   Fair point, by "usual" way I'm just referring to the way credentials are [generally managed](https://docs.aws.amazon.com/boto3/latest/guide/credentials.html) by aws. Users need not concern themselves with this here, so I'm just dropping that sentence.



##########
providers/amazon/docs/executors/eks-executor.rst:
##########
@@ -0,0 +1,232 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. |executorName| replace:: EKS
+
+.. _eks_executor:
+
+================
+AWS EKS Executor
+================
+
+The EKS executor runs each Airflow task in its own pod on an Amazon EKS cluster.
+
+This executor extends the Kubernetes executor that ships in the ``cncf.kubernetes``
+provider. Pod scheduling, pod templates, per-task pod overrides, and log handling all
+behave the same way they do under
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor`. The part this executor
+adds is authentication. It builds the Kubernetes client for your EKS cluster from an
+Airflow AWS connection, so the scheduler does not need a kubeconfig file on disk and
+you do not have to refresh cluster credentials yourself.
+
+Because the executor inherits its pod behavior, everything written about the
+Kubernetes executor still applies, including the requirement for a database backend
+other than SQLite.
+
+For a quick start guide please see :ref:`here <eks_setup_guide>`.
+
+Requirements
+------------
+
+The executor needs ``apache-airflow-providers-cncf-kubernetes`` version 10.24.0 or
+newer, the release that added the ``client_factory`` setting the executor is built on.
+Install the ``cncf.kubernetes`` extra of the Amazon provider together with that
+version:
+
+.. code-block:: bash
+
+    pip install 'apache-airflow-providers-amazon[cncf.kubernetes]' \
+        'apache-airflow-providers-cncf-kubernetes>=10.24.0'
+
+The extra by itself declares a much older floor, because the Amazon provider's other
+Kubernetes integrations still work with it. A fresh install resolves to the newest
+``cncf.kubernetes`` regardless, so naming the version matters when an older one is
+already pinned in your environment. If the installed version is too old, the executor
+raises an error when it starts up rather than falling back to a client it cannot
+authenticate with.
+
+The AWS credentials the executor uses must be allowed to call ``eks:DescribeCluster``
+on the cluster, and the IAM principal behind those credentials must be granted
+access inside the cluster itself. On modern clusters this means an EKS access entry;
+on older clusters it means an entry in the ``aws-auth`` config map. Without that
+in-cluster grant the scheduler can read the cluster description but every Kubernetes
+API call is rejected.

Review Comment:
   It is already being checked on startup
   
   https://github.com/apache/airflow/blob/a947f0014996f3b99ec2cf10083745c9f01e69f8/providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py#L103-L109
   
   Reworded the paragraph to say so, and the error now also listing the permissions it needs.



##########
providers/amazon/docs/executors/eks-executor.rst:
##########
@@ -0,0 +1,232 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. |executorName| replace:: EKS
+
+.. _eks_executor:
+
+================
+AWS EKS Executor
+================
+
+The EKS executor runs each Airflow task in its own pod on an Amazon EKS cluster.
+
+This executor extends the Kubernetes executor that ships in the ``cncf.kubernetes``
+provider. Pod scheduling, pod templates, per-task pod overrides, and log handling all
+behave the same way they do under
+:doc:`apache-airflow-providers-cncf-kubernetes:kubernetes_executor`. The part this executor
+adds is authentication. It builds the Kubernetes client for your EKS cluster from an
+Airflow AWS connection, so the scheduler does not need a kubeconfig file on disk and
+you do not have to refresh cluster credentials yourself.
+
+Because the executor inherits its pod behavior, everything written about the
+Kubernetes executor still applies, including the requirement for a database backend
+other than SQLite.
+
+For a quick start guide please see :ref:`here <eks_setup_guide>`.
+
+Requirements
+------------
+
+The executor needs ``apache-airflow-providers-cncf-kubernetes`` version 10.24.0 or
+newer, the release that added the ``client_factory`` setting the executor is built on.
+Install the ``cncf.kubernetes`` extra of the Amazon provider together with that
+version:
+
+.. code-block:: bash
+
+    pip install 'apache-airflow-providers-amazon[cncf.kubernetes]' \
+        'apache-airflow-providers-cncf-kubernetes>=10.24.0'
+
+The extra by itself declares a much older floor, because the Amazon provider's other
+Kubernetes integrations still work with it. A fresh install resolves to the newest
+``cncf.kubernetes`` regardless, so naming the version matters when an older one is
+already pinned in your environment. If the installed version is too old, the executor
+raises an error when it starts up rather than falling back to a client it cannot
+authenticate with.
+

Review Comment:
   Good idea, dropped it and checking at import time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4299822487


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py:
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kubernetes client factories for the AwsEksExecutor."""
+
+# Internal to AwsEksExecutor, which writes these functions' import paths into
+# [kubernetes_executor] client_factory and async_client_factory.
+
+from __future__ import annotations
+
+import os
+import tempfile
+import time
+from base64 import b64decode
+from typing import TYPE_CHECKING
+
+import kubernetes
+from packaging.version import Version
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.aws.hooks.eks import EksHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.providers.amazon.aws.utils.eks_get_token import (
+    TOKEN_EXPIRATION_MINUTES,
+    fetch_access_token_for_cluster,
+)
+from airflow.providers.common.compat.sdk import conf
+
+if TYPE_CHECKING:
+    from kubernetes import client
+    from kubernetes_asyncio import client as async_client
+
+# UPDATING still serves the Kubernetes API, so every other status is refused.
+_USABLE_CLUSTER_STATUSES = ("ACTIVE", "UPDATING")
+
+
+def _get_sync_token_key(kubernetes_version: str) -> str:
+    # kubernetes 36 moved the bearer token from the "authorization" api_key to "BearerToken".
+    return "BearerToken" if Version(kubernetes_version).major >= 36 else "authorization"
+
+
+_SYNC_TOKEN_KEY = _get_sync_token_key(kubernetes.__version__)
+
+
+def _get_eks_kube_client() -> client.CoreV1Api:
+    """Build a Kubernetes client for the configured EKS cluster, in memory and without a kubeconfig."""
+    from kubernetes import client
+
+    configuration = client.Configuration()
+    _configure_eks_auth(configuration, _SYNC_TOKEN_KEY)
+    return client.CoreV1Api(client.ApiClient(configuration=configuration))
+
+
+def _get_eks_async_kube_client() -> async_client.CoreV1Api:
+    """Build the asynchronous Kubernetes client used when ``async_pod_creation`` is enabled."""
+    from kubernetes_asyncio import client as async_client
+
+    configuration = async_client.Configuration()
+    # kubernetes_asyncio has always used "BearerToken".
+    _configure_eks_auth(configuration, "BearerToken")
+    return async_client.CoreV1Api(async_client.ApiClient(configuration=configuration))
+
+
+def _configure_eks_auth(
+    configuration: client.Configuration | async_client.Configuration, token_key: str
+) -> None:
+    cluster_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None)
+    if not cluster_name:
+        raise ValueError(
+            f"[{CONFIG_GROUP_NAME}] cluster_name is required to build an EKS client. "
+            "Set it in airflow.cfg or with AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME."
+        )
+    region_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.REGION_NAME, fallback=None)
+    conn_id = conf.get(
+        CONFIG_GROUP_NAME,
+        AllEksConfigKeys.AWS_CONN_ID,
+        fallback=CONFIG_DEFAULTS[AllEksConfigKeys.AWS_CONN_ID],
+    )
+
+    eks_hook = EksHook(aws_conn_id=conn_id, region_name=region_name)
+    cluster = eks_hook.conn.describe_cluster(name=cluster_name)["cluster"]
+    if cluster["status"] not in _USABLE_CLUSTER_STATUSES:
+        raise ValueError(
+            f"EKS cluster {cluster_name} is {cluster['status']}; the executor needs it to be ACTIVE or "
+            f"UPDATING. Wait for it to become ACTIVE, or point [{CONFIG_GROUP_NAME}] cluster_name at a "
+            "usable cluster."
+        )
+    session = eks_hook.get_session()
+
+    # EKS only accepts tokens presigned against the regional STS endpoint.
+    os.environ["AWS_STS_REGIONAL_ENDPOINTS"] = "regional"
+    try:
+        sts_endpoint = StsHook(
+            aws_conn_id=conn_id, region_name=session.region_name
+        ).conn_client_meta.endpoint_url
+    finally:
+        del os.environ["AWS_STS_REGIONAL_ENDPOINTS"]
+    sts_url = f"{sts_endpoint}/?Action=GetCallerIdentity&Version=2011-06-15"
+
+    configuration.host = cluster["endpoint"]
+    configuration.ssl_ca_cert = _write_cluster_ca_file(cluster["certificateAuthority"]["data"])
+    configuration.api_key_prefix[token_key] = "Bearer"
+
+    # EKS tokens expire after about 15 minutes, so get a new one shortly before the current one expires.
+    token: str | None = None
+    refresh_at = 0.0

Review Comment:
   Yeah, done on purpose. This instructs the client_factory to mint the first token right away on the first call, and then subsequently when refresh_at is set to:
   
   `refresh_at = time.monotonic() + TOKEN_EXPIRATION_MINUTES * 60`
   
   It will, from here on out, refresh tokens at the 14 minute mark (expiry is at 15 minutes).
   
   So, it is a bit misleading that the first "refresh" is actually just the first time the token ever gets minted. But, it's clever, and allows us to write the call to `fetch_access_token_for_cluster` only once in function `refresh_api_key`, which I quite like, so I think it's reasonable to keep.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4299990669


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py:
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kubernetes client factories for the AwsEksExecutor."""
+
+# Internal to AwsEksExecutor, which writes these functions' import paths into
+# [kubernetes_executor] client_factory and async_client_factory.
+
+from __future__ import annotations
+
+import os
+import tempfile
+import time
+from base64 import b64decode
+from typing import TYPE_CHECKING
+
+import kubernetes
+from packaging.version import Version
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.aws.hooks.eks import EksHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.providers.amazon.aws.utils.eks_get_token import (
+    TOKEN_EXPIRATION_MINUTES,
+    fetch_access_token_for_cluster,
+)
+from airflow.providers.common.compat.sdk import conf
+
+if TYPE_CHECKING:
+    from kubernetes import client
+    from kubernetes_asyncio import client as async_client
+
+# UPDATING still serves the Kubernetes API, so every other status is refused.
+_USABLE_CLUSTER_STATUSES = ("ACTIVE", "UPDATING")
+
+
+def _get_sync_token_key(kubernetes_version: str) -> str:
+    # kubernetes 36 moved the bearer token from the "authorization" api_key to "BearerToken".
+    return "BearerToken" if Version(kubernetes_version).major >= 36 else "authorization"
+
+
+_SYNC_TOKEN_KEY = _get_sync_token_key(kubernetes.__version__)
+
+
+def _get_eks_kube_client() -> client.CoreV1Api:
+    """Build a Kubernetes client for the configured EKS cluster, in memory and without a kubeconfig."""
+    from kubernetes import client
+
+    configuration = client.Configuration()
+    _configure_eks_auth(configuration, _SYNC_TOKEN_KEY)
+    return client.CoreV1Api(client.ApiClient(configuration=configuration))
+
+
+def _get_eks_async_kube_client() -> async_client.CoreV1Api:
+    """Build the asynchronous Kubernetes client used when ``async_pod_creation`` is enabled."""
+    from kubernetes_asyncio import client as async_client
+
+    configuration = async_client.Configuration()
+    # kubernetes_asyncio has always used "BearerToken".
+    _configure_eks_auth(configuration, "BearerToken")
+    return async_client.CoreV1Api(async_client.ApiClient(configuration=configuration))
+
+
+def _configure_eks_auth(
+    configuration: client.Configuration | async_client.Configuration, token_key: str
+) -> None:
+    cluster_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None)

Review Comment:
   Since at the moment all teams have to use the same cluster (same as KubernetesExecutor, issue tracked in #74844 ), all the `AwsEksExecutor` config parameters are shared too. For now, on startup, I added a check for team-specific `AwsEksExecutor` config options and, if detected, an error is thrown.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4320261935


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py:
##########
@@ -0,0 +1,137 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""AwsEksExecutor: run Airflow tasks as pods on an Amazon EKS cluster."""
+
+from __future__ import annotations
+
+import os
+from typing import TYPE_CHECKING
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException, conf
+
+_CNCF_KUBERNETES_REQUIRED = "AwsEksExecutor requires apache-airflow-providers-cncf-kubernetes>=10.24.0"
+
+try:
+    from kubernetes.client.rest import ApiException
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+    from airflow.providers.cncf.kubernetes.get_provider_info import (
+        get_provider_info as get_cncf_kubernetes_provider_info,
+    )
+except ImportError as e:
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED) from e
+
+# Check for the option, not the version, since unreleased trees report the previous version.
+if "client_factory" not in (
+    get_cncf_kubernetes_provider_info().get("config", {}).get("kubernetes_executor", {}).get("options", {})
+):
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED)
+
+_FACTORY_MODULE = "airflow.providers.amazon.aws.executors.eks._client_factory"
+_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_kube_client"
+_ASYNC_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_async_kube_client"
+
+
+class AwsEksExecutor(KubernetesExecutor):
+    """
+    A KubernetesExecutor that authenticates against an Amazon EKS cluster.
+
+    Builds the Kubernetes client from ``[aws_eks_executor]`` configuration and keeps the
+    short-lived EKS token fresh in-process. All pod-level behaviour comes unchanged from the
+    KubernetesExecutor and its ``[kubernetes_executor]`` configuration.
+    """
+
+    supports_multi_team: bool = True
+
+    def __init__(self, *args, **kwargs):
+        self._validate_eks_config()
+        super().__init__(*args, **kwargs)
+        # Needs team_name, which super().__init__ sets.
+        self._ensure_client_factory()
+
+    def start(self) -> None:
+        """Call this when the Executor is run for the first time by the scheduler."""
+        super().start()
+        check_health = conf.getboolean(
+            CONFIG_GROUP_NAME,
+            AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP,
+            fallback=CONFIG_DEFAULTS[AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP],
+        )
+
+        if not check_health:
+            return
+
+        self.log.info("Starting EKS Executor and determining health...")
+        try:
+            self.check_health()
+        except RuntimeError:
+            self.log.error("Stopping the Airflow Scheduler from starting until the issue is resolved.")
+            raise
+
+    def check_health(self) -> None:
+        """
+        Make a test Kubernetes API call to check the health of the EKS Executor.
+
+        Building the client only proves the cluster exists; without this, a missing EKS access
+        entry or RBAC binding only shows up later as a watcher error loop.
+        """
+        if TYPE_CHECKING:
+            assert self.kube_client
+        namespace = self.kube_config.kube_namespace
+        try:
+            self.kube_client.list_namespaced_pod(namespace, limit=1)
+        except ApiException as e:
+            raise RuntimeError(
+                f"EKS Executor health check has failed because: cannot list pods in namespace {namespace} "
+                f"({e.status} {e.reason}). The IAM principal behind the AWS connection needs an EKS access "
+                "entry (or aws-auth mapping) on the cluster that allows create, get, list, watch, patch and "
+                f"delete on pods and get on pods/log in namespace {namespace}. See 'Grant access to the "
+                "cluster' in the AwsEksExecutor docs."
+            ) from e
+        self.log.info("EKS Executor health check has succeeded.")
+
+    @staticmethod
+    def _validate_eks_config() -> None:
+        if not conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None):

Review Comment:
   That's right, nothing blocking that cncf change from happening now. Working on writing/testing that PR now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4299822487


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py:
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kubernetes client factories for the AwsEksExecutor."""
+
+# Internal to AwsEksExecutor, which writes these functions' import paths into
+# [kubernetes_executor] client_factory and async_client_factory.
+
+from __future__ import annotations
+
+import os
+import tempfile
+import time
+from base64 import b64decode
+from typing import TYPE_CHECKING
+
+import kubernetes
+from packaging.version import Version
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.aws.hooks.eks import EksHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.providers.amazon.aws.utils.eks_get_token import (
+    TOKEN_EXPIRATION_MINUTES,
+    fetch_access_token_for_cluster,
+)
+from airflow.providers.common.compat.sdk import conf
+
+if TYPE_CHECKING:
+    from kubernetes import client
+    from kubernetes_asyncio import client as async_client
+
+# UPDATING still serves the Kubernetes API, so every other status is refused.
+_USABLE_CLUSTER_STATUSES = ("ACTIVE", "UPDATING")
+
+
+def _get_sync_token_key(kubernetes_version: str) -> str:
+    # kubernetes 36 moved the bearer token from the "authorization" api_key to "BearerToken".
+    return "BearerToken" if Version(kubernetes_version).major >= 36 else "authorization"
+
+
+_SYNC_TOKEN_KEY = _get_sync_token_key(kubernetes.__version__)
+
+
+def _get_eks_kube_client() -> client.CoreV1Api:
+    """Build a Kubernetes client for the configured EKS cluster, in memory and without a kubeconfig."""
+    from kubernetes import client
+
+    configuration = client.Configuration()
+    _configure_eks_auth(configuration, _SYNC_TOKEN_KEY)
+    return client.CoreV1Api(client.ApiClient(configuration=configuration))
+
+
+def _get_eks_async_kube_client() -> async_client.CoreV1Api:
+    """Build the asynchronous Kubernetes client used when ``async_pod_creation`` is enabled."""
+    from kubernetes_asyncio import client as async_client
+
+    configuration = async_client.Configuration()
+    # kubernetes_asyncio has always used "BearerToken".
+    _configure_eks_auth(configuration, "BearerToken")
+    return async_client.CoreV1Api(async_client.ApiClient(configuration=configuration))
+
+
+def _configure_eks_auth(
+    configuration: client.Configuration | async_client.Configuration, token_key: str
+) -> None:
+    cluster_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None)
+    if not cluster_name:
+        raise ValueError(
+            f"[{CONFIG_GROUP_NAME}] cluster_name is required to build an EKS client. "
+            "Set it in airflow.cfg or with AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME."
+        )
+    region_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.REGION_NAME, fallback=None)
+    conn_id = conf.get(
+        CONFIG_GROUP_NAME,
+        AllEksConfigKeys.AWS_CONN_ID,
+        fallback=CONFIG_DEFAULTS[AllEksConfigKeys.AWS_CONN_ID],
+    )
+
+    eks_hook = EksHook(aws_conn_id=conn_id, region_name=region_name)
+    cluster = eks_hook.conn.describe_cluster(name=cluster_name)["cluster"]
+    if cluster["status"] not in _USABLE_CLUSTER_STATUSES:
+        raise ValueError(
+            f"EKS cluster {cluster_name} is {cluster['status']}; the executor needs it to be ACTIVE or "
+            f"UPDATING. Wait for it to become ACTIVE, or point [{CONFIG_GROUP_NAME}] cluster_name at a "
+            "usable cluster."
+        )
+    session = eks_hook.get_session()
+
+    # EKS only accepts tokens presigned against the regional STS endpoint.
+    os.environ["AWS_STS_REGIONAL_ENDPOINTS"] = "regional"
+    try:
+        sts_endpoint = StsHook(
+            aws_conn_id=conn_id, region_name=session.region_name
+        ).conn_client_meta.endpoint_url
+    finally:
+        del os.environ["AWS_STS_REGIONAL_ENDPOINTS"]
+    sts_url = f"{sts_endpoint}/?Action=GetCallerIdentity&Version=2011-06-15"
+
+    configuration.host = cluster["endpoint"]
+    configuration.ssl_ca_cert = _write_cluster_ca_file(cluster["certificateAuthority"]["data"])
+    configuration.api_key_prefix[token_key] = "Bearer"
+
+    # EKS tokens expire after about 15 minutes, so get a new one shortly before the current one expires.
+    token: str | None = None
+    refresh_at = 0.0

Review Comment:
   Yeah, done on purpose. This instructs the client_factory to mint the first token right away on the first call, and then subsequently when refresh_at is set to:
   
   `refresh_at = time.monotonic() + TOKEN_EXPIRATION_MINUTES * 60`
   
   It will, from here on out, refresh tokens at the 14 minute mark (expiry is at 15 minutes).
   
   So, it is a bit misleading that the first "refresh" is actually just the first time the token ever gets minted. But, it allows us to write the call to `fetch_access_token_for_cluster` only once in function `refresh_api_key`, which I quite like, so I think it's reasonable to keep.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4291715024


##########
providers/amazon/tests/unit/amazon/aws/executors/eks/test_eks_executor.py:
##########
@@ -0,0 +1,154 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import importlib
+import os
+import sys
+from unittest import mock
+
+import pytest
+from kubernetes.client import CoreV1Api
+from kubernetes.client.rest import ApiException
+
+from airflow.providers.amazon.aws.executors.eks import AwsEksExecutor, _client_factory
+from airflow.providers.amazon.aws.executors.eks.eks_executor import (
+    _ASYNC_CLIENT_FACTORY_PATH,
+    _CLIENT_FACTORY_PATH,
+)
+from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException, conf
+
+from tests_common.test_utils.config import conf_vars
+from tests_common.test_utils.version_compat import AIRFLOW_V_3_2_PLUS
+
+EKS_EXECUTOR_MODULE = "airflow.providers.amazon.aws.executors.eks.eks_executor"
+CLIENT_FACTORY_ENV_VAR = "AIRFLOW__KUBERNETES_EXECUTOR__CLIENT_FACTORY"
+TEAMS = [
+    pytest.param(None, "AIRFLOW__", id="global"),
+    pytest.param(
+        "team_a",
+        "AIRFLOW__TEAM_A___",
+        id="team",
+        marks=pytest.mark.skipif(not AIRFLOW_V_3_2_PLUS, reason="Multi-team requires Airflow 3.2+"),
+    ),
+]
+
+
+def _team_kwargs(team_name: str | None) -> dict:
+    return {"team_name": team_name} if team_name else {}
+
+
+@pytest.fixture(autouse=True)
+def eks_config():
+    with mock.patch.dict(os.environ), conf_vars({("aws_eks_executor", "cluster_name"): "test-eks-cluster"}):
+        for prefix in ("AIRFLOW__", "AIRFLOW__TEAM_A___"):
+            for key in ("CLIENT_FACTORY", "ASYNC_CLIENT_FACTORY"):
+                os.environ.pop(f"{prefix}KUBERNETES_EXECUTOR__{key}", None)
+        yield
+
+
+class TestAwsEksExecutor:

Review Comment:
   Similar to the above comment, `KubernetesExecutor` and `AwsEksExecutor` use the same cluster for all teams as of right now. I will deal with that limitation as a follow-on, described by the issue https://github.com/apache/airflow/issues/74844 and then I will add these tests.
   
   Resolving this comment for now as it your first comment captures the same sentiment, so let's direct reviewer attention to that comment to agree on an approach.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["o-nikolas (via GitHub)" <gi...@apache.org>]

o-nikolas commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4317766281


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py:
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kubernetes client factories for the AwsEksExecutor."""
+
+# Internal to AwsEksExecutor, which writes these functions' import paths into
+# [kubernetes_executor] client_factory and async_client_factory.
+
+from __future__ import annotations
+
+import os
+import tempfile
+import time
+from base64 import b64decode
+from typing import TYPE_CHECKING
+
+import kubernetes
+from packaging.version import Version
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.aws.hooks.eks import EksHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.providers.amazon.aws.utils.eks_get_token import (
+    TOKEN_EXPIRATION_MINUTES,
+    fetch_access_token_for_cluster,
+)
+from airflow.providers.common.compat.sdk import conf
+
+if TYPE_CHECKING:
+    from kubernetes import client
+    from kubernetes_asyncio import client as async_client
+
+# UPDATING still serves the Kubernetes API, so every other status is refused.
+_USABLE_CLUSTER_STATUSES = ("ACTIVE", "UPDATING")
+
+
+def _get_sync_token_key(kubernetes_version: str) -> str:
+    # kubernetes 36 moved the bearer token from the "authorization" api_key to "BearerToken".
+    return "BearerToken" if Version(kubernetes_version).major >= 36 else "authorization"
+
+
+_SYNC_TOKEN_KEY = _get_sync_token_key(kubernetes.__version__)
+
+
+def _get_eks_kube_client() -> client.CoreV1Api:
+    """Build a Kubernetes client for the configured EKS cluster, in memory and without a kubeconfig."""
+    from kubernetes import client
+
+    configuration = client.Configuration()
+    _configure_eks_auth(configuration, _SYNC_TOKEN_KEY)
+    return client.CoreV1Api(client.ApiClient(configuration=configuration))
+
+
+def _get_eks_async_kube_client() -> async_client.CoreV1Api:
+    """Build the asynchronous Kubernetes client used when ``async_pod_creation`` is enabled."""
+    from kubernetes_asyncio import client as async_client
+
+    configuration = async_client.Configuration()
+    # kubernetes_asyncio has always used "BearerToken".
+    _configure_eks_auth(configuration, "BearerToken")
+    return async_client.CoreV1Api(async_client.ApiClient(configuration=configuration))
+
+
+def _configure_eks_auth(
+    configuration: client.Configuration | async_client.Configuration, token_key: str
+) -> None:
+    cluster_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None)
+    if not cluster_name:
+        raise ValueError(
+            f"[{CONFIG_GROUP_NAME}] cluster_name is required to build an EKS client. "
+            "Set it in airflow.cfg or with AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME."
+        )
+    region_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.REGION_NAME, fallback=None)
+    conn_id = conf.get(
+        CONFIG_GROUP_NAME,
+        AllEksConfigKeys.AWS_CONN_ID,
+        fallback=CONFIG_DEFAULTS[AllEksConfigKeys.AWS_CONN_ID],
+    )
+
+    eks_hook = EksHook(aws_conn_id=conn_id, region_name=region_name)
+    cluster = eks_hook.conn.describe_cluster(name=cluster_name)["cluster"]
+    if cluster["status"] not in _USABLE_CLUSTER_STATUSES:
+        raise ValueError(
+            f"EKS cluster {cluster_name} is {cluster['status']}; the executor needs it to be ACTIVE or "
+            f"UPDATING. Wait for it to become ACTIVE, or point [{CONFIG_GROUP_NAME}] cluster_name at a "
+            "usable cluster."
+        )
+    session = eks_hook.get_session()
+
+    # EKS only accepts tokens presigned against the regional STS endpoint.
+    os.environ["AWS_STS_REGIONAL_ENDPOINTS"] = "regional"
+    try:
+        sts_endpoint = StsHook(
+            aws_conn_id=conn_id, region_name=session.region_name
+        ).conn_client_meta.endpoint_url
+    finally:
+        del os.environ["AWS_STS_REGIONAL_ENDPOINTS"]
+    sts_url = f"{sts_endpoint}/?Action=GetCallerIdentity&Version=2011-06-15"
+
+    configuration.host = cluster["endpoint"]
+    configuration.ssl_ca_cert = _write_cluster_ca_file(cluster["certificateAuthority"]["data"])
+    configuration.api_key_prefix[token_key] = "Bearer"
+
+    # EKS tokens expire after about 15 minutes, so get a new one shortly before the current one expires.
+    token: str | None = None
+    refresh_at = 0.0

Review Comment:
   Okay, maybe just worth a comment or doc string update explaining this!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4320141016


##########
providers/amazon/src/airflow/providers/amazon/get_provider_info.py:
##########
@@ -1567,6 +1567,39 @@ def get_provider_info():
                     },
                 },
             },
+            "aws_eks_executor": {
+                "description": "This section only applies if you are using the AwsEksExecutor in\nAirflow's ``[core]`` configuration. It requires the ``cncf.kubernetes``\nextra of this provider. The executor supplies the Kubernetes client for\nthe configured Amazon EKS cluster; all pod-level settings are still read\nfrom the ``[kubernetes_executor]`` section.\nFor boto3 credential management, see\nhttps://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html\n",
+                "options": {
+                    "conn_id": {
+                        "description": "The Airflow connection (i.e. credentials) used by the EKS executor to\nmake API calls to Amazon EKS.\n",
+                        "version_added": "9.38.0",

Review Comment:
   Done, bumped to 9.39.0



##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py:
##########
@@ -0,0 +1,226 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""AwsEksExecutor: run Airflow tasks as pods on an Amazon EKS cluster."""
+
+from __future__ import annotations
+
+import os
+from http import HTTPStatus
+from typing import TYPE_CHECKING
+
+from airflow.executors.executor_loader import ExecutorLoader
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.version_compat import AIRFLOW_V_3_2_PLUS
+from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException, conf
+
+if TYPE_CHECKING:
+    from kubernetes.client import models as k8s
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor_types import KubernetesJob
+
+_CNCF_KUBERNETES_REQUIRED = "AwsEksExecutor requires apache-airflow-providers-cncf-kubernetes>=10.24.0"
+
+try:
+    from kubernetes.client.rest import ApiException
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+    from airflow.providers.cncf.kubernetes.get_provider_info import (
+        get_provider_info as get_cncf_kubernetes_provider_info,
+    )
+except ImportError as e:
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED) from e
+
+# Check for the option, not the version, since unreleased trees report the previous version.
+if "client_factory" not in (
+    get_cncf_kubernetes_provider_info().get("config", {}).get("kubernetes_executor", {}).get("options", {})
+):
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED)
+
+_FACTORY_MODULE = "airflow.providers.amazon.aws.executors.eks._client_factory"
+_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_kube_client"
+_ASYNC_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_async_kube_client"
+# Airflow 2 hybrid executors that wrap a KubernetesExecutor.
+_HYBRID_KUBERNETES_EXECUTORS = ("CeleryKubernetesExecutor", "LocalKubernetesExecutor")
+
+
+class AwsEksExecutor(KubernetesExecutor):
+    """
+    A KubernetesExecutor that authenticates against an Amazon EKS cluster.
+
+    Builds the Kubernetes client from ``[aws_eks_executor]`` configuration and keeps the
+    short-lived EKS token fresh in-process. All pod-level behaviour comes unchanged from the
+    KubernetesExecutor and its ``[kubernetes_executor]`` configuration.
+    """
+
+    supports_multi_team: bool = True
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        # These need team_name, which super().__init__ sets.
+        self._check_no_team_eks_config()
+        self._validate_eks_config()
+        self._check_no_kubernetes_executor_in_scope()
+        self._ensure_client_factory()
+
+    def start(self) -> None:
+        """Call this when the Executor is run for the first time by the scheduler."""
+        super().start()
+        if TYPE_CHECKING:
+            assert self.kube_scheduler
+        # Both pod creation paths build every worker pod here, so check the image before it is sent.
+        build_pod_request = self.kube_scheduler._build_pod_request
+
+        def _build_pod_request_with_image(job: KubernetesJob) -> k8s.V1Pod:
+            pod = build_pod_request(job)
+            self._check_worker_image(pod)
+            return pod
+
+        self.kube_scheduler._build_pod_request = _build_pod_request_with_image  # type: ignore[method-assign,assignment]
+
+        check_health = conf.getboolean(
+            CONFIG_GROUP_NAME,
+            AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP,
+            fallback=CONFIG_DEFAULTS[AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP],
+        )
+
+        if not check_health:
+            return
+
+        self.log.info("Starting EKS Executor and determining health...")
+        try:
+            self.check_health()
+        except RuntimeError:
+            self.log.error("Stopping the Airflow Scheduler from starting until the issue is resolved.")
+            raise
+
+    def check_health(self) -> None:
+        """
+        Make a test Kubernetes API call to check the health of the EKS Executor.
+
+        Building the client only proves the cluster exists; without this, a missing EKS access
+        entry or RBAC binding only shows up later as a watcher error loop.
+        """
+        if TYPE_CHECKING:
+            assert self.kube_client
+        namespace = self.kube_config.kube_namespace
+        try:
+            self.kube_client.list_namespaced_pod(namespace, limit=1)
+        except ApiException as e:
+            raise RuntimeError(
+                f"EKS Executor health check has failed because: cannot list pods in namespace {namespace} "
+                f"({e.status} {e.reason}). The IAM principal behind the AWS connection needs an EKS access "
+                "entry (or aws-auth mapping) on the cluster that allows create, get, list, watch, patch and "
+                f"delete on pods and get on pods/log in namespace {namespace}. See 'Grant access to the "
+                "cluster' in the AwsEksExecutor docs."
+            ) from e
+        self.log.info("EKS Executor health check has succeeded.")
+
+    def _check_worker_image(self, pod: k8s.V1Pod) -> None:
+        if pod.spec.containers and pod.spec.containers[0].image:
+            return
+        section = (
+            f"team {self.team_name}'s [kubernetes_executor]" if self.team_name else "[kubernetes_executor]"
+        )
+        annotations = pod.metadata.annotations or {}
+        # Raised as the 422 the API server would return, so KubernetesExecutor fails the task.
+        raise ApiException(
+            status=HTTPStatus.UNPROCESSABLE_ENTITY,
+            reason=(
+                f"Task {annotations.get('dag_id')}.{annotations.get('task_id')} has no worker image, so its "
+                "pod was not created. Set an image in one of three ways: on the base container of the pod "
+                f"template ({section} pod_template_file, or the task's executor_config pod_template_file), "
+                f"in the task's executor_config pod_override, or with {section} "
+                "worker_container_repository and worker_container_tag."
+            ),
+        )
+
+    @staticmethod
+    def _validate_eks_config() -> None:
+        if not conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None):
+            raise ValueError(
+                f"AwsEksExecutor requires [{CONFIG_GROUP_NAME}] cluster_name to be set. "
+                "Set it in airflow.cfg or with AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME."
+            )
+
+    def _check_no_team_eks_config(self) -> None:
+        # [aws_eks_executor] is only read globally for now, so refuse team values rather than ignore them.
+        if not self.team_name:
+            return
+        # self.conf looks options up for this team only, falling back to the built-in defaults.
+        team_keys = sorted(
+            key
+            for key in AllEksConfigKeys()
+            if self.conf.get(CONFIG_GROUP_NAME, key, fallback=None) not in (None, CONFIG_DEFAULTS.get(key))
+        )
+        if team_keys:
+            raise ValueError(
+                f"AwsEksExecutor doesn't support team-specific [{CONFIG_GROUP_NAME}] config yet, but team "
+                f"{self.team_name} sets {', '.join(team_keys)}. Set these in the global "
+                f"[{CONFIG_GROUP_NAME}] section instead. Per-team clusters are tracked at "
+                "https://github.com/apache/airflow/issues/74844"
+            )
+
+    def _check_no_kubernetes_executor_in_scope(self) -> None:
+        executor_names = (
+            ExecutorLoader.get_executor_names(validate_teams=False)
+            if AIRFLOW_V_3_2_PLUS
+            else ExecutorLoader.get_executor_names()
+        )
+        for executor_name in executor_names:
+            # ExecutorName has no team_name before Airflow 3.1.
+            if getattr(executor_name, "team_name", None) != self.team_name:
+                continue
+            executor_cls, _ = ExecutorLoader.import_executor_cls(executor_name)
+            if issubclass(executor_cls, AwsEksExecutor) or not (
+                issubclass(executor_cls, KubernetesExecutor)
+                or executor_cls.__name__ in _HYBRID_KUBERNETES_EXECUTORS
+            ):
+                continue
+            scope = f"team {self.team_name}" if self.team_name else "the global scope"
+            raise ValueError(

Review Comment:
   Agreed, currently writing and testing the fix for this, which is spelled out in https://github.com/apache/airflow/issues/74844. We will have per-team kubernetes cluster support, in my estimation, by the next provider release (not the one tomorrow, but the next one after in 2 weeks)!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4225649968


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py:
##########
@@ -0,0 +1,130 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kubernetes client factories for the AwsEksExecutor."""
+
+# Internal to AwsEksExecutor, not a public API. The executor writes the import paths of
+# _get_eks_kube_client and _get_eks_async_kube_client into [kubernetes_executor] client_factory
+# and async_client_factory, and cncf.kubernetes re-resolves them by that path in every process
+# that builds a client, including the spawned pod watcher. Keep eks_executor's paths in step.
+
+from __future__ import annotations
+
+import os
+import tempfile
+from base64 import b64decode
+from typing import TYPE_CHECKING
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.aws.hooks.eks import EksHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.providers.amazon.aws.utils.eks_get_token import fetch_access_token_for_cluster
+from airflow.providers.common.compat.sdk import conf
+
+if TYPE_CHECKING:
+    from kubernetes import client
+    from kubernetes_asyncio import client as async_client
+
+# UPDATING still serves the Kubernetes API, so only creating, deleting and failed clusters are refused.
+_USABLE_CLUSTER_STATUSES = ("ACTIVE", "UPDATING")
+
+
+def _get_eks_kube_client() -> client.CoreV1Api:
+    """Build a Kubernetes client for the configured EKS cluster, in memory and without a kubeconfig."""
+    from kubernetes import client
+
+    configuration = client.Configuration()
+    _configure_eks_auth(configuration)
+    return client.CoreV1Api(client.ApiClient(configuration=configuration))
+
+
+def _get_eks_async_kube_client() -> async_client.CoreV1Api:
+    """Build the asynchronous Kubernetes client used when ``async_pod_creation`` is enabled."""
+    from kubernetes_asyncio import client as async_client
+
+    configuration = async_client.Configuration()
+    _configure_eks_auth(configuration)
+    return async_client.CoreV1Api(async_client.ApiClient(configuration=configuration))
+
+
+def _configure_eks_auth(configuration: client.Configuration | async_client.Configuration) -> None:
+    cluster_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None)
+    if not cluster_name:
+        raise ValueError(f"[{CONFIG_GROUP_NAME}] cluster_name is required to build an EKS client")
+    region_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.REGION_NAME, fallback=None)
+    conn_id = conf.get(
+        CONFIG_GROUP_NAME,
+        AllEksConfigKeys.AWS_CONN_ID,
+        fallback=CONFIG_DEFAULTS[AllEksConfigKeys.AWS_CONN_ID],
+    )
+
+    eks_hook = EksHook(aws_conn_id=conn_id, region_name=region_name)
+    cluster = eks_hook.conn.describe_cluster(name=cluster_name)["cluster"]
+    if cluster["status"] not in _USABLE_CLUSTER_STATUSES:
+        raise ValueError(
+            f"EKS cluster {cluster_name} is {cluster['status']}; the executor needs it to be ACTIVE"
+        )
+    session = eks_hook.get_session()
+
+    # EKS only accepts tokens presigned against the regional STS endpoint; some regions
+    # otherwise default to the global one. Same dance as EksHook.generate_config_file.
+    os.environ["AWS_STS_REGIONAL_ENDPOINTS"] = "regional"
+    try:
+        sts_endpoint = StsHook(
+            aws_conn_id=conn_id, region_name=session.region_name
+        ).conn_client_meta.endpoint_url
+    finally:
+        del os.environ["AWS_STS_REGIONAL_ENDPOINTS"]
+    sts_url = f"{sts_endpoint}/?Action=GetCallerIdentity&Version=2011-06-15"
+
+    configuration.host = cluster["endpoint"]
+    configuration.ssl_ca_cert = _write_cluster_ca_file(cluster["certificateAuthority"]["data"])
+    # Key the bearer auth under both identifiers. kubernetes-client >= 36 looks the token up under
+    # "BearerToken" and only aliases the api_key (not api_key_prefix) back to the legacy
+    # "authorization" slot (see kubernetes-client/python#2595); older clients use "authorization".
+    # Setting the prefix only under "authorization" makes the newer client emit the raw token with
+    # no "Bearer " prefix, which the API server rejects with 401, so set both slots.
+    for identifier in ("BearerToken", "authorization"):
+        configuration.api_key_prefix[identifier] = "Bearer"
+
+    # The EKS token is a presigned STS URL valid for ~15 minutes, but the scheduler holds
+    # one client for its whole lifetime. refresh_api_key_hook runs on every authenticated
+    # request (Configuration.get_api_key_with_prefix), and minting is a local SigV4 signing
+    # with no network round-trip, so re-minting per request keeps the token fresh at
+    # negligible cost. Botocore refreshes the session's own credentials when they rotate.
+    def refresh_api_key(config: client.Configuration | async_client.Configuration) -> None:
+        token = fetch_access_token_for_cluster(
+            cluster_name, sts_url, region_name=session.region_name, session=session
+        )

Review Comment:
   No need for a sidecar, this is running inside the scheduler and just runs via the Kubernetes client's refresh hook. This happens right before each request, so there's no concern about it going stale.
   
   https://github.com/apache/airflow/blob/a947f0014996f3b99ec2cf10083745c9f01e69f8/providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py#L112-L120
   
   The Kubernetes executor is doing the same:
   
   https://github.com/kubernetes-client/python/blob/v35.0.0/kubernetes/base/config/incluster_config.py#L95-L100
   
   Reworded the comment to be clearer about that.



##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py:
##########
@@ -0,0 +1,130 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kubernetes client factories for the AwsEksExecutor."""
+
+# Internal to AwsEksExecutor, not a public API. The executor writes the import paths of
+# _get_eks_kube_client and _get_eks_async_kube_client into [kubernetes_executor] client_factory
+# and async_client_factory, and cncf.kubernetes re-resolves them by that path in every process
+# that builds a client, including the spawned pod watcher. Keep eks_executor's paths in step.
+
+from __future__ import annotations
+
+import os
+import tempfile
+from base64 import b64decode
+from typing import TYPE_CHECKING
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.aws.hooks.eks import EksHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.providers.amazon.aws.utils.eks_get_token import fetch_access_token_for_cluster
+from airflow.providers.common.compat.sdk import conf
+
+if TYPE_CHECKING:
+    from kubernetes import client
+    from kubernetes_asyncio import client as async_client
+
+# UPDATING still serves the Kubernetes API, so only creating, deleting and failed clusters are refused.
+_USABLE_CLUSTER_STATUSES = ("ACTIVE", "UPDATING")
+
+
+def _get_eks_kube_client() -> client.CoreV1Api:
+    """Build a Kubernetes client for the configured EKS cluster, in memory and without a kubeconfig."""
+    from kubernetes import client
+
+    configuration = client.Configuration()
+    _configure_eks_auth(configuration)
+    return client.CoreV1Api(client.ApiClient(configuration=configuration))
+
+
+def _get_eks_async_kube_client() -> async_client.CoreV1Api:
+    """Build the asynchronous Kubernetes client used when ``async_pod_creation`` is enabled."""
+    from kubernetes_asyncio import client as async_client
+
+    configuration = async_client.Configuration()
+    _configure_eks_auth(configuration)
+    return async_client.CoreV1Api(async_client.ApiClient(configuration=configuration))
+
+
+def _configure_eks_auth(configuration: client.Configuration | async_client.Configuration) -> None:
+    cluster_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None)
+    if not cluster_name:
+        raise ValueError(f"[{CONFIG_GROUP_NAME}] cluster_name is required to build an EKS client")
+    region_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.REGION_NAME, fallback=None)
+    conn_id = conf.get(
+        CONFIG_GROUP_NAME,
+        AllEksConfigKeys.AWS_CONN_ID,
+        fallback=CONFIG_DEFAULTS[AllEksConfigKeys.AWS_CONN_ID],
+    )
+
+    eks_hook = EksHook(aws_conn_id=conn_id, region_name=region_name)
+    cluster = eks_hook.conn.describe_cluster(name=cluster_name)["cluster"]
+    if cluster["status"] not in _USABLE_CLUSTER_STATUSES:
+        raise ValueError(
+            f"EKS cluster {cluster_name} is {cluster['status']}; the executor needs it to be ACTIVE"
+        )

Review Comment:
   Good catch



##########
providers/amazon/docs/executors/eks-executor.rst:
##########


Review Comment:
   Good idea, added a "Choosing an executor" comparison table.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4215297882


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py:
##########
@@ -0,0 +1,145 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""AwsEksExecutor: run Airflow tasks as pods on an Amazon EKS cluster."""
+
+from __future__ import annotations
+
+import os
+from typing import TYPE_CHECKING
+
+from airflow.providers.common.compat.sdk import conf
+
+try:
+    from kubernetes.client.rest import ApiException
+
+    from airflow.providers.cncf.kubernetes import __version__ as cncf_kubernetes_version
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+    from airflow.providers.cncf.kubernetes.get_provider_info import (
+        get_provider_info as get_cncf_kubernetes_provider_info,
+    )
+except ImportError as e:
+    raise ImportError(
+        "AwsEksExecutor requires the cncf.kubernetes provider; install it with "
+        "pip install 'apache-airflow-providers-amazon[cncf.kubernetes]'"
+    ) from e
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+
+# Import paths, because cncf.kubernetes re-resolves the factories in each process (see _client_factory).
+_FACTORY_MODULE = "airflow.providers.amazon.aws.executors.eks._client_factory"
+_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_kube_client"
+_ASYNC_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_async_kube_client"
+# Only used in the error message below. The check looks for the client_factory option instead of
+# comparing versions, because an unreleased source tree still reports the previous release.
+MIN_CNCF_KUBERNETES_VERSION = "10.24.0"
+
+
+class AwsEksExecutor(KubernetesExecutor):

Review Comment:
   Yes, I went with `AwsEksExecutor` to stay consistent with the `Aws` prefix convention across the other executors:
   
   https://github.com/apache/airflow/blob/9c9346c8065ec5e4f1bbd5f8d17b931d1d88981a/providers/amazon/src/airflow/providers/amazon/aws/executors/ecs/ecs_executor.py#L88
   
   https://github.com/apache/airflow/blob/9c9346c8065ec5e4f1bbd5f8d17b931d1d88981a/providers/amazon/src/airflow/providers/amazon/aws/executors/batch/batch_executor.py#L82
   
   https://github.com/apache/airflow/blob/9c9346c8065ec5e4f1bbd5f8d17b931d1d88981a/providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/lambda_executor.py#L76
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["ferruzzi (via GitHub)" <gi...@apache.org>]

ferruzzi commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4288802873


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py:
##########
@@ -0,0 +1,137 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""AwsEksExecutor: run Airflow tasks as pods on an Amazon EKS cluster."""
+
+from __future__ import annotations
+
+import os
+from typing import TYPE_CHECKING
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException, conf
+
+_CNCF_KUBERNETES_REQUIRED = "AwsEksExecutor requires apache-airflow-providers-cncf-kubernetes>=10.24.0"
+
+try:
+    from kubernetes.client.rest import ApiException
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+    from airflow.providers.cncf.kubernetes.get_provider_info import (
+        get_provider_info as get_cncf_kubernetes_provider_info,
+    )
+except ImportError as e:
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED) from e
+
+# Check for the option, not the version, since unreleased trees report the previous version.
+if "client_factory" not in (
+    get_cncf_kubernetes_provider_info().get("config", {}).get("kubernetes_executor", {}).get("options", {})
+):
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED)
+
+_FACTORY_MODULE = "airflow.providers.amazon.aws.executors.eks._client_factory"
+_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_kube_client"
+_ASYNC_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_async_kube_client"
+
+
+class AwsEksExecutor(KubernetesExecutor):
+    """
+    A KubernetesExecutor that authenticates against an Amazon EKS cluster.
+
+    Builds the Kubernetes client from ``[aws_eks_executor]`` configuration and keeps the
+    short-lived EKS token fresh in-process. All pod-level behaviour comes unchanged from the
+    KubernetesExecutor and its ``[kubernetes_executor]`` configuration.
+    """
+
+    supports_multi_team: bool = True
+
+    def __init__(self, *args, **kwargs):
+        self._validate_eks_config()
+        super().__init__(*args, **kwargs)
+        # Needs team_name, which super().__init__ sets.
+        self._ensure_client_factory()
+
+    def start(self) -> None:
+        """Call this when the Executor is run for the first time by the scheduler."""
+        super().start()
+        check_health = conf.getboolean(
+            CONFIG_GROUP_NAME,
+            AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP,
+            fallback=CONFIG_DEFAULTS[AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP],
+        )
+
+        if not check_health:
+            return
+
+        self.log.info("Starting EKS Executor and determining health...")
+        try:
+            self.check_health()
+        except RuntimeError:
+            self.log.error("Stopping the Airflow Scheduler from starting until the issue is resolved.")
+            raise
+
+    def check_health(self) -> None:
+        """
+        Make a test Kubernetes API call to check the health of the EKS Executor.
+
+        Building the client only proves the cluster exists; without this, a missing EKS access
+        entry or RBAC binding only shows up later as a watcher error loop.
+        """
+        if TYPE_CHECKING:
+            assert self.kube_client
+        namespace = self.kube_config.kube_namespace
+        try:
+            self.kube_client.list_namespaced_pod(namespace, limit=1)
+        except ApiException as e:
+            raise RuntimeError(
+                f"EKS Executor health check has failed because: cannot list pods in namespace {namespace} "
+                f"({e.status} {e.reason}). The IAM principal behind the AWS connection needs an EKS access "
+                "entry (or aws-auth mapping) on the cluster that allows create, get, list, watch, patch and "
+                f"delete on pods and get on pods/log in namespace {namespace}. See 'Grant access to the "
+                "cluster' in the AwsEksExecutor docs."
+            ) from e
+        self.log.info("EKS Executor health check has succeeded.")
+
+    @staticmethod
+    def _validate_eks_config() -> None:
+        if not conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None):

Review Comment:
   This is maybe a @vincbeck multi-team config question... 
   
   ECS, Batch, and Lambda executors all handle `conf.get` calls through `self.conf` as part of the multi-team support.  Your docs say this new executor supports multi-team but you are not following that same pattern here and elsewhere.  Is that a miss, or is there a reason EKS doesn't need to match the same pattern?



##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py:
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kubernetes client factories for the AwsEksExecutor."""
+
+# Internal to AwsEksExecutor, which writes these functions' import paths into
+# [kubernetes_executor] client_factory and async_client_factory.
+
+from __future__ import annotations
+
+import os
+import tempfile
+import time
+from base64 import b64decode
+from typing import TYPE_CHECKING
+
+import kubernetes
+from packaging.version import Version
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.aws.hooks.eks import EksHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.providers.amazon.aws.utils.eks_get_token import (
+    TOKEN_EXPIRATION_MINUTES,
+    fetch_access_token_for_cluster,
+)
+from airflow.providers.common.compat.sdk import conf
+
+if TYPE_CHECKING:
+    from kubernetes import client
+    from kubernetes_asyncio import client as async_client
+
+# UPDATING still serves the Kubernetes API, so every other status is refused.
+_USABLE_CLUSTER_STATUSES = ("ACTIVE", "UPDATING")
+
+
+def _get_sync_token_key(kubernetes_version: str) -> str:
+    # kubernetes 36 moved the bearer token from the "authorization" api_key to "BearerToken".
+    return "BearerToken" if Version(kubernetes_version).major >= 36 else "authorization"
+
+
+_SYNC_TOKEN_KEY = _get_sync_token_key(kubernetes.__version__)
+
+
+def _get_eks_kube_client() -> client.CoreV1Api:
+    """Build a Kubernetes client for the configured EKS cluster, in memory and without a kubeconfig."""
+    from kubernetes import client
+
+    configuration = client.Configuration()
+    _configure_eks_auth(configuration, _SYNC_TOKEN_KEY)
+    return client.CoreV1Api(client.ApiClient(configuration=configuration))
+
+
+def _get_eks_async_kube_client() -> async_client.CoreV1Api:
+    """Build the asynchronous Kubernetes client used when ``async_pod_creation`` is enabled."""
+    from kubernetes_asyncio import client as async_client
+
+    configuration = async_client.Configuration()
+    # kubernetes_asyncio has always used "BearerToken".
+    _configure_eks_auth(configuration, "BearerToken")
+    return async_client.CoreV1Api(async_client.ApiClient(configuration=configuration))
+
+
+def _configure_eks_auth(
+    configuration: client.Configuration | async_client.Configuration, token_key: str
+) -> None:
+    cluster_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None)
+    if not cluster_name:
+        raise ValueError(
+            f"[{CONFIG_GROUP_NAME}] cluster_name is required to build an EKS client. "
+            "Set it in airflow.cfg or with AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME."
+        )
+    region_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.REGION_NAME, fallback=None)
+    conn_id = conf.get(
+        CONFIG_GROUP_NAME,
+        AllEksConfigKeys.AWS_CONN_ID,
+        fallback=CONFIG_DEFAULTS[AllEksConfigKeys.AWS_CONN_ID],
+    )
+
+    eks_hook = EksHook(aws_conn_id=conn_id, region_name=region_name)
+    cluster = eks_hook.conn.describe_cluster(name=cluster_name)["cluster"]
+    if cluster["status"] not in _USABLE_CLUSTER_STATUSES:
+        raise ValueError(
+            f"EKS cluster {cluster_name} is {cluster['status']}; the executor needs it to be ACTIVE or "
+            f"UPDATING. Wait for it to become ACTIVE, or point [{CONFIG_GROUP_NAME}] cluster_name at a "
+            "usable cluster."
+        )
+    session = eks_hook.get_session()
+
+    # EKS only accepts tokens presigned against the regional STS endpoint.
+    os.environ["AWS_STS_REGIONAL_ENDPOINTS"] = "regional"
+    try:
+        sts_endpoint = StsHook(
+            aws_conn_id=conn_id, region_name=session.region_name
+        ).conn_client_meta.endpoint_url
+    finally:
+        del os.environ["AWS_STS_REGIONAL_ENDPOINTS"]

Review Comment:
   I'm not sure if anything else uses "AWS_STS_REGIONAL_ENDPOINTS".  Is it possible that this had a value before L108 that we should restore here instead of deleting?  



##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py:
##########
@@ -0,0 +1,137 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""AwsEksExecutor: run Airflow tasks as pods on an Amazon EKS cluster."""
+
+from __future__ import annotations
+
+import os
+from typing import TYPE_CHECKING
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException, conf
+
+_CNCF_KUBERNETES_REQUIRED = "AwsEksExecutor requires apache-airflow-providers-cncf-kubernetes>=10.24.0"
+
+try:
+    from kubernetes.client.rest import ApiException
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+    from airflow.providers.cncf.kubernetes.get_provider_info import (
+        get_provider_info as get_cncf_kubernetes_provider_info,
+    )
+except ImportError as e:
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED) from e
+
+# Check for the option, not the version, since unreleased trees report the previous version.
+if "client_factory" not in (
+    get_cncf_kubernetes_provider_info().get("config", {}).get("kubernetes_executor", {}).get("options", {})
+):
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED)
+
+_FACTORY_MODULE = "airflow.providers.amazon.aws.executors.eks._client_factory"
+_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_kube_client"
+_ASYNC_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_async_kube_client"
+
+
+class AwsEksExecutor(KubernetesExecutor):
+    """
+    A KubernetesExecutor that authenticates against an Amazon EKS cluster.
+
+    Builds the Kubernetes client from ``[aws_eks_executor]`` configuration and keeps the
+    short-lived EKS token fresh in-process. All pod-level behaviour comes unchanged from the
+    KubernetesExecutor and its ``[kubernetes_executor]`` configuration.
+    """
+
+    supports_multi_team: bool = True
+
+    def __init__(self, *args, **kwargs):
+        self._validate_eks_config()
+        super().__init__(*args, **kwargs)
+        # Needs team_name, which super().__init__ sets.
+        self._ensure_client_factory()
+
+    def start(self) -> None:
+        """Call this when the Executor is run for the first time by the scheduler."""
+        super().start()
+        check_health = conf.getboolean(
+            CONFIG_GROUP_NAME,
+            AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP,
+            fallback=CONFIG_DEFAULTS[AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP],
+        )
+
+        if not check_health:
+            return
+
+        self.log.info("Starting EKS Executor and determining health...")
+        try:
+            self.check_health()
+        except RuntimeError:
+            self.log.error("Stopping the Airflow Scheduler from starting until the issue is resolved.")
+            raise
+
+    def check_health(self) -> None:
+        """
+        Make a test Kubernetes API call to check the health of the EKS Executor.
+
+        Building the client only proves the cluster exists; without this, a missing EKS access
+        entry or RBAC binding only shows up later as a watcher error loop.
+        """
+        if TYPE_CHECKING:
+            assert self.kube_client
+        namespace = self.kube_config.kube_namespace
+        try:
+            self.kube_client.list_namespaced_pod(namespace, limit=1)
+        except ApiException as e:
+            raise RuntimeError(
+                f"EKS Executor health check has failed because: cannot list pods in namespace {namespace} "
+                f"({e.status} {e.reason}). The IAM principal behind the AWS connection needs an EKS access "
+                "entry (or aws-auth mapping) on the cluster that allows create, get, list, watch, patch and "
+                f"delete on pods and get on pods/log in namespace {namespace}. See 'Grant access to the "
+                "cluster' in the AwsEksExecutor docs."
+            ) from e
+        self.log.info("EKS Executor health check has succeeded.")
+
+    @staticmethod
+    def _validate_eks_config() -> None:
+        if not conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None):
+            raise ValueError(
+                f"AwsEksExecutor requires [{CONFIG_GROUP_NAME}] cluster_name to be set. "
+                "Set it in airflow.cfg or with AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME."
+            )
+
+    def _ensure_client_factory(self) -> None:
+        # A team's factory is looked up in team config only, with no global fallback.
+        team_name = self.team_name
+        team_kwargs = {"team_name": team_name} if team_name else {}
+        env_var_prefix = f"AIRFLOW__{team_name.upper()}___" if team_name else "AIRFLOW__"
+        for key, path in (
+            ("client_factory", _CLIENT_FACTORY_PATH),
+            ("async_client_factory", _ASYNC_CLIENT_FACTORY_PATH),
+        ):
+            configured = conf.get("kubernetes_executor", key, fallback=None, **team_kwargs)
+            if configured and configured != path:
+                raise ValueError(
+                    f"AwsEksExecutor sets [kubernetes_executor] {key} itself, so leave it unset; "
+                    f"it is set to {configured}"
+                )
+            if not configured:
+                # An env var, not conf.set, so the spawned pod watcher process sees it too.
+                os.environ[f"{env_var_prefix}KUBERNETES_EXECUTOR__{key.upper()}"] = path

Review Comment:
   Won't this cause conflicts in a multi-executor setup with EKS and K8S executors running side by side?  It looks liken we're stealing their `KUBERNETES_EXECUTOR__` namespace and overwriting it.



##########
providers/amazon/tests/unit/amazon/aws/executors/eks/test_eks_executor.py:
##########
@@ -0,0 +1,154 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import importlib
+import os
+import sys
+from unittest import mock
+
+import pytest
+from kubernetes.client import CoreV1Api
+from kubernetes.client.rest import ApiException
+
+from airflow.providers.amazon.aws.executors.eks import AwsEksExecutor, _client_factory
+from airflow.providers.amazon.aws.executors.eks.eks_executor import (
+    _ASYNC_CLIENT_FACTORY_PATH,
+    _CLIENT_FACTORY_PATH,
+)
+from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException, conf
+
+from tests_common.test_utils.config import conf_vars
+from tests_common.test_utils.version_compat import AIRFLOW_V_3_2_PLUS
+
+EKS_EXECUTOR_MODULE = "airflow.providers.amazon.aws.executors.eks.eks_executor"
+CLIENT_FACTORY_ENV_VAR = "AIRFLOW__KUBERNETES_EXECUTOR__CLIENT_FACTORY"
+TEAMS = [
+    pytest.param(None, "AIRFLOW__", id="global"),
+    pytest.param(
+        "team_a",
+        "AIRFLOW__TEAM_A___",
+        id="team",
+        marks=pytest.mark.skipif(not AIRFLOW_V_3_2_PLUS, reason="Multi-team requires Airflow 3.2+"),
+    ),
+]
+
+
+def _team_kwargs(team_name: str | None) -> dict:
+    return {"team_name": team_name} if team_name else {}
+
+
+@pytest.fixture(autouse=True)
+def eks_config():
+    with mock.patch.dict(os.environ), conf_vars({("aws_eks_executor", "cluster_name"): "test-eks-cluster"}):
+        for prefix in ("AIRFLOW__", "AIRFLOW__TEAM_A___"):
+            for key in ("CLIENT_FACTORY", "ASYNC_CLIENT_FACTORY"):
+                os.environ.pop(f"{prefix}KUBERNETES_EXECUTOR__{key}", None)
+        yield
+
+
+class TestAwsEksExecutor:

Review Comment:
   `test_team_config` in `test_ecs_executor.py` shows both global and team-scoped values getting tested and I don't see the equivalent here, can you add them?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4291655987


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py:
##########
@@ -0,0 +1,137 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""AwsEksExecutor: run Airflow tasks as pods on an Amazon EKS cluster."""
+
+from __future__ import annotations
+
+import os
+from typing import TYPE_CHECKING
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException, conf
+
+_CNCF_KUBERNETES_REQUIRED = "AwsEksExecutor requires apache-airflow-providers-cncf-kubernetes>=10.24.0"
+
+try:
+    from kubernetes.client.rest import ApiException
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+    from airflow.providers.cncf.kubernetes.get_provider_info import (
+        get_provider_info as get_cncf_kubernetes_provider_info,
+    )
+except ImportError as e:
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED) from e
+
+# Check for the option, not the version, since unreleased trees report the previous version.
+if "client_factory" not in (
+    get_cncf_kubernetes_provider_info().get("config", {}).get("kubernetes_executor", {}).get("options", {})
+):
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED)
+
+_FACTORY_MODULE = "airflow.providers.amazon.aws.executors.eks._client_factory"
+_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_kube_client"
+_ASYNC_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_async_kube_client"
+
+
+class AwsEksExecutor(KubernetesExecutor):
+    """
+    A KubernetesExecutor that authenticates against an Amazon EKS cluster.
+
+    Builds the Kubernetes client from ``[aws_eks_executor]`` configuration and keeps the
+    short-lived EKS token fresh in-process. All pod-level behaviour comes unchanged from the
+    KubernetesExecutor and its ``[kubernetes_executor]`` configuration.
+    """
+
+    supports_multi_team: bool = True
+
+    def __init__(self, *args, **kwargs):
+        self._validate_eks_config()
+        super().__init__(*args, **kwargs)
+        # Needs team_name, which super().__init__ sets.
+        self._ensure_client_factory()
+
+    def start(self) -> None:
+        """Call this when the Executor is run for the first time by the scheduler."""
+        super().start()
+        check_health = conf.getboolean(
+            CONFIG_GROUP_NAME,
+            AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP,
+            fallback=CONFIG_DEFAULTS[AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP],
+        )
+
+        if not check_health:
+            return
+
+        self.log.info("Starting EKS Executor and determining health...")
+        try:
+            self.check_health()
+        except RuntimeError:
+            self.log.error("Stopping the Airflow Scheduler from starting until the issue is resolved.")
+            raise
+
+    def check_health(self) -> None:
+        """
+        Make a test Kubernetes API call to check the health of the EKS Executor.
+
+        Building the client only proves the cluster exists; without this, a missing EKS access
+        entry or RBAC binding only shows up later as a watcher error loop.
+        """
+        if TYPE_CHECKING:
+            assert self.kube_client
+        namespace = self.kube_config.kube_namespace
+        try:
+            self.kube_client.list_namespaced_pod(namespace, limit=1)
+        except ApiException as e:
+            raise RuntimeError(
+                f"EKS Executor health check has failed because: cannot list pods in namespace {namespace} "
+                f"({e.status} {e.reason}). The IAM principal behind the AWS connection needs an EKS access "
+                "entry (or aws-auth mapping) on the cluster that allows create, get, list, watch, patch and "
+                f"delete on pods and get on pods/log in namespace {namespace}. See 'Grant access to the "
+                "cluster' in the AwsEksExecutor docs."
+            ) from e
+        self.log.info("EKS Executor health check has succeeded.")
+
+    @staticmethod
+    def _validate_eks_config() -> None:
+        if not conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None):

Review Comment:
   It's a good point to bring up, and it's something I did consider explicitly when writing #73014. To keep the changes to `cncf` minimal, I decided to bring `AwsEksExecutor` to multi-team feature parity with `KubernetesExecutor` ([docs](https://github.com/apache/airflow/blob/313255f8db6543f8482bd1c391d97a46567da89f/providers/amazon/docs/executors/eks-executor.rst?plain=1#L203-L207)), which reads its cluster settings globally, so teams share one cluster ([kube_client.py](https://github.com/apache/airflow/blob/313255f8db6543f8482bd1c391d97a46567da89f/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/kube_client.py#L180-L181)):
   
   ~~~python
       if in_cluster is None:
           in_cluster = conf.getboolean("kubernetes_executor", "in_cluster")
   ~~~
   
   On the other hand, the AWS executors read their whole section per team, so each team can have its own cluster ([ecs_executor.py](https://github.com/apache/airflow/blob/313255f8db6543f8482bd1c391d97a46567da89f/providers/amazon/src/airflow/providers/amazon/aws/executors/ecs/ecs_executor.py#L142)):
   
   ~~~python
           self.cluster = self.conf.get(CONFIG_GROUP_NAME, AllEcsConfigKeys.CLUSTER)
   ~~~
   
   Getting EKS to that level needs cncf to pass the team name to the client factory, which is a small change I'll do as a follow-up. I'll file an issue for it and target for it to be included in the next cncf release.
   
   If you, @o-nikolas, and @vincbeck are happy with that plan, then we'll treat this as a follow-on.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["ferruzzi (via GitHub)" <gi...@apache.org>]

ferruzzi commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4297134542


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py:
##########
@@ -0,0 +1,137 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""AwsEksExecutor: run Airflow tasks as pods on an Amazon EKS cluster."""
+
+from __future__ import annotations
+
+import os
+from typing import TYPE_CHECKING
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException, conf
+
+_CNCF_KUBERNETES_REQUIRED = "AwsEksExecutor requires apache-airflow-providers-cncf-kubernetes>=10.24.0"
+
+try:
+    from kubernetes.client.rest import ApiException
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+    from airflow.providers.cncf.kubernetes.get_provider_info import (
+        get_provider_info as get_cncf_kubernetes_provider_info,
+    )
+except ImportError as e:
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED) from e
+
+# Check for the option, not the version, since unreleased trees report the previous version.
+if "client_factory" not in (
+    get_cncf_kubernetes_provider_info().get("config", {}).get("kubernetes_executor", {}).get("options", {})
+):
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED)
+
+_FACTORY_MODULE = "airflow.providers.amazon.aws.executors.eks._client_factory"
+_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_kube_client"
+_ASYNC_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_async_kube_client"
+
+
+class AwsEksExecutor(KubernetesExecutor):
+    """
+    A KubernetesExecutor that authenticates against an Amazon EKS cluster.
+
+    Builds the Kubernetes client from ``[aws_eks_executor]`` configuration and keeps the
+    short-lived EKS token fresh in-process. All pod-level behaviour comes unchanged from the
+    KubernetesExecutor and its ``[kubernetes_executor]`` configuration.
+    """
+
+    supports_multi_team: bool = True
+
+    def __init__(self, *args, **kwargs):
+        self._validate_eks_config()
+        super().__init__(*args, **kwargs)
+        # Needs team_name, which super().__init__ sets.
+        self._ensure_client_factory()
+
+    def start(self) -> None:
+        """Call this when the Executor is run for the first time by the scheduler."""
+        super().start()
+        check_health = conf.getboolean(
+            CONFIG_GROUP_NAME,
+            AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP,
+            fallback=CONFIG_DEFAULTS[AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP],
+        )
+
+        if not check_health:
+            return
+
+        self.log.info("Starting EKS Executor and determining health...")
+        try:
+            self.check_health()
+        except RuntimeError:
+            self.log.error("Stopping the Airflow Scheduler from starting until the issue is resolved.")
+            raise
+
+    def check_health(self) -> None:
+        """
+        Make a test Kubernetes API call to check the health of the EKS Executor.
+
+        Building the client only proves the cluster exists; without this, a missing EKS access
+        entry or RBAC binding only shows up later as a watcher error loop.
+        """
+        if TYPE_CHECKING:
+            assert self.kube_client
+        namespace = self.kube_config.kube_namespace
+        try:
+            self.kube_client.list_namespaced_pod(namespace, limit=1)
+        except ApiException as e:
+            raise RuntimeError(
+                f"EKS Executor health check has failed because: cannot list pods in namespace {namespace} "
+                f"({e.status} {e.reason}). The IAM principal behind the AWS connection needs an EKS access "
+                "entry (or aws-auth mapping) on the cluster that allows create, get, list, watch, patch and "
+                f"delete on pods and get on pods/log in namespace {namespace}. See 'Grant access to the "
+                "cluster' in the AwsEksExecutor docs."
+            ) from e
+        self.log.info("EKS Executor health check has succeeded.")
+
+    @staticmethod
+    def _validate_eks_config() -> None:
+        if not conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None):

Review Comment:
   Ah, I didn't realize that was a limitation in the K8s executor as well, I only recognized the pattern from the other AWS executors.  If that's the case, then I'm fine with that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["o-nikolas (via GitHub)" <gi...@apache.org>]

o-nikolas commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4317755257


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py:
##########
@@ -0,0 +1,137 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""AwsEksExecutor: run Airflow tasks as pods on an Amazon EKS cluster."""
+
+from __future__ import annotations
+
+import os
+from typing import TYPE_CHECKING
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException, conf
+
+_CNCF_KUBERNETES_REQUIRED = "AwsEksExecutor requires apache-airflow-providers-cncf-kubernetes>=10.24.0"
+
+try:
+    from kubernetes.client.rest import ApiException
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+    from airflow.providers.cncf.kubernetes.get_provider_info import (
+        get_provider_info as get_cncf_kubernetes_provider_info,
+    )
+except ImportError as e:
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED) from e
+
+# Check for the option, not the version, since unreleased trees report the previous version.
+if "client_factory" not in (
+    get_cncf_kubernetes_provider_info().get("config", {}).get("kubernetes_executor", {}).get("options", {})
+):
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED)
+
+_FACTORY_MODULE = "airflow.providers.amazon.aws.executors.eks._client_factory"
+_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_kube_client"
+_ASYNC_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_async_kube_client"
+
+
+class AwsEksExecutor(KubernetesExecutor):
+    """
+    A KubernetesExecutor that authenticates against an Amazon EKS cluster.
+
+    Builds the Kubernetes client from ``[aws_eks_executor]`` configuration and keeps the
+    short-lived EKS token fresh in-process. All pod-level behaviour comes unchanged from the
+    KubernetesExecutor and its ``[kubernetes_executor]`` configuration.
+    """
+
+    supports_multi_team: bool = True
+
+    def __init__(self, *args, **kwargs):
+        self._validate_eks_config()
+        super().__init__(*args, **kwargs)
+        # Needs team_name, which super().__init__ sets.
+        self._ensure_client_factory()
+
+    def start(self) -> None:
+        """Call this when the Executor is run for the first time by the scheduler."""
+        super().start()
+        check_health = conf.getboolean(
+            CONFIG_GROUP_NAME,
+            AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP,
+            fallback=CONFIG_DEFAULTS[AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP],
+        )
+
+        if not check_health:
+            return
+
+        self.log.info("Starting EKS Executor and determining health...")
+        try:
+            self.check_health()
+        except RuntimeError:
+            self.log.error("Stopping the Airflow Scheduler from starting until the issue is resolved.")
+            raise
+
+    def check_health(self) -> None:
+        """
+        Make a test Kubernetes API call to check the health of the EKS Executor.
+
+        Building the client only proves the cluster exists; without this, a missing EKS access
+        entry or RBAC binding only shows up later as a watcher error loop.
+        """
+        if TYPE_CHECKING:
+            assert self.kube_client
+        namespace = self.kube_config.kube_namespace
+        try:
+            self.kube_client.list_namespaced_pod(namespace, limit=1)
+        except ApiException as e:
+            raise RuntimeError(
+                f"EKS Executor health check has failed because: cannot list pods in namespace {namespace} "
+                f"({e.status} {e.reason}). The IAM principal behind the AWS connection needs an EKS access "
+                "entry (or aws-auth mapping) on the cluster that allows create, get, list, watch, patch and "
+                f"delete on pods and get on pods/log in namespace {namespace}. See 'Grant access to the "
+                "cluster' in the AwsEksExecutor docs."
+            ) from e
+        self.log.info("EKS Executor health check has succeeded.")
+
+    @staticmethod
+    def _validate_eks_config() -> None:
+        if not conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None):

Review Comment:
   I like that plan, please do follow through on that, we can start on that immediately without waiting for this PR as far as I understand it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["o-nikolas (via GitHub)" <gi...@apache.org>]

o-nikolas commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4317781754


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/_client_factory.py:
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kubernetes client factories for the AwsEksExecutor."""
+
+# Internal to AwsEksExecutor, which writes these functions' import paths into
+# [kubernetes_executor] client_factory and async_client_factory.
+
+from __future__ import annotations
+
+import os
+import tempfile
+import time
+from base64 import b64decode
+from typing import TYPE_CHECKING
+
+import kubernetes
+from packaging.version import Version
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.aws.hooks.eks import EksHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.providers.amazon.aws.utils.eks_get_token import (
+    TOKEN_EXPIRATION_MINUTES,
+    fetch_access_token_for_cluster,
+)
+from airflow.providers.common.compat.sdk import conf
+
+if TYPE_CHECKING:
+    from kubernetes import client
+    from kubernetes_asyncio import client as async_client
+
+# UPDATING still serves the Kubernetes API, so every other status is refused.
+_USABLE_CLUSTER_STATUSES = ("ACTIVE", "UPDATING")
+
+
+def _get_sync_token_key(kubernetes_version: str) -> str:
+    # kubernetes 36 moved the bearer token from the "authorization" api_key to "BearerToken".
+    return "BearerToken" if Version(kubernetes_version).major >= 36 else "authorization"
+
+
+_SYNC_TOKEN_KEY = _get_sync_token_key(kubernetes.__version__)
+
+
+def _get_eks_kube_client() -> client.CoreV1Api:
+    """Build a Kubernetes client for the configured EKS cluster, in memory and without a kubeconfig."""
+    from kubernetes import client
+
+    configuration = client.Configuration()
+    _configure_eks_auth(configuration, _SYNC_TOKEN_KEY)
+    return client.CoreV1Api(client.ApiClient(configuration=configuration))
+
+
+def _get_eks_async_kube_client() -> async_client.CoreV1Api:
+    """Build the asynchronous Kubernetes client used when ``async_pod_creation`` is enabled."""
+    from kubernetes_asyncio import client as async_client
+
+    configuration = async_client.Configuration()
+    # kubernetes_asyncio has always used "BearerToken".
+    _configure_eks_auth(configuration, "BearerToken")
+    return async_client.CoreV1Api(async_client.ApiClient(configuration=configuration))
+
+
+def _configure_eks_auth(
+    configuration: client.Configuration | async_client.Configuration, token_key: str
+) -> None:
+    cluster_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None)
+    if not cluster_name:
+        raise ValueError(
+            f"[{CONFIG_GROUP_NAME}] cluster_name is required to build an EKS client. "
+            "Set it in airflow.cfg or with AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME."
+        )
+    region_name = conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.REGION_NAME, fallback=None)
+    conn_id = conf.get(
+        CONFIG_GROUP_NAME,
+        AllEksConfigKeys.AWS_CONN_ID,
+        fallback=CONFIG_DEFAULTS[AllEksConfigKeys.AWS_CONN_ID],
+    )
+
+    eks_hook = EksHook(aws_conn_id=conn_id, region_name=region_name)
+    cluster = eks_hook.conn.describe_cluster(name=cluster_name)["cluster"]
+    if cluster["status"] not in _USABLE_CLUSTER_STATUSES:
+        raise ValueError(
+            f"EKS cluster {cluster_name} is {cluster['status']}; the executor needs it to be ACTIVE or "
+            f"UPDATING. Wait for it to become ACTIVE, or point [{CONFIG_GROUP_NAME}] cluster_name at a "
+            "usable cluster."
+        )
+    session = eks_hook.get_session()
+
+    # EKS only accepts tokens presigned against the regional STS endpoint.
+    os.environ["AWS_STS_REGIONAL_ENDPOINTS"] = "regional"
+    try:
+        sts_endpoint = StsHook(
+            aws_conn_id=conn_id, region_name=session.region_name
+        ).conn_client_meta.endpoint_url
+    finally:
+        del os.environ["AWS_STS_REGIONAL_ENDPOINTS"]

Review Comment:
   Okay, so does that mean this executor only works now with min 1.42? That's at least worth putting in the docs and/or validating at runtime.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4317213748


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py:
##########
@@ -0,0 +1,137 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""AwsEksExecutor: run Airflow tasks as pods on an Amazon EKS cluster."""
+
+from __future__ import annotations
+
+import os
+from typing import TYPE_CHECKING
+
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException, conf
+
+_CNCF_KUBERNETES_REQUIRED = "AwsEksExecutor requires apache-airflow-providers-cncf-kubernetes>=10.24.0"
+
+try:
+    from kubernetes.client.rest import ApiException
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+    from airflow.providers.cncf.kubernetes.get_provider_info import (
+        get_provider_info as get_cncf_kubernetes_provider_info,
+    )
+except ImportError as e:
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED) from e
+
+# Check for the option, not the version, since unreleased trees report the previous version.
+if "client_factory" not in (
+    get_cncf_kubernetes_provider_info().get("config", {}).get("kubernetes_executor", {}).get("options", {})
+):
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED)
+
+_FACTORY_MODULE = "airflow.providers.amazon.aws.executors.eks._client_factory"
+_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_kube_client"
+_ASYNC_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_async_kube_client"
+
+
+class AwsEksExecutor(KubernetesExecutor):
+    """
+    A KubernetesExecutor that authenticates against an Amazon EKS cluster.
+
+    Builds the Kubernetes client from ``[aws_eks_executor]`` configuration and keeps the
+    short-lived EKS token fresh in-process. All pod-level behaviour comes unchanged from the
+    KubernetesExecutor and its ``[kubernetes_executor]`` configuration.
+    """
+
+    supports_multi_team: bool = True
+
+    def __init__(self, *args, **kwargs):
+        self._validate_eks_config()
+        super().__init__(*args, **kwargs)
+        # Needs team_name, which super().__init__ sets.
+        self._ensure_client_factory()
+
+    def start(self) -> None:
+        """Call this when the Executor is run for the first time by the scheduler."""
+        super().start()
+        check_health = conf.getboolean(
+            CONFIG_GROUP_NAME,
+            AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP,
+            fallback=CONFIG_DEFAULTS[AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP],
+        )
+
+        if not check_health:
+            return
+
+        self.log.info("Starting EKS Executor and determining health...")
+        try:
+            self.check_health()
+        except RuntimeError:
+            self.log.error("Stopping the Airflow Scheduler from starting until the issue is resolved.")
+            raise
+
+    def check_health(self) -> None:
+        """
+        Make a test Kubernetes API call to check the health of the EKS Executor.
+
+        Building the client only proves the cluster exists; without this, a missing EKS access
+        entry or RBAC binding only shows up later as a watcher error loop.
+        """
+        if TYPE_CHECKING:
+            assert self.kube_client
+        namespace = self.kube_config.kube_namespace
+        try:
+            self.kube_client.list_namespaced_pod(namespace, limit=1)
+        except ApiException as e:
+            raise RuntimeError(
+                f"EKS Executor health check has failed because: cannot list pods in namespace {namespace} "
+                f"({e.status} {e.reason}). The IAM principal behind the AWS connection needs an EKS access "
+                "entry (or aws-auth mapping) on the cluster that allows create, get, list, watch, patch and "
+                f"delete on pods and get on pods/log in namespace {namespace}. See 'Grant access to the "
+                "cluster' in the AwsEksExecutor docs."
+            ) from e
+        self.log.info("EKS Executor health check has succeeded.")
+
+    @staticmethod
+    def _validate_eks_config() -> None:
+        if not conf.get(CONFIG_GROUP_NAME, AllEksConfigKeys.CLUSTER_NAME, fallback=None):

Review Comment:
   Synced offline and we agreed that the capability of per-team kubernetes clusters will be a follow-on.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["seanghaeli (via GitHub)" <gi...@apache.org>]

seanghaeli commented on code in PR #74399:
URL: https://github.com/apache/airflow/pull/74399#discussion_r4320207160


##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py:
##########
@@ -0,0 +1,226 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""AwsEksExecutor: run Airflow tasks as pods on an Amazon EKS cluster."""
+
+from __future__ import annotations
+
+import os
+from http import HTTPStatus
+from typing import TYPE_CHECKING
+
+from airflow.executors.executor_loader import ExecutorLoader
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.version_compat import AIRFLOW_V_3_2_PLUS
+from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException, conf
+
+if TYPE_CHECKING:
+    from kubernetes.client import models as k8s
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor_types import KubernetesJob
+
+_CNCF_KUBERNETES_REQUIRED = "AwsEksExecutor requires apache-airflow-providers-cncf-kubernetes>=10.24.0"
+
+try:
+    from kubernetes.client.rest import ApiException
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+    from airflow.providers.cncf.kubernetes.get_provider_info import (
+        get_provider_info as get_cncf_kubernetes_provider_info,
+    )
+except ImportError as e:
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED) from e
+
+# Check for the option, not the version, since unreleased trees report the previous version.
+if "client_factory" not in (
+    get_cncf_kubernetes_provider_info().get("config", {}).get("kubernetes_executor", {}).get("options", {})
+):
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED)
+
+_FACTORY_MODULE = "airflow.providers.amazon.aws.executors.eks._client_factory"
+_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_kube_client"
+_ASYNC_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_async_kube_client"
+# Airflow 2 hybrid executors that wrap a KubernetesExecutor.
+_HYBRID_KUBERNETES_EXECUTORS = ("CeleryKubernetesExecutor", "LocalKubernetesExecutor")
+
+
+class AwsEksExecutor(KubernetesExecutor):
+    """
+    A KubernetesExecutor that authenticates against an Amazon EKS cluster.
+
+    Builds the Kubernetes client from ``[aws_eks_executor]`` configuration and keeps the
+    short-lived EKS token fresh in-process. All pod-level behaviour comes unchanged from the
+    KubernetesExecutor and its ``[kubernetes_executor]`` configuration.
+    """
+
+    supports_multi_team: bool = True
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        # These need team_name, which super().__init__ sets.
+        self._check_no_team_eks_config()
+        self._validate_eks_config()
+        self._check_no_kubernetes_executor_in_scope()
+        self._ensure_client_factory()
+
+    def start(self) -> None:
+        """Call this when the Executor is run for the first time by the scheduler."""
+        super().start()
+        if TYPE_CHECKING:
+            assert self.kube_scheduler
+        # Both pod creation paths build every worker pod here, so check the image before it is sent.
+        build_pod_request = self.kube_scheduler._build_pod_request
+
+        def _build_pod_request_with_image(job: KubernetesJob) -> k8s.V1Pod:
+            pod = build_pod_request(job)
+            self._check_worker_image(pod)
+            return pod
+
+        self.kube_scheduler._build_pod_request = _build_pod_request_with_image  # type: ignore[method-assign,assignment]
+
+        check_health = conf.getboolean(
+            CONFIG_GROUP_NAME,
+            AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP,
+            fallback=CONFIG_DEFAULTS[AllEksConfigKeys.CHECK_HEALTH_ON_STARTUP],
+        )
+
+        if not check_health:
+            return
+
+        self.log.info("Starting EKS Executor and determining health...")
+        try:
+            self.check_health()
+        except RuntimeError:
+            self.log.error("Stopping the Airflow Scheduler from starting until the issue is resolved.")
+            raise
+
+    def check_health(self) -> None:
+        """
+        Make a test Kubernetes API call to check the health of the EKS Executor.
+
+        Building the client only proves the cluster exists; without this, a missing EKS access
+        entry or RBAC binding only shows up later as a watcher error loop.
+        """
+        if TYPE_CHECKING:
+            assert self.kube_client
+        namespace = self.kube_config.kube_namespace
+        try:
+            self.kube_client.list_namespaced_pod(namespace, limit=1)
+        except ApiException as e:
+            raise RuntimeError(
+                f"EKS Executor health check has failed because: cannot list pods in namespace {namespace} "
+                f"({e.status} {e.reason}). The IAM principal behind the AWS connection needs an EKS access "
+                "entry (or aws-auth mapping) on the cluster that allows create, get, list, watch, patch and "
+                f"delete on pods and get on pods/log in namespace {namespace}. See 'Grant access to the "
+                "cluster' in the AwsEksExecutor docs."
+            ) from e
+        self.log.info("EKS Executor health check has succeeded.")
+
+    def _check_worker_image(self, pod: k8s.V1Pod) -> None:

Review Comment:
   Agreed with this, I'm taking this check out of Eks and making a small cncf PR to put it in the base kubernetes classes



##########
providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py:
##########
@@ -0,0 +1,226 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""AwsEksExecutor: run Airflow tasks as pods on an Amazon EKS cluster."""
+
+from __future__ import annotations
+
+import os
+from http import HTTPStatus
+from typing import TYPE_CHECKING
+
+from airflow.executors.executor_loader import ExecutorLoader
+from airflow.providers.amazon.aws.executors.eks.utils import (
+    CONFIG_DEFAULTS,
+    CONFIG_GROUP_NAME,
+    AllEksConfigKeys,
+)
+from airflow.providers.amazon.version_compat import AIRFLOW_V_3_2_PLUS
+from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException, conf
+
+if TYPE_CHECKING:
+    from kubernetes.client import models as k8s
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor_types import KubernetesJob
+
+_CNCF_KUBERNETES_REQUIRED = "AwsEksExecutor requires apache-airflow-providers-cncf-kubernetes>=10.24.0"
+
+try:
+    from kubernetes.client.rest import ApiException
+
+    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import KubernetesExecutor
+    from airflow.providers.cncf.kubernetes.get_provider_info import (
+        get_provider_info as get_cncf_kubernetes_provider_info,
+    )
+except ImportError as e:
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED) from e
+
+# Check for the option, not the version, since unreleased trees report the previous version.
+if "client_factory" not in (
+    get_cncf_kubernetes_provider_info().get("config", {}).get("kubernetes_executor", {}).get("options", {})
+):
+    raise AirflowOptionalProviderFeatureException(_CNCF_KUBERNETES_REQUIRED)
+
+_FACTORY_MODULE = "airflow.providers.amazon.aws.executors.eks._client_factory"
+_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_kube_client"
+_ASYNC_CLIENT_FACTORY_PATH = f"{_FACTORY_MODULE}._get_eks_async_kube_client"
+# Airflow 2 hybrid executors that wrap a KubernetesExecutor.
+_HYBRID_KUBERNETES_EXECUTORS = ("CeleryKubernetesExecutor", "LocalKubernetesExecutor")
+
+
+class AwsEksExecutor(KubernetesExecutor):
+    """
+    A KubernetesExecutor that authenticates against an Amazon EKS cluster.
+
+    Builds the Kubernetes client from ``[aws_eks_executor]`` configuration and keeps the
+    short-lived EKS token fresh in-process. All pod-level behaviour comes unchanged from the
+    KubernetesExecutor and its ``[kubernetes_executor]`` configuration.
+    """
+
+    supports_multi_team: bool = True
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        # These need team_name, which super().__init__ sets.
+        self._check_no_team_eks_config()
+        self._validate_eks_config()
+        self._check_no_kubernetes_executor_in_scope()
+        self._ensure_client_factory()
+
+    def start(self) -> None:
+        """Call this when the Executor is run for the first time by the scheduler."""
+        super().start()
+        if TYPE_CHECKING:
+            assert self.kube_scheduler
+        # Both pod creation paths build every worker pod here, so check the image before it is sent.
+        build_pod_request = self.kube_scheduler._build_pod_request
+
+        def _build_pod_request_with_image(job: KubernetesJob) -> k8s.V1Pod:
+            pod = build_pod_request(job)
+            self._check_worker_image(pod)
+            return pod
+
+        self.kube_scheduler._build_pod_request = _build_pod_request_with_image  # type: ignore[method-assign,assignment]

Review Comment:
   Agreed again, as with the above, I'm taking this check out of Eks and making a small cncf PR to put it in the base kubernetes classes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add AwsEksExecutor to the Amazon provider [airflow]

["o-nikolas (via GitHub)" <gi...@apache.org>]

o-nikolas merged PR #74399:
URL: https://github.com/apache/airflow/pull/74399


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org