You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "valereColleville (via GitHub)" <gi...@apache.org> on 2025/07/21 08:14:15 UTC
[I] EKSPodOperator authentication issue using connections from metadata-database in Airflow 3.0.3 [airflow]
valereColleville opened a new issue, #53578:
URL: https://github.com/apache/airflow/issues/53578
### Apache Airflow Provider(s)
amazon
### Versions of Apache Airflow Providers
_No response_
### Apache Airflow version
3.0.3
### Operating System
Linux (amazon linux AL2023), EKS 1.32
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### What happened
Since upgrading to Airflow 3.0.3, our DAG using EKSPodOperator is no longer able to authenticate with our EKS cluster.
When the DAG runs, it fails with a 401 Unauthorized error when trying to list pods:
https://github.com/apache/airflow/blob/3.0.3/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py#L627
From the EKS side, we can see in the audit logs that the API call is made with no user identity:
```json
{
"kind": "Event",
"apiVersion": "audit.k8s.io/v1",
"level": "Request",
"auditID": "5007765e-80ab-475b-a30d-6bd769ea71f2",
"stage": "ResponseStarted",
"requestURI": "/api/v1/namespaces/function-execution-dev/pods?labelSelector=dag_id%3Dwoke_eks%2Ckubernetes_pod_operator%3DTrue%2Crun_id%3Dmanual__2025-07-18T213454.6384250000-30e0ce312%2Ctask_id%3Dexecute_k8s_eks_work%2Calready_checked%21%3DTrue%2C%21airflow-worker",
"verb": "list",
"user": {},
"sourceIPs": ["10.6.113.97"],
"userAgent": "OpenAPI-Generator/32.0.1/python",
"objectRef": {
"resource": "pods",
"namespace": "function-execution-dev",
"apiVersion": "v1"
},
"responseStatus": {
"metadata": {},
"status": "Failure",
"message": "Unauthorized",
"reason": "Unauthorized",
"code": 401
},
"requestReceivedTimestamp": "2025-07-18T21:35:02.934260Z",
"stageTimestamp": "2025-07-18T21:35:02.934497Z"
}
```
We found a potential fix here:
https://github.com/apache/airflow/pull/51333#issuecomment-3089951506
We tried backporting the commit from that PR into our 3.0.3 deployment. Although it removed the ORM stack trace from the logs, the authentication issue still persists.
Based on the explanation in the PR, **we suspect the issue might be related to how the aws_connection is retrieved**. We store our AWS connection in the Airflow metadata database, and if the system using TaskSDK is no more able to initialised the ORM, it could be unable to access the metadata-database in the same time?
Here are some tests we performed and their results:
From the worker, we manually ran the token generation:
```bash
python3.12 -m airflow.providers.amazon.aws.utils.eks_get_token \
--cluster-name CLUSTER \
--region-name eu-central-1 \
--aws-conn-id val_aws_assume_test
```
We used the generated token to query the Kubernetes API:
```bash
curl -s --header "Authorization: Bearer $TOKEN" \
--insecure \
"$APISERVER/api/v1/namespaces/function-execution-dev/pods?labelSelector=dag_id%3Dwoke_eks%2Ckubernetes_pod_operator%3DTrue%2Crun_id%3Dmanual__2025-07-18T134438.5412460000-53e546ffb%2Ctask_id%3Dexecute_k8s_eks_work%2Calready_checked%21%3DTrue%2C%21airflow-worker"
```
✅ This works as expected (so our connection and roles are correctly configured).
However, when we export PYTHON_OPERATORS_VIRTUAL_ENV_MODE=1, the generated token is invalid and results in a 401 error.
We also tried blocking the temporary kubeconfig deletion to debug from the worker pod. During local testing with the kubeconfig, we faced multiple issues:
unsupported API versions (alpha1)
regex overmatch problems
errors when setting PYTHON_OPERATORS_VIRTUAL_ENV_MODE=1
None of these tests gave conclusive insights. We are currently unable to use the generated kubeconfig reliably for local validation.
### What you think should happen instead
Should works with connection the same way than before in 2.X.
### How to reproduce
EKS setup, Create a connection of AWS type (we have tryied both assume role or user static credentials)
The following DAG:
```python
from __future__ import annotations
import os
import datetime
import logging
from pathlib import Path
from airflow.sdk import DAG
from airflow.models.param import Param
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
from airflow.providers.amazon.aws.operators.eks import EksPodOperator
from kubernetes.client import models as k8s
with DAG(
dag_id=Path(__file__).stem.removeprefix("dag_"),
dag_display_name="woke pod",
schedule=None,
catchup=False,
# concurrency=1,
max_active_runs=1,
start_date=datetime.datetime(2022, 3, 4),
tags=["poc"],
is_paused_upon_creation=True,
doc_md=__doc__,
params={
"containerImage": Param(
"",
type="string",
title="containerImage",
examples=["<IMAGE_TO_DEPLOY>"],
),
},
) as dag:
logger = logging.getLogger(__name__)
logger.info("This is a log message")
securityContextRunAsUser = int(os.getenv("COAM_K8S_SECURITYCONTEXT_RUNASUSER", 0))
volumeClaim = os.getenv("COAM_K8S_VOLUMECLAIM")
serviceAccount = os.getenv("COAM_K8S_SERVICE_ACCOUNT_NAME")
tolerationsIsolatedWorkload = os.getenv("COAM_K8S_TOLERATIONS_ISOLATEDWORKLOAD")
clusterName = os.getenv("COAM_K8S_CLUSTER_NAME")
k8sNamespace = os.getenv("COAM_K8S_NAMESPACE")
exec_k8s_eks = EksPodOperator(
task_id="execute_k8s_eks_work",
pod_name="worky_eks_pod_name",
cluster_name=clusterName,
aws_conn_id="<YOUR_CONNECTION_ID>",
namespace=k8sNamespace,
image="{{ params.containerImage }}",
cmds=["sleep", "60"],
get_logs=True,
deferrable=False,
container_logs=True, # debugging: to actually see stdout for containers without airflow UI (nice for dev)
tolerations=[
k8s.V1Toleration(
key="IsolatedWorkload", operator="Equal", value=tolerationsIsolatedWorkload, effect="NoSchedule"
)
],
volumes=(
[
k8s.V1Volume(
name=volumeClaim,
persistent_volume_claim=k8s.V1PersistentVolumeClaimVolumeSource(claim_name=volumeClaim),
)
]
if volumeClaim
else []
),
volume_mounts=(
[k8s.V1VolumeMount(name=volumeClaim, mount_path=f"/{volumeClaim}", sub_path=None, read_only=False)]
if volumeClaim
else []
),
service_account_name=serviceAccount,
security_context=k8s.V1PodSecurityContext(
fs_group=securityContextRunAsUser,
run_as_group=securityContextRunAsUser,
run_as_user=securityContextRunAsUser,
),
)
exec_k8s_eks
if __name__ == "__main__":
dag.test()
```
Run the Dag in airflow and you should see the 401 log.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] EKSPodOperator authentication issue using connections from metadata-database in Airflow 3.0.3 [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas closed issue #53578: EKSPodOperator authentication issue using connections from metadata-database in Airflow 3.0.3
URL: https://github.com/apache/airflow/issues/53578
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] EKSPodOperator authentication issue using connections from metadata-database in Airflow 3.0.3 [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #53578:
URL: https://github.com/apache/airflow/issues/53578#issuecomment-3095680236
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] EKSPodOperator authentication issue using connections from metadata-database in Airflow 3.0.3 [airflow]
Posted by "vinitpayal (via GitHub)" <gi...@apache.org>.
vinitpayal commented on issue #53578:
URL: https://github.com/apache/airflow/issues/53578#issuecomment-3192557380
@viiccwen any leads we are also seeing the same issue in our setup :( I can help if required to fix if you have some leads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] EKSPodOperator authentication issue using connections from metadata-database in Airflow 3.0.3 [airflow]
Posted by "viiccwen (via GitHub)" <gi...@apache.org>.
viiccwen commented on issue #53578:
URL: https://github.com/apache/airflow/issues/53578#issuecomment-3138458978
@eladkal Hello,
I’m willing to take this issue and work on a fix.
Could you please assign it to me? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] EKSPodOperator authentication issue using connections from metadata-database in Airflow 3.0.3 [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #53578:
URL: https://github.com/apache/airflow/issues/53578#issuecomment-3097559397
cc @o-nikolas @vincbeck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] EKSPodOperator authentication issue using connections from metadata-database in Airflow 3.0.3 [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on issue #53578:
URL: https://github.com/apache/airflow/issues/53578#issuecomment-3139889507
Done :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] EKSPodOperator authentication issue using connections from metadata-database in Airflow 3.0.3 [airflow]
Posted by "viiccwen (via GitHub)" <gi...@apache.org>.
viiccwen commented on issue #53578:
URL: https://github.com/apache/airflow/issues/53578#issuecomment-3193536156
Hello @vinitpayal ,
Thank you for the kind offer. I'll take this on tomorrow and will reach out if I require any assistance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] EKSPodOperator authentication issue using connections from metadata-database in Airflow 3.0.3 [airflow]
Posted by "valereColleville (via GitHub)" <gi...@apache.org>.
valereColleville commented on issue #53578:
URL: https://github.com/apache/airflow/issues/53578#issuecomment-3097683955
A bit more log after some digging (which seems to confirm that using EKSPodOperator with Metastore connection is not possible):
In the first level (reation of the operator), the system work, aws connection are retreived using taskSDK api without issue (i have backport the fix on main to select TaskSDK if setup inside AWS_Base
```python
@classmethod
def get_connection(cls, conn_id: str) -> Connection:
"""
Get connection, given connection id.
:param conn_id: connection id
:return: connection
"""
import sys
# if SUPERVISOR_COMMS is set, we're in task sdk context
if hasattr(sys.modules.get("airflow.sdk.execution_time.task_runner"), "SUPERVISOR_COMMS"):
log.info("Connection search from (via task-sdk) - airflow.sdk.definitions.connection")
from airflow.sdk.definitions.connection import Connection
conn = Connection.get(conn_id)
log.info("Connection Retrieved '%s' (via task-sdk)", conn.conn_id)
log.info("Connection Retrieved '%s' (via task-sdk)", conn.extra)
log.info("Connection Retrieved '%s' (via task-sdk)", conn.conn_type)
return conn
from airflow.models.connection import Connection as ConnectionModel
log.info("Connection search from ConnectionModel get_connection_from_secrets - airflow.models.connection")
conn = ConnectionModel.get_connection_from_secrets(conn_id)
log.info("Connection Retrieved '%s' (via core Airflow)", conn.conn_id)
return conn
````
The issue happen in the second level, once the kubeconfig is generated by EKSHook with the commande to use eks_get_token, when using a connection_id so that conneciton is pass to the underlaying sh exec command of the kubeconfig (make sens).
But when the exec run, it doesn't detect the supervisor_comms (probably not loaded), so it use the default "not-tasksdk api" to solve the connection_id when creating the EKSHook (from the kubectl exec call that time).
https://github.com/apache/airflow/blob/3.0.3/airflow-core/src/airflow/models/connection.py#L466
So from there it test all backend_secret, but failling over metastore one so not finding any connection_id.
Logs from the exec sh -c commande from generated kubeconfig
```
[2025-07-21T17:07:12.555+0000] {base.py:75} INFO - Connection search from ConnectionModel get_connection_from_secrets - airflow.models.connection
[2025-07-21T17:07:12.555+0000] {connection.py:489} INFO - VAC-MOD-LOG: enabled only if SecretCache.init() has been called first
[2025-07-21T17:07:12.555+0000] {connection.py:496} INFO - VAC-MOD-LOG: Iterate over backends
[2025-07-21T17:07:12.556+0000] {connection.py:498} INFO - VAC-MOD-LOG: Checking secrets backend (EnvironmentVariablesBackend) .
[2025-07-21T17:07:12.556+0000] {connection.py:498} INFO - VAC-MOD-LOG: Checking secrets backend (MetastoreBackend) .
[2025-07-21T17:07:12.556+0000] {connection.py:510} INFO - VAC-MOD-LOG: Unable to retrieve connection from secrets backend (MetastoreBackend). Checking subsequent secrets backend.
[2025-07-21T17:07:12.556+0000] {base_aws.py:623} WARNING - Unable to find AWS Connection ID 'val_aws_assume_test', switching to empty.
[2025-07-21T17:07:12.556+0000] {base_aws.py:189} INFO - No connection ID provided. Fallback on boto3 credential strategy (region_name='eu-central-1'). See: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
[2025-07-21T17:07:12.563+0000] {base_aws.py:189} INFO - No connection ID provided. Fallback on boto3 credential strategy (region_name='eu-central-1'). See: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
[2025-07-21T17:07:13.064+0000] {base.py:75} INFO - Connection search from ConnectionModel get_connection_from_secrets - airflow.models.connection
[2025-07-21T17:07:13.064+0000] {connection.py:489} INFO - VAC-MOD-LOG: enabled only if SecretCache.init() has been called first
[2025-07-21T17:07:13.064+0000] {connection.py:496} INFO - VAC-MOD-LOG: Iterate over backends
[2025-07-21T17:07:13.064+0000] {connection.py:498} INFO - VAC-MOD-LOG: Checking secrets backend (EnvironmentVariablesBackend) .
[2025-07-21T17:07:13.065+0000] {connection.py:498} INFO - VAC-MOD-LOG: Checking secrets backend (MetastoreBackend) .
[2025-07-21T17:07:13.065+0000] {connection.py:510} INFO - VAC-MOD-LOG: Unable to retrieve connection from secrets backend (MetastoreBackend). Checking subsequent secrets backend.
[2025-07-21T17:07:13.065+0000] {base_aws.py:623} WARNING - Unable to find AWS Connection ID 'aws_default', switching to empty.
[2025-07-21T17:07:13.065+0000] {base_aws.py:189} INFO - No connection ID provided. Fallback on boto3 credential strategy (region_name='eu-central-1'). See: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
expirationTimestamp: 2025-07-21T17:21:13Z, token: k8s-aws-v1.xxxx
```
Where the exact same commande run from inside the worker directly generate the following logs:
```
[2025-07-21T17:12:27.099+0000] {base.py:75} INFO - Connection search from ConnectionModel get_connection_from_secrets - airflow.models.connection
[2025-07-21T17:12:27.100+0000] {connection.py:489} INFO - VAC-MOD-LOG: enabled only if SecretCache.init() has been called first
[2025-07-21T17:12:27.100+0000] {connection.py:496} INFO - VAC-MOD-LOG: Iterate over backends
[2025-07-21T17:12:27.101+0000] {connection.py:498} INFO - VAC-MOD-LOG: Checking secrets backend (EnvironmentVariablesBackend) .
[2025-07-21T17:12:27.101+0000] {connection.py:498} INFO - VAC-MOD-LOG: Checking secrets backend (MetastoreBackend) .
[2025-07-21T17:12:27.101+0000] {metastore.py:51} INFO - VAC-MOD-LOG: METASTORE GET CONNECTION val_aws_assume_test
[2025-07-21T17:12:28.668+0000] {base.py:77} INFO - Connection Retrieved 'val_aws_assume_test' (via core Airflow)
[2025-07-21T17:12:29.547+0000] {base.py:75} INFO - Connection search from ConnectionModel get_connection_from_secrets - airflow.models.connection
[2025-07-21T17:12:29.547+0000] {connection.py:489} INFO - VAC-MOD-LOG: enabled only if SecretCache.init() has been called first
[2025-07-21T17:12:29.547+0000] {connection.py:496} INFO - VAC-MOD-LOG: Iterate over backends
[2025-07-21T17:12:29.548+0000] {connection.py:498} INFO - VAC-MOD-LOG: Checking secrets backend (EnvironmentVariablesBackend) .
[2025-07-21T17:12:29.548+0000] {connection.py:498} INFO - VAC-MOD-LOG: Checking secrets backend (MetastoreBackend) .
[2025-07-21T17:12:29.548+0000] {metastore.py:51} INFO - VAC-MOD-LOG: METASTORE GET CONNECTION aws_default
[2025-07-21T17:12:29.564+0000] {base.py:77} INFO - Connection Retrieved 'aws_default' (via core Airflow)
[2025-07-21T17:12:29.564+0000] {connection_wrapper.py:325} INFO - AWS Connection (conn_id='aws_default', conn_type='aws') credentials retrieved from login and password.
expirationTimestamp: 2025-07-21T17:26:29Z, token:
```
(from the worker we run the commande without export PYTHON_OPERATORS_VIRTUAL_ENV_MODE=1, if we use it, we have the same result as the sub-exec from kubeconfig)
Hope that addition of details will help!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org