You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "alvarogaroconstella (via GitHub)" <gi...@apache.org> on 2025/02/18 09:12:29 UTC
[I] 403 permission Denied KubernetesPodOperator [airflow]
alvarogaroconstella opened a new issue, #46855:
URL: https://github.com/apache/airflow/issues/46855
### Apache Airflow version
2.10.5
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
I am trying to pass a .yaml file with the kubeconfig of my cluster and create a pod using a dag, the fact is that is The error kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden keeps appearing.
```
HTTP response headers: HTTPHeaderDict({'Audit-Id': '16328c7b-8057-4a30-9625-206bd8c052ae', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '05f64062-139b-4118-9727-4804f796adbe', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'f6addaaf-d023-4008-b9a5-bd8507e0f00c', 'Date': 'Tue, 18 Feb 2025 09:01:05 GMT', 'Content-Length': '266'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" in the namespace \"airflow-idti\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
```
I have tried setting the service_account_name parameter both with and without it, but it doesn't work. The file I pass is a kubeconfig mounted inside a path of the Airflow pods, and I’ve confirmed that its value is correct and present in all the pods. When I use in_cluster: true, it works, but when in_cluster: false, this error always appears and I can't log in. I've followed the documentation, but I still encounter this error.
### What you think should happen instead?
I think that with in_cluster = false and with the kubeconfig file should work
### How to reproduce
```
import logging
from plugins.utils.region import (
get_kubeconfig_path_from_region,
get_region_from_sqs_url,
print_kubeconfig_info
)
from airflow.utils.dates import days_ago
from airflow.decorators import dag
from airflow.models.param import Param
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
from airflow.operators.python import PythonOperator
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
logging.getLogger("elasticsearch").setLevel(logging.ERROR)
DEFAULT_ARGS = {
"owner": "airflow",
"retries": 1,
"start_date": days_ago(1),
}
@dag(
"trigger_alerts",
default_args=DEFAULT_ARGS,
schedule_interval=None,
catchup=False,
params={
"breach_id": Param("a11111111111111111111111", type="string"),
"user_id": Param("590073c764e160715e845f37", type="string"),
"queue_name": Param(
"http://sqs.us-west-2.localhost.localstack.icloud:4566/000000000000/notify-docker-retrospective",
type="string",
),
"slice_id": Param(0, type="integer"),
"num_slices": Param(0, type="integer"),
},
)
def alert_delivery_isolation():
"""
Orchestrates the process of fetching alerts, filtering them based on provisioning client,
and sending the filtered alerts to the appropriate SQS queue.
"""
region = PythonOperator(
task_id="region",
python_callable=get_region_from_sqs_url,
op_args=["{{ params.queue_name }}"],
)
kubeconfig_path = PythonOperator(
task_id="kubeconfig_path",
python_callable=get_kubeconfig_path_from_region,
op_args=["{{ ti.xcom_pull(task_ids='region') }}"],
)
print_kubeconfig_task = PythonOperator(
task_id="print_kubeconfig",
python_callable=print_kubeconfig_info,
provide_context=True,
)
write_alert_operator = KubernetesPodOperator(
name="write_alerts_in_sqs",
namespace="airflow-idti",
image="867215330947.dkr.ecr.us-east-1.amazonaws.com/idtheft-airflow:airflowtesting",
cmds=["python"],
arguments=[
"/opt/airflow/plugins/scripts/trigger_alerts.py",
"--source_key",
"{{ params.breach_id }}",
"--user_id",
"{{ params.user_id }}",
"--queue_name",
"{{ params.queue_name }}",
"--slice_id",
"{{ params.slice_id }}",
"--num_slices",
"{{ params.num_slices }}",
],
task_id="write_alerts_in_sqs",
in_cluster=False,
config_file = "/opt/airflow/kubeconfig/kubeconfig-us.yaml",
get_logs=True,
service_account_name='airflow-idti-worker',
# pod_template_file="/opt/airflow/config/kubernetes/pod_template.yaml",
)
region >> kubeconfig_path >> print_kubeconfig_task >> write_alert_operator
dag = alert_delivery_isolation()
```
Kubeconfig file in route /opt/airflow/kubeconfig/kubeconfig-us.yaml:
```
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tL....
server: https://xxxxxxxxxxxxxxxxxxxxxxxx.gr7.us-east-1.eks.amazonaws.com
name: arn:aws:eks:us-east-1:867215330947:cluster/NitoAirflowCluster
contexts:
- context:
cluster: arn:aws:eks:us-east-1:867215330947:cluster/NitoAirflowCluster
user: arn:aws:eks:us-east-1:867215330947:cluster/NitoAirflowCluster
name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:867215330947:cluster/NitoAirflowCluster
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- --region
- us-east-1
- eks
- get-token
- --cluster-name
- NitoAirflowCluster
- --output
- json
command: aws
```
Full traceback error:
```
[2025-02-18, 09:06:06 UTC] {kube_config.py:520} ERROR - [Errno 13] Permission denied: 'aws'
[2025-02-18, 09:06:06 UTC] {taskinstance.py:3311} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 767, in _execute_task
result = _execute_callable(context=context, **execute_callable_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 733, in _execute_callable
return ExecutionCallableRunner(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/operator_helpers.py", line 252, in run
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/baseoperator.py", line 422, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 583, in execute
return self.execute_sync(context)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 593, in execute_sync
self.pod = self.get_or_create_pod( # must set `self.pod` for `on_kill`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 551, in get_or_create_pod
pod = self.find_pod(pod_request_obj.metadata.namespace, context=context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 524, in find_pod
pod_list = self.client.list_namespaced_pod(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py", line 15823, in list_namespaced_pod
return self.list_namespaced_pod_with_http_info(namespace, **kwargs) # noqa: E501
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py", line 15942, in list_namespaced_pod_with_http_info
return self.api_client.call_api(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET
return self.request("GET", url,
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '0e63b26a-4ad2-4baf-8bbb-26f4c0c1876b', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '05f64062-139b-4118-9727-4804f796adbe', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'f6addaaf-d023-4008-b9a5-bd8507e0f00c', 'Date': 'Tue, 18 Feb 2025 09:06:06 GMT', 'Content-Length': '266'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" in the namespace \"airflow-idti\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
```
### Operating System
"Debian GNU/Linux 12 (bookworm)"
### Versions of Apache Airflow Providers
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
I used the default values but with some custom values to add some configmaps
### Anything else?
Thanks in advance, I think it is a bug, if not I can move to other place
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 403 permission Denied KubernetesPodOperator [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #46855:
URL: https://github.com/apache/airflow/issues/46855#issuecomment-2665023916
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 403 permission Denied KubernetesPodOperator [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #46855:
URL: https://github.com/apache/airflow/issues/46855#issuecomment-4493479824
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 403 permission Denied KubernetesPodOperator [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #46855: 403 permission Denied KubernetesPodOperator
URL: https://github.com/apache/airflow/issues/46855
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 403 permission Denied KubernetesPodOperator [airflow]
Posted by "andrewhharmon (via GitHub)" <gi...@apache.org>.
andrewhharmon commented on issue #46855:
URL: https://github.com/apache/airflow/issues/46855#issuecomment-3877305088
This seems more like it is failing to call the `aws` cli command. Can you confirm the aws cli is installed, its in PATH and the user airflow is running under has permissions to call it? It is finding your kubeconfig file, bc it's using it to try to fetch aws creds. it does that via the aws cli. It runs the `command` in your kubeconfig file. It's saying it doesn't have permissions to run that command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 403 permission Denied KubernetesPodOperator [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #46855:
URL: https://github.com/apache/airflow/issues/46855#issuecomment-4559858537
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org