You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "raphaelauv (via GitHub)" <gi...@apache.org> on 2023/02/12 10:00:18 UTC
[GitHub] [airflow] raphaelauv opened a new issue, #29488: KPO - deferrable - Invalid kube-config file. Expected key contexts in kube-config
raphaelauv opened a new issue, #29488:
URL: https://github.com/apache/airflow/issues/29488
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
5.2.0
### Apache Airflow version
2.5.1
### Operating System
linux 5.15.0-60-generic - Ubuntu 22.04
### Deployment
Docker-Compose
### Deployment details
_No response_
### What happened
airflow connection - kubernetes_default ->
```json
{
"conn_type": "kubernetes",
"extra": "{\"extra__kubernetes__in_cluster\": false, \"extra__kubernetes__kube_config_path\": \"/opt/airflow/include/.kube/config\", \"extra__kubernetes__namespace\": \"default\", \"extra__kubernetes__cluster_context\": \"kind-kind\", \"extra__kubernetes__disable_verify_ssl\": false, \"extra__kubernetes__disable_tcp_keepalive\": false}"
}
```
```python
from pendulum import today
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
from kubernetes.client import V1ResourceRequirements
dag = DAG(
dag_id="kubernetes_dag",
schedule_interval="0 0 * * *",
start_date=today("UTC").add(days=-1)
)
with dag:
cmd = "echo toto && sleep 10 && echo finish"
KubernetesPodOperator(
task_id="task-a",
namespace="default",
kubernetes_conn_id="kubernetes_default",
name="airflow-test-pod",
image="alpine:3.16.2",
cmds=["sh", "-c", cmd],
is_delete_operator_pod=True,
deferrable=True,
get_logs=True,
)
KubernetesPodOperator(
task_id="task-B",
namespace="default",
kubernetes_conn_id="kubernetes_default",
name="airflow-test-pod",
image="alpine:3.16.2",
cmds=["sh", "-c", cmd],
is_delete_operator_pod=True,
get_logs=True,
)
```
```log
[2023-02-12, 09:53:24 UTC] {taskinstance.py:1768} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 611, in execute_complete
raise AirflowException(event["message"])
airflow.exceptions.AirflowException: Invalid kube-config file. Expected key contexts in kube-config
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 630, in execute_complete
self.post_complete_action(
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 654, in post_complete_action
self.cleanup(
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 673, in cleanup
raise AirflowException(
airflow.exceptions.AirflowException: Pod airflow-test-pod-vw8fxf25 returned a failure:
```
```
```
### What you think should happen instead
deferrable KPO should work same as KPO and not fail
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #29488: KPO - deferrable - Invalid kube-config file. Expected key contexts in kube-config
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #29488:
URL: https://github.com/apache/airflow/issues/29488#issuecomment-1427133882
This exception is raised by the kubernetes client, the version used in your Airflow worker may be not compatible with the kubectl version which you used to generate the config file, can you check what are the used version for kubernetes client and kubectl? can you add the config file to check why it's not parsed by the client?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv commented on issue #29488: KPO - deferrable - Invalid kube-config file. Expected key contexts in kube-config
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on issue #29488:
URL: https://github.com/apache/airflow/issues/29488#issuecomment-1426991163
maybe same problem than this one -> https://github.com/astronomer/astronomer-providers/issues/745
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #29488: KPO - deferrable - Invalid kube-config file. Expected key contexts in kube-config
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #29488:
URL: https://github.com/apache/airflow/issues/29488#issuecomment-1427147130
Indeed, I think there is a missing check for this path when you provide it in connection, I will try to add a unit test and maybe refacto my PR, can you please test it on your dag if this is possible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv commented on issue #29488: KPO - deferrable - Invalid kube-config file. Expected key contexts in kube-config
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on issue #29488:
URL: https://github.com/apache/airflow/issues/29488#issuecomment-1427139550
if I set the parameter `config_file` the deferable task then work
```python
KubernetesPodOperator(
task_id="task-a",
namespace="default",
kubernetes_conn_id="kubernetes_default",
config_file="/opt/airflow/include/.kube/config",
deferrable=True,
...
```
hello @hussein-awala, the config file is not the problem , it's the deferrable code not fully using the kubernetes_conn_id , (exactly like the issue I shared in my previous message)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #29488: KPO - deferrable - Invalid kube-config file. Expected key contexts in kube-config
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #29488: KPO - deferrable - Invalid kube-config file. Expected key contexts in kube-config
URL: https://github.com/apache/airflow/issues/29488
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org