You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/27 21:57:11 UTC
[GitHub] [airflow] dszakallas opened a new issue #22560: KubernetesExecutor does not work with kubelogin exec authentication
dszakallas opened a new issue #22560:
URL: https://github.com/apache/airflow/issues/22560
### Apache Airflow version
2.2.4 (latest released)
### What happened
When using KubernetesExecutor with `in_cluster = False` in conjunction with a Config with an `exec` user using `kubelogin`, something wrong happens and authentication does not take place:
```
hack-scheduler-1 | File "/airflow/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 747, in _adopt_completed_pods
hack-scheduler-1 | pod_list = kube_client.list_namespaced_pod(namespace=self.kube_config.kube_namespace, **kwargs)
hack-scheduler-1 | File "/airflow/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15697, in list_namespaced_pod
hack-scheduler-1 | return self.list_namespaced_pod_with_http_info(namespace, **kwargs) # noqa: E501
hack-scheduler-1 | File "/airflow/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15812, in list_namespaced_pod_with_http_info
hack-scheduler-1 | return self.api_client.call_api(
hack-scheduler-1 | File "/airflow/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
hack-scheduler-1 | return self.__call_api(resource_path, method,
hack-scheduler-1 | File "/airflow/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
hack-scheduler-1 | response_data = self.request(
hack-scheduler-1 | File "/airflow/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 373, in request
hack-scheduler-1 | return self.rest_client.GET(url,
hack-scheduler-1 | File "/airflow/lib/python3.8/site-packages/kubernetes/client/rest.py", line 240, in GET
hack-scheduler-1 | return self.request("GET", url,
hack-scheduler-1 | File "/airflow/lib/python3.8/site-packages/kubernetes/client/rest.py", line 234, in request
hack-scheduler-1 | raise ApiException(http_resp=r)
hack-scheduler-1 | kubernetes.client.exceptions.ApiException: (401)
hack-scheduler-1 | Reason: Unauthorized
hack-scheduler-1 | HTTP response headers: HTTPHeaderDict({'Audit-Id': '9d09a92f-d294-4a82-9aac-bbafe9573469', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Sun, 27 Mar 2022 21:16:39 GMT', 'Content-Length': '129'})
```
I managed to trace back to the source of the error to `RefreshConfiguration`, and create a workaround.
https://github.com/apache/airflow/blob/ee9049c0566b2539a247687de05f9cffa008f871/airflow/kubernetes/kube_client.py#L45-L46
Bypassing the `RefreshConfiguration` by changing the above two lines to:
```python
config.load_kube_config(context=cluster_context, config_file=configfile)
cfg = None
```
resolves the problem. I am still debugging what exactly the problem is with `RefreshConfiguration` and `kubelogin`.
Factoids:
- I am facing this issue with the [service principal authentication flow](https://github.com/Azure/kubelogin#spn-login-with-secret) with AKS cluster. I didn't try other flows so they might or might not work.
- I am using kubelogin v0.0.11
### What you think should happen instead
Authentication should work without a problem.
### How to reproduce
It's hard to reproduce given the specificity of the problem.
1. Create a service principal and assign permissions to be able to create resources on the AKS cluster.
2. Install [`kubelogin`](https://github.com/Azure/kubelogin)
3. Create a Config file that uses the `kubelogin` exec authentication flow with service principal authentication with the correct values filled in. See [docs](https://github.com/Azure/kubelogin#spn-login-with-secret)
4. Confirm it works by running
```python
from kubernetes import client, config
config.load_kube_config()
print(client.CoreV1Api().list_namespaced_pod('default'))
```
5. Try out with Airflow and get lots of 401 errors.
### Operating System
Debian
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
Proof of concept deployment with Docker compose for local development purposes using KubernetesExecutor to schedule worker pods in an AKS cluster.
### Anything else
This issue happens every time.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish closed issue #22560: KubernetesExecutor does not work with exec authentication on kubernetes client >= 22.6.0
Posted by GitBox <gi...@apache.org>.
dstandish closed issue #22560:
URL: https://github.com/apache/airflow/issues/22560
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #22560: KubernetesExecutor does not work with exec authentication on kubernetes client >= 22.6.0
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #22560:
URL: https://github.com/apache/airflow/issues/22560#issuecomment-1080153353
@dszakallas can you open PR with the suggested fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dszakallas commented on issue #22560: KubernetesExecutor does not work with kubelogin exec authentication
Posted by GitBox <gi...@apache.org>.
dszakallas commented on issue #22560:
URL: https://github.com/apache/airflow/issues/22560#issuecomment-1080038043
I did a bit more of debugging, it seems that the error is not specific to kubelogin. I am renaming the ticket as it has little to do with kubelogin and Azure.
I found this in the logs:
```
{{refresh_config.py:73}} ERROR - __init__() missing 1 required positional argument: 'cwd'
```
Changing
https://github.com/apache/airflow/blob/ee9049c0566b2539a247687de05f9cffa008f871/airflow/kubernetes/refresh_config.py#L61
to
```python
status = ExecProvider(self._user['exec'], os.getcwd()).run()
```
resolves the issue.
I am using the following version of `kubernetes`:
```
kubernetes 23.3.0
```
Here's the relevant diff in the upstream:
https://github.com/kubernetes-client/python/commit/1c5bf586f0882c81c03181588830887345703ca5
So it seems like this API breaks in [v22.6.0](https://github.com/kubernetes-client/python/releases/tag/v22.6.0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #22560: KubernetesExecutor does not work with exec authentication on kubernetes client >= 22.6.0
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #22560:
URL: https://github.com/apache/airflow/issues/22560#issuecomment-1080308464
That might be another one (@ephraimbuddy @dstandish) that might make 2.2.5 needing an RC2
There is this huge bump of K8S client version from 2.2.4 (11.0.0 -> 23.3.0) so I think we should be extra careful and possibly do some more testing to prevent regressions there, otherwise we might need to release 2.2.6 reallly quickly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #22560: KubernetesExecutor does not work with exec authentication on kubernetes client >= 22.6.0
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #22560:
URL: https://github.com/apache/airflow/issues/22560#issuecomment-1080828902
@dszakallas, not sure what version of the kubernetes provider you have, but try with this combo:
```
kubernetes==11.0.0
apache-airflow-providers-cncf-kubernetes==3.1.1
```
Basically, 2.2.4 only works with `11.0.0`, and provider `3.1.2` only works with k8s >=21.7.0. The "common ground" is the above versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #22560: KubernetesExecutor does not work with exec authentication on kubernetes client >= 22.6.0
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #22560:
URL: https://github.com/apache/airflow/issues/22560#issuecomment-1080934827
closing for the following reasons
this issue should be resolved by using constraints file as suggested by @jedcunningham
And from airflow 2.3.0, we remove anyway the “refresh configuration” code that is the problem here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #22560: KubernetesExecutor does not work with kubelogin exec authentication
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #22560:
URL: https://github.com/apache/airflow/issues/22560#issuecomment-1080026975
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dszakallas edited a comment on issue #22560: KubernetesExecutor does not work with exec authentication on kubernetes >= 22.6.0
Posted by GitBox <gi...@apache.org>.
dszakallas edited a comment on issue #22560:
URL: https://github.com/apache/airflow/issues/22560#issuecomment-1080038043
I did a bit more of debugging, it seems that the error is not specific to kubelogin and a much more general problem. I am renaming the ticket as it has little to do with kubelogin and Azure.
I found this in the logs:
```
{{refresh_config.py:73}} ERROR - __init__() missing 1 required positional argument: 'cwd'
```
I am using the following version of `kubernetes` client:
```
kubernetes 23.3.0
```
Here's the relevant diff in the upstream:
https://github.com/kubernetes-client/python/commit/1c5bf586f0882c81c03181588830887345703ca5
So it seems like this API breaks in [v22.6.0](https://github.com/kubernetes-client/python/releases/tag/v22.6.0).
Changing
https://github.com/apache/airflow/blob/ee9049c0566b2539a247687de05f9cffa008f871/airflow/kubernetes/refresh_config.py#L61
to
```python
status = ExecProvider(self._user['exec'], os.getcwd()).run()
```
resolves the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #22560: KubernetesExecutor does not work with exec authentication on kubernetes client >= 22.6.0
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #22560:
URL: https://github.com/apache/airflow/issues/22560#issuecomment-1080831612
I'll also add, you should be using constraints, which gets you on versions all tested together:
https://github.com/apache/airflow/blob/constraints-2.2.4/constraints-3.7.txt
https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#constraints-files
That would have gotten you `11.0.0` and `3.0.2` which would work together as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org