You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/14 22:51:05 UTC
[GitHub] [airflow] Coqueiro opened a new issue #13076: Getting "Invalid kube-config file" when trying to run SparkKubernetesOperator
Coqueiro opened a new issue #13076:
URL: https://github.com/apache/airflow/issues/13076
**Apache Airflow version**: 1.10.12 with Python3.7
**Kubernetes version**:
```
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-27T00:38:11Z", GoVersion:"go1.14.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:04:18Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
```
I'm running an Airflow cluster using CeleryExecutor inside a Kubernetes cluster, after installing `cncf.kubernetes` backport package. I already did some testing with the Spark Operator inside the cluster and I'm able to run `SparkApplications` smoothly by applying them with `kubectl`. I'm trying now to make an Airflow DAG execute them. When doing so, I ran into this message:
```
[2020-12-14 22:18:10,609] {taskinstance.py:1150} ERROR - Invalid kube-config file. No configuration found.
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
result = task_copy.execute(context=context)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py", line 67, in execute
namespace=self.namespace,
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 127, in create_custom_object
api = client.CustomObjectsApi(self.api_client)
File "/home/airflow/.local/lib/python3.7/site-packages/cached_property.py", line 35, in __get__
value = obj.__dict__[self.func.__name__] = self.func(obj)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 108, in api_client
return self.get_conn()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 102, in get_conn
config.load_kube_config(client_configuration=self.client_configuration)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/config/kube_config.py", line 739, in load_kube_config
persist_config=persist_config)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/config/kube_config.py", line 701, in _get_kube_config_loader_for_yaml_file
'Invalid kube-config file. '
kubernetes.config.config_exception.ConfigException: Invalid kube-config file. No configuration found.
```
Here's my DAG code:
```
import os
from datetime import datetime, timedelta
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.spark_kubernetes import SparkKubernetesOperator
from airflow.operators.dummy_operator import DummyOperator
DAG_ID = "spark_operator_test"
DESCRIPTION = ""
SCHEDULE_INTERVAL = "0 10 * * *"
default_args = {
"owner": "airflow",
"depends_on_past": False,
"start_date": datetime(2020, 11, 18),
"retries": 1,
"retry_delay": timedelta(minutes=5),
"provide_context": True
}
with DAG(
dag_id=DAG_ID,
schedule_interval=SCHEDULE_INTERVAL,
description=DESCRIPTION,
default_args=default_args,
catchup=False,
concurrency=1
) as dag:
end_dag = DummyOperator(task_id='end_dag')
t1 = SparkKubernetesOperator(
task_id='spark_operator_execute',
namespace="analytics-airflow",
application_file="resources/spark/spark-operator-test.yaml",
kubernetes_conn_id="kubernetes_default",
do_xcom_push=True
)
t1 >> end_dag
```
I tried creating a connection with the following extra:
```
{"extra__kubernetes__in_cluster":True}
```
But it didn't work either. How can I correctly configure my airflow deployment so that I can make a task that can use the `SparkKubernetesOperator`, creating a `SparkApplication` within the same cluster?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Iamcerba commented on issue #13076: Getting "Invalid kube-config file" when trying to run SparkKubernetesOperator
Posted by GitBox <gi...@apache.org>.
Iamcerba commented on issue #13076:
URL: https://github.com/apache/airflow/issues/13076#issuecomment-765204373
@Coqueiro it actually not completely correct, but will work in your case.
Underneath this connection string converted to dict using "json.loads" and eventually in the code the following check applied:
```
in_cluster = extras.get("extra__kubernetes__in_cluster")
if in_cluster:
self.log.debug("loading kube_config from: in_cluster configuration")
config.load_incluster_config()
return client.ApiClient()
```
If you provide any string in the value of {"extra__kubernetes__in_cluster":"<any non empty string>"} you will enter IF clause as soon as bool("<any non empty string>") == True.
The correct connection string will be:
```
{"extra__kubernetes__in_cluster":true}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Coqueiro closed issue #13076: Getting "Invalid kube-config file" when trying to run SparkKubernetesOperator
Posted by GitBox <gi...@apache.org>.
Coqueiro closed issue #13076:
URL: https://github.com/apache/airflow/issues/13076
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Coqueiro commented on issue #13076: Getting "Invalid kube-config file" when trying to run SparkKubernetesOperator
Posted by GitBox <gi...@apache.org>.
Coqueiro commented on issue #13076:
URL: https://github.com/apache/airflow/issues/13076#issuecomment-744895585
It worked once I changed the conn to the following extra:
```
{"extra__kubernetes__in_cluster":"True"}
```
It seems that the `True` on the json I originally used isn't deserialized correctly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #13076: Getting "Invalid kube-config file" when trying to run SparkKubernetesOperator
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #13076:
URL: https://github.com/apache/airflow/issues/13076#issuecomment-744761189
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Iamcerba edited a comment on issue #13076: Getting "Invalid kube-config file" when trying to run SparkKubernetesOperator
Posted by GitBox <gi...@apache.org>.
Iamcerba edited a comment on issue #13076:
URL: https://github.com/apache/airflow/issues/13076#issuecomment-765204373
@Coqueiro it actually not completely correct, but will work in your case.
Underneath this connection string converted to dict using "json.loads" and eventually in the code the following check applied:
```
in_cluster = extras.get("extra__kubernetes__in_cluster")
if in_cluster:
self.log.debug("loading kube_config from: in_cluster configuration")
config.load_incluster_config()
return client.ApiClient()
```
If you provide any string in the value of {"extra__kubernetes__in_cluster":"<any non empty string>"} you will enter IF clause as soon as bool("<any non empty string>") == True.
The correct connection string will be:
```
{"extra__kubernetes__in_cluster":true}
```
And I personally use the following version of the config:
```
{"extra__kubernetes__in_cluster": true, "extra__kubernetes__kube_config_path": "", "extra__kubernetes__kube_config": "", "extra__kubernetes__namespace": "spark-jobs"}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org