You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/09 11:19:04 UTC

[GitHub] [airflow] jpatel-fpmx opened a new issue #16909: Unable to connect to Kubernetes cluster from Airflow

jpatel-fpmx opened a new issue #16909:
URL: https://github.com/apache/airflow/issues/16909

<!--

Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)

If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>

Please delete these comment blocks before submitting the issue.

-->

**Apache Airflow version**: v2.1.1

**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):-
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:52:14Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.8-eks-96780e", GitCommit:"96780e1b30acbf0a52c38b6030d7853e575bcdf3", GitTreeState:"clean", BuildDate:"2021-03-10T21:32:29Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}

**Environment**: Development

- **Cloud provider or hardware configuration**: AWS
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:

**What happened**:
I am unable to connect to my Kubernetes cluster from Airflow while using SparkKubernetesOperator. I have hosted Airflow and Spark-operator on EKS. I have created a connection on Airflow to connect to kubernetes cluster by using "in cluster configuration".
I am just running the sample application just to check the execution of spark on Kubernetes through Airflow. Please find below the sample application link:-
https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes.py

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"sparkapplications.sparkoperator.k8s.io is forbidden: User \"system:serviceaccount:***:***-cluster\" cannot create resource \"sparkapplications\" in API group \"sparkoperator.k8s.io\" in the namespace \"spark-operator\"","reason":"Forbidden","details":{"group":"sparkoperator.k8s.io","kind":"sparkapplications"},"code":403}

**What you expected to happen**:
Kubernetes Airflow should schedule and run spark job using spark operator.

The Airflow Kubernetes connection is not working.

**How to reproduce it**:
1. Deploy Spark operator using helm on Kubernetes cluster.
2. Deploy Airflow using helm on Kubernetes cluster.
3. Deploy below application by writing Airflow DAG:-
https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes.py

**Anything else we need to know**:

<!--

How often does this problem occur? Once? Every time etc? Everytime

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"sparkapplications.sparkoperator.k8s.io is forbidden: User \"system:serviceaccount:***:***-cluster\" cannot create resource \"sparkapplications\" in API group \"sparkoperator.k8s.io\" in the namespace \"spark-operator\"","reason":"Forbidden","details":{"group":"sparkoperator.k8s.io","kind":"sparkapplications"},"code":403}

-->

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org