You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/11 07:27:23 UTC
[GitHub] [airflow] sakshi-bansal opened a new issue #13006: Setting in_cluster to True is failing with invalid tube config
sakshi-bansal opened a new issue #13006:
URL: https://github.com/apache/airflow/issues/13006
Airflow version: 1.10.12
Deploying the example https://github.com/apache/airflow/blob/master/airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes.py and
Added parameter **in_cluster=True** in the dag.
Expected Result:
Dag gets executed and spark application is deployed in the local cluster where airflow is deployed
Actual Result:
Error: {taskinstance.py:1150} ERROR - Invalid kube-config file. No configuration found.
Work-around:
Works after manually adding a kube config in the worker pod.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Vivekdjango commented on issue #13006: Setting in_cluster to True is failing with invalid kube config
Posted by GitBox <gi...@apache.org>.
Vivekdjango commented on issue #13006:
URL: https://github.com/apache/airflow/issues/13006#issuecomment-812850041
> > Where did you pass 'extra__kubernetes__in_cluster' param?
>
> In connection confiugraitton, see: http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow-providers-cncf-kubernetes/latest/connections/kubernetes.html
> http://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html
Thanks @mik-laj : I created the connection from UI , now how to pass it to 'SparkKubernetesOperator'?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #13006: Setting in_cluster to True is failing with invalid kube config
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #13006:
URL: https://github.com/apache/airflow/issues/13006#issuecomment-812852492
You should use `*_conn_id` parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sakshi-bansal commented on issue #13006: Setting in_cluster to True is failing with invalid kube config
Posted by GitBox <gi...@apache.org>.
sakshi-bansal commented on issue #13006:
URL: https://github.com/apache/airflow/issues/13006#issuecomment-748893169
One possible solution that worked for me:
1. Add in_cluster in the kubernetes_default connection in the extras as **{"extra__kubernetes__in_cluster": "True"}**
2. Create ClusterRole granting the serviceaccount airflow access to create spark applications
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Vivekdjango commented on issue #13006: Setting in_cluster to True is failing with invalid kube config
Posted by GitBox <gi...@apache.org>.
Vivekdjango commented on issue #13006:
URL: https://github.com/apache/airflow/issues/13006#issuecomment-812846631
> `in_cluster` isn't a kwarg for `SparkKubernetesOperator` or `SparkKubernetesSensor`. Putting `extra__kubernetes__in_cluster` in the connection extras is the right way currently.
>
> That said, `KubernetesPodOperator` does support `in_cluster` making operators in this same provider inconsistent. Makes me wonder if KPO should also use the connection like the hook does? Or, at the very least have `extra__kubernetes__in_cluster` on the default connection? @dimberman, any input?
@jedcunningham : I am new to this skill so could you please help me out? Where did you pass 'extra__kubernetes__in_cluster' param? Inside 'SparkKubernetesOperator' or under some yaml file? please reply.
As far I tried to add under 'SparkKubernetesOperator' but didn't work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13006: Setting in_cluster to True is failing with invalid kube config
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13006:
URL: https://github.com/apache/airflow/issues/13006#issuecomment-797152831
> `in_cluster` isn't a kwarg for `SparkKubernetesOperator` or `SparkKubernetesSensor`. Putting `extra__kubernetes__in_cluster` in the connection extras is the right way currently.
>
> That said, `KubernetesPodOperator` does support `in_cluster` making operators in this same provider inconsistent. Makes me wonder if KPO should also use the connection like the hook does? Or, at the very least have `extra__kubernetes__in_cluster` on the default connection? @dimberman, any input?
As long as it is consistent I am happy with either approach.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Vivekdjango commented on issue #13006: Setting in_cluster to True is failing with invalid kube config
Posted by GitBox <gi...@apache.org>.
Vivekdjango commented on issue #13006:
URL: https://github.com/apache/airflow/issues/13006#issuecomment-813194068
Your help is really appreciable. Thanks for all your guidance it resolved the issue. @mik-laj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #13006: Setting in_cluster to True is failing with invalid kube config
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #13006:
URL: https://github.com/apache/airflow/issues/13006#issuecomment-794523604
`in_cluster` isn't a kwarg for `SparkKubernetesOperator` or `SparkKubernetesOperator`. Putting `extra__kubernetes__in_cluster` in the connection extras is the right way way currently.
That said, `KubernetesPodOperator` does support `in_cluster` making operators in this same provider inconsistent. Makes me wonder if KPO should also use the connection like the hook does? Or, at the very least have `extra__kubernetes__in_cluster` on the default connection? @dimberman, any input?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Vivekdjango commented on issue #13006: Setting in_cluster to True is failing with invalid kube config
Posted by GitBox <gi...@apache.org>.
Vivekdjango commented on issue #13006:
URL: https://github.com/apache/airflow/issues/13006#issuecomment-813054034
@mik-laj : I tried but no luck, I created connection with two ways:
1)Conn_ID: <NAME>
Type:Kubernetes Cluster Connection
In Cluster Config: Checked
Kube config (JSON format): {"extra__kubernetes__in_cluster": "True"}
Error:"Invalid connection configuration. Options extra__kubernetes__kube_config_path, "
"extra__kubernetes__kube_config, extra__kubernetes__in_cluster are mutually exclusive. "
"You can only use one option at a time."
2)Conn_ID: <NAME>
Type:Kubernetes Cluster Connection
Kube config (JSON format): {"extra__kubernetes__in_cluster": "True"}
Error: Invalid kube-config file. No configuration found.
Could you please guide where I am doing wrong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #13006: Setting in_cluster to True is failing with invalid kube config
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #13006:
URL: https://github.com/apache/airflow/issues/13006#issuecomment-813054500
Kube config field should be empty. In Cluster Config should be checked
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #13006: Setting in_cluster to True is failing with invalid tube config
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #13006:
URL: https://github.com/apache/airflow/issues/13006#issuecomment-743025500
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #13006: Setting in_cluster to True is failing with invalid kube config
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #13006:
URL: https://github.com/apache/airflow/issues/13006#issuecomment-808592821
#14954 provides a little more clarity, but we still should make it consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Vivekdjango edited a comment on issue #13006: Setting in_cluster to True is failing with invalid kube config
Posted by GitBox <gi...@apache.org>.
Vivekdjango edited a comment on issue #13006:
URL: https://github.com/apache/airflow/issues/13006#issuecomment-813054034
@mik-laj : I tried but no luck, I created connection with two ways:
1)Conn_ID: <NAME>
Type:Kubernetes Cluster Connection
In Cluster Config: Checked
Kube config (JSON format): {"extra__kubernetes__in_cluster": "True"}
Error:"Invalid connection configuration. Options extra__kubernetes__kube_config_path, "
"extra__kubernetes__kube_config, extra__kubernetes__in_cluster are mutually exclusive. "
"You can only use one option at a time."
2)Conn_ID: <NAME>
Type:Kubernetes Cluster Connection
Kube config (JSON format): {"extra__kubernetes__in_cluster": "True"}
Error: Invalid kube-config file. No configuration found.
Could you please guide where I am doing wrong?
Under Operator I am passing connection as below:
kubernetes_conn_id='NAME',
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #13006: Setting in_cluster to True is failing with invalid kube config
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #13006:
URL: https://github.com/apache/airflow/issues/13006#issuecomment-812848567
> Where did you pass 'extra__kubernetes__in_cluster' param?
In connection confiugraitton, see: http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow-providers-cncf-kubernetes/latest/connections/kubernetes.html
http://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman closed issue #13006: Setting in_cluster to True is failing with invalid kube config
Posted by GitBox <gi...@apache.org>.
dimberman closed issue #13006:
URL: https://github.com/apache/airflow/issues/13006
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham edited a comment on issue #13006: Setting in_cluster to True is failing with invalid kube config
Posted by GitBox <gi...@apache.org>.
jedcunningham edited a comment on issue #13006:
URL: https://github.com/apache/airflow/issues/13006#issuecomment-794523604
`in_cluster` isn't a kwarg for `SparkKubernetesOperator` or `SparkKubernetesSensor`. Putting `extra__kubernetes__in_cluster` in the connection extras is the right way currently.
That said, `KubernetesPodOperator` does support `in_cluster` making operators in this same provider inconsistent. Makes me wonder if KPO should also use the connection like the hook does? Or, at the very least have `extra__kubernetes__in_cluster` on the default connection? @dimberman, any input?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #13006: Setting in_cluster to True is failing with invalid kube config
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #13006:
URL: https://github.com/apache/airflow/issues/13006
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org