You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/06/29 16:28:49 UTC
[GitHub] [airflow] mdcsaenz opened a new issue #16716: Service token file does not exists in GKEStartPodOperator 2.1.0 airflow
mdcsaenz opened a new issue #16716:
URL: https://github.com/apache/airflow/issues/16716
**Apache Airflow version**:
Version: v2.1.0
Git Version: .release:2.1.0+304e174674ff6921cb7ed79c0158949b50eff8fe
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.10-gke.1600", GitCommit:"7b8e568a7fb4c9d199c2ba29a5f7d76f6b4341c2", GitTreeState:"clean", BuildDate:"2021-05-07T09:18:53Z", GoVersion:"go1.15.10b5", Compiler:"gc", Platform:"linux/amd64"}
**Environment**: Prod
- **Cloud provider or hardware configuration**: GCP/GKE , machines: n1-standard-4 (4 vCPUs, 15 GB memory)
-
- **OS** (e.g. from /etc/os-release):
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
- **Kernel** (e.g. `uname -a`):
- Linux 5.4.89+ #1 SMP Sat Feb 13 19:45:14 PST 2021 x86_64 GNU/Linux
**What happened**:
[2021-06-29 16:06:15,708] {credentials_provider.py:300} INFO - Getting connection using `google.auth.default()` since no key file is defined for hook.
[2021-06-29 16:06:18,357] {process_utils.py:135} INFO - Executing cmd: gcloud container clusters get-credentials <the_cluster_name> --zone us-central1 --project <ourproject>
[2021-06-29 16:06:18,365] {process_utils.py:139} INFO - Output:
[2021-06-29 16:06:20,643] {process_utils.py:143} INFO - Fetching cluster endpoint and auth data.
[2021-06-29 16:06:20,836] {process_utils.py:143} INFO - kubeconfig entry generated for the_cluster_name.
[2021-06-29 16:06:20,991] {taskinstance.py:1481} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1137, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
result = task_copy.execute(context=context)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/google/cloud/operators/kubernetes_engine.py", line 339, in execute
return super().execute(context)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 337, in execute
cluster_context=self.cluster_context, config_file=self.config_file
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/kubernetes/kube_client.py", line 148, in get_kube_client
client_conf = _get_kube_config(in_cluster, cluster_context, config_file)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/kubernetes/kube_client.py", line 43, in _get_kube_config
config.load_incluster_config()
File "/home/airflow/.local/lib/python3.6/site-packages/kubernetes/config/incluster_config.py", line 94, in load_incluster_config
cert_filename=SERVICE_CERT_FILENAME).load_and_set()
File "/home/airflow/.local/lib/python3.6/site-packages/kubernetes/config/incluster_config.py", line 45, in load_and_set
self._load_config()
File "/home/airflow/.local/lib/python3.6/site-packages/kubernetes/config/incluster_config.py", line 62, in _load_config
raise ConfigException("Service token file does not exists.")
kubernetes.config.config_exception.ConfigException: Service token file does not exists.
[2021-06-29 16:06:20,994] {taskinstance.py:1531} INFO - Marking task as FAILED. dag_id=example_gcp_gke, task_id=pod_task, execution_date=20210629T160613, start_date=20210629T160615, end_date=20210629T160620
[2021-06-29 16:06:21,034] {local_task_job.py:151} INFO - Task exited with return code 1
<!-- (please include exact error messages if you can) -->
**What you expected to happen**:
To actually create a pod in that specific namespace and run the perl image
<!-- What do you think went wrong? -->
In 1.10.15 version we were able to use this operator without issue when trying to create pod in the namespace. With the upgrade in 2.0, with the same parms, it keeps saying service token does not exist.
**How to reproduce it**:
`import os
from airflow import models
from airflow.operators.bash import BashOperator
from airflow.providers.google.cloud.operators.kubernetes_engine import (
GKEStartPodOperator,
)
from airflow.utils.dates import days_ago
from kubernetes.client import models as k8s
GCP_PROJECT_ID = os.environ.get("GCP_PROJECT_ID")
GCP_LOCATION = os.environ.get("GCP_GKE_LOCATION")
CLUSTER_NAME = os.environ.get("GCP_GKE_CLUSTER_NAME")
with models.DAG(
"example_gcp_gke",
schedule_interval=None, # Override to match your needs
start_date=days_ago(1),
tags=['example'],
) as dag:
pod_task = GKEStartPodOperator(
task_id="pod_task",
project_id=GCP_PROJECT_ID,
location=GCP_LOCATION,
cluster_name=CLUSTER_NAME,
namespace='default',
image="perl",
name="test-pod",
)
# [START howto_operator_gke_start_pod_xcom]
pod_task_xcom = GKEStartPodOperator(
task_id="pod_task_xcom",
project_id=GCP_PROJECT_ID,
location=GCP_LOCATION,
cluster_name=CLUSTER_NAME,
do_xcom_push=True,
namespace="default",
image="alpine",
cmds=["sh", "-c", 'mkdir -p /airflow/xcom/;echo \'[1,2,3,4]\' > /airflow/xcom/return.json'],
name="test-pod-xcom",
)
pod_task_xcom_result = BashOperator(
bash_command="echo \"{{ task_instance.xcom_pull('pod_task_xcom')[0] }}\"",
task_id="pod_task_xcom_result",
)
pod_task >> pod_task_xcom >> pod_task_xcom_result
`
**Anything else we need to know**: None
How often does this problem occur? Once? Every time etc? Every time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mdcsaenz commented on issue #16716: Service token file does not exists in GKEStartPodOperator 2.1.0 airflow
Posted by GitBox <gi...@apache.org>.
mdcsaenz commented on issue #16716:
URL: https://github.com/apache/airflow/issues/16716#issuecomment-871627345
The SA was not mounted on the worker pod in order for it to actually create the pod.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #16716: Service token file does not exists in GKEStartPodOperator 2.1.0 airflow
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #16716:
URL: https://github.com/apache/airflow/issues/16716#issuecomment-870845539
Can you try to pass in_cluster=False to GKEStartPodOperator?
https://github.com/apache/airflow/blob/4b14d93f25f237b8cca17f57352c253f292a111b/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py#L95-L96
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mdcsaenz closed issue #16716: Service token file does not exists in GKEStartPodOperator 2.1.0 airflow
Posted by GitBox <gi...@apache.org>.
mdcsaenz closed issue #16716:
URL: https://github.com/apache/airflow/issues/16716
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org