You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Sunny Gupta (JIRA)" <ji...@apache.org> on 2018/10/24 20:37:00 UTC
[jira] [Created] (AIRFLOW-3253) KubernetesPodOperator Unauthorized
Code 401
Sunny Gupta created AIRFLOW-3253:
------------------------------------
Summary: KubernetesPodOperator Unauthorized Code 401
Key: AIRFLOW-3253
URL: https://issues.apache.org/jira/browse/AIRFLOW-3253
Project: Apache Airflow
Issue Type: Bug
Components: authentication, gcp, kubernetes
Affects Versions: 1.10.0
Reporter: Sunny Gupta
apache-airflow==1.10.0
kubernetes==7.0.0 (Tried)
kubernetes==8.0.0b1 (Tried)
Everytime after couple of successful scheduled runs, some runs failed and throw below error.
Error looks related to k8s authorization and it seems like a pattern in my case, everytime expiry comes near, job fails and after new expiry updates it runs for a while and fails.
Above speculation could be wrong, need help to fix this issue. I am running one sample python hello DAG and planning to move production workload but this is blocker for me.
Tried :(
* ~/.kube folder clearing and regenerate token by `gcloud container clusters get-credentials ***` even tried setting as cron to force update tokens.
* Tried kubernetes==7.0.0 to latest beta version.
Below is my kubectl config. When I run *kubectl* cli to do GET ops on pods, nodes resources,no issues.
{code:java}
$ kubectl config view
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://XX.XX.XX.XX
name: gke_us-central1-b_dev-kube-cluster
contexts:
- context:
cluster: gke_us-central1-b_dev-kube-cluster
user: gke_us-central1-b_dev-kube-cluster
name: gke_us-central1-b_dev-kube-cluster
current-context: gke_us-central1-b_dev-kube-cluster
kind: Config
preferences: {}
users:
- name: gke_us-central1-b_dev-kube-cluster
user:
auth-provider:
config:
access-token: ya29.c.****TOKEN****5EREdigv
cmd-args: config config-helper --format=json
cmd-path: /usr/lib/google-cloud-sdk/bin/gcloud
expiry: 2018-10-24T20:54:37Z
expiry-key: '{.credential.token_expiry}'
token-key: '{.credential.access_token}'
name: gcp
{code}
In an hour, running every */5 min, 2-3 jobs fails. with below error.
{code:java}
kubernetes.client.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04 GMT', 'Content-Length': '129', 'Audit-Id': '89dcda61-a60f-4b23-85d6-9d28a6bfeed0', 'Www-Authenticate': 'Basic realm="kubernetes-master"', 'Content-Type': 'application/json'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}{code}
{code:java}
// complete logs
====================================================
*** Log file does not exist: /root/airflow/logs/pyk8s.v3/python-hello/2018-10-24T06:16:00+00:00/1.log
*** Fetching from: http://aflow-worker.internal:8793/log/pyk8s.v3/python-hello/2018-10-24T06:16:00+00:00/1.log
[2018-10-24 06:20:02,947] {models.py:1335} INFO - Dependencies all met for <TaskInstance: pyk8s.v3.python-hello 2018-10-24T06:16:00+00:00 [queued]>
[2018-10-24 06:20:02,952] {models.py:1335} INFO - Dependencies all met for <TaskInstance: pyk8s.v3.python-hello 2018-10-24T06:16:00+00:00 [queued]>
[2018-10-24 06:20:02,952] {models.py:1547} INFO -
--------------------------------------------------------------------------------
Starting attempt 1 of 1
--------------------------------------------------------------------------------
[2018-10-24 06:20:02,966] {models.py:1569} INFO - Executing <Task(KubernetesPodOperator): python-hello> on 2018-10-24T06:16:00+00:00
[2018-10-24 06:20:02,967] {base_task_runner.py:124} INFO - Running: ['bash', '-c', 'airflow run pyk8s.v3 python-hello 2018-10-24T06:16:00+00:00 --job_id 354 --raw -sd DAGS_FOLDER/pyk8s.v3.py --cfg_path /tmp/tmpf0saygt7']
[2018-10-24 06:20:03,405] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello [2018-10-24 06:20:03,404] {settings.py:174} INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
[2018-10-24 06:20:03,808] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello [2018-10-24 06:20:03,807] {__init__.py:51} INFO - Using executor CeleryExecutor
[2018-10-24 06:20:03,970] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello [2018-10-24 06:20:03,970] {models.py:258} INFO - Filling up the DagBag from /root/airflow/dags/pyk8s.v3.py
[2018-10-24 06:20:04,255] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello [2018-10-24 06:20:04,255] {cli.py:492} INFO - Running <TaskInstance: pyk8s.v3.python-hello 2018-10-24T06:16:00+00:00 [running]> on host aflow-worker.c.internal
[2018-10-24 06:20:04,337] {logging_mixin.py:95} INFO - [2018-10-24 06:20:04,332] {pod_launcher.py:58} ERROR - Exception when attempting to create Namespaced Pod.
Traceback (most recent call last):
File "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py", line 55, in run_pod_async
resp = self._client.create_namespaced_pod(body=req, namespace=pod.namespace)
File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 6057, in create_namespaced_pod
(data) = self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)
File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 6142, in create_namespaced_pod_with_http_info
collection_formats=collection_formats)
File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 321, in call_api
_return_http_data_only, collection_formats, _preload_content, _request_timeout)
File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 155, in __call_api
_request_timeout=_request_timeout)
File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 364, in request
body=body)
File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py", line 266, in POST
body=body)
File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py", line 222, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04 GMT', 'Content-Length': '129', 'Audit-Id': '89dcda61-a60f-4b23-85d6-9d45a6bfeed0', 'Www-Authenticate': 'Basic realm="kubernetes-master"', 'Content-Type': 'application/json'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
[2018-10-24 06:20:04,337] {logging_mixin.py:95} INFO -
[2018-10-24 06:20:04,337] {models.py:1736} ERROR - (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04 GMT', 'Content-Length': '129', 'Audit-Id': '89dcda61-a60f-4b23-85d6-9d45a6bfeed0', 'Www-Authenticate': 'Basic realm="kubernetes-master"', 'Content-Type': 'application/json'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
Traceback (most recent call last):
File "/root/airflow/venv/lib/python3.5/site-packages/airflow/models.py", line 1633, in _run_raw_task
result = task_copy.execute(context=context)
File "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 115, in execute
get_logs=self.get_logs)
File "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py", line 71, in run_pod
resp = self.run_pod_async(pod)
File "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py", line 55, in run_pod_async
resp = self._client.create_namespaced_pod(body=req, namespace=pod.namespace)
File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 6057, in create_namespaced_pod
(data) = self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)
File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 6142, in create_namespaced_pod_with_http_info
collection_formats=collection_formats)
File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 321, in call_api
_return_http_data_only, collection_formats, _preload_content, _request_timeout)
File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 155, in __call_api
_request_timeout=_request_timeout)
File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 364, in request
body=body)
File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py", line 266, in POST
body=body)
File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py", line 222, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04 GMT', 'Content-Length': '129', 'Audit-Id': '89dcda61-a60f-4b23-85d6-9d45a6bfeed0', 'Www-Authenticate': 'Basic realm="kubernetes-master"', 'Content-Type': 'application/json'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
[2018-10-24 06:20:04,339] {models.py:1764} INFO - Marking task as FAILED.
[2018-10-24 06:20:04,493] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello Traceback (most recent call last):
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello File "/root/airflow/venv/bin/airflow", line 32, in <module>
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello args.func(args)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello File "/root/airflow/venv/lib/python3.5/site-packages/airflow/utils/cli.py", line 74, in wrapper
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello return f(*args, **kwargs)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello File "/root/airflow/venv/lib/python3.5/site-packages/airflow/bin/cli.py", line 498, in run
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello _run(args, dag, ti)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello File "/root/airflow/venv/lib/python3.5/site-packages/airflow/bin/cli.py", line 402, in _run
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello pool=args.pool,
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello File "/root/airflow/venv/lib/python3.5/site-packages/airflow/utils/db.py", line 74, in wrapper
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello return func(*args, **kwargs)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello File "/root/airflow/venv/lib/python3.5/site-packages/airflow/models.py", line 1633, in _run_raw_task
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello result = task_copy.execute(context=context)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello File "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 115, in execute
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello get_logs=self.get_logs)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello File "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py", line 71, in run_pod
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello resp = self.run_pod_async(pod)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello File "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py", line 55, in run_pod_async
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello resp = self._client.create_namespaced_pod(body=req, namespace=pod.namespace)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 6057, in create_namespaced_pod
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello (data) = self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 6142, in create_namespaced_pod_with_http_info
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello collection_formats=collection_formats)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 321, in call_api
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello _return_http_data_only, collection_formats, _preload_content, _request_timeout)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 155, in __call_api
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello _request_timeout=_request_timeout)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 364, in request
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello body=body)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py", line 266, in POST
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello body=body)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello File "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py", line 222, in request
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello raise ApiException(http_resp=r)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello kubernetes.client.rest.ApiException: (401)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello Reason: Unauthorized
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04 GMT', 'Content-Length': '129', 'Audit-Id': '89dcda61-a60f-4b23-85d6-9d45a6bfeed0', 'Www-Authenticate': 'Basic realm="kubernetes-master"', 'Content-Type': 'application/json'})
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask python-hello
[2018-10-24 06:20:07,938] {logging_mixin.py:95} INFO - [2018-10-24 06:20:07,937] {jobs.py:2612} INFO - Task exited with return code 1
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)