[GitHub] [airflow] seancaulfield commented on issue #18989: CeleryKubernetesExecutor does not shutdown gracefully in 2.2.0

[GitBox <gi...@apache.org>]

seancaulfield commented on issue #18989:
URL: https://github.com/apache/airflow/issues/18989#issuecomment-973280537


   We also have this issue, but with the `KubernetesExecutor`. Airflow version is 2.1.4. Seems to happen whenever the scheduler is being restarted by the deployment.
   
   Here is the stack trace, but it's likely a little jumbled as I had to pull it from DataDog and the lines don't always arrive in order there:
   
   ```
   [2021-11-18 17:24:30,593] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 60
   [2021-11-18 17:24:30,593] {scheduler_job.py:722} INFO - Exited execute loop
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 552, in sync
       self._change_state(key, state, pod_id, namespace)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 634, in _change_state
       self.kube_scheduler.delete_pod(pod_id, namespace)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 340, in delete_pod
       **self.kube_config.kube_client_request_args,
     File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 10173, in delete_namespaced_pod
       (data) = self.delete_namespaced_pod_with_http_info(name, namespace, **kwargs)  # noqa: E501
     File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 10273, in delete_namespaced_pod_with_http_info
       collection_formats=collection_formats)
     File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 345, in call_api
       _preload_content, _request_timeout)
     File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 176, in __call_api
       _request_timeout=_request_timeout)
     File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 411, in request
       body=body)
     File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 268, in DELETE
       body=body)
     File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 170, in request
       headers=headers)
     File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/request.py", line 75, in request
       method, url, fields=fields, headers=headers, **urlopen_kw
     File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/request.py", line 96, in request_encode_url
       return self.urlopen(method, url, **extra_kw)
     File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/poolmanager.py", line 375, in urlopen
       response = conn.urlopen(method, u.request_uri, **kw)
     File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
       chunked=chunked,
     File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request
       six.raise_from(e, None)
     File "<string>", line 3, in raise_from
     File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request
       httplib_response = conn.getresponse()
     File "/usr/local/lib/python3.7/http/client.py", line 1373, in getresponse
       response.begin()
     File "/usr/local/lib/python3.7/http/client.py", line 319, in begin
       version, status, reason = self._read_status()
     File "/usr/local/lib/python3.7/http/client.py", line 280, in _read_status
       line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
     File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
       return self._sock.recv_into(b)
     File "/usr/local/lib/python3.7/ssl.py", line 1071, in recv_into
       return self.read(nbytes, buffer)
     File "/usr/local/lib/python3.7/ssl.py", line 929, in read
       return self._sslobj.read(len, buffer)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 155, in _exit_gracefully
       sys.exit(os.EX_OK)
   SystemExit: 0
   
   During handling of the above exception, another exception occurred:
   Traceback (most recent call last):
     File "/home/airflow/.local/bin/airflow", line 8, in <module>
       sys.exit(main())
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main
       args.func(args)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
       return func(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 92, in wrapper
       return f(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/scheduler_command.py", line 70, in scheduler
       job.run()
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 245, in run
       self._execute()
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 695, in _execute
       self._run_scheduler_loop()
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 790, in _run_scheduler_loop
       self.executor.heartbeat()
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/base_executor.py", line 162, in heartbeat
       self.sync()
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 562, in sync
       self.result_queue.task_done()
     File "<string>", line 2, in task_done
     File "/usr/local/lib/python3.7/multiprocessing/managers.py", line 818, in _callmethod
       conn.send((self._id, methodname, args, kwds))
     File "/usr/local/lib/python3.7/multiprocessing/connection.py", line 206, in send
       self._send_bytes(_ForkingPickler.dumps(obj))
     File "/usr/local/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
       self._send(header + buf)
     File "/usr/local/lib/python3.7/multiprocessing/connection.py", line 368, in _send
       n = write(self._handle, buf)
   BrokenPipeError: [Errno 32] Broken pipe
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org