You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "zchunhai (via GitHub)" <gi...@apache.org> on 2024/09/04 11:24:27 UTC

[I] airflow-scheduler deadlock and crash [airflow]

zchunhai opened a new issue, #41992:
URL: https://github.com/apache/airflow/issues/41992

   ### Apache Airflow version
   
   2.10.0
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   [2024-09-04T16:08:33.735+0800] {kubernetes_executor.py:356} INFO - Changing state of (TaskInstanceKey(dag_id='7402a0ed-4746-44b6-9a91-5e1682f32cf0', task_id='table_counts-child-3', run_id='scheduled__2024-09-03T08:03:23.150918+00:00', try_number=1, map_index=-1), None, '7402a0ed-4746-44b6-9a91-5e1682f32cf0-table-counts-child-3-p3lreggg', 'airflow', '41210824') to None
   [2024-09-04T16:08:33.741+0800] {kubernetes_executor.py:441} INFO - Deleted pod: TaskInstanceKey(dag_id='7402a0ed-4746-44b6-9a91-5e1682f32cf0', task_id='table_counts-child-3', run_id='scheduled__2024-09-03T08:03:23.150918+00:00', try_number=1, map_index=-1) in namespace airflow
   [2024-09-04T16:08:33.744+0800] {kubernetes_executor.py:360} ERROR - Exception: None is not a valid TaskInstanceState when attempting to change state of (TaskInstanceKey(dag_id='7402a0ed-4746-44b6-9a91-5e1682f32cf0', task_id='table_counts-child-3', run_id='scheduled__2024-09-03T08:03:23.150918+00:00', try_number=1, map_index=-1), None, '7402a0ed-4746-44b6-9a91-5e1682f32cf0-table-counts-child-3-p3lreggg', 'airflow', '41210824') to None, re-queueing.
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 358, in sync
       self._change_state(key, state, pod_name, namespace)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/session.py", line 97, in wrapper
       return func(*args, session=session, **kwargs)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 456, in _change_state
       state = TaskInstanceState(state)
     File "/usr/local/lib/python3.10/enum.py", line 385, in __call__
       return cls.__new__(cls, value)
     File "/usr/local/lib/python3.10/enum.py", line 710, in __new__
       raise ve_exc
   ValueError: None is not a valid TaskInstanceState
   [2024-09-04T16:08:33.746+0800] {kubernetes_executor.py:356} INFO - Changing state of (TaskInstanceKey(dag_id='7402a0ed-4746-44b6-9a91-5e1682f32cf0', task_id='table_counts-child-3', run_id='scheduled__2024-09-03T08:03:23.150918+00:00', try_number=1, map_index=-1), None, '7402a0ed-4746-44b6-9a91-5e1682f32cf0-table-counts-child-3-p3lreggg', 'airflow', '41210824') to None
   [2024-09-04T16:08:33.747+0800] {scheduler_job_runner.py:260} INFO - Exiting gracefully upon receiving signal 15
   [2024-09-04T16:08:34.750+0800] {process_utils.py:132} INFO - Sending Signals.SIGTERM to group 2540. PIDs of all processes in the group: [20146, 20284, 2540]
   [2024-09-04T16:08:34.750+0800] {process_utils.py:87} INFO - Sending the signal Signals.SIGTERM to group 2540
   [2024-09-04T16:08:37.565+0800] {process_utils.py:266} INFO - Waiting up to 5 seconds for processes to exit...
   [2024-09-04T16:08:37.572+0800] {process_utils.py:80} INFO - Process psutil.Process(pid=20146, status='terminated', started='16:08:33') (20146) terminated with exit code None
   [2024-09-04T16:08:37.586+0800] {process_utils.py:266} INFO - Waiting up to 5 seconds for processes to exit...
   [2024-09-04T16:08:37.586+0800] {process_utils.py:80} INFO - Process psutil.Process(pid=20284, status='terminated', started='16:08:34') (20284) terminated with exit code None
   [2024-09-04T16:08:37.613+0800] {process_utils.py:80} INFO - Process psutil.Process(pid=2540, status='terminated', exitcode=0, started='16:04:42') (2540) terminated with exit code 0
   [2024-09-04T16:08:37.615+0800] {scheduler_job_runner.py:1001} ERROR - Exception when executing SchedulerJob._run_scheduler_loop
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 358, in sync
       self._change_state(key, state, pod_name, namespace)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/session.py", line 97, in wrapper
       return func(*args, session=session, **kwargs)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 440, in _change_state
       self.kube_scheduler.delete_pod(pod_name=pod_name, namespace=namespace)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py", line 416, in delete_pod
       body=client.V1DeleteOptions(**self.kube_config.delete_option_kwargs),
     File "/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/models/v1_delete_options.py", line 58, in __init__
       local_vars_configuration = Configuration()
     File "/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/configuration.py", line 126, in __init__
       self.debug = False
     File "/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/configuration.py", line 271, in debug
       logger.setLevel(logging.WARNING)
     File "/usr/local/lib/python3.10/logging/__init__.py", line 1453, in setLevel
       self.manager._clear_cache()
     File "/usr/local/lib/python3.10/logging/__init__.py", line 1412, in _clear_cache
       logger._cache.clear()
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/scheduler_job_runner.py", line 263, in _exit_gracefully
       sys.exit(os.EX_OK)
   SystemExit: 0
   
   During handling of the above exception, another exception occurred:
   
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/scheduler_job_runner.py", line 984, in _execute
       self._run_scheduler_loop()
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/scheduler_job_runner.py", line 1127, in _run_scheduler_loop
       executor.heartbeat()
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/traces/tracer.py", line 58, in wrapper
       return func(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/base_executor.py", line 247, in heartbeat
       self.sync()
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 368, in sync
       self.result_queue.task_done()
     File "<string>", line 2, in task_done
     File "/usr/local/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod
       conn.send((self._id, methodname, args, kwds))
     File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 206, in send
       self._send_bytes(_ForkingPickler.dumps(obj))
     File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes
       self._send(header + buf)
     File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 368, in _send
       n = write(self._handle, buf)
   BrokenPipeError: [Errno 32] Broken pipe
   [2024-09-04T16:08:37.619+0800] {kubernetes_executor.py:695} INFO - Shutting down Kubernetes executor
   [2024-09-04T16:08:37.620+0800] {scheduler_job_runner.py:1008} ERROR - Exception when executing Executor.end on KubernetesExecutor(parallelism=32)
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 358, in sync
       self._change_state(key, state, pod_name, namespace)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/session.py", line 97, in wrapper
       return func(*args, session=session, **kwargs)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 440, in _change_state
       self.kube_scheduler.delete_pod(pod_name=pod_name, namespace=namespace)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py", line 416, in delete_pod
       body=client.V1DeleteOptions(**self.kube_config.delete_option_kwargs),
     File "/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/models/v1_delete_options.py", line 58, in __init__
       local_vars_configuration = Configuration()
     File "/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/configuration.py", line 126, in __init__
       self.debug = False
     File "/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/configuration.py", line 271, in debug
       logger.setLevel(logging.WARNING)
     File "/usr/local/lib/python3.10/logging/__init__.py", line 1453, in setLevel
       self.manager._clear_cache()
     File "/usr/local/lib/python3.10/logging/__init__.py", line 1412, in _clear_cache
       logger._cache.clear()
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/scheduler_job_runner.py", line 263, in _exit_gracefully
       sys.exit(os.EX_OK)
   SystemExit: 0
   
   During handling of the above exception, another exception occurred:
   
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/scheduler_job_runner.py", line 984, in _execute
       self._run_scheduler_loop()
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/scheduler_job_runner.py", line 1127, in _run_scheduler_loop
       executor.heartbeat()
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/traces/tracer.py", line 58, in wrapper
       return func(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/base_executor.py", line 247, in heartbeat
       self.sync()
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 368, in sync
       self.result_queue.task_done()
     File "<string>", line 2, in task_done
     File "/usr/local/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod
       conn.send((self._id, methodname, args, kwds))
     File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 206, in send
       self._send_bytes(_ForkingPickler.dumps(obj))
     File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes
       self._send(header + buf)
     File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 368, in _send
       n = write(self._handle, buf)
   BrokenPipeError: [Errno 32] Broken pipe
   
   During handling of the above exception, another exception occurred:
   
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/scheduler_job_runner.py", line 1006, in _execute
       executor.end()
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 698, in end
       self._flush_task_queue()
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 654, in _flush_task_queue
       self.log.debug("Executor shutting down, task_queue approximate size=%d", self.task_queue.qsize())
     File "<string>", line 2, in qsize
     File "/usr/local/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod
       conn.send((self._id, methodname, args, kwds))
     File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 206, in send
       self._send_bytes(_ForkingPickler.dumps(obj))
     File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes
       self._send(header + buf)
     File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 368, in _send
       n = write(self._handle, buf)
   BrokenPipeError: [Errno 32] Broken pipe
   [2024-09-04T16:08:37.624+0800] {process_utils.py:132} INFO - Sending Signals.SIGTERM to group 2540. PIDs of all processes in the group: []
   [2024-09-04T16:08:37.625+0800] {process_utils.py:87} INFO - Sending the signal Signals.SIGTERM to group 2540
   [2024-09-04T16:08:37.625+0800] {process_utils.py:101} INFO - Sending the signal Signals.SIGTERM to process 2540 as process group is missing.
   [2024-09-04T16:08:37.626+0800] {scheduler_job_runner.py:1014} INFO - Exited execute loop
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 358, in sync
       self._change_state(key, state, pod_name, namespace)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/session.py", line 97, in wrapper
       return func(*args, session=session, **kwargs)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 440, in _change_state
       self.kube_scheduler.delete_pod(pod_name=pod_name, namespace=namespace)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py", line 416, in delete_pod
       body=client.V1DeleteOptions(**self.kube_config.delete_option_kwargs),
     File "/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/models/v1_delete_options.py", line 58, in __init__
       local_vars_configuration = Configuration()
     File "/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/configuration.py", line 126, in __init__
       self.debug = False
     File "/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/configuration.py", line 271, in debug
       logger.setLevel(logging.WARNING)
     File "/usr/local/lib/python3.10/logging/__init__.py", line 1453, in setLevel
       self.manager._clear_cache()
     File "/usr/local/lib/python3.10/logging/__init__.py", line 1412, in _clear_cache
       logger._cache.clear()
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/scheduler_job_runner.py", line 263, in _exit_gracefully
       sys.exit(os.EX_OK)
   SystemExit: 0
   
   During handling of the above exception, another exception occurred:
   
   Traceback (most recent call last):
     File "/home/airflow/.local/bin/airflow", line 8, in <module>
       sys.exit(main())
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/__main__.py", line 62, in main
       args.func(args)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/cli/cli_config.py", line 49, in command
       return func(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/cli.py", line 115, in wrapper
       return f(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
       return func(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 59, in scheduler
       run_command_with_daemon_option(
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/cli/commands/daemon_utils.py", line 86, in run_command_with_daemon_option
       callback()
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 62, in <lambda>
       callback=lambda: _run_scheduler_job(args),
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 48, in _run_scheduler_job
       run_job(job=job_runner.job, execute_callable=job_runner._execute)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/session.py", line 97, in wrapper
       return func(*args, session=session, **kwargs)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/job.py", line 421, in run_job
       return execute_job(job, execute_callable=execute_callable)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/job.py", line 450, in execute_job
       ret = execute_callable()
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/scheduler_job_runner.py", line 984, in _execute
       self._run_scheduler_loop()
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/scheduler_job_runner.py", line 1127, in _run_scheduler_loop
       executor.heartbeat()
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/traces/tracer.py", line 58, in wrapper
       return func(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/base_executor.py", line 247, in heartbeat
       self.sync()
     File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 368, in sync
       self.result_queue.task_done()
     File "<string>", line 2, in task_done
     File "/usr/local/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod
       conn.send((self._id, methodname, args, kwds))
     File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 206, in send
       self._send_bytes(_ForkingPickler.dumps(obj))
     File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes
       self._send(header + buf)
     File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 368, in _send
       n = write(self._handle, buf)
   BrokenPipeError: [Errno 32] Broken pipe
   
   ### What you think should happen instead?
   
   _No response_
   
   ### How to reproduce
   
   Use the Kubernetes scheduler of Airflow to execute a DAG that includes a PythonOperator.
   
   
   ### Operating System
   
   Centos 7.9
   
   ### Versions of Apache Airflow Providers
   
   apache_airflow_providers_cncf_kubernetes-7.8.0
   
   ### Deployment
   
   Other 3rd-party Helm chart
   
   ### Deployment details
   
   https://github.com/open-metadata/openmetadata-helm-charts
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] airflow-scheduler deadlock and crash [airflow]

Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #41992:
URL: https://github.com/apache/airflow/issues/41992#issuecomment-2329283057

   This was fixed in https://github.com/apache/airflow/pull/35891 and released as part of apache_airflow_providers_cncf_kubernetes-7.11.0 . Please upgrade the provider package and test it.
   
   https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/7.11.0/changelog.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] airflow-scheduler deadlock and crash [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #41992:
URL: https://github.com/apache/airflow/issues/41992#issuecomment-2328665126

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] airflow-scheduler deadlock and crash [airflow]

Posted by "zchunhai (via GitHub)" <gi...@apache.org>.
zchunhai closed issue #41992: airflow-scheduler deadlock and crash
URL: https://github.com/apache/airflow/issues/41992


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org