You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/26 14:41:19 UTC
[GitHub] [airflow] bparhy commented on issue #18041: Tasks intermittently gets terminated with SIGTERM on kubernetes executor
bparhy commented on issue #18041:
URL: https://github.com/apache/airflow/issues/18041#issuecomment-1022261032
I am seeing similar issue in our Airflow with kubernetes environments.
Airflow Version : 2.1.3
Kubernetes Version : v1.20.5
What Happens
This happens intermittently.
I am seeing tasks are not deleting at the kubernetes side. Even though they are completed successfully in the UI they have an Error status at the pod.
We have 1000s of task and this happens only for couple of tasks so far.
Tasks Log:
[2022-01-26 00:51:50,823] {local_task_job.py:209} WARNING - State of this instance has been externally set to success. Terminating instance.
[2022-01-26 00:51:50,825] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 55
[2022-01-26 00:51:50,827] {taskinstance.py:1235} ERROR - Received SIGTERM. Terminating subprocesses.
[2022-01-26 00:51:52,468] {process_utils.py:66} INFO - Process psutil.Process(pid=55, status='terminated', exitcode=1, started='00:51:45') (55) terminated with exit code 1
Task Pod Log
[2022-01-26 00:51:37,973] {dagbag.py:496} INFO - Filling up the DagBag from /usr/local/airflow/dags/name/name1/name2/name3_dag.py
/usr/local/lib/python3.6/site-packages/sqlalchemy/ext/declarative/clsregistry.py:129 SAWarning: This declarative base already contains a class with the same class name and module name as bi_plugin.DagRun, and will be replaced in the string-lookup table.
Running <TaskInstance: dagname.task_name 2022-01-25T00:50:00+00:00 [queued]> on host dagnametaskname.457ebbfa477643b289bceef4b0fbc1ab
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/usr/local/lib/python3.6/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/airflow/utils/cli.py", line 91, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/airflow/cli/commands/task_command.py", line 238, in task_run
_run_task_by_selected_method(args, dag, ti)
File "/usr/local/lib/python3.6/site-packages/airflow/cli/commands/task_command.py", line 64, in _run_task_by_selected_method
_run_task_by_local_task_job(args, ti)
File "/usr/local/lib/python3.6/site-packages/airflow/cli/commands/task_command.py", line 121, in _run_task_by_local_task_job
run_job.run()
File "/usr/local/lib/python3.6/site-packages/airflow/jobs/base_job.py", line 245, in run
self._execute()
File "/usr/local/lib/python3.6/site-packages/airflow/jobs/local_task_job.py", line 100, in _execute
self.task_runner.start()
File "/usr/local/lib/python3.6/site-packages/airflow/task/task_runner/standard_task_runner.py", line 41, in start
self.process = self._start_by_fork()
File "/usr/local/lib/python3.6/site-packages/airflow/task/task_runner/standard_task_runner.py", line 92, in _start_by_fork
logging.shutdown()
File "/usr/local/lib/python3.6/logging/__init__.py", line 1946, in shutdown
h.close()
File "/usr/local/lib/python3.6/logging/__init__.py", line 1048, in close
stream.close()
File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1237, in signal_handler
raise AirflowException("Task received SIGTERM signal")
airflow.exceptions.AirflowException: Task received SIGTERM signal
[2022-01-26 00:51:51,249] {connection.py:499} INFO - closed
[2022-01-26 00:51:51,249] {connection.py:502} INFO - No async queries seem to be running, deleting session
Running <TaskInstance: dag_name.task_name 2022-01-25T00:50:00+00:00 [queued]> on host dagnametaskname.457ebbfa477643b289bceef4b0fbc1ab
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/usr/local/lib/python3.6/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/airflow/utils/cli.py", line 91, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/airflow/cli/commands/task_command.py", line 238, in task_run
_run_task_by_selected_method(args, dag, ti)
File "/usr/local/lib/python3.6/site-packages/airflow/cli/commands/task_command.py", line 64, in _run_task_by_selected_method
_run_task_by_local_task_job(args, ti)
File "/usr/local/lib/python3.6/site-packages/airflow/cli/commands/task_command.py", line 121, in _run_task_by_local_task_job
run_job.run()
File "/usr/local/lib/python3.6/site-packages/airflow/jobs/base_job.py", line 245, in run
self._execute()
File "/usr/local/lib/python3.6/site-packages/airflow/jobs/local_task_job.py", line 145, in _execute
self.on_kill()
File "/usr/local/lib/python3.6/site-packages/airflow/jobs/local_task_job.py", line 171, in on_kill
self.task_runner.on_finish()
File "/usr/local/lib/python3.6/site-packages/airflow/task/task_runner/base_task_runner.py", line 178, in on_finish
self._error_file.close()
File "/usr/local/lib/python3.6/tempfile.py", line 511, in close
self._closer.close()
File "/usr/local/lib/python3.6/tempfile.py", line 448, in close
unlink(self.name)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpfabi4n96'
I have not set configuration params and all have default value. I see a CPU spike but unable to relate it.
Thanks in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org