You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/09/07 11:34:33 UTC
[GitHub] [airflow] laserpedro commented on issue #18041: Tasks intermittently gets terminated with SIGTERM on kubernetes executor
laserpedro commented on issue #18041:
URL: https://github.com/apache/airflow/issues/18041#issuecomment-914227769
Hello,
I am facing the same issue:
airflow 2.1.3 (tested also with 2.1.2)
backend: postgresql
executor: LocalExecutor
I have modified the variables killed_task_cleanup_time and schedule_after_task_execution to resp. 100000 and False.
I have also installed airflow as non root user and set the default run_as_user to be airflow.
My tasks are getting constantly killed in backfill mode with the traceback:
[2021-09-07 10:21:20,185] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 73011
Honestly, I am a bit discouraged at this point, could you help me please ? tks
tasks logs:
`*** Reading local file: /home/airflow/airflow/logs/import_forex_unwind_trades/detect_book_referential/2021-08-05T00:00:00+00:00/28.log
[2021-09-07 11:18:19,826] {taskinstance.py:896} INFO - Dependencies all met for <TaskInstance: import_forex_unwind_trades.detect_book_referential 2021-08-05T00:00:00+00:00 [queued]>
[2021-09-07 11:18:21,644] {taskinstance.py:896} INFO - Dependencies all met for <TaskInstance: import_forex_unwind_trades.detect_book_referential 2021-08-05T00:00:00+00:00 [queued]>
[2021-09-07 11:18:21,644] {taskinstance.py:1087} INFO -
--------------------------------------------------------------------------------
[2021-09-07 11:18:21,644] {taskinstance.py:1088} INFO - Starting attempt 28 of 28
[2021-09-07 11:18:21,644] {taskinstance.py:1089} INFO -
--------------------------------------------------------------------------------
[2021-09-07 11:18:21,661] {taskinstance.py:1107} INFO - Executing <Task(HttpSensor): detect_book_referential> on 2021-08-05T00:00:00+00:00
[2021-09-07 11:18:21,662] {base_task_runner.py:133} INFO - Running on host: gvasrv-airflow
[2021-09-07 11:18:21,662] {base_task_runner.py:134} INFO - Running: ['airflow', 'tasks', 'run', 'import_forex_unwind_trades', 'detect_book_referential', '2021-08-05T00:00:00+00:00', '--job-id', '723490', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/import_fx_unwind_trades.py', '--cfg-path', '/tmp/tmpe43x01zl', '--error-file', '/tmp/tmpohmu9qb7']
[2021-09-07 11:18:24,493] {base_task_runner.py:118} INFO - Job 723490: Subtask detect_book_referential [2021-09-07 11:18:24,492] {dagbag.py:496} INFO - Filling up the DagBag from /home/airflow/airflow/dags/import_fx_unwind_trades.py
[2021-09-07 11:18:26,045] {base_task_runner.py:118} INFO - Job 723490: Subtask detect_book_referential /home/airflow/anaconda3/envs/airflow_dev_py37/lib/python3.7/site-packages/airflow/providers/http/sensors/http.py:26 DeprecationWarning: This decorator is deprecated.
[2021-09-07 11:18:26,045] {base_task_runner.py:118} INFO - Job 723490: Subtask detect_book_referential
[2021-09-07 11:18:26,045] {base_task_runner.py:118} INFO - Job 723490: Subtask detect_book_referential In previous versions, all subclasses of BaseOperator must use apply_default decorator for the`default_args` feature to work properly.
[2021-09-07 11:18:26,045] {base_task_runner.py:118} INFO - Job 723490: Subtask detect_book_referential
[2021-09-07 11:18:26,045] {base_task_runner.py:118} INFO - Job 723490: Subtask detect_book_referential In current version, it is optional. The decorator is applied automatically using the metaclass.
[2021-09-07 11:18:26,045] {base_task_runner.py:118} INFO - Job 723490: Subtask detect_book_referential
[2021-09-07 11:18:32,281] {base_task_runner.py:118} INFO - Job 723490: Subtask detect_book_referential [2021-09-07 11:18:32,280] {base.py:78} INFO - Using connection to: id: alpflow_symph_conn. Host: https://as.symphony.com/integration/v1/whi/simpleWebHookIntegration/, Port: None, Schema: None, Login: None, Password: None, extra: {'webhook_token': '***'}
[2021-09-07 11:18:33,857] {local_task_job.py:194} WARNING - Recorded pid 60749 does not match the current pid 98677
[2021-09-07 11:18:33,863] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 98677
[2021-09-07 11:18:33,870] {process_utils.py:66} INFO - Process psutil.Process(pid=98677, status='terminated', exitcode=<Negsignal.SIGTERM: -15>, started='11:18:21') (98677) terminated with exit code Negsignal.SIGTERM
`
scheduler log:
`
[2021-09-07 11:33:13,489] {dagrun.py:429} ERROR - Marking run <DagRun import_forex_unwind_trades @ 2021-08-11 00:00:00+00:00: backfill__2021-08-11T00:00:00+00:00, externally triggered: False> failed
[2021-09-07 11:33:13,527] {backfill_job.py:388} INFO - [backfill progress] | finished run 7 of 16 | tasks waiting: 0 | succeeded: 0 | running: 9 | failed: 39 | skipped: 0 | deadlocked: 0 | not ready: 0
[2021-09-07 11:33:14,819] {local_executor.py:124} ERROR - Failed to execute task PID of job runner does not match.
`
I have impresonated my Runner so that they are not runned with sudo priviledges (apparently this could cause the PID mismatch) but still it does not solve the issue.
Tks,
Pierre
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org