You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/21 07:11:00 UTC
[GitHub] [airflow] vinit-tribes edited a comment on issue #16227: LocalTaskJob heartbeat race condition with finishing task causing SIGTERM
vinit-tribes edited a comment on issue #16227:
URL: https://github.com/apache/airflow/issues/16227#issuecomment-1018241064
Thanks @ephraimbuddy for the response, now no we aren't using `run_as_user` and we already tested with what's suggested in above documentation after seeing your previous comment on this comment i.e. adding airflow `ALL=(ALL) NOPASSWD: ALL` to sudoers file but without any luck :(
Pasting full error log below
```
*** Reading remote log from gs://dev_etl_airflow_logs/Test_indexes_and_constraints/wait_for_a_minute/2022-01-20T15:56:40.994067+00:00/2.log.
*** Previous log discarded: 404 GET https://storage.googleapis.com/download/storage/v1/b/dev_etl_airflow_logs/o/Test_indexes_and_constraints%2Fwait_for_a_minute%2F2022-01-20T15%3A56%3A40.994067%2B00%3A00%2F2.log?alt=media: No such object: dev_etl_airflow_logs/Test_indexes_and_constraints/wait_for_a_minute/2022-01-20T15:56:40.994067+00:00/2.log: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)
[2022-01-20, 15:57:15 UTC] {taskinstance.py:1032} INFO - Dependencies all met for <TaskInstance: Test_indexes_and_constraints.wait_for_a_minute manual__2022-01-20T15:56:40.994067+00:00 [queued]>
[2022-01-20, 15:57:15 UTC] {taskinstance.py:1032} INFO - Dependencies all met for <TaskInstance: Test_indexes_and_constraints.wait_for_a_minute manual__2022-01-20T15:56:40.994067+00:00 [queued]>
[2022-01-20, 15:57:15 UTC] {taskinstance.py:1238} INFO -
--------------------------------------------------------------------------------
[2022-01-20, 15:57:15 UTC] {taskinstance.py:1239} INFO - Starting attempt 2 of 2
[2022-01-20, 15:57:15 UTC] {taskinstance.py:1240} INFO -
--------------------------------------------------------------------------------
[2022-01-20, 15:57:15 UTC] {taskinstance.py:1259} INFO - Executing <Task(PythonOperator): wait_for_a_minute> on 2022-01-20 15:56:40.994067+00:00
[2022-01-20, 15:57:15 UTC] {base_task_runner.py:136} INFO - Running on host: dev-airflow
[2022-01-20, 15:57:15 UTC] {base_task_runner.py:137} INFO - Running: ['airflow', 'tasks', 'run', 'Test_indexes_and_constraints', 'wait_for_a_minute', 'manual__2022-01-20T15:56:40.994067+00:00', '--job-id', '129997', '--raw', '--subdir', 'DAGS_FOLDER/test_ind_const.py', '--cfg-path', '/tmp/tmpon04h4a3', '--error-file', '/tmp/tmp13o8vd12']
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361: DeprecationWarning: The logging_level option in [core] has been moved to the logging_level option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute option = self._get_option_from_config_file(deprecated_key, deprecated_section, key, kwargs, section)
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361: DeprecationWarning: The fab_logging_level option in [core] has been moved to the fab_logging_level option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute option = self._get_option_from_config_file(deprecated_key, deprecated_section, key, kwargs, section)
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:303: FutureWarning: The hostname_callable setting in [core] has the old default value of 'socket:getfqdn'. This value has been changed to 'socket.getfqdn' in the running config, but please update your config before Apache Airflow 2.1.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute FutureWarning,
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:303: FutureWarning: The navbar_color setting in [webserver] has the old default value of '#007A87'. This value has been changed to '#fff' in the running config, but please update your config before Apache Airflow 2.1.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute FutureWarning,
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361: DeprecationWarning: The log_format option in [core] has been moved to the log_format option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute option = self._get_option_from_config_file(deprecated_key, deprecated_section, key, kwargs, section)
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361: DeprecationWarning: The simple_log_format option in [core] has been moved to the simple_log_format option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute option = self._get_option_from_config_file(deprecated_key, deprecated_section, key, kwargs, section)
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The hide_sensitive_variable_fields option in [admin] has been moved to the hide_sensitive_var_conn_fields option in [core] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The logging_config_class option in [core] has been moved to the logging_config_class option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The colored_log_format option in [core] has been moved to the colored_log_format option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The colored_console_log option in [core] has been moved to the colored_console_log option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The colored_formatter_class option in [core] has been moved to the colored_formatter_class option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The base_log_folder option in [core] has been moved to the base_log_folder option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The dag_processor_manager_log_location option in [core] has been moved to the dag_processor_manager_log_location option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The log_filename_template option in [core] has been moved to the log_filename_template option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The log_processor_filename_template option in [core] has been moved to the log_processor_filename_template option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The remote_logging option in [core] has been moved to the remote_logging option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The remote_base_log_folder option in [core] has been moved to the remote_base_log_folder option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The task_log_reader option in [core] has been moved to the task_log_reader option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:17 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The statsd_on option in [scheduler] has been moved to the statsd_on option in [metrics] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:17 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:17 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The dag_concurrency option in [core] has been renamed to max_active_tasks_per_dag - the old setting has been used, but please update your config.
[2022-01-20, 15:57:17 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The processor_poll_interval option in [scheduler] has been renamed to scheduler_idle_sleep_time - the old setting has been used, but please update your config.
[2022-01-20, 15:57:17 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute [�[34m2022-01-20, 15:57:17 UTC�[0m] {�[34mdagbag.py:�[0m500} INFO�[0m - Filling up the DagBag from /home/airflow/airflow-dags-compiled/dags/test_ind_const.py�[0m
[2022-01-20, 15:57:20 UTC] {local_task_job.py:207} WARNING - Recorded pid 3944469 does not match the current pid 3944473
[2022-01-20, 15:57:20 UTC] {process_utils.py:124} INFO - Sending Signals.SIGTERM to group 3944473. PIDs of all processes in the group: [3944473]
[2022-01-20, 15:57:20 UTC] {process_utils.py:75} INFO - Sending the signal Signals.SIGTERM to group 3944473
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute Running <TaskInstance: Test_indexes_and_constraints.wait_for_a_minute manual__2022-01-20T15:56:40.994067+00:00 [running]> on host dev-airflow
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute Traceback (most recent call last):
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/bin/airflow", line 8, in <module>
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute sys.exit(main())
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/__main__.py", line 48, in main
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute args.func(args)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute return func(*args, **kwargs)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/utils/cli.py", line 92, in wrapper
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute return f(*args, **kwargs)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 298, in task_run
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute _run_task_by_selected_method(args, dag, ti)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 107, in _run_task_by_selected_method
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute _run_raw_task(args, ti)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 184, in _run_raw_task
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute error_file=args.error_file,
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/utils/session.py", line 70, in wrapper
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute return func(*args, session=session, **kwargs)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1329, in _run_raw_task
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute self._execute_task_with_callbacks(context)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1455, in _execute_task_with_callbacks
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute result = self._execute_task(context, self.task)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1511, in _execute_task
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute result = execute_callable(context=context)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/operators/python.py", line 174, in execute
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute return_value = self.execute_callable()
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/operators/python.py", line 185, in execute_callable
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute return self.python_callable(*self.op_args, **self.op_kwargs)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/home/airflow/airflow-dags-compiled/dags/python_callables/neo4j_extraction_helper.py", line 105, in short_break
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute sleep(60)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1410, in signal_handler
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute raise AirflowException("Task received SIGTERM signal")
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute airflow.exceptions.AirflowException: Task received SIGTERM signal
[2022-01-20, 15:57:21 UTC] {process_utils.py:70} INFO - Process psutil.Process(pid=3944473, status='terminated', exitcode=1, started='15:57:15') (3944473) terminated with exit code 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org