You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/20 16:04:21 UTC
[GitHub] [airflow] vinit-tribes opened a new issue #20992: Recorded pid<> does not match the current pid <>
vinit-tribes opened a new issue #20992:
URL: https://github.com/apache/airflow/issues/20992
### Apache Airflow version
2.2.3 (latest released)
### What happened
After upgrading to airflow 2.2.3 for every task we are getting, celery executor is being used in our case
Recorded pid 3944469 does not match the current pid 3944473
### What you expected to happen
_No response_
### How to reproduce
_No response_
### Operating System
ubuntu 20.04.1 LTS
### Versions of Apache Airflow Providers
apache-airflow-providers-celery==1.0.1
apache-airflow-providers-ftp==1.0.1
apache-airflow-providers-google==5.1.0
apache-airflow-providers-http==1.1.0
apache-airflow-providers-imap==1.0.1
apache-airflow-providers-neo4j==1.0.0
apache-airflow-providers-postgres==1.0.2
apache-airflow-providers-sqlite==1.0.1
apache-airflow-providers-ssh==2.1.0
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20992: Recorded pid<> does not match the current pid <>
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20992:
URL: https://github.com/apache/airflow/issues/20992#issuecomment-1052430894
I am not 100% sure but it is very likely it was caused by a race condition i've fixed in #19935 . It's been merged in 2.1.3 so either migrating to 2.1.4 or 2.2.3 (recommended) shoudl solve it - both are now supported in Composer.
If this is the reason then migration should help @jcasal-ixl - can you please migrate and let us know if the problem is solved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] PApostol commented on issue #20992: Recorded pid<> does not match the current pid <>
Posted by GitBox <gi...@apache.org>.
PApostol commented on issue #20992:
URL: https://github.com/apache/airflow/issues/20992#issuecomment-1017697200
I think there's no definite solution yet, since the original issue is still open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #20992: Recorded pid<> does not match the current pid <>
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #20992:
URL: https://github.com/apache/airflow/issues/20992#issuecomment-1017660926
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20992: Recorded pid<> does not match the current pid <>
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20992:
URL: https://github.com/apache/airflow/issues/20992#issuecomment-1019559979
I think what would be very useful to solve the problem is that everyone who has similar problem, would provide a more detailed information on how they get this error. Rather than "it also is the problem in 2.2.3" or saying "mee too", the best way you can help maintainers to investigate the issue is to prepare a very detailed description of your environment and post as many logs as you can find. This would be really helpful if everyone can do it and - for example - prepare a gist, or gists with their logs and environment descriptions.
Airflow is a free product and we value that our users (who get it for free) could help us to determine the root causes for the issues.
For the start - seems that this issue is connected with "default impersonation", so I think if anyone here has this issue without - impersonation, let us know. And if you do use impersonation - please (additionally to logs) make sure that you've followed this https://airflow.apache.org/docs/apache-airflow/stable/security/workload.html#impersonation and provide detailed description on how your impersonated user is configured.
Ideally if you could provide a step-by-step on how we can reproduce the right configuration and steps to lead to the problem (I guess you must have some details on what has been changed vs. standard airflow configuration). If you could perform some step-by-step recreation of your environment, and trace down when it breaks that could be super-helpful.
We won't be able to help as maintainers, if we do not get a really reproducible case when it happens - and getting as much information as possible is crucial to diagnose and fix this one.
Looking forward to getting more details!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jcasal-ixl commented on issue #20992: Recorded pid<> does not match the current pid <>
Posted by GitBox <gi...@apache.org>.
jcasal-ixl commented on issue #20992:
URL: https://github.com/apache/airflow/issues/20992#issuecomment-1041902311
For what is worth, I do not use "default impersonation", mine is set to blank with source of default, I do not use run_as_user.
I run only one scheduler. I'm on version 2.1.2 on Cloud Composer. It happens to me intermittently, it isn't a huge problem, but still an annoyance. Seems to me to only happen in Dynamically generated DAGs, wonder if thats the same for everyone else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vinit-tribes commented on issue #20992: Recorded pid<> does not match the current pid <>
Posted by GitBox <gi...@apache.org>.
vinit-tribes commented on issue #20992:
URL: https://github.com/apache/airflow/issues/20992#issuecomment-1017674730
Screenshot for the error traceback
Entire error traceback
`*** Reading remote log from gs://dev_etl_airflow_logs/Test_indexes_and_constraints/wait_for_a_minute/2022-01-20T15:56:40.994067+00:00/2.log.
*** Previous log discarded: 404 GET https://storage.googleapis.com/download/storage/v1/b/dev_etl_airflow_logs/o/Test_indexes_and_constraints%2Fwait_for_a_minute%2F2022-01-20T15%3A56%3A40.994067%2B00%3A00%2F2.log?alt=media: No such object: dev_etl_airflow_logs/Test_indexes_and_constraints/wait_for_a_minute/2022-01-20T15:56:40.994067+00:00/2.log: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)
[2022-01-20, 15:57:15 UTC] {taskinstance.py:1032} INFO - Dependencies all met for <TaskInstance: Test_indexes_and_constraints.wait_for_a_minute manual__2022-01-20T15:56:40.994067+00:00 [queued]>
[2022-01-20, 15:57:15 UTC] {taskinstance.py:1032} INFO - Dependencies all met for <TaskInstance: Test_indexes_and_constraints.wait_for_a_minute manual__2022-01-20T15:56:40.994067+00:00 [queued]>
[2022-01-20, 15:57:15 UTC] {taskinstance.py:1238} INFO -
--------------------------------------------------------------------------------
[2022-01-20, 15:57:15 UTC] {taskinstance.py:1239} INFO - Starting attempt 2 of 2
[2022-01-20, 15:57:15 UTC] {taskinstance.py:1240} INFO -
--------------------------------------------------------------------------------
[2022-01-20, 15:57:15 UTC] {taskinstance.py:1259} INFO - Executing <Task(PythonOperator): wait_for_a_minute> on 2022-01-20 15:56:40.994067+00:00
[2022-01-20, 15:57:15 UTC] {base_task_runner.py:136} INFO - Running on host: dev-airflow
[2022-01-20, 15:57:15 UTC] {base_task_runner.py:137} INFO - Running: ['airflow', 'tasks', 'run', 'Test_indexes_and_constraints', 'wait_for_a_minute', 'manual__2022-01-20T15:56:40.994067+00:00', '--job-id', '129997', '--raw', '--subdir', 'DAGS_FOLDER/test_ind_const.py', '--cfg-path', '/tmp/tmpon04h4a3', '--error-file', '/tmp/tmp13o8vd12']
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361: DeprecationWarning: The logging_level option in [core] has been moved to the logging_level option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute option = self._get_option_from_config_file(deprecated_key, deprecated_section, key, kwargs, section)
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361: DeprecationWarning: The fab_logging_level option in [core] has been moved to the fab_logging_level option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute option = self._get_option_from_config_file(deprecated_key, deprecated_section, key, kwargs, section)
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:303: FutureWarning: The hostname_callable setting in [core] has the old default value of 'socket:getfqdn'. This value has been changed to 'socket.getfqdn' in the running config, but please update your config before Apache Airflow 2.1.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute FutureWarning,
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:303: FutureWarning: The navbar_color setting in [webserver] has the old default value of '#007A87'. This value has been changed to '#fff' in the running config, but please update your config before Apache Airflow 2.1.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute FutureWarning,
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361: DeprecationWarning: The log_format option in [core] has been moved to the log_format option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute option = self._get_option_from_config_file(deprecated_key, deprecated_section, key, kwargs, section)
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361: DeprecationWarning: The simple_log_format option in [core] has been moved to the simple_log_format option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute option = self._get_option_from_config_file(deprecated_key, deprecated_section, key, kwargs, section)
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The hide_sensitive_variable_fields option in [admin] has been moved to the hide_sensitive_var_conn_fields option in [core] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The logging_config_class option in [core] has been moved to the logging_config_class option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The colored_log_format option in [core] has been moved to the colored_log_format option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The colored_console_log option in [core] has been moved to the colored_console_log option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The colored_formatter_class option in [core] has been moved to the colored_formatter_class option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The base_log_folder option in [core] has been moved to the base_log_folder option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The dag_processor_manager_log_location option in [core] has been moved to the dag_processor_manager_log_location option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The log_filename_template option in [core] has been moved to the log_filename_template option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The log_processor_filename_template option in [core] has been moved to the log_processor_filename_template option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The remote_logging option in [core] has been moved to the remote_logging option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The remote_base_log_folder option in [core] has been moved to the remote_base_log_folder option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The task_log_reader option in [core] has been moved to the task_log_reader option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:17 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The statsd_on option in [scheduler] has been moved to the statsd_on option in [metrics] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:17 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:17 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The dag_concurrency option in [core] has been renamed to max_active_tasks_per_dag - the old setting has been used, but please update your config.
[2022-01-20, 15:57:17 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The processor_poll_interval option in [scheduler] has been renamed to scheduler_idle_sleep_time - the old setting has been used, but please update your config.
[2022-01-20, 15:57:17 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute [[34m2022-01-20, 15:57:17 UTC[0m] {[34mdagbag.py:[0m500} INFO[0m - Filling up the DagBag from /home/airflow/airflow-dags-compiled/dags/test_ind_const.py[0m
[2022-01-20, 15:57:20 UTC] {local_task_job.py:207} WARNING - Recorded pid 3944469 does not match the current pid 3944473
[2022-01-20, 15:57:20 UTC] {process_utils.py:124} INFO - Sending Signals.SIGTERM to group 3944473. PIDs of all processes in the group: [3944473]
[2022-01-20, 15:57:20 UTC] {process_utils.py:75} INFO - Sending the signal Signals.SIGTERM to group 3944473
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute Running <TaskInstance: Test_indexes_and_constraints.wait_for_a_minute manual__2022-01-20T15:56:40.994067+00:00 [running]> on host dev-airflow
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute Traceback (most recent call last):
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/bin/airflow", line 8, in <module>
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute sys.exit(main())
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/__main__.py", line 48, in main
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute args.func(args)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute return func(*args, **kwargs)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/utils/cli.py", line 92, in wrapper
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute return f(*args, **kwargs)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 298, in task_run
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute _run_task_by_selected_method(args, dag, ti)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 107, in _run_task_by_selected_method
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute _run_raw_task(args, ti)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 184, in _run_raw_task
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute error_file=args.error_file,
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/utils/session.py", line 70, in wrapper
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute return func(*args, session=session, **kwargs)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1329, in _run_raw_task
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute self._execute_task_with_callbacks(context)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1455, in _execute_task_with_callbacks
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute result = self._execute_task(context, self.task)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1511, in _execute_task
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute result = execute_callable(context=context)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/operators/python.py", line 174, in execute
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute return_value = self.execute_callable()
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/operators/python.py", line 185, in execute_callable
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute return self.python_callable(*self.op_args, **self.op_kwargs)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/home/airflow/airflow-dags-compiled/dags/python_callables/neo4j_extraction_helper.py", line 105, in short_break
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute sleep(60)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1410, in signal_handler
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute raise AirflowException("Task received SIGTERM signal")
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute airflow.exceptions.AirflowException: Task received SIGTERM signal
[2022-01-20, 15:57:21 UTC] {process_utils.py:70} INFO - Process psutil.Process(pid=3944473, status='terminated', exitcode=1, started='15:57:15') (3944473) terminated with exit code 1
`
<img width="1920" alt="Screenshot 2022-01-20 at 9 46 49 PM" src="https://user-images.githubusercontent.com/51072190/150378173-036a886c-e51e-41db-ab02-9ba311b4fe4b.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] PApostol commented on issue #20992: Recorded pid<> does not match the current pid <>
Posted by GitBox <gi...@apache.org>.
PApostol commented on issue #20992:
URL: https://github.com/apache/airflow/issues/20992#issuecomment-1017682778
This has also been reported in #17507, there is a long discussion about it in that thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vinit-tribes edited a comment on issue #20992: Recorded pid<> does not match the current pid <>
Posted by GitBox <gi...@apache.org>.
vinit-tribes edited a comment on issue #20992:
URL: https://github.com/apache/airflow/issues/20992#issuecomment-1017674730
Screenshot for the error traceback
Entire error traceback
```
*** Reading remote log from gs://dev_etl_airflow_logs/Test_indexes_and_constraints/wait_for_a_minute/2022-01-20T15:56:40.994067+00:00/2.log.
*** Previous log discarded: 404 GET https://storage.googleapis.com/download/storage/v1/b/dev_etl_airflow_logs/o/Test_indexes_and_constraints%2Fwait_for_a_minute%2F2022-01-20T15%3A56%3A40.994067%2B00%3A00%2F2.log?alt=media: No such object: dev_etl_airflow_logs/Test_indexes_and_constraints/wait_for_a_minute/2022-01-20T15:56:40.994067+00:00/2.log: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)
[2022-01-20, 15:57:15 UTC] {taskinstance.py:1032} INFO - Dependencies all met for <TaskInstance: Test_indexes_and_constraints.wait_for_a_minute manual__2022-01-20T15:56:40.994067+00:00 [queued]>
[2022-01-20, 15:57:15 UTC] {taskinstance.py:1032} INFO - Dependencies all met for <TaskInstance: Test_indexes_and_constraints.wait_for_a_minute manual__2022-01-20T15:56:40.994067+00:00 [queued]>
[2022-01-20, 15:57:15 UTC] {taskinstance.py:1238} INFO -
--------------------------------------------------------------------------------
[2022-01-20, 15:57:15 UTC] {taskinstance.py:1239} INFO - Starting attempt 2 of 2
[2022-01-20, 15:57:15 UTC] {taskinstance.py:1240} INFO -
--------------------------------------------------------------------------------
[2022-01-20, 15:57:15 UTC] {taskinstance.py:1259} INFO - Executing <Task(PythonOperator): wait_for_a_minute> on 2022-01-20 15:56:40.994067+00:00
[2022-01-20, 15:57:15 UTC] {base_task_runner.py:136} INFO - Running on host: dev-airflow
[2022-01-20, 15:57:15 UTC] {base_task_runner.py:137} INFO - Running: ['airflow', 'tasks', 'run', 'Test_indexes_and_constraints', 'wait_for_a_minute', 'manual__2022-01-20T15:56:40.994067+00:00', '--job-id', '129997', '--raw', '--subdir', 'DAGS_FOLDER/test_ind_const.py', '--cfg-path', '/tmp/tmpon04h4a3', '--error-file', '/tmp/tmp13o8vd12']
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361: DeprecationWarning: The logging_level option in [core] has been moved to the logging_level option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute option = self._get_option_from_config_file(deprecated_key, deprecated_section, key, kwargs, section)
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361: DeprecationWarning: The fab_logging_level option in [core] has been moved to the fab_logging_level option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute option = self._get_option_from_config_file(deprecated_key, deprecated_section, key, kwargs, section)
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:303: FutureWarning: The hostname_callable setting in [core] has the old default value of 'socket:getfqdn'. This value has been changed to 'socket.getfqdn' in the running config, but please update your config before Apache Airflow 2.1.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute FutureWarning,
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:303: FutureWarning: The navbar_color setting in [webserver] has the old default value of '#007A87'. This value has been changed to '#fff' in the running config, but please update your config before Apache Airflow 2.1.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute FutureWarning,
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361: DeprecationWarning: The log_format option in [core] has been moved to the log_format option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute option = self._get_option_from_config_file(deprecated_key, deprecated_section, key, kwargs, section)
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361: DeprecationWarning: The simple_log_format option in [core] has been moved to the simple_log_format option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute option = self._get_option_from_config_file(deprecated_key, deprecated_section, key, kwargs, section)
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The hide_sensitive_variable_fields option in [admin] has been moved to the hide_sensitive_var_conn_fields option in [core] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The logging_config_class option in [core] has been moved to the logging_config_class option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The colored_log_format option in [core] has been moved to the colored_log_format option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The colored_console_log option in [core] has been moved to the colored_console_log option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The colored_formatter_class option in [core] has been moved to the colored_formatter_class option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The base_log_folder option in [core] has been moved to the base_log_folder option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The dag_processor_manager_log_location option in [core] has been moved to the dag_processor_manager_log_location option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The log_filename_template option in [core] has been moved to the log_filename_template option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The log_processor_filename_template option in [core] has been moved to the log_processor_filename_template option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The remote_logging option in [core] has been moved to the remote_logging option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The remote_base_log_folder option in [core] has been moved to the remote_base_log_folder option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The task_log_reader option in [core] has been moved to the task_log_reader option in [logging] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:16 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:17 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The statsd_on option in [scheduler] has been moved to the statsd_on option in [metrics] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:17 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
[2022-01-20, 15:57:17 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The dag_concurrency option in [core] has been renamed to max_active_tasks_per_dag - the old setting has been used, but please update your config.
[2022-01-20, 15:57:17 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute /srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/configuration.py:361 DeprecationWarning: The processor_poll_interval option in [scheduler] has been renamed to scheduler_idle_sleep_time - the old setting has been used, but please update your config.
[2022-01-20, 15:57:17 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute [[34m2022-01-20, 15:57:17 UTC[0m] {[34mdagbag.py:[0m500} INFO[0m - Filling up the DagBag from /home/airflow/airflow-dags-compiled/dags/test_ind_const.py[0m
[2022-01-20, 15:57:20 UTC] {local_task_job.py:207} WARNING - Recorded pid 3944469 does not match the current pid 3944473
[2022-01-20, 15:57:20 UTC] {process_utils.py:124} INFO - Sending Signals.SIGTERM to group 3944473. PIDs of all processes in the group: [3944473]
[2022-01-20, 15:57:20 UTC] {process_utils.py:75} INFO - Sending the signal Signals.SIGTERM to group 3944473
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute Running <TaskInstance: Test_indexes_and_constraints.wait_for_a_minute manual__2022-01-20T15:56:40.994067+00:00 [running]> on host dev-airflow
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute Traceback (most recent call last):
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/bin/airflow", line 8, in <module>
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute sys.exit(main())
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/__main__.py", line 48, in main
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute args.func(args)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute return func(*args, **kwargs)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/utils/cli.py", line 92, in wrapper
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute return f(*args, **kwargs)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 298, in task_run
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute _run_task_by_selected_method(args, dag, ti)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 107, in _run_task_by_selected_method
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute _run_raw_task(args, ti)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 184, in _run_raw_task
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute error_file=args.error_file,
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/utils/session.py", line 70, in wrapper
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute return func(*args, session=session, **kwargs)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1329, in _run_raw_task
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute self._execute_task_with_callbacks(context)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1455, in _execute_task_with_callbacks
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute result = self._execute_task(context, self.task)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1511, in _execute_task
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute result = execute_callable(context=context)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/operators/python.py", line 174, in execute
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute return_value = self.execute_callable()
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/operators/python.py", line 185, in execute_callable
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute return self.python_callable(*self.op_args, **self.op_kwargs)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/home/airflow/airflow-dags-compiled/dags/python_callables/neo4j_extraction_helper.py", line 105, in short_break
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute sleep(60)
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute File "/srv/airflow_venv2.2.3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1410, in signal_handler
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute raise AirflowException("Task received SIGTERM signal")
[2022-01-20, 15:57:20 UTC] {base_task_runner.py:121} INFO - Job 129997: Subtask wait_for_a_minute airflow.exceptions.AirflowException: Task received SIGTERM signal
[2022-01-20, 15:57:21 UTC] {process_utils.py:70} INFO - Process psutil.Process(pid=3944473, status='terminated', exitcode=1, started='15:57:15') (3944473) terminated with exit code 1
```
<img width="1920" alt="Screenshot 2022-01-20 at 9 46 49 PM" src="https://user-images.githubusercontent.com/51072190/150378173-036a886c-e51e-41db-ab02-9ba311b4fe4b.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vinit-tribes commented on issue #20992: Recorded pid<> does not match the current pid <>
Posted by GitBox <gi...@apache.org>.
vinit-tribes commented on issue #20992:
URL: https://github.com/apache/airflow/issues/20992#issuecomment-1017742867
This is a serious issue I think and so far not found anyone talking confidently about even any workarounds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vinit-tribes commented on issue #20992: Recorded pid<> does not match the current pid <>
Posted by GitBox <gi...@apache.org>.
vinit-tribes commented on issue #20992:
URL: https://github.com/apache/airflow/issues/20992#issuecomment-1017689116
Yes @PApostol reviewed that but the solution presented there isn't working for us
The issue there was for older version of airflow but for us it's happening in airflow 2.2.3 which is the latest released version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] easontm commented on issue #20992: Recorded pid<> does not match the current pid <>
Posted by GitBox <gi...@apache.org>.
easontm commented on issue #20992:
URL: https://github.com/apache/airflow/issues/20992#issuecomment-1056148016
Here's my datapoint. If there are other process logs that matter, please let me know.
Airflow version: 2.2.3
Executor: CeleryKubernetes
Deployment pattern: Other docker-based
- Using HA scheduling (2)
- 6 Celery workers
- RabbitMQ as the message queue (3 replicas, RabbitMQClusterOperator)
The task in question is a TimeDeltaSensor. After this failure the Celery worker continued to function and has processed other tasks, and this task itself was completed by a different Celery worker. I'm not setting `default_impersonation`. This issue is also not consistent for me as well -- I cannot repro on demand.
Task definition, the default args applied to it only concern scheduling, emails, and retries.
```
my_sensor = TimeDeltaSensor(
task_id="my_sensor", delta=timedelta(minutes=30), dag=dag
)
```
Celery worker log:
<details>
```
[2022-03-01 16:00:07,634: WARNING/ForkPoolWorker-5] Running <TaskInstance: MY_DAG.MY_TASK scheduled__2022-03-01T15:00:00+00:00 [queued]> on host airflow-celery-worker-6bb54c97cb-gks65
[2022-03-01 16:00:10,831: ERROR/ForkPoolWorker-4] Failed to execute task PID of job runner does not match.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/airflow/executors/celery_executor.py", line 121, in _execute_in_fork
args.func(args)
File "/usr/local/lib/python3.7/dist-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/airflow/utils/cli.py", line 92, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/airflow/cli/commands/task_command.py", line 298, in task_run
_run_task_by_selected_method(args, dag, ti)
File "/usr/local/lib/python3.7/dist-packages/airflow/cli/commands/task_command.py", line 105, in _run_task_by_selected_method
_run_task_by_local_task_job(args, ti)
File "/usr/local/lib/python3.7/dist-packages/airflow/cli/commands/task_command.py", line 163, in _run_task_by_local_task_job
run_job.run()
File "/usr/local/lib/python3.7/dist-packages/airflow/jobs/base_job.py", line 245, in run
self._execute()
File "/usr/local/lib/python3.7/dist-packages/airflow/jobs/local_task_job.py", line 134, in _execute
self.heartbeat()
File "/usr/local/lib/python3.7/dist-packages/airflow/jobs/base_job.py", line 226, in heartbeat
self.heartbeat_callback(session=session)
File "/usr/local/lib/python3.7/dist-packages/airflow/utils/session.py", line 67, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/airflow/jobs/local_task_job.py", line 209, in heartbeat_callback
raise AirflowException("PID of job runner does not match")
airflow.exceptions.AirflowException: PID of job runner does not match
```
</details>
Slightly abbreviated task logs:
<details>
```
--------------------------------------------------------------------------------
[2022-03-01, 16:00:05 UTC] {taskinstance.py:1239} INFO - Starting attempt 1 of 3
[2022-03-01, 16:00:05 UTC] {taskinstance.py:1240} INFO -
--------------------------------------------------------------------------------
[2022-03-01, 16:00:05 UTC] {taskinstance.py:1259} INFO - Executing <Task(TimeDeltaSensor): MY_TASK> on 2022-03-01 15:00:00+00:00
[2022-03-01, 16:00:05 UTC] {standard_task_runner.py:52} INFO - Started process 5939 to run task
[2022-03-01, 16:00:05 UTC] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'MY_DAG', 'MY_TASK', 'scheduled__2022-03-01T15:00:00+00:00', '--job-id', '6384572', '--raw', '--subdir', 'DAGS_FOLDER/MY_DAG.py', '--cfg-path', '/tmp/tmp84eym25r', '--error-file', '/tmp/tmpyb039unj']
[2022-03-01, 16:00:05 UTC] {standard_task_runner.py:77} INFO - Job 6384572: Subtask MY_TASK
[2022-03-01, 16:00:05 UTC] {logging_mixin.py:109} INFO - Running <TaskInstance: MY_DAG.MY_TASK scheduled__2022-03-01T15:00:00+00:00 [running]> on host airflow-celery-worker-6bb54c97cb-gks65
[2022-03-01, 16:00:05 UTC] {taskinstance.py:1426} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=MY_OWNER
AIRFLOW_CTX_DAG_ID=MY_DAG
AIRFLOW_CTX_TASK_ID=MY_TASK
AIRFLOW_CTX_EXECUTION_DATE=2022-03-01T15:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2022-03-01T15:00:00+00:00
[2022-03-01, 16:00:05 UTC] {time_delta.py:39} INFO - Checking if the time (2022-03-01T16:30:00+00:00) has come
[2022-03-01, 16:00:10 UTC] {local_task_job.py:207} WARNING - Recorded pid 5949 does not match the current pid 5939
[2022-03-01, 16:00:10 UTC] {process_utils.py:124} INFO - Sending Signals.SIGTERM to group 5939. PIDs of all processes in the group: [5939]
[2022-03-01, 16:00:10 UTC] {process_utils.py:75} INFO - Sending the signal Signals.SIGTERM to group 5939
[2022-03-01, 16:00:10 UTC] {taskinstance.py:1408} ERROR - Received SIGTERM. Terminating subprocesses.
...
usual stacktrace
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #20992: Recorded pid<> does not match the current pid <>
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #20992:
URL: https://github.com/apache/airflow/issues/20992#issuecomment-1086423938
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org