You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/05/26 19:09:13 UTC
[GitHub] [airflow] svncsvr opened a new issue, #23948: Airflow Scheduler Keeps Crashing
svncsvr opened a new issue, #23948:
URL: https://github.com/apache/airflow/issues/23948
### Apache Airflow version
2.2.3
### What happened
First of all, It is a very strange error and happened all of a sudden when running some tasks within a DAG which was switched off
So basically within this DAG, we had a sensor running and a few downstream after that in a clean state. After those were in a clean state another sensor was also running. At this point, the scheduler started crashing
We are using CeleryExecutor and running Airflow v2.2.3
### What you think should happen instead
_No response_
### How to reproduce
I don't really know but here is the error stack trace
=1653590702662, https://airflow.apache.org/docs/docker-stack/build.html
1653590716300,BACKEND=redis
1653590716301,DB_HOST=airflow-de-preprod.ns1d7y.0001.euc1.cache.amazonaws.com
1653590716301,DB_PORT=6379
1653590719192, ____________ _____________
1653590719192, ____ |__( )_________ __/__ /________ __
1653590719192,____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
1653590719192,___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
1653590719192, _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
1653590719192,"[[34m2022-05-26 18:45:19,191[0m] {[34mscheduler_job.py:[0m596} INFO[0m - Starting the scheduler[0m"
1653590719192,"[[34m2022-05-26 18:45:19,192[0m] {[34mscheduler_job.py:[0m601} INFO[0m - Processing each file at most -1 times[0m"
1653590719417,"[[34m2022-05-26 18:45:19,412[0m] {[34mmanager.py:[0m163} INFO[0m - Launched DagFileProcessorManager with pid: 38[0m"
1653590719417,"[[34m2022-05-26 18:45:19,415[0m] {[34mscheduler_job.py:[0m1114} INFO[0m - Resetting orphaned tasks for active dag runs[0m"
1653590719421,"[[34m2022-05-26 18:45:19,420[0m] {[34msettings.py:[0m52} INFO[0m - Configured default timezone Timezone('Europe/Berlin')[0m"
1653590719423,"[[34m2022-05-26 18:45:19,423[0m] {[34msettings.py:[0m471} INFO[0m - Loaded airflow_local_settings from /opt/airflow/config/airflow_local_settings.py .[0m"
1653590719563,"[[34m2022-05-26 18:45:19,559[0m] {[34mscheduler_job.py:[0m644} ERROR[0m - Exception when executing SchedulerJob._run_scheduler_loop[0m"
1653590719563,Traceback (most recent call last):
1653590719563," File ""/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py"", line 628, in _execute"
1653590719563, self._run_scheduler_loop()
1653590719563," File ""/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py"", line 681, in _run_scheduler_loop"
1653590719563, self.adopt_or_reset_orphaned_tasks()
1653590719563," File ""/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py"", line 70, in wrapper"
1653590719563," return func(*args, session=session, **kwargs)"
1653590719563," File ""/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py"", line 1117, in adopt_or_reset_orphaned_tasks"
1653590719563, for attempt in run_with_db_retries(logger=self.log):
1653590719563," File ""/home/airflow/.local/lib/python3.7/site-packages/tenacity/__init__.py"", line 382, in __iter__"
1653590719563, do = self.iter(retry_state=retry_state)
1653590719563," File ""/home/airflow/.local/lib/python3.7/site-packages/tenacity/__init__.py"", line 349, in iter"
1653590719563, return fut.result()
1653590719563," File ""/usr/local/lib/python3.7/concurrent/futures/_base.py"", line 428, in result"
1653590719563, return self.__get_result()
1653590719563," File ""/usr/local/lib/python3.7/concurrent/futures/_base.py"", line 384, in __get_result"
1653590719563, raise self._exception
1653590719563," File ""/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py"", line 1162, in adopt_or_reset_orphaned_tasks"
1653590719563, to_reset = self.executor.try_adopt_task_instances(tis_to_reset_or_adopt)
1653590719563," File ""/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/celery_executor.py"", line 485, in try_adopt_task_instances"
1653590719563, self.adopted_task_timeouts[ti.key] = ti.queued_dttm + self.task_adoption_timeout
1653590719563,TypeError: unsupported operand type(s) for +: 'NoneType' and 'datetime.timedelta'
1653590720596,"[[34m2022-05-26 18:45:20,596[0m] {[34mprocess_utils.py:[0m124} INFO[0m - Sending Signals.SIGTERM to group 38. PIDs of all processes in the group: [39, 40, 38][0m"
1653590720597,"[[34m2022-05-26 18:45:20,596[0m] {[34mprocess_utils.py:[0m75} INFO[0m - Sending the signal Signals.SIGTERM to group 38[0m"
1653590721273,"[2022-05-26 18:45:21,272] {process_utils.py:237} INFO - Waiting up to 5 seconds for processes to exit..."
1653590721276,"[[34m2022-05-26 18:45:21,276[0m] {[34mprocess_utils.py:[0m70} INFO[0m - Process psutil.Process(pid=40, status='terminated', started='18:45:19') (40) terminated with exit code None[0m"
1653590721280,"[[34m2022-05-26 18:45:21,280[0m] {[34mprocess_utils.py:[0m70} INFO[0m - Process psutil.Process(pid=38, status='terminated', exitcode=0, started='18:45:18') (38) terminated with exit code 0[0m"
1653590721281,"[[34m2022-05-26 18:45:21,280[0m] {[34mprocess_utils.py:[0m70} INFO[0m - Process psutil.Process(pid=39, status='terminated', started='18:45:19') (39) terminated with exit code None[0m"
1653590721281,"[[34m2022-05-26 18:45:21,281[0m] {[34mscheduler_job.py:[0m655} INFO[0m - Exited execute loop[0m"
1653590721294,Traceback (most recent call last):
1653590721294," File ""/home/airflow/.local/bin/airflow"", line 8, in <module>"
1653590721294, sys.exit(main())
1653590721294," File ""/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py"", line 48, in main"
1653590721294, args.func(args)
1653590721294," File ""/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py"", line 48, in command"
1653590721294," return func(*args, **kwargs)"
1653590721294," File ""/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py"", line 92, in wrapper"
1653590721294," return f(*args, **kwargs)"
1653590721294," File ""/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/scheduler_command.py"", line 75, in scheduler"
1653590721294, _run_scheduler_job(args=args)
1653590721294," File ""/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/scheduler_command.py"", line 46, in _run_scheduler_job"
1653590721294, job.run()
1653590721294," File ""/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/base_job.py"", line 245, in run"
1653590721294, self._execute()
1653590721294," File ""/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py"", line 628, in _execute"
1653590721294, self._run_scheduler_loop()
1653590721294," File ""/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py"", line 681, in _run_scheduler_loop"
1653590721294, self.adopt_or_reset_orphaned_tasks()
1653590721294," File ""/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py"", line 70, in wrapper"
1653590721294," return func(*args, session=session, **kwargs)"
1653590721294," File ""/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py"", line 1117, in adopt_or_reset_orphaned_tasks"
1653590721294, for attempt in run_with_db_retries(logger=self.log):
1653590721294," File ""/home/airflow/.local/lib/python3.7/site-packages/tenacity/__init__.py"", line 382, in __iter__"
1653590721294, do = self.iter(retry_state=retry_state)
1653590721294," File ""/home/airflow/.local/lib/python3.7/site-packages/tenacity/__init__.py"", line 349, in iter"
1653590721294, return fut.result()
1653590721294," File ""/usr/local/lib/python3.7/concurrent/futures/_base.py"", line 428, in result"
1653590721294, return self.__get_result()
1653590721294," File ""/usr/local/lib/python3.7/concurrent/futures/_base.py"", line 384, in __get_result"
1653590721294, raise self._exception
1653590721294," File ""/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py"", line 1162, in adopt_or_reset_orphaned_tasks"
1653590721294, to_reset = self.executor.try_adopt_task_instances(tis_to_reset_or_adopt)
1653590721294," File ""/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/celery_executor.py"", line 485, in try_adopt_task_instances"
1653590721294, self.adopted_task_timeouts[ti.key] = ti.queued_dttm + self.task_adoption_timeout
1653590721294,TypeError: unsupported operand type(s) for +: 'NoneType' and 'datetime.timedelta'
### Operating System
Amazon Linux 2
### Versions of Apache Airflow Providers
Airflow v2.2.3
apache-airflow-providers-http==2.0.0
apache-airflow-providers-microsoft-mssql==2.1.0
apache-airflow-providers-microsoft-winrm==2.0.2
apache-airflow-providers-mongo==2.3.0
apache-airflow-providers-ssh==2.4.0
apache-airflow-providers-celery==2.1.0
### Deployment
Other Docker-based deployment
### Deployment details
Deployed to AWS ECS
### Anything else
We have been using Airflow since version 1.10.2 and never had this issue before.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #23948: Airflow Scheduler Keeps Crashing
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #23948:
URL: https://github.com/apache/airflow/issues/23948#issuecomment-1139261539
Duplicate of https://github.com/apache/airflow/issues/22160
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed issue #23948: Airflow Scheduler Keeps Crashing
Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #23948: Airflow Scheduler Keeps Crashing
URL: https://github.com/apache/airflow/issues/23948
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #23948: Airflow Scheduler Keeps Crashing
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #23948:
URL: https://github.com/apache/airflow/issues/23948#issuecomment-1138918695
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org