You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/09/07 04:55:32 UTC

[GitHub] [airflow] r-richmond opened a new issue #18051: Airflow scheduling only 1 dag at a time leaving other dags in a queued state

r-richmond opened a new issue #18051:
URL: https://github.com/apache/airflow/issues/18051


   ### Apache Airflow version
   
   2.1.3 (latest released)
   
   ### Operating System
   
   Linux
   
   ### Versions of Apache Airflow Providers
   
   I don't think it is relevant but can provide them upon request.
   
   ### Deployment
   
   Other Docker-based deployment
   
   ### Deployment details
   
   _No response_
   
   ### What happened
   
   Completed an update to airflow 2.1.3 on Thursday and received an error report over the weekend that jobs were not being run. Upon investigating I discovered that only 1 DAG was running with everything else stuck in the queued state.
   Glancing at the 1 running dag was a long running backfill (30 days, ETA 2 weeks left, ~180 dag runs) that has `max_active_runs=1`
   
   The logs on airflow worker were normal however the logs on the scheduler were displaying the following error `raise OSError("handle is closed")` (See below for complete Logs).
   
   Anyways restarting the scheduler & workers did nothing. However upon turning off this long running dag (another 1 just like it started after turning off that one too) all of the dags began scheduling normally. 
   
   ### What you expected to happen
   
   Airflow should be able to continue scheduling dags normally without the existence of a single long running backfill.
   
   Note this may be related to one of the following:
   https://github.com/apache/airflow/issues/17975
   https://github.com/apache/airflow/issues/13542
   https://github.com/apache/airflow/pull/17945
   
   
   ### How to reproduce
   
   Steps to attempt to reproduce (Apologies for not going this far).
   
   1. Introduce a dag that has a large amount dag runs needed to catch up to current (catch_up=true)
   2. set the max_active_runs = 1 for this dag
   3. (probably a simple dag that sleeps for an hour on a task)
   4. submit the dag with a start date 2 years ago and runs daily
   5. Test and see if the dag can schedule other dags while the original dag is turned on.
   
   ### Anything else
   
   <details>
   <summary>Airflow Scheduler Logs</summary>
   ```
     ____________       _____________
    ____    |__( )_________  __/__  /________      __
   ____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
   ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
    _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
   [2021-09-07 02:55:51,944] {scheduler_job.py:661} INFO - Starting the scheduler
   [2021-09-07 02:55:51,944] {scheduler_job.py:666} INFO - Processing each file at most -1 times
   [2021-09-07 02:55:52,055] {manager.py:254} INFO - Launched DagFileProcessorManager with pid: 11
   [2021-09-07 02:55:52,058] {scheduler_job.py:1197} INFO - Resetting orphaned tasks for active dag runs
   [2021-09-07 02:55:52,062] {settings.py:51} INFO - Configured default timezone Timezone('UTC')
   [2021-09-07 02:59:34,363] {processor.py:243} WARNING - Killing DAGFileProcessorProcess (PID=612)
   Process ForkProcess-1:
   Traceback (most recent call last):
     File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
       self.run()
     File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
       self._target(*self._args, **self._kwargs)
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/manager.py", line 370, in _run_processor_manager
       processor_manager.start()
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/manager.py", line 610, in start
       return self._run_parsing_loop()
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/manager.py", line 671, in _run_parsing_loop
       self._collect_results_from_processor(processor)
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/manager.py", line 981, in _collect_results_from_processor
       if processor.result is not None:
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/processor.py", line 321, in result
       if not self.done:
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/processor.py", line 286, in done
       if self._parent_channel.poll():
     File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 255, in poll
       self._check_closed()
     File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 136, in _check_closed
       raise OSError("handle is closed")
   OSError: handle is closed
   [2021-09-07 02:59:35,405] {manager.py:401} WARNING - DagFileProcessorManager (PID=11) exited with exit code 1 - re-launching
   [2021-09-07 02:59:35,410] {manager.py:254} INFO - Launched DagFileProcessorManager with pid: 819
   [2021-09-07 02:59:35,418] {settings.py:51} INFO - Configured default timezone Timezone('UTC')
   [2021-09-07 03:00:52,433] {scheduler_job.py:1197} INFO - Resetting orphaned tasks for active dag runs
   [2021-09-07 03:00:52,441] {scheduler_job.py:1219} INFO - Marked 1 SchedulerJob instances as failed
   [2021-09-07 03:00:52,889] {celery_executor.py:483} INFO - Adopted the following 1 tasks from a dead executor
   	<TaskInstance: wow_mainline_queue_backfill.f_queue_time_cdc 2020-12-24 05:00:00+00:00 [running]> in state STARTED
   [2021-09-07 03:03:22,623] {processor.py:243} WARNING - Killing DAGFileProcessorProcess (PID=1626)
   [2021-09-07 03:03:22,624] {processor.py:243} WARNING - Killing DAGFileProcessorProcess (PID=1626)
   Process ForkProcess-2:
   Traceback (most recent call last):
     File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
       self.run()
     File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
       self._target(*self._args, **self._kwargs)
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/manager.py", line 370, in _run_processor_manager
       processor_manager.start()
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/manager.py", line 610, in start
       return self._run_parsing_loop()
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/manager.py", line 671, in _run_parsing_loop
       self._collect_results_from_processor(processor)
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/manager.py", line 981, in _collect_results_from_processor
       if processor.result is not None:
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/processor.py", line 321, in result
       if not self.done:
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/processor.py", line 286, in done
       if self._parent_channel.poll():
     File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 255, in poll
       self._check_closed()
     File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 136, in _check_closed
       raise OSError("handle is closed")
   OSError: handle is closed
   [2021-09-07 03:03:23,543] {manager.py:401} WARNING - DagFileProcessorManager (PID=819) exited with exit code 1 - re-launching
   [2021-09-07 03:03:23,548] {manager.py:254} INFO - Launched DagFileProcessorManager with pid: 1806
   [2021-09-07 03:03:23,557] {settings.py:51} INFO - Configured default timezone Timezone('UTC')
   [2021-09-07 03:05:54,184] {scheduler_job.py:1197} INFO - Resetting orphaned tasks for active dag runs
   [2021-09-07 03:07:42,551] {processor.py:243} WARNING - Killing DAGFileProcessorProcess (PID=2711)
   Process ForkProcess-3:
   Traceback (most recent call last):
     File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
       self.run()
     File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
       self._target(*self._args, **self._kwargs)
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/manager.py", line 370, in _run_processor_manager
       processor_manager.start()
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/manager.py", line 610, in start
       return self._run_parsing_loop()
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/manager.py", line 671, in _run_parsing_loop
       self._collect_results_from_processor(processor)
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/manager.py", line 981, in _collect_results_from_processor
       if processor.result is not None:
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/processor.py", line 321, in result
       if not self.done:
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/processor.py", line 286, in done
       if self._parent_channel.poll():
     File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 255, in poll
       self._check_closed()
     File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 136, in _check_closed
       raise OSError("handle is closed")
   OSError: handle is closed
   [2021-09-07 03:07:42,784] {manager.py:401} WARNING - DagFileProcessorManager (PID=1806) exited with exit code 1 - re-launching
   ```
   </details>
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] r-richmond commented on issue #18051: Airflow scheduling only 1 dag at a time leaving other dags in a queued state

Posted by GitBox <gi...@apache.org>.
r-richmond commented on issue #18051:
URL: https://github.com/apache/airflow/issues/18051#issuecomment-924580052


   Going to close this out since I assume #18061 tackled it and I'm on 2.1.4. now. If this issue presents itself again I'll re-open.
   
   Thanks again @ephraimbuddy 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] r-richmond commented on issue #18051: Airflow scheduling only 1 dag at a time leaving other dags in a queued state

Posted by GitBox <gi...@apache.org>.
r-richmond commented on issue #18051:
URL: https://github.com/apache/airflow/issues/18051#issuecomment-916603493


   Ty @ephraimbuddy for the update. I was about to proceed with cherry picking #17945 and your comment saved me some extra experimentation work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] datarian commented on issue #18051: Airflow scheduling only 1 dag at a time leaving other dags in a queued state

Posted by GitBox <gi...@apache.org>.
datarian commented on issue #18051:
URL: https://github.com/apache/airflow/issues/18051#issuecomment-914044948


   I am also observing the same issue.
   
   I can add that it is not limited to `max_active_runs = 1`. We tried to increase `max_active_runs` in the hope to speed up things, but the same deadlock for other DAGs was still present.
   
   We have Airflow 2.1.3 deployed on k8s using the unofficial helm chart at https://github.com/airflow-helm/charts.
   
   We mostly have default config settings. Changes related to the scheduler are these:
   ```
   AIRFLOW__SCHEDULER__PARSING_PROCESSES: "4"
   AIRFLOW__SCHEDULER__PROCESSOR_POLL_INTERVAL: "3"
   AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL: "60"
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] r-richmond closed issue #18051: Airflow scheduling only 1 dag at a time leaving other dags in a queued state

Posted by GitBox <gi...@apache.org>.
r-richmond closed issue #18051:
URL: https://github.com/apache/airflow/issues/18051


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] r-richmond edited a comment on issue #18051: Airflow scheduling only 1 dag at a time leaving other dags in a queued state

Posted by GitBox <gi...@apache.org>.
r-richmond edited a comment on issue #18051:
URL: https://github.com/apache/airflow/issues/18051#issuecomment-924580052


   Going to close this out since I assume #18061 tackled it and I'm on 2.1.4. now. If this issue presents itself again I'll re-open.
   
   Thanks again @ephraimbuddy 
   
   note: the only thing that I find still strange is the error logs from the scheduler that I originally had in particular the error
   ```
   Traceback (most recent call last):
     File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
       self.run()
     File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
       self._target(*self._args, **self._kwargs)
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/manager.py", line 370, in _run_processor_manager
       processor_manager.start()
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/manager.py", line 610, in start
       return self._run_parsing_loop()
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/manager.py", line 671, in _run_parsing_loop
       self._collect_results_from_processor(processor)
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/manager.py", line 981, in _collect_results_from_processor
       if processor.result is not None:
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/processor.py", line 321, in result
       if not self.done:
     File "/usr/local/lib/python3.8/site-packages/airflow/dag_processing/processor.py", line 286, in done
       if self._parent_channel.poll():
     File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 255, in poll
       self._check_closed()
     File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 136, in _check_closed
       raise OSError("handle is closed")
   ```
   however, that hasn't come up again so... fingers crossed it won't happen again?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on issue #18051: Airflow scheduling only 1 dag at a time leaving other dags in a queued state

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #18051:
URL: https://github.com/apache/airflow/issues/18051#issuecomment-915037157


   For now, Increase `AIRFLOW__SCHEDULER__MAX_DAGRUNS_PER_LOOP_TO_SCHEDULE`  to a higher number.
   
   @r-richmond , #17945 will resolve it but would introduce another bug that execution_order won't be followed. We are working on a better fix: https://github.com/apache/airflow/pull/18061


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org