[jira] [Created] (AIRFLOW-812) Scheduler job terminates when there is no dag file despite run_numbers and run_duration set to unlimited.

["Feng Lu (JIRA)" <ji...@apache.org>]

Feng Lu created AIRFLOW-812:
-------------------------------

             Summary: Scheduler job terminates when there is no dag file despite run_numbers and run_duration set to unlimited.
                 Key: AIRFLOW-812
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-812
             Project: Apache Airflow
          Issue Type: Bug
          Components: utils
            Reporter: Feng Lu
            Assignee: Feng Lu


When scheduler run_numbers and run_duration are set to -1 (unlimited), we expect the scheduler to run continuously. However, when dag folder is empty, the scheduler terminates immediately. This is due to a bug in utils/dag_processing when checking max_runs_reached. 

Details:

airflow scheduler
[2017-01-27 08:20:50,433] {__init__.py:57} INFO - Using executor SequentialExecutor
[2017-01-27 08:20:50,508] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2017-01-27 08:20:50,529] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/

[2017-01-27 08:20:50,771] {jobs.py:514} ERROR - Cannot use more than 1 thread when using sqlite. Setting max_threads to 1
[2017-01-27 08:20:50,786] {jobs.py:1222} INFO - Starting the scheduler
[2017-01-27 08:20:50,787] {jobs.py:1238} INFO - Processing files using up to 1 processes at a time
[2017-01-27 08:20:50,787] {jobs.py:1240} INFO - Running execute loop for -1 seconds
[2017-01-27 08:20:50,787] {jobs.py:1242} INFO - Processing each file at most None times
[2017-01-27 08:20:50,787] {jobs.py:1244} INFO - Process each file at most once every 0 seconds
[2017-01-27 08:20:50,787] {jobs.py:1246} INFO - Checking for new files in /home/fenglu/airflow/dags every 300 seconds
[2017-01-27 08:20:50,787] {jobs.py:1249} INFO - Searching for files in /home/fenglu/airflow/dags
[2017-01-27 08:20:50,787] {jobs.py:1252} INFO - There are 0 files in /home/fenglu/airflow/dags
[2017-01-27 08:20:50,787] {jobs.py:1314} INFO - Resetting state for orphaned tasks
[2017-01-27 08:20:50,789] {jobs.py:1326} INFO - Removing old import errors
[2017-01-27 08:20:50,795] {jobs.py:1363} INFO - Heartbeating the process manager
[2017-01-27 08:20:50,796] {jobs.py:1400} INFO - Heartbeating the executor
[2017-01-27 08:20:51,797] {jobs.py:1432} INFO - Exiting loop as all files have been processed None times
[2017-01-27 08:20:51,797] {jobs.py:1448} INFO - Deactivating DAGs that haven't been touched since 2017-01-27T08:20:50.795782
[2017-01-27 08:20:51,801] {jobs.py:1271} INFO - Exited execute loop



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)