[jira] [Resolved] (AIRFLOW-812) Scheduler job terminates when there is no dag file despite run_numbers and run_duration set to unlimited.

["Bolke de Bruin (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bolke de Bruin resolved AIRFLOW-812.
------------------------------------
       Resolution: Fixed
    Fix Version/s: 1.8.1

Issue resolved by pull request #2027
[https://github.com/apache/incubator-airflow/pull/2027]

> Scheduler job terminates when there is no dag file despite run_numbers and run_duration set to unlimited.
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-812
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-812
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: utils
>            Reporter: Feng Lu
>            Assignee: Feng Lu
>             Fix For: 1.8.1
>
>
> When scheduler run_numbers and run_duration are set to -1 (unlimited), we expect the scheduler to run continuously. However, when dag folder is empty, the scheduler terminates immediately. This is due to a bug in utils/dag_processing when checking max_runs_reached. 
> Details:
> airflow scheduler
> [2017-01-27 08:20:50,433] {__init__.py:57} INFO - Using executor SequentialExecutor
> [2017-01-27 08:20:50,508] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2017-01-27 08:20:50,529] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
>   ____________       _____________
>  ____    |__( )_________  __/__  /________      __
> ____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
> ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
>  _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
> [2017-01-27 08:20:50,771] {jobs.py:514} ERROR - Cannot use more than 1 thread when using sqlite. Setting max_threads to 1
> [2017-01-27 08:20:50,786] {jobs.py:1222} INFO - Starting the scheduler
> [2017-01-27 08:20:50,787] {jobs.py:1238} INFO - Processing files using up to 1 processes at a time
> [2017-01-27 08:20:50,787] {jobs.py:1240} INFO - Running execute loop for -1 seconds
> [2017-01-27 08:20:50,787] {jobs.py:1242} INFO - Processing each file at most None times
> [2017-01-27 08:20:50,787] {jobs.py:1244} INFO - Process each file at most once every 0 seconds
> [2017-01-27 08:20:50,787] {jobs.py:1246} INFO - Checking for new files in /home/fenglu/airflow/dags every 300 seconds
> [2017-01-27 08:20:50,787] {jobs.py:1249} INFO - Searching for files in /home/fenglu/airflow/dags
> [2017-01-27 08:20:50,787] {jobs.py:1252} INFO - There are 0 files in /home/fenglu/airflow/dags
> [2017-01-27 08:20:50,787] {jobs.py:1314} INFO - Resetting state for orphaned tasks
> [2017-01-27 08:20:50,789] {jobs.py:1326} INFO - Removing old import errors
> [2017-01-27 08:20:50,795] {jobs.py:1363} INFO - Heartbeating the process manager
> [2017-01-27 08:20:50,796] {jobs.py:1400} INFO - Heartbeating the executor
> [2017-01-27 08:20:51,797] {jobs.py:1432} INFO - Exiting loop as all files have been processed None times
> [2017-01-27 08:20:51,797] {jobs.py:1448} INFO - Deactivating DAGs that haven't been touched since 2017-01-27T08:20:50.795782
> [2017-01-27 08:20:51,801] {jobs.py:1271} INFO - Exited execute loop



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)