[jira] [Commented] (AIRFLOW-2167) Scheduler's clear_nonexistent_import_errors function should be called on first iteration

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927771#comment-16927771 ] 

ASF GitHub Bot commented on AIRFLOW-2167:
-----------------------------------------

stale[bot] commented on pull request #3088: [AIRFLOW-2167] cleanup nonexistent import errors on the first iteration in addition to periodically
URL: https://github.com/apache/airflow/pull/3088
 
 
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Scheduler's clear_nonexistent_import_errors function should be called on first iteration
> ----------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-2167
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2167
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: 1.9.0
>            Reporter: Casey
>            Assignee: Casey
>            Priority: Minor
>         Attachments: Screen Shot 2018-03-02 at 2.08.29 PM.png
>
>
> In `airflow/jobs.py`, the `**clear_nonexistent_import_errors` function is not called until the amount of seconds defined by `dag_dir_list_interval` has elapsed.  If the scheduler is not alive for the duration of `dag_dir_list_interval` (300 seconds) this cleanup never occurs.  In some environments this could result in error messages displaying on the UI permanently, even if the DAG has been removed from the environment.
> It was previously an Airflow best practice to have the scheduler run N runtimes and terminate.  Then, the scheduler would started again by an auxiliary process like Docker or Supervisor.  This situation is what brought the bug to my attention.
> My suggested fix is to tweak jobs.py to run the import error cleanup on the first iteration and periodically as defined by `dag_dir_list_interval`.  This way, a scheduler setup with a small number of runs will still have old errors cleaned up.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)