[jira] [Updated] (AIRFLOW-4485) All tasks stop running when using reschedule mode due to some tasks having negative a try_number

["Kaxil Naik (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-4485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kaxil Naik updated AIRFLOW-4485:
--------------------------------
    Fix Version/s: 1.10.4

> All tasks stop running when using reschedule mode due to some tasks having negative a try_number
> ------------------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-4485
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4485
>             Project: Apache Airflow
>          Issue Type: Bug
>    Affects Versions: 1.10.3
>            Reporter: Teresa Martyny
>            Priority: Major
>             Fix For: 1.10.4
>
>
> When we use reschedule mode for our sensors, about an hour into our core pipeline running, the following happens:
> 1. Negative try_number: We begin to see on the Scheduler `Executor reports execution of [task info here] exited with status success for try_number -1` .... this then proceeds to continue to decrement until it reaches try_number -4 - With each run, -4 is the number where the following steps proceed to play out:
> 2. We see a spike(and then stop) in this error message on the Scheduler: `ERROR - Executor reports task instance {} finished ({}) although the task says its {}. Was the task killed externally?` coming from `airflow/jobs.py#_process_executor_events`
> 3. Sometimes followed by a few instances of the error on a single Worker: `Celery command failed` coming from `airflow/executors/celery_executor.py#execute_command`
> 4. Followed on the Worker by one instance of the error: `ZeroDivisionError` 
> 5. Followed by a spike in `ZeroDivisionError` on the Scheduler originating from `airflow/models/__init__.py#next_retry_datetime` line 1183
> 6. The pipeline then grinds to a halt. Tasks sit in a scheduled state in the scheduler, celery won't touch them. If try_numbers go negative, but never make it to negative 4, it doesn't grind to a halt. 
>  
> We identified that the reschedule mode decrements the try_number in `airflow/models/__init__.py#_handle_reschedule` 
> We did not identify why it never re-increments the `try_number` again to ostensibly do what the code is attempting: use the same `try_number` and write to the same log file.
> When we switched the sensors to use poke instead all of the above problems stopped. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)