[GitHub] [airflow] taylorfinnell commented on pull request #17819: Properly handle ti state difference between executor and scheduler

[GitBox <gi...@apache.org>]

taylorfinnell commented on pull request #17819:
URL: https://github.com/apache/airflow/pull/17819#issuecomment-925228694


   @ephraimbuddy @kaxil 
   
   We applied this patch directly on top of the 2.1.4 tag and noticed issues almost instantly. 
   
   - Tasks would queue
   - Tasks would throw an exception - the exception in this case was from our DAG code signaling we are not ready to process, so retry later. It was not an internal airflow exception
   - Task logs would indicate that state was being set to `UP_FOR_RETRY`
   - The UI would show the task still `RUNNING`. The task would have ~16 retries left to go and would sit on attempt 1
   - Clearing the task would result in `shutdown` state
   - To get the task to run again we first had to fail it, then clear it again
   
   We also tried the `main` branch up to this commit and saw very similar issues. 
   
   Here are some logs.
   
   ```
   ...
   [2021-09-22 04:47:15,214] {taskinstance.py:1463} ERROR - Task failed with exception
   ...
   raise FileNotFoundError(f"{file} does not exist")
   
   [2021-09-22 04:47:15,299] {logging_mixin.py:109} WARNING - /opt/app-root/lib64/python3.8/site-packages/sqlalchemy/orm/strategies.py:911 SAWarning: Multiple rows returned with uselist=False for lazily-loaded attribute 'DagRun.task_instances'
   [2021-09-22 04:47:15,301] {taskinstance.py:1512} INFO - Marking task as UP_FOR_RETRY. dag_id=foo_dag_split, task_id=foo_task, execution_date=20210921T043000, start_date=20210922T044714, end_date=20210922T044715
   [2021-09-22 04:47:15,302] {logging_mixin.py:109} WARNING - /opt/app-root/lib64/python3.8/site-packages/sqlalchemy/orm/session.py:2193 SAWarning: Instance <TaskInstance at 0x7f01f5adc070> is already pending in this Session yet is being merged again; this is probably not what you want to do
   [2021-09-22 04:47:15,367] {local_task_job.py:151} INFO - Task exited with return code 1
   [2021-09-22 04:47:15,517] {logging_mixin.py:109} WARNING - /opt/app-root/lib64/python3.8/site-packages/sqlalchemy/orm/strategies.py:911 SAWarning: Multiple rows returned with uselist=False for lazily-loaded attribute 'DagRun.task_instances'
   [2021-09-22 04:47:15,517] {taskinstance.py:1512} INFO - Marking task as UP_FOR_RETRY. dag_id=foo_dag_split, task_id=foo_task, execution_date=20210921T043000, start_date=20210922T044714, end_date=20210922T044715
   [2021-09-22 04:47:15,518] {logging_mixin.py:109} WARNING - /opt/app-root/lib64/python3.8/site-packages/sqlalchemy/orm/session.py:2193 SAWarning: Instance <TaskInstance at 0x7f01f5ad6580> is already pending in this Session yet is being merged again; this is probably not what you want to do
   
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org