[GitHub] [airflow] yermalov-here commented on issue #34204: Deferrable ExternalTaskSensor times out even if DAG is running

["yermalov-here (via GitHub)" <gi...@apache.org>]

yermalov-here commented on issue #34204:
URL: https://github.com/apache/airflow/issues/34204#issuecomment-1719185340

   i took a look at the related issue #34205  and the root-cause is in the logic that differs from the `non deferrable` behavior. i was able to fix the very exact problem described there, but the behavior looks weird.
   https://github.com/apache/airflow/compare/main...yermalov-here:airflow:fix_ExternalTaskSensor_deferrable
   
   imho this trigger is worthy of a refactoring and the `non deferrable` behavior should be the guideline. i've noted the following issues:
   0. the above-mentioned issue with hardcoded timeout. i think the sensor `timeout` value should be passed there (and the `poke_interval` as well)
   1. if it was the task not completing on time, the log will still say "Dag was not started within 1 minute, assuming fail.", which is confusing if you are waiting for a task or a task group, not for the full dag. see the `execute_complete` method of the sensor
   2. on each iteration the trigger checks whether a dag_run exists, which i don't see happening in the sensor poke
   
   in general to me it looks like it is quite an issue, that logic needs to be duplicated for `deferrable` and `non deferrable` sensor and operator versions. it leads to them having different logic and different sets of features.
   being an Airflow user I wouldn't expect behavior to differ for the same Operator/Sensor being run in two different modes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org