[GitHub] [airflow] robg-eb commented on issue #32638: Find a solution to handle the execution timeout in Triggers

["robg-eb (via GitHub)" <gi...@apache.org>]

robg-eb commented on issue #32638:
URL: https://github.com/apache/airflow/issues/32638#issuecomment-1681340414

   @hussein-awala - This may be somewhat related to what you're working on, but there are a few Async Sensors that DO respect timeout, like `BigQueryTableExistenceAsyncSensor`, but then they have a different problem in that the async sensor acts differently than sync sensors do with respect to how long they wait for timeout.
   
   Take these two:
   
   ```
       check_for_data = BigQueryTableExistenceSensor(
           task_id="check_for_data",
           project_id=GCP_PROJECT,
           dataset_id="searchconsole",
           table_id="searchdata_url_impression_robtest",
           timeout=60,
           retries=3,
       )
   
       check_for_data_async = BigQueryTableExistenceAsyncSensor(
           task_id="check_for_data_async",
           project_id=GCP_PROJECT,
           dataset_id="searchconsole",
           table_id="searchdata_url_impression_robtest",
           timeout=60,
           retries=3,
       )
   ```
   
   These are two Sensors with exact same configuration params, only difference is that one is the Async sensor that is supposed to be a drop-in replacement for the Sync sensor. When you let this run, looking for a table that it doesnt find, the Sync sensor times out as it should after 60 seconds. The Async sensor retries 3 times and ends up waiting a total of (60 seconds * 4 attempts) + (retry wait interval) * 3.    That is not expected.   We'd want that `retries` don't occur past the total timeout interval for Async, the same way it works for Sync.  Is that something you're incorporating into your update?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org