[GitHub] [airflow] potiuk edited a comment on pull request #19740: Added wait mechanizm to the DataprocJobSensor to avoid errors when Job is not available

[GitBox <gi...@apache.org>]

potiuk edited a comment on pull request #19740:
URL: https://github.com/apache/airflow/pull/19740#issuecomment-989872894


   I think both solutions are good in different scenarios- this really depends on how "transient" the message is and the context in which it is done. Both has pros and cons. And having both options is good.
   
   If this is really a message that appears randomly for 1/1000 requests out of a blue, then doing retries in the API call makes sense as it will be quickly retried and will succeed and the user will not even see it as an error (only logs will contain the information).  
   
   Also "API retry" scenario is good when you have custom operator which uses several operations sequentially using different Hooks. The retry mechanism of Airflow works in the way that whole custom operator will be retried from the beginning and it might mean delays, reprocessing data, bigger transfers etc. etc. 
   
   However if this just "standard" operator with single Hook and single opertion and especially if this is caused some "periods of unavailability" - like the API will start returning errors continuously for 5 minutes, using the standard "retry" mechanism of Airflow makes sense, because it will not block the worker for the 5 minutes.
   
   So adding an option to retry on API layer is a good idea additionally to "retry" functionality of Airflow.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org