[GitHub] [airflow] potiuk edited a comment on pull request #21852: [DRAFT] Databricks hook - retry on HTTP Status 429 as well

[GitBox <gi...@apache.org>]

potiuk edited a comment on pull request #21852:
URL: https://github.com/apache/airflow/pull/21852#issuecomment-1055258678


   > > Exponential backoff still tries too early in most situations. A client receiving 429 is supposed to wait at least until the date specified in the `Retry-After` response header before any retries.
   > 
   > Sure. Retry-After should be the source of first retry time - but exponential back-off after that does not hurt.
   
   Just to add a bit more reasoning. The problem is that Retry-After is only a hint, not a "source of truth". It relies on the fact the server "knows" what it is doing. Which is not necessary valid:
   a) it might be based on past information (which might be outdated and might not include the mounting spike properly)  - this happens often
   b) it might be simply not there. often you will not get 429 but 5XX in similar situations because it's not only the server that gets flooded but also some gateways on the way or simply the server might timeout or run out of memory or other resources. 
   
   So IMHO it needs to be client to decide how to behave. One added value of exponential backoff is that it is still helpful in all retriable conditions that are "unknown" - i.e. 5XX. Those do not contain "Retry-After" to base your decision on. 
   
   So I think exponential back-off with some initial timeout (if Retry-After is available - it should be a starting point) should be the right approach. Additionally if even including exponential back-off, you get 429 with Retry-After where your exponential back-off next step is not long enough - the timeout should be re-adjusted to the "Retry-After" received. But if it is longer, then we should continue exponential back-off (because server information might be already out-dated and not include mounting traffic.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org