You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "0BVer (via GitHub)" <gi...@apache.org> on 2025/10/10 07:36:38 UTC

[I] Support configurable multiplier for retry exponential backoff [airflow]

0BVer opened a new issue, #56537:
URL: https://github.com/apache/airflow/issues/56537

   ### Description
   
   I love using Airflow to build workflow logic, but I've hit a limitation with the current retry options.
   Right now, I need to implement a customizable retry logic — specifically, I want to set a configurable multiplier for the retry exponential backoff, but the multiplier is currently hardcoded to 2 in Airflow.
   
   ```python
   def next_retry_datetime(self):
       ...
       min_backoff = math.ceil(delay.total_seconds() * (2 ** (self.try_number - 1)))
       ...
   ```
   
   So I suggest adding a new parameter such as retry_delay_multiplier to allow users to configure the exponential backoff.
   
   ```python
   # like this!
   default_args = {
       'depends_on_past': False,
       'retries': 2,
       'retry_exponential_backoff': True,
       'retry_delay': timedelta(minutes=4),
       'retry_delay_multiplier': 5,
   }
   ```
   It will retry after 4 min, then 20 min, and then 100 min.
   
   ### Use case/motivation
   
   Many data pipelines or APIs have different retry requirements: the fixed exponential backoff with a multiplier of 2 is sometimes too aggressive or too slow depending on the job's nature. If Airflow provides a configurable multiplier for exponential backoff, users can fine-tune retry intervals for different workloads. This is especially useful for critical external integrations, error-prone batch jobs, or systems with rate limits, where controlling the growth rate of delay is essential for stability, throughput, and SLA adherence.
   
   Customizing the multiplier would make Airflow's retry system much more versatile and suitable for a wider variety of operational scenarios.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Support configurable multiplier for retry exponential backoff [airflow]

Posted by "steveahnahn (via GitHub)" <gi...@apache.org>.
steveahnahn commented on issue #56537:
URL: https://github.com/apache/airflow/issues/56537#issuecomment-3420151788

   Opened a PR to address this: https://github.com/apache/airflow/pull/56866


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Support configurable multiplier for retry exponential backoff [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl closed issue #56537: Support configurable multiplier for retry exponential backoff
URL: https://github.com/apache/airflow/issues/56537


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Support configurable multiplier for retry exponential backoff [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #56537:
URL: https://github.com/apache/airflow/issues/56537#issuecomment-3388663664

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Support configurable multiplier for retry exponential backoff [airflow]

Posted by "0BVer (via GitHub)" <gi...@apache.org>.
0BVer commented on issue #56537:
URL: https://github.com/apache/airflow/issues/56537#issuecomment-3421093376

   oops @steveahnahn is faster then me. never mind


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org