You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "alealandreev (via GitHub)" <gi...@apache.org> on 2025/03/19 18:27:34 UTC
[I] Retry exponential backoff max float overflow [airflow]
alealandreev opened a new issue, #47971:
URL: https://github.com/apache/airflow/issues/47971
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.10.3
### What happened?
Hello,
I encountered with a bug. My DAG configs were: retries=1000, retry_delay=5 min (300 seconds), max_retry_delay=1h (3600 seconds). My DAG failed ~1000 times and after that Scheduler broke down. After that retries exceeded 1000 and stopped on 1017 retry attempt.
I did my research on this problem and found that this happened due to formula **min_backoff = math.ceil(delay.total_seconds() * (2 ** (self.try_number - 1)))** in **taskinstance.py** file. So retry_exponential_backoff has no limit of try_number and during calculations it can overflow max Float value. So even if max_retry_delay is set formula is still calculating. And during calculations on very large retry number it crashes.
Please fix bug.
I also did pull request with my possible solution:
https://github.com/apache/airflow/pull/47967
From Airflow logs:
2024-12-09 02:16:39.825 OverflowError: cannot convert float infinity to integer
2024-12-09 02:16:39.825 min_backoff = int(math.ceil(delay.total_seconds() * (2 ** (self.try_number - 2))))
2024-12-08 09:29:14.583 [2024-12-08T06:29:14.583+0000] {scheduler_job_runner.py:705} INFO - Executor reports execution of mydag.spark_submit run_id=manual__2024-11-02T10:19:30.618008+00:00 exited with status up_for_retry for try_number 470
Configs:
_with DAG(
dag_id=DAG_ID,
start_date=MYDAG_START_DATE,
schedule_interval="@daily",
catchup=AIRFLOW_CATCHUP,
default_args={
'depends_on_past': True,
"retries": 1000,
"retry_delay": duration(minutes=5),
"retry_exponential_backoff": True,
"max_retry_delay": duration(hours=1),
},
) as dag:_
<img width="947" alt="Image" src="https://github.com/user-attachments/assets/f3307b23-0307-4b4d-b968-3e1984fbe93c" />
<img width="1050" alt="Image" src="https://github.com/user-attachments/assets/f161329c-155c-4d92-b3b4-cf442d6ed036" />
### What you think should happen instead?
My pull request: https://github.com/apache/airflow/pull/47967
### How to reproduce
Use configs from above. Example:
with DAG(
dag_id=DAG_ID,
start_date=MY_AIRFLOW_START_DATE,
schedule_interval="@daily",
catchup=AIRFLOW_CATCHUP,
default_args={
'depends_on_past': True,
"retries": 1000,
"retry_delay": duration(minutes=5),
"retry_exponential_backoff": True,
"max_retry_delay": duration(hours=1),
},
) as dag
### Operating System
Ubuntu 22.04
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry exponential backoff max float overflow [airflow]
Posted by "alealandreev (via GitHub)" <gi...@apache.org>.
alealandreev commented on issue #47971:
URL: https://github.com/apache/airflow/issues/47971#issuecomment-2803903248
Dear all,
Please see [additional PR](https://github.com/apache/airflow/pull/49274)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry exponential backoff max float overflow [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #47971:
URL: https://github.com/apache/airflow/issues/47971#issuecomment-2745390622
> Yes it is rare case, but It leads scheduler crash. From the provided configuration this failure will happen in ~41 days. If max_retry_delay will be 15 minutes it will be in 10+ days... Scheduler will be failed and you can't understand why without touching logs.
Which does not make it realistic case to be honest. That's why it's super niche. You probably can get hundreds of unrealistic cases like that. And I am not saying it does not need to be fixed, it might, but looking at the PR, the code implemented there is far too long for the functionality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry exponential backoff max float overflow [airflow]
Posted by "perry2of5 (via GitHub)" <gi...@apache.org>.
perry2of5 commented on issue #47971:
URL: https://github.com/apache/airflow/issues/47971#issuecomment-3025807213
I addressed the comments on my PR which should solve the issue. You can back port it to whatever version you are using, @netogerbi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry exponential backoff max float overflow [airflow]
Posted by "perry2of5 (via GitHub)" <gi...@apache.org>.
perry2of5 commented on issue #47971:
URL: https://github.com/apache/airflow/issues/47971#issuecomment-2906103070
Is there a problem with my approach in PR #48557?
https://github.com/apache/airflow/pull/48557
If this isn't worth fixing then I can close the PR. To me it seems worth fixing even though it is a niche case. I can update the PR if someone wants me to.....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry exponential backoff max float overflow [airflow]
Posted by "alealandreev (via GitHub)" <gi...@apache.org>.
alealandreev commented on issue #47971:
URL: https://github.com/apache/airflow/issues/47971#issuecomment-2760638969
Dear @potiuk ,
What will be done with this bug? I provided different possible solutions for it, including simple patch to limit try_number to 500 in function, which calculates next retry delay.
It will be nice to fix this bug in upcoming releases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry exponential backoff max float overflow [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil closed issue #47971: Retry exponential backoff max float overflow
URL: https://github.com/apache/airflow/issues/47971
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry exponential backoff max float overflow [airflow]
Posted by "kenmy (via GitHub)" <gi...@apache.org>.
kenmy commented on issue #47971:
URL: https://github.com/apache/airflow/issues/47971#issuecomment-2743542863
Yes it is rare case, but It leads scheduler crash. From the provided configuration this failure will happen in ~41 days. If max_retry_delay will be 15 minutes it will be in 10+ days... Scheduler will be failed and you can't understand why without touching logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry exponential backoff max float overflow [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #47971:
URL: https://github.com/apache/airflow/issues/47971#issuecomment-2741928364
It's a very, very, very niche case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry exponential backoff max float overflow [airflow]
Posted by "alealandreev (via GitHub)" <gi...@apache.org>.
alealandreev commented on issue #47971:
URL: https://github.com/apache/airflow/issues/47971#issuecomment-2747881219
Dear all,
Please see [commit](https://github.com/alealandreev/airflow/commit/9ea065b7df74158646e1913f83b39738269543cd)
Pull request updated with simplified logic: https://github.com/apache/airflow/pull/48057
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry exponential backoff max float overflow [airflow]
Posted by "alealandreev (via GitHub)" <gi...@apache.org>.
alealandreev commented on issue #47971:
URL: https://github.com/apache/airflow/issues/47971#issuecomment-2743087624
> It's a very, very, very niche case.
Yes, but it is definitely a bug, which should be fixed.
I did new pull request: https://github.com/apache/airflow/pull/48051
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry exponential backoff max float overflow [airflow]
Posted by "perry2of5 (via GitHub)" <gi...@apache.org>.
perry2of5 commented on issue #47971:
URL: https://github.com/apache/airflow/issues/47971#issuecomment-2764298014
I think it is better to catch the overflow and use the maximum delay from the task or the environment. I submitted a PR against @alealandreev's fork...feel free to merge it. If you don't want to, I'm happy to just submit a patch against main. Let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry exponential backoff max float overflow [airflow]
Posted by "perry2of5 (via GitHub)" <gi...@apache.org>.
perry2of5 commented on issue #47971:
URL: https://github.com/apache/airflow/issues/47971#issuecomment-2927655060
PR 48557 is ready for review again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry exponential backoff max float overflow [airflow]
Posted by "alealandreev (via GitHub)" <gi...@apache.org>.
alealandreev commented on issue #47971:
URL: https://github.com/apache/airflow/issues/47971#issuecomment-2753487887
Dear all,
Please see [commit](https://github.com/apache/airflow/pull/48378) patch in /main, not only in 2.10.3
If it is not correct by form please do correct fix by yourself, if possible.
Thank you in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry exponential backoff max float overflow [airflow]
Posted by "netogerbi (via GitHub)" <gi...@apache.org>.
netogerbi commented on issue #47971:
URL: https://github.com/apache/airflow/issues/47971#issuecomment-3019428328
I just had this issue in prod! is there a workaround to make it work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org