You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "tim-x-y-z (via GitHub)" <gi...@apache.org> on 2025/06/30 16:07:55 UTC
[I] `retries` in `CloudRunExecuteJobOperator` does not superseed `Number of retries per failed task` parameter in job definition [airflow]
tim-x-y-z opened a new issue, #52580:
URL: https://github.com/apache/airflow/issues/52580
### Apache Airflow version
2.11.0
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Hi team,
This is related to the google cloud run operator
```
from airflow.providers.google.cloud.operators.cloud_run import (
CloudRunExecuteJobOperator,
)
```
When passing a number of retries to that operator i exepect the task to retry that number of times. However if the `Number of retries per failed task` parameter is set in the jobs definition, then the task will retry x times and report it as a single try on airflow.
### What you think should happen instead?
I think passing `retries` to the `CloudRunExecuteJobOperator` should override the `Number of retries per failed task` parameter in the job definition, probably setting it to 0 ? Such that all the tries are co-ordinated by airflow.
### How to reproduce
Create a new cloud run job and have this parameter set:
Make sure the code in the job will fail.
```
CloudRunExecuteJobOperator(
task_id="example",
project_id="example",
region="europe-north1",
job_name="example_job_name",
overrides={
"container_overrides": [
{
"args": container_args,
}
]
},
dag=dag,
deferrable=True,
retries=1
)
```
The task will retry 3 times, then report only one failure, get triggered again by airflow and retry 3 times again.
### Operating System
Linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Google Cloud Composer
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `retries` in `CloudRunExecuteJobOperator` does not superseed `Number of retries per failed task` parameter in job definition [airflow]
Posted by "tim-x-y-z (via GitHub)" <gi...@apache.org>.
tim-x-y-z commented on issue #52580:
URL: https://github.com/apache/airflow/issues/52580#issuecomment-3022690214
> you should pass your config with the kwarg overrides.
I'm fairly confident i can't do that - but i can give it a go.
> That seems a good idea. However, afaik, we can't override the cloud run's 'retry' :( Correct me if I'm wrong
Yes i think this is the problem, it seems google run don't expose that in any of their api's - which to me was counter intuitive cause they do expose the `timeout` parameter in that way..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `retries` in `CloudRunExecuteJobOperator` does not superseed `Number of retries per failed task` parameter in job definition [airflow]
Posted by "ramonvermeulen (via GitHub)" <gi...@apache.org>.
ramonvermeulen commented on issue #52580:
URL: https://github.com/apache/airflow/issues/52580#issuecomment-3027459290
> afaik you couldn't override cloud run initialized retry number. so to make it work how you want it to work, i think you should set retry number 0 since you create the cloud run job
AFAIK this is indeed the way to go. It is also not an option in the [underlaying API](https://cloud.google.com/run/docs/reference/rest/v2/projects.locations.jobs/run) nor UI when triggering a cloud run job invocation.
See:
<img width="577" alt="Image" src="https://github.com/user-attachments/assets/88f0b052-70c4-4c83-a712-fcef836e515d" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `retries` in `CloudRunExecuteJobOperator` does not superseed `Number of retries per failed task` parameter in job definition [airflow]
Posted by "kgw7401 (via GitHub)" <gi...@apache.org>.
kgw7401 commented on issue #52580:
URL: https://github.com/apache/airflow/issues/52580#issuecomment-3021067101
Let me take a look this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `retries` in `CloudRunExecuteJobOperator` does not superseed `Number of retries per failed task` parameter in job definition [airflow]
Posted by "kevinhongzl (via GitHub)" <gi...@apache.org>.
kevinhongzl commented on issue #52580:
URL: https://github.com/apache/airflow/issues/52580#issuecomment-3021503091
> I think passing retries to the CloudRunExecuteJobOperator should override the Number of retries per failed task parameter in the job definition
`retry` is an argument for AIRFLOW controlling how many time your task will restart if it fails. If you would like to control how cloud run jobs behave, you should pass your config with the kwarg `overrides`.
> ... probably setting it to 0 ? Such that all the tries are co-ordinated by airflow.
That seems a good idea. However, afaik, we can't override the cloud run's 'retry' :( Correct me if I'm wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `retries` in `CloudRunExecuteJobOperator` does not superseed `Number of retries per failed task` parameter in job definition [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #52580:
URL: https://github.com/apache/airflow/issues/52580#issuecomment-3031743766
> Yes i think this is the problem, it seems google run don't expose that in any of their api's
If the needed parameter isn't exposed I don't think Airflow can handle this.
I suggest to open a feature request to Google issue tracker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `retries` in `CloudRunExecuteJobOperator` does not superseed `Number of retries per failed task` parameter in job definition [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on issue #52580:
URL: https://github.com/apache/airflow/issues/52580#issuecomment-3795313906
As written above, there's nothing much we can do about it.
The best you could do is either set Airflow's total tries to 1 and handle the retries in Cloud Run jobs, or vice versa (that's what we do in my work).
Even if it is implemented in the future by GCP, I'm not sure if it's best to mix them up (maybe with an additional flag, but I wouldn't make it the go-to behavior because it's not trivial).
Closing this issue as `won't fix`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `retries` in `CloudRunExecuteJobOperator` does not superseed `Number of retries per failed task` parameter in job definition [airflow]
Posted by "kgw7401 (via GitHub)" <gi...@apache.org>.
kgw7401 commented on issue #52580:
URL: https://github.com/apache/airflow/issues/52580#issuecomment-3024230418
afaik you couldn't override cloud run initialized retry number. so to make it work how you want it to work, i think you should set retry number 0 since you create the cloud run job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `retries` in `CloudRunExecuteJobOperator` does not superseed `Number of retries per failed task` parameter in job definition [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 closed issue #52580: `retries` in `CloudRunExecuteJobOperator` does not superseed `Number of retries per failed task` parameter in job definition
URL: https://github.com/apache/airflow/issues/52580
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org