You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "aiqc (via GitHub)" <gi...@apache.org> on 2024/03/19 19:31:32 UTC
[I] AWS Batch job stuck in runnable state with no retry [airflow]
aiqc opened a new issue, #38298:
URL: https://github.com/apache/airflow/issues/38298
### Description
I am running AWS Batch jobs using the BatchOperator.
When I use Spot Instances, sometimes the job gets stuck in a "Runnable" status in the AWS Batch Queue. Probably because there are no spot capacity instances available. Eventually, the job will show `statusReason`:`UNDETERMINED - Batch job is blocked, root cause is undetermined.`
Meanwhile, Airflow shows these stuck jobs in the green "running" state and keeps polling them. I have both `retries` and `retry_delay` defined, but I don't think they are being triggered because of this running state.
### Use case/motivation
Cost savings -- I'd like a bit more retry functionality baked into the BatchOperator so that I can run spot instance jobs affordably
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] AWS Batch job stuck in runnable state with no retry [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #38298:
URL: https://github.com/apache/airflow/issues/38298#issuecomment-2007990805
I guess it something related to the [Jobs stuck in a `RUNNABLE` status ](https://docs.aws.amazon.com/batch/latest/userguide/troubleshooting.html#job_stuck_in_runnable)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] AWS Batch job stuck in runnable state with no retry [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #38298:
URL: https://github.com/apache/airflow/issues/38298#issuecomment-2008061623
Oh that is old issue with Batch but a least it nowadays provide a bit more information rather than couple years ago.
Right now BatchOperator use waiters in some places so it would be nice to get example of the output https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/batch/client/describe_jobs.html at the moment when job are stuck (do not forget to replace any sensible information by placeholders) and after that someone could try to check is it possible to detect this case by waiters and fail the job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] AWS Batch job stuck in runnable state with no retry [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on issue #38298:
URL: https://github.com/apache/airflow/issues/38298#issuecomment-2059594745
> Circling back here. The root problem was service quota limits on AWS Fargate.
Sounds good, closing the issue then :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] AWS Batch job stuck in runnable state with no retry [airflow]
Posted by "aiqc (via GitHub)" <gi...@apache.org>.
aiqc commented on issue #38298:
URL: https://github.com/apache/airflow/issues/38298#issuecomment-2033387481
Circling back here. The root problem was service quota limits on AWS Fargate.
On Tue, Mar 19, 2024 at 4:22 PM, Andrey Anshin ***@***.***(mailto:On Tue, Mar 19, 2024 at 4:22 PM, Andrey Anshin <<a href=)> wrote:
> Oh that is old issue with Batch but a least it nowadays provide a bit more information rather than couple years ago.
>
> Right now BatchOperator use waiters in some places so it would be nice to get example of the output https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/batch/client/describe_jobs.html at the moment when job are stuck (do not forget to replace any sensible information by placeholders) and after that someone could try to check is it possible to detect this case by waiters and fail the job.
>
> —
> Reply to this email directly, [view it on GitHub](https://github.com/apache/airflow/issues/38298#issuecomment-2008061623), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AR4EIMXOZRVCV7TBPFGUJQLYZCNARAVCNFSM6AAAAABE6GHFL2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBYGA3DCNRSGM).
> You are receiving this because you authored the thread.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] AWS Batch job stuck in runnable state with no retry [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas closed issue #38298: AWS Batch job stuck in runnable state with no retry
URL: https://github.com/apache/airflow/issues/38298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] AWS Batch job stuck in runnable state with no retry [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #38298:
URL: https://github.com/apache/airflow/issues/38298#issuecomment-2033310670
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] AWS Batch job stuck in runnable state with no retry [airflow]
Posted by "aiqc (via GitHub)" <gi...@apache.org>.
aiqc commented on issue #38298:
URL: https://github.com/apache/airflow/issues/38298#issuecomment-2008021205
@Taragolis Indeed, it's surprising that the `statusReason` does not mention "capacity"
I should mention that as soon as I rerun the DAG with **reserved** (non-spot) instances, it works as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org