You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "rishi-kulkarni (via GitHub)" <gi...@apache.org> on 2023/09/01 15:31:31 UTC

[GitHub] [airflow] rishi-kulkarni opened a new issue, #34017: Deferred BatchOperator does not recognize "FAILED" as an end state, hanging until it times out

rishi-kulkarni opened a new issue, #34017:
URL: https://github.com/apache/airflow/issues/34017

   ### Apache Airflow version
   
   2.7.0
   
   ### What happened
   
   When a Batch job fails, the trigger remains in the deferred state, noting that the status in "FAILED" but not marking the task as done. The task only is marked as failed once it times out. 
   
   ```[2023-08-26, 23:30:28 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:31:28 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:32:29 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:33:29 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:34:29 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:35:29 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:36:29 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:37:29 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:38:29 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:39:29 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:40:29 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:41:29 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:42:29 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:43:29 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:44:29 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:45:29 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:46:29 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:47:30 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:48:30 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:49:30 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:50:30 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:51:30 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:52:30 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:53:30 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:54:30 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:55:30 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:56:30 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:57:30 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:58:30 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-26, 23:59:30 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-27, 00:00:30 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-27, 00:01:31 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   [2023-08-27, 00:02:31 UTC] {waiter_with_logging.py:129} INFO - Batch job 85f49c11-c0ab-4a05-b361-1d06a0c0b81d not ready yet: ['FAILED']
   ```
   
   ### What you think should happen instead
   
   When a Batch job has failed, the Airflow task should be marked as failed immediately. 
   
   ### How to reproduce
   
   Set `deferrable=True` on a batch operator that starts a task that fails. The BatchOperator will continue checking the task until the operator times out. 
   
   ### Operating System
   
   Linux
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Virtualenv installation
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] rishi-kulkarni commented on issue #34017: Deferred BatchOperator does not recognize "FAILED" as an end state, hanging until it times out

Posted by "rishi-kulkarni (via GitHub)" <gi...@apache.org>.
rishi-kulkarni commented on issue #34017:
URL: https://github.com/apache/airflow/issues/34017#issuecomment-1702951655

   Ah, great - yep, we were on 8.4.0. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Taragolis commented on issue #34017: Deferred BatchOperator does not recognize "FAILED" as an end state, hanging until it times out

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #34017:
URL: https://github.com/apache/airflow/issues/34017#issuecomment-1702948126

   Which version of provider do you use?
   I guess it should be fixed in v8.6.0 of provider
   
   - https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/changelog.html#bug-fixes
   - https://github.com/apache/airflow/pull/33656


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] rishi-kulkarni closed issue #34017: Deferred BatchOperator does not recognize "FAILED" as an end state, hanging until it times out

Posted by "rishi-kulkarni (via GitHub)" <gi...@apache.org>.
rishi-kulkarni closed issue #34017: Deferred BatchOperator does not recognize "FAILED" as an end state, hanging until it times out
URL: https://github.com/apache/airflow/issues/34017


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org