You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/05 19:25:05 UTC

[GitHub] [airflow] enniomorricone123 opened a new issue #18756: Kubernete executor: fetch log failed, no retries on task when task pod is DELETED while running

enniomorricone123 opened a new issue #18756:
URL: https://github.com/apache/airflow/issues/18756


   ### Apache Airflow version
   
   2.1.3
   
   ### Operating System
   
   linux
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   use helmchart and deployed on amazon eks with Kubernetes Executor
   
   ### What happened
   
   When Task was submitted and new pod launched, and was running. If the task pod was somehow deleted right after it turned to running state, airflow webUI log will show errors:
   
   ```
   *** Trying to get logs (last 100 lines) from worker pod  ***
   
   *** Unable to fetch logs from worker pod  ***
   (400)
   Reason: Bad Request
   HTTP response headers: HTTPHeaderDict({'Audit-Id': 'bcdf197b-2c51-4837-9d3a-833078203c54', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 05 Oct 2021 19:22:32 GMT', 'Content-Length': '136'})
   HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"name must be provided","reason":"BadRequest","code":400}\n'
   
   ```
   And no retries will be triggered even retried are configured to this task.
   
   
   
   
   
   ### What you expected to happen
   
   _No response_
   
   ### How to reproduce
   
   How to reproduce?
   
   Using kuberneteExecutor and launch a dummy task. You can manually delete a running task pod right after it turned to RUNNING state.
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] boring-cyborg[bot] commented on issue #18756: Kubernete executor: fetch log failed, no retries on task when task pod is DELETED while running

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #18756:
URL: https://github.com/apache/airflow/issues/18756#issuecomment-934706696


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] easontm commented on issue #18756: Kubernete executor: fetch log failed, no retries on task when task pod is DELETED while running

Posted by GitBox <gi...@apache.org>.
easontm commented on issue #18756:
URL: https://github.com/apache/airflow/issues/18756#issuecomment-942882070


   I believe I'm experiencing a similar issue. Occasionally in my deployment, task pods fail to create due to k8s environmental reasons. To a degree this can't be helped by me, and I would prefer if the scheduler could automatically retry/reschedule on this event (because it can at least tell that the task failed).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org