You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/05 19:25:05 UTC
[GitHub] [airflow] enniomorricone123 opened a new issue #18756: Kubernete executor: fetch log failed, no retries on task when task pod is DELETED while running
enniomorricone123 opened a new issue #18756:
URL: https://github.com/apache/airflow/issues/18756
### Apache Airflow version
2.1.3
### Operating System
linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
use helmchart and deployed on amazon eks with Kubernetes Executor
### What happened
When Task was submitted and new pod launched, and was running. If the task pod was somehow deleted right after it turned to running state, airflow webUI log will show errors:
```
*** Trying to get logs (last 100 lines) from worker pod ***
*** Unable to fetch logs from worker pod ***
(400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'bcdf197b-2c51-4837-9d3a-833078203c54', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 05 Oct 2021 19:22:32 GMT', 'Content-Length': '136'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"name must be provided","reason":"BadRequest","code":400}\n'
```
And no retries will be triggered even retried are configured to this task.
### What you expected to happen
_No response_
### How to reproduce
How to reproduce?
Using kuberneteExecutor and launch a dummy task. You can manually delete a running task pod right after it turned to RUNNING state.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #18756: Kubernete executor: fetch log failed, no retries on task when task pod is DELETED while running
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #18756:
URL: https://github.com/apache/airflow/issues/18756#issuecomment-934706696
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] easontm commented on issue #18756: Kubernete executor: fetch log failed, no retries on task when task pod is DELETED while running
Posted by GitBox <gi...@apache.org>.
easontm commented on issue #18756:
URL: https://github.com/apache/airflow/issues/18756#issuecomment-942882070
I believe I'm experiencing a similar issue. Occasionally in my deployment, task pods fail to create due to k8s environmental reasons. To a degree this can't be helped by me, and I would prefer if the scheduler could automatically retry/reschedule on this event (because it can at least tell that the task failed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org