You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dengpenn (via GitHub)" <gi...@apache.org> on 2024/04/24 04:27:08 UTC
[I] Improve KubernetesExecutor Observability [airflow]
dengpenn opened a new issue, #39215:
URL: https://github.com/apache/airflow/issues/39215
### Description
During our adoption of Airflow, the scheduler might create hundreds of pods during main scheduling loop. I propose to add two kind of metrics: the response code of k8s client and latency of creating/patching/deleting the pod.
### Use case/motivation
Airflow executor create one pod for each individual task. During peak time, we saw 800+ tasks were scheduled and the latency of underlying K8s API increased. The executor's heartbeat might be delayed due to the creation of task pods, potentially affecting the scheduler's heartbeat. It will be good to have metrics to monitor the response code and the latency of k8s API for creating/patching/deleting the pod.
### Related issues
N/A
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Improve KubernetesExecutor Observability [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #39215:
URL: https://github.com/apache/airflow/issues/39215#issuecomment-2074002153
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Improve KubernetesExecutor Observability [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #39215:
URL: https://github.com/apache/airflow/issues/39215#issuecomment-2143640836
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Improve KubernetesExecutor Observability [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on issue #39215:
URL: https://github.com/apache/airflow/issues/39215#issuecomment-2099354527
What is the [worker pods creation batch size](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/configurations-ref.html#worker-pods-creation-batch-size)? This limits the number of pods created during a given scheduler loop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Improve KubernetesExecutor Observability [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #39215: Improve KubernetesExecutor Observability
URL: https://github.com/apache/airflow/issues/39215
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Improve KubernetesExecutor Observability [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #39215:
URL: https://github.com/apache/airflow/issues/39215#issuecomment-2131753503
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Improve KubernetesExecutor Observability [airflow]
Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on issue #39215:
URL: https://github.com/apache/airflow/issues/39215#issuecomment-2105527232
We do have two metrics for the same. Can you check?
```
kubernetes_executor.clear_not_launched_queued_tasks.duration
kubernetes_executor.adopt_task_instances.duration
```
https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org