You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dirrao (via GitHub)" <gi...@apache.org> on 2023/10/17 01:52:15 UTC
[PR] clear_not_launched_queued_tasks optimization using kube list pods bat… [airflow]
dirrao opened a new pull request, #34985:
URL: https://github.com/apache/airflow/pull/34985
Problem: Airflow running the clear_not_launched_queued_tasks function on a certain frequency (default 30 seconds). When we run the airflow on a large Kube cluster (pods more than > 5K). Internally the clear_not_launched_queued_tasks function loops through each queued task and checks the corresponding worker pod existence in the Kube cluster. Right this existence check using list pods Kube API. The API is taking more than 1s. if there are 120 queued tasks, then it will take ~ 120 seconds (1s * 120). So, this leads the scheduler to spend most of its time in this function rather than scheduling the tasks. It leads to none of the jobs being scheduled or degraded scheduler performance.
Solution: Use k8 list pods batch api call to get all the worker pod owned by scheduler. Prepare the set of searchable strings using pod labels. Use this set data structure and identify whether the task associated pod exists or not.
set elements string format:
(dag_id=<dag_id>,task_id=<task_id>,airflow-worker=[,map_index=<map_index>],[run_id=<run_id>]|[execution_date=<execution_date>])
The details for the issue is mentioned in the below ticket.
https://github.com/apache/airflow/issues/34877
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve clear_not_launched_queued_tasks call duration [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #34985:
URL: https://github.com/apache/airflow/pull/34985#discussion_r1363230300
##########
airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py:
##########
@@ -224,6 +224,36 @@ def clear_not_launched_queued_tasks(self, session: Session = NEW_SESSION) -> Non
if time.time() - timestamp > allowed_age:
del self.last_handled[key]
+ if len(queued_tis) == 0:
+ return
+
+ # airflow worker label selector batch call
+ kwargs = dict(label_selector=f"airflow-worker={self._make_safe_label_value(str(self.job_id))}")
Review Comment:
```suggestion
kwargs = {"label_selector": f"airflow-worker={self._make_safe_label_value(str(self.job_id))}"}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve clear_not_launched_queued_tasks call duration [airflow]
Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on PR #34985:
URL: https://github.com/apache/airflow/pull/34985#issuecomment-1783640156
@uranusjr / @jedcunningham,
can you review it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve clear_not_launched_queued_tasks call duration [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #34985:
URL: https://github.com/apache/airflow/pull/34985#discussion_r1363229837
##########
airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py:
##########
@@ -224,6 +224,36 @@ def clear_not_launched_queued_tasks(self, session: Session = NEW_SESSION) -> Non
if time.time() - timestamp > allowed_age:
del self.last_handled[key]
+ if len(queued_tis) == 0:
Review Comment:
```suggestion
if not queued_tis:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve clear_not_launched_queued_tasks call duration [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #34985:
URL: https://github.com/apache/airflow/pull/34985#issuecomment-1788668495
Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] clear_not_launched_queued_tasks optimization using kube list pods bat… [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #34985:
URL: https://github.com/apache/airflow/pull/34985#issuecomment-1767612736
Please consider reformatting the commit message so it fits the standard style.
https://cbea.ms/git-commit/#seven-rules
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve clear_not_launched_queued_tasks call duration [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal merged PR #34985:
URL: https://github.com/apache/airflow/pull/34985
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve clear_not_launched_queued_tasks call duration [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #34985:
URL: https://github.com/apache/airflow/pull/34985#discussion_r1363231341
##########
airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py:
##########
@@ -224,6 +224,36 @@ def clear_not_launched_queued_tasks(self, session: Session = NEW_SESSION) -> Non
if time.time() - timestamp > allowed_age:
del self.last_handled[key]
+ if len(queued_tis) == 0:
+ return
+
+ # airflow worker label selector batch call
+ kwargs = dict(label_selector=f"airflow-worker={self._make_safe_label_value(str(self.job_id))}")
+ if self.kube_config.kube_client_request_args:
+ kwargs.update(**self.kube_config.kube_client_request_args)
Review Comment:
```suggestion
kwargs.update(self.kube_config.kube_client_request_args)
```
`dict.update()` can take a dict directly without unpacking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve clear_not_launched_queued_tasks call duration [airflow]
Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on PR #34985:
URL: https://github.com/apache/airflow/pull/34985#issuecomment-1773974950
@hussein-awala Did you get a chance to test it? Any findings?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org