You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Aizhamal Nurmamat kyzy (JIRA)" <ji...@apache.org> on 2019/05/18 02:46:02 UTC
[jira] [Updated] (AIRFLOW-3516) Improve K8 Executors Pod creation
Rate
[ https://issues.apache.org/jira/browse/AIRFLOW-3516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aizhamal Nurmamat kyzy updated AIRFLOW-3516:
--------------------------------------------
Labels: kubernetes (was: )
Component/s: (was: kubernetes)
(was: executor)
operators
Moving to operators component, and labeling with kubernetes as part of the component refactor.
> Improve K8 Executors Pod creation Rate
> --------------------------------------
>
> Key: AIRFLOW-3516
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3516
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
> Affects Versions: 1.10.1
> Reporter: raman
> Assignee: raman
> Priority: Major
> Labels: kubernetes
> Fix For: 1.10.3
>
>
> It seems that Airflow scheduler is creating worker pod sequentially(one pod per scheduler loop) in Kubernetes executors sync function which is called from base executors.heartbeat() function.
> One enhancement could be to create the k8 pod in batches where batch size can be configured through airflow.cfg file
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)