[jira] [Created] (AIRFLOW-6996) Control KubernetesExecutor delete_worker_pods setting per task

["Barend (Jira)" <ji...@apache.org>]

Barend created AIRFLOW-6996:
-------------------------------

             Summary: Control KubernetesExecutor delete_worker_pods setting per task
                 Key: AIRFLOW-6996
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6996
             Project: Apache Airflow
          Issue Type: Improvement
          Components: executor-kubernetes
    Affects Versions: 1.10.9, 2.0.0
            Reporter: Barend
            Assignee: Daniel Imberman


h3. Context

The global {{airflow.cfg}} defines a boolean flag that controls whether the KubernetesExecutor should delete worker pods:
{code:none}
[kubernetes]
# If True (default), worker pods will be deleted upon termination
delete_worker_pods = True
{code}
You generally want this flag to be {{True}}, unless diagnosing specific kinds of task launch problems.

It's currently an all-or-nothing setting that affects all tasks in all DAGs.
h3. Improvement

When including a task in a DAG, I have the option to provide executor configuration specifically for that task:
{code:python}
sometask = DummyOperator(
        task_id="example",
        executor_config={ "KubernetesExecutor": { "image": "..." }  }
)
{code}
This does not currently (v1.10.9) give me the option to override pod deletion for that specific task. I think this would be an improvement for two reasons:
 # you're generally stopping deletion to debug the launch of a specifc task, making this the tightest possible scope where you'd want to control this behaviour
 # you can control the setting without restarting Airflow, by reloading the DAG



--
This message was sent by Atlassian Jira
(v8.3.4#803005)