[jira] [Commented] (AIRFLOW-6996) Control KubernetesExecutor delete_worker_pods setting per task

["Roster (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082989#comment-17082989 ] 

Roster commented on AIRFLOW-6996:
---------------------------------

is it smart to allow such free overwrite?
In some companies you don't want programmers to overwrite what devops set as expected behavior.

> Control KubernetesExecutor delete_worker_pods setting per task
> --------------------------------------------------------------
>
>                 Key: AIRFLOW-6996
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6996
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: executor-kubernetes
>    Affects Versions: 2.0.0, 1.10.9
>            Reporter: Barend
>            Assignee: Daniel Imberman
>            Priority: Major
>
> h3. Context
> The global {{airflow.cfg}} defines a boolean flag that controls whether the KubernetesExecutor should delete worker pods:
> {code:none}
> [kubernetes]
> # If True (default), worker pods will be deleted upon termination
> delete_worker_pods = True
> {code}
> You generally want this flag to be {{True}}, unless diagnosing specific kinds of task launch problems.
> It's currently an all-or-nothing setting that affects all tasks in all DAGs.
> h3. Improvement
> When including a task in a DAG, I have the option to provide executor configuration specifically for that task:
> {code:python}
> sometask = DummyOperator(
>         task_id="example",
>         executor_config={ "KubernetesExecutor": { "image": "..." }  }
> )
> {code}
> This does not currently (v1.10.9) give me the option to override pod deletion for that specific task. I think this would be an improvement for two reasons:
>  # you're generally stopping deletion to debug the launch of a specifc task, making this the tightest possible scope where you'd want to control this behaviour
>  # you can control the setting without restarting Airflow, by reloading the DAG



--
This message was sent by Atlassian Jira
(v8.3.4#803005)