[jira] [Commented] (AIRFLOW-6843) Add grace_period_seconds config option for delete_namespaced_pod kube client request

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-6843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17178028#comment-17178028 ] 

ASF subversion and git services commented on AIRFLOW-6843:
----------------------------------------------------------

Commit 881c271bf6d57467ed78ee30d0b9e191a7971111 in airflow's branch refs/heads/v1-10-test from Pete DeJoy
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=881c271 ]

[AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod (#7523)

(cherry picked from commit 676c8515f7bdc8d2767a89484261056dc4bf4bed)


> Add grace_period_seconds config option for delete_namespaced_pod kube client request
> ------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-6843
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6843
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: executor-kubernetes
>    Affects Versions: 1.10.9
>            Reporter: Pete DeJoy
>            Assignee: Pete DeJoy
>            Priority: Major
>             Fix For: 2.0.0
>
>
> If you're using the Kubernetes executor, Amazon EKS deletes pods very quickly after the tasks complete. This can be an issue if you're scraping Airflow logs from a service like FluentD, as it means that the task pod gets deleted before FluentD can pick up logs for fast-running (or fast-failing) tasks.
> The [`kube_client_request_args` |#kube-client-request-args]environment variable is passed to the [`delete_namespaced_pod`|#L443] client request, but passing the necessary [`grace_period_seconds`|#delete_namespaced_pod] as a [`post_param`|#L108] on that object causes other client requests to fail, as that `kube_client_request_args` is passed as a kwarg to all client requests, even those that don't have a `post_params` option for `grace_period_seconds`.
> In order to provide a config option that fixes this issue, there needs to be a new configurable environment variable `AIRFLOW__KUBERNETES____DELETE_POD_GRACE_PERIOD_SECONDS` option that we pass to the [`delete_namespaced_pod`|#L443]] client request as its own argument. This will be defaulted to 0.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)