You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/02/24 15:34:49 UTC

[GitHub] [airflow] petedejoy opened a new pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to `V1DeleteOptions` of kube client `delete_namespaced_pod` request

petedejoy opened a new pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to `V1DeleteOptions`  of kube client `delete_namespaced_pod` request
URL: https://github.com/apache/airflow/pull/7523
 
 
   ---
   Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   If you're using the Kubernetes executor, Amazon EKS deletes pods very quickly after tasks complete. This can be an issue if you're scraping Airflow logs from a service like FluentD, as it means that the task pod gets deleted before FluentD can pick up logs for fast-running (or fast-failing) tasks.
   
   The `kube_client_request_args` environment variable is passed to the `delete_namespaced_pod` client request, but we need a way to pass the `grace_period_seconds` param (and potentially others) to the `V1DeleteOptions` included in the body of the request. Towards that objective, I've created a `delete_option_kwargs` object that can be set via an Airflow environment variable. This is object is then passed to the `V1DeleteOptions` of the `delete_namespaced_pods` request.
   
   This can be set via an `AIRFLOW__KUBERNETES__DELETE_OPTION_KWARGS` env var. The value should be JSON and can contain any of the options available in the [Kube client `V1DeleteOptions` class.](https://github.com/kubernetes-client/python/blob/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/models/v1_delete_options.py#L19)
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   <sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod
URL: https://github.com/apache/airflow/pull/7523#discussion_r383944608
 
 

 ##########
 File path: airflow/config_templates/config.yml
 ##########
 @@ -2035,6 +2035,17 @@
       type: string
       example: ~
       default: ""
+    - name: delete_option_kwargs
+      description: |
+        Optional keyword arguments to pass to the ``delete_namespaced_pod`` kubernetes client
+        ``core_v1_api`` method when using the Kubernetes Executor.
+        This should be an object and can contain any of the options listed in the ``v1DeleteOptions``
+        class defined here:
+        https://github.com/kubernetes-client/python/blob/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/models/v1_delete_options.py#L19
+      version_added: ~
+      type: string
+      example: "{{\"grace_period_seconds\": 10}}"
 
 Review comment:
   It is tricky because of `task_log_prefix_template = {{ti.dag_id}}-{{ti.task_id}}-{{execution_date}}-{{try_number}}`. It would replcae it too `task_log_prefix_template = {{{{ti.dag_id}}}}-{{{{ti.task_id}}}}-{{{{execution_date}}}}-{{{{try_number}}}}`
   
   > Maybe we need to update the pre-commit script to replace `{` with `{{` and `}` with `}}` so that this and the default_airflow.cfg are correct.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod
URL: https://github.com/apache/airflow/pull/7523#issuecomment-590610951
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7523?src=pr&el=h1) Report
   > :exclamation: No coverage uploaded for pull request base (`master@c9408a8`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `100%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7523/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7523?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff            @@
   ##             master    #7523   +/-   ##
   =========================================
     Coverage          ?   86.52%           
   =========================================
     Files             ?      893           
     Lines             ?    42270           
     Branches          ?        0           
   =========================================
     Hits              ?    36575           
     Misses            ?     5695           
     Partials          ?        0
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7523?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7523/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `57.17% <100%> (ø)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7523?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7523?src=pr&el=footer). Last update [c9408a8...b6a11c1](https://codecov.io/gh/apache/airflow/pull/7523?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] petedejoy commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod

Posted by GitBox <gi...@apache.org>.
petedejoy commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod
URL: https://github.com/apache/airflow/pull/7523#discussion_r383442931
 
 

 ##########
 File path: airflow/executors/kubernetes_executor.py
 ##########
 @@ -71,6 +71,8 @@ def __init__(self):  # pylint: disable=too-many-statements
                                                'env_from_configmap_ref')
         self.env_from_secret_ref = conf.get(self.kubernetes_section,
                                             'env_from_secret_ref')
+        self.delete_options_kwargs = conf.get(self.kubernetes_section,
 
 Review comment:
   Good catch, @ashb! Pushing up a quick fix now.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod
URL: https://github.com/apache/airflow/pull/7523#discussion_r383936665
 
 

 ##########
 File path: airflow/config_templates/config.yml
 ##########
 @@ -2035,6 +2035,17 @@
       type: string
       example: ~
       default: ""
+    - name: delete_option_kwargs
+      description: |
+        Optional keyword arguments to pass to the ``delete_namespaced_pod`` kubernetes client
+        ``core_v1_api`` method when using the Kubernetes Executor.
+        This should be an object and can contain any of the options listed in the ``v1DeleteOptions``
+        class defined here:
+        https://github.com/kubernetes-client/python/blob/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/models/v1_delete_options.py#L19
+      version_added: ~
+      type: string
+      example: "{{\"grace_period_seconds\": 10}}"
 
 Review comment:
   We need `{{"grace_period_seconds": 10}}` in airflow.cfg because ConfigParser does not parse `{` properly

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil merged pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod

Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod
URL: https://github.com/apache/airflow/pull/7523
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod
URL: https://github.com/apache/airflow/pull/7523#discussion_r383432557
 
 

 ##########
 File path: airflow/executors/kubernetes_executor.py
 ##########
 @@ -71,6 +71,8 @@ def __init__(self):  # pylint: disable=too-many-statements
                                                'env_from_configmap_ref')
         self.env_from_secret_ref = conf.get(self.kubernetes_section,
                                             'env_from_secret_ref')
+        self.delete_options_kwargs = conf.get(self.kubernetes_section,
 
 Review comment:
   This is a string isn't it, but the doc says kwargs. Are we missing a `json.loads` here?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod
URL: https://github.com/apache/airflow/pull/7523#issuecomment-590610951
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7523?src=pr&el=h1) Report
   > :exclamation: No coverage uploaded for pull request base (`master@c9408a8`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `100%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7523/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7523?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff            @@
   ##             master    #7523   +/-   ##
   =========================================
     Coverage          ?   86.34%           
   =========================================
     Files             ?      893           
     Lines             ?    42270           
     Branches          ?        0           
   =========================================
     Hits              ?    36499           
     Misses            ?     5771           
     Partials          ?        0
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7523?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7523/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `57.17% <100%> (ø)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7523?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7523?src=pr&el=footer). Last update [c9408a8...b6a11c1](https://codecov.io/gh/apache/airflow/pull/7523?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod
URL: https://github.com/apache/airflow/pull/7523#discussion_r383567643
 
 

 ##########
 File path: airflow/config_templates/default_airflow.cfg
 ##########
 @@ -980,6 +980,14 @@ tolerations =
 # https://raw.githubusercontent.com/kubernetes-client/python/master/kubernetes/client/apis/core_v1_api.py
 kube_client_request_args =
 
+# Optional keyword arguments to pass to the ``delete_namespaced_pod`` kubernetes client
+# ``core_v1_api`` method when using the Kubernetes Executor.
+# This should be an object and can contain any of the options listed in the ``v1DeleteOptions``
+# class defined here:
+# https://github.com/kubernetes-client/python/blob/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/models/v1_delete_options.py#L19
+# Example: delete_option_kwargs = '{"grace_period_seconds": 10}'
 
 Review comment:
   Updated and fixed

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod
URL: https://github.com/apache/airflow/pull/7523#discussion_r383966365
 
 

 ##########
 File path: airflow/config_templates/config.yml
 ##########
 @@ -2035,6 +2035,17 @@
       type: string
       example: ~
       default: ""
+    - name: delete_option_kwargs
+      description: |
+        Optional keyword arguments to pass to the ``delete_namespaced_pod`` kubernetes client
+        ``core_v1_api`` method when using the Kubernetes Executor.
+        This should be an object and can contain any of the options listed in the ``v1DeleteOptions``
+        class defined here:
+        https://github.com/kubernetes-client/python/blob/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/models/v1_delete_options.py#L19
+      version_added: ~
+      type: string
+      example: "{{\"grace_period_seconds\": 10}}"
 
 Review comment:
   Updated !

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod
URL: https://github.com/apache/airflow/pull/7523#discussion_r383963718
 
 

 ##########
 File path: setup.py
 ##########
 @@ -309,7 +309,7 @@ def write_version(filename: str = os.path.join(*[dirname(__file__), "airflow", "
     'pinotdb==0.1.1',
 ]
 postgres = [
-    'psycopg2-binary>=2.7.4',
 
 Review comment:
   Why this ? Just wondering?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod
URL: https://github.com/apache/airflow/pull/7523#discussion_r383935377
 
 

 ##########
 File path: airflow/config_templates/config.yml
 ##########
 @@ -2035,6 +2035,17 @@
       type: string
       example: ~
       default: ""
+    - name: delete_option_kwargs
+      description: |
+        Optional keyword arguments to pass to the ``delete_namespaced_pod`` kubernetes client
+        ``core_v1_api`` method when using the Kubernetes Executor.
+        This should be an object and can contain any of the options listed in the ``v1DeleteOptions``
+        class defined here:
+        https://github.com/kubernetes-client/python/blob/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/models/v1_delete_options.py#L19
+      version_added: ~
+      type: string
+      example: "{{\"grace_period_seconds\": 10}}"
 
 Review comment:
   Doesn't affect the output, but could be done as:
   
   ```suggestion
         example: '{{"grace_period_seconds": 10}}'
   ```
   
   Also: OH GOD. Do we show this as `{` or `{{` when we render the docs. Becasuse it needs to be `{` -- `{{` is only needed in default_airflow.cfg because that is a template, but we should show in our docs is not templated/formated.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod
URL: https://github.com/apache/airflow/pull/7523#discussion_r383542166
 
 

 ##########
 File path: airflow/config_templates/default_airflow.cfg
 ##########
 @@ -980,6 +980,14 @@ tolerations =
 # https://raw.githubusercontent.com/kubernetes-client/python/master/kubernetes/client/apis/core_v1_api.py
 kube_client_request_args =
 
+# Optional keyword arguments to pass to the ``delete_namespaced_pod`` kubernetes client
+# ``core_v1_api`` method when using the Kubernetes Executor.
+# This should be an object and can contain any of the options listed in the ``v1DeleteOptions``
+# class defined here:
+# https://github.com/kubernetes-client/python/blob/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/models/v1_delete_options.py#L19
+# Example: delete_option_kwargs = '{"grace_period_seconds": 10}'
 
 Review comment:
   Are quotes right here @kaxil? 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod
URL: https://github.com/apache/airflow/pull/7523#discussion_r383502808
 
 

 ##########
 File path: airflow/config_templates/config.yml
 ##########
 @@ -2035,6 +2035,14 @@
       type: string
       example: ~
       default: ""
+    - name: delete_option_kwargs
+      description: |
+        Optional keyword arguments to pass to the `delete_namespaced_pod` kubernetes client core_v1_api method when using the Kubernetes Executor. This should be an object and can contain any of the options listed in the `v1DeleteOptions` class defined here:
+        https://github.com/kubernetes-client/python/blob/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/models/v1_delete_options.py#L19
+      version_added: 1.10.10
+      type: string
+      example: ~
 
 Review comment:
   Added

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod
URL: https://github.com/apache/airflow/pull/7523#issuecomment-590956652
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7523?src=pr&el=h1) Report
   > :exclamation: No coverage uploaded for pull request base (`master@b6a11c1`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `93.18%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7523/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7523?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff            @@
   ##             master    #7523   +/-   ##
   =========================================
     Coverage          ?   86.85%           
   =========================================
     Files             ?      896           
     Lines             ?    42638           
     Branches          ?        0           
   =========================================
     Hits              ?    37035           
     Misses            ?     5603           
     Partials          ?        0
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7523?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/utils/helpers.py](https://codecov.io/gh/apache/airflow/pull/7523/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5) | `82.6% <100%> (ø)` | |
   | [airflow/settings.py](https://codecov.io/gh/apache/airflow/pull/7523/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZXR0aW5ncy5weQ==) | `93.66% <100%> (ø)` | |
   | [airflow/models/dagbag.py](https://codecov.io/gh/apache/airflow/pull/7523/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnYmFnLnB5) | `89.61% <100%> (ø)` | |
   | [...s/google/cloud/example\_dags/example\_stackdriver.py](https://codecov.io/gh/apache/airflow/pull/7523/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX3N0YWNrZHJpdmVyLnB5) | `100% <100%> (ø)` | |
   | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/7523/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `95.17% <100%> (ø)` | |
   | [...ow/providers/microsoft/azure/hooks/azure\_cosmos.py](https://codecov.io/gh/apache/airflow/pull/7523/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbWljcm9zb2Z0L2F6dXJlL2hvb2tzL2F6dXJlX2Nvc21vcy5weQ==) | `76.72% <100%> (ø)` | |
   | [airflow/providers/amazon/aws/hooks/s3.py](https://codecov.io/gh/apache/airflow/pull/7523/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9zMy5weQ==) | `96.6% <100%> (ø)` | |
   | [...ow/providers/google/cloud/operators/stackdriver.py](https://codecov.io/gh/apache/airflow/pull/7523/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9zdGFja2RyaXZlci5weQ==) | `100% <100%> (ø)` | |
   | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7523/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `91.37% <100%> (ø)` | |
   | [airflow/utils/log/logging\_mixin.py](https://codecov.io/gh/apache/airflow/pull/7523/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvbG9nZ2luZ19taXhpbi5weQ==) | `95.38% <100%> (ø)` | |
   | ... and [7 more](https://codecov.io/gh/apache/airflow/pull/7523/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7523?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7523?src=pr&el=footer). Last update [b6a11c1...e13f6c1](https://codecov.io/gh/apache/airflow/pull/7523?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman commented on issue #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod

Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod
URL: https://github.com/apache/airflow/pull/7523#issuecomment-590438501
 
 
   @petedejoy if you want a unit test for this, you can set the timeout to 15 seconds and make sure that once a pod completes it stays up for 15 seconds. @kaxil do you think a test is necessary here?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod
URL: https://github.com/apache/airflow/pull/7523#discussion_r383497075
 
 

 ##########
 File path: airflow/config_templates/config.yml
 ##########
 @@ -2035,6 +2035,14 @@
       type: string
       example: ~
       default: ""
+    - name: delete_option_kwargs
+      description: |
+        Optional keyword arguments to pass to the `delete_namespaced_pod` kubernetes client core_v1_api method when using the Kubernetes Executor. This should be an object and can contain any of the options listed in the `v1DeleteOptions` class defined here:
+        https://github.com/kubernetes-client/python/blob/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/models/v1_delete_options.py#L19
+      version_added: 1.10.10
+      type: string
+      example: ~
 
 Review comment:
   We should add an example !

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod
URL: https://github.com/apache/airflow/pull/7523#discussion_r383935977
 
 

 ##########
 File path: airflow/config_templates/config.yml
 ##########
 @@ -2035,6 +2035,17 @@
       type: string
       example: ~
       default: ""
+    - name: delete_option_kwargs
+      description: |
+        Optional keyword arguments to pass to the ``delete_namespaced_pod`` kubernetes client
+        ``core_v1_api`` method when using the Kubernetes Executor.
+        This should be an object and can contain any of the options listed in the ``v1DeleteOptions``
+        class defined here:
+        https://github.com/kubernetes-client/python/blob/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/models/v1_delete_options.py#L19
+      version_added: ~
+      type: string
+      example: "{{\"grace_period_seconds\": 10}}"
 
 Review comment:
   Maybe we need to update the pre-commit script to replace `{` with `{{` and `}` with `}}` so that this and the default_airflow.cfg are correct.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod
URL: https://github.com/apache/airflow/pull/7523#discussion_r383964107
 
 

 ##########
 File path: setup.py
 ##########
 @@ -309,7 +309,7 @@ def write_version(filename: str = os.path.join(*[dirname(__file__), "airflow", "
     'pinotdb==0.1.1',
 ]
 postgres = [
-    'psycopg2-binary>=2.7.4',
 
 Review comment:
   Whoops this was not meant for this PR, removing it

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman commented on issue #7523: [AIRFLOW-6843] Add delete_option_kwargs to `V1DeleteOptions` of kube client `delete_namespaced_pod` request

Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #7523: [AIRFLOW-6843] Add delete_option_kwargs to `V1DeleteOptions`  of kube client `delete_namespaced_pod` request
URL: https://github.com/apache/airflow/pull/7523#issuecomment-590417939
 
 
   @petedejoy can you fix the conflicts?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod
URL: https://github.com/apache/airflow/pull/7523#discussion_r383936665
 
 

 ##########
 File path: airflow/config_templates/config.yml
 ##########
 @@ -2035,6 +2035,17 @@
       type: string
       example: ~
       default: ""
+    - name: delete_option_kwargs
+      description: |
+        Optional keyword arguments to pass to the ``delete_namespaced_pod`` kubernetes client
+        ``core_v1_api`` method when using the Kubernetes Executor.
+        This should be an object and can contain any of the options listed in the ``v1DeleteOptions``
+        class defined here:
+        https://github.com/kubernetes-client/python/blob/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/models/v1_delete_options.py#L19
+      version_added: ~
+      type: string
+      example: "{{\"grace_period_seconds\": 10}}"
 
 Review comment:
   We need `{{"grace_period_seconds": 10}}` in airflow.cfg because ConfigParser does not parse `{` properly

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services