You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/02/15 00:34:28 UTC
[GitHub] [airflow] brandonwillard opened a new pull request #7423:
[AIRFLOW-3126] Add option to specify additional K8s volumes
brandonwillard opened a new pull request #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423
This PR introduces a new config option, `kubernetes.extra_volume_mounts`, that allows users to specify multiple Kubernetes volumes to be mounted in each generated worker pod.
---
Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
<sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] brandonwillard commented on issue #7423: [AIRFLOW-3126]
Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
brandonwillard commented on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-586559290
How did this get tagged as a GCP-related PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7423:
[AIRFLOW-3126] Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-586568480
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=h1) Report
> Merging [#7423](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1982c3fdca1f04cfc41fc5b5e285d8f01c6b76ab&el=desc) will **decrease** coverage by `22.22%`.
> The diff coverage is `4.34%`.
[](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7423 +/- ##
===========================================
- Coverage 86.97% 64.75% -22.23%
===========================================
Files 927 927
Lines 44963 45009 +46
===========================================
- Hits 39108 29145 -9963
- Misses 5855 15864 +10009
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5) | `16.88% <0.00%> (-82.43%)` | :arrow_down: |
| [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `16.27% <5.55%> (-40.60%)` | :arrow_down: |
| [airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/pig\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9waWdfaG9vay5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/druid\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kcnVpZF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/mssql\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9tc3NxbF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [495 more](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=footer). Last update [1982c3f...2aa2c86](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7423:
[AIRFLOW-3126] Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-586568480
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=h1) Report
> Merging [#7423](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1982c3fdca1f04cfc41fc5b5e285d8f01c6b76ab&el=desc) will **decrease** coverage by `54.34%`.
> The diff coverage is `4.34%`.
[](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7423 +/- ##
===========================================
- Coverage 86.97% 32.62% -54.35%
===========================================
Files 927 927
Lines 44963 45009 +46
===========================================
- Hits 39108 14686 -24422
- Misses 5855 30323 +24468
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5) | `16.88% <0.00%> (-82.43%)` | :arrow_down: |
| [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `16.27% <5.55%> (-40.60%)` | :arrow_down: |
| [airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/pig\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9waWdfaG9vay5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/druid\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kcnVpZF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/mssql\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9tc3NxbF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [794 more](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=footer). Last update [1982c3f...2aa2c86](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] brandonwillard commented on issue #7423: [AIRFLOW-3126]
Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
brandonwillard commented on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-590955205
It does not appear to help.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] brandonwillard edited a comment on issue #7423:
[AIRFLOW-3126] Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
brandonwillard edited a comment on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-586637145
All right, I just checked locally (with `breeze`) and the [uncovered sections of those unrelated MySql files](https://codecov.io/gh/apache/airflow/pull/7423/changes) are definitely being called by their respective tests, which are also passing. As well, the [Travis log messages for those tests](https://travis-ci.org/apache/airflow/jobs/650753318#L1810) say they passed, and the warnings on Travis look the same as the local ones, so I'm not seeing an obvious divergence between local and Travis testing.
In other words, I'm not seeing a reason for those coverage decreases in those untouched and unrelated files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] brandonwillard removed a comment on issue #7423:
[AIRFLOW-3126] Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
brandonwillard removed a comment on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-586559488
Also, why is [`isort` failing](https://travis-ci.org/apache/airflow/jobs/650737702#L1742) on an untouched (and seemingly GCP-related) file?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] davlum commented on issue #7423: [AIRFLOW-3126] Add
option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
davlum commented on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-591443251
If that field is set it won't use the other fields as indicated by the [comment](https://github.com/apache/airflow/blob/746d8de2fcaa9554b4ce7dbf261e4ab148233222/airflow/config_templates/default_airflow.cfg#L755). You're right in that it definitely won't help with #7405, but I don't see why what you're trying to accomplish in this PR can't be done with that field. Do you have some example YAML that you want to create and we can test with that? Ftr I'm not invalidating this PR, I was just wondering if you had considered the option.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] brandonwillard removed a comment on issue #7423:
[AIRFLOW-3126] Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
brandonwillard removed a comment on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-590955205
It does not appear to help.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7423:
[AIRFLOW-3126] Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-586568480
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=h1) Report
> Merging [#7423](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1982c3fdca1f04cfc41fc5b5e285d8f01c6b76ab&el=desc) will **decrease** coverage by `54.38%`.
> The diff coverage is `4.34%`.
[](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7423 +/- ##
===========================================
- Coverage 86.97% 32.58% -54.39%
===========================================
Files 927 927
Lines 44963 45009 +46
===========================================
- Hits 39108 14668 -24440
- Misses 5855 30341 +24486
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5) | `16.88% <0.00%> (-82.43%)` | :arrow_down: |
| [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `16.27% <5.55%> (-40.60%)` | :arrow_down: |
| [airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/pig\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9waWdfaG9vay5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/druid\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kcnVpZF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/mssql\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9tc3NxbF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [795 more](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=footer). Last update [1982c3f...2aa2c86](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] brandonwillard commented on issue #7423: [AIRFLOW-3126]
Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
brandonwillard commented on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-590958733
@davlum, I've tried it and It did not appear to help. Can you give an example of how it can be used to add volumes in this way without sacrificing any of the existing functionality/configurability of worker pods?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] davlum commented on issue #7423: [AIRFLOW-3126] Add
option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
davlum commented on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-590936689
I've noticed you've added several PRs recently to add missing functionality to the `airflow.cfg` for configuring the KubernetesExecutor. There is also a [new field](https://github.com/apache/airflow/blob/746d8de2fcaa9554b4ce7dbf261e4ab148233222/airflow/config_templates/default_airflow.cfg#L756) which allows you to pass the YAML directly if that helps at all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7423: [AIRFLOW-3126] Add
option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-586568480
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=h1) Report
> Merging [#7423](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/edcad79b8d1ee359586c843c6e685cc58d6abacf?src=pr&el=desc) will **decrease** coverage by `0.16%`.
> The diff coverage is `97.82%`.
[](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7423 +/- ##
=========================================
- Coverage 86.66% 86.5% -0.17%
=========================================
Files 878 878
Lines 41163 41209 +46
=========================================
- Hits 35673 35646 -27
- Misses 5490 5563 +73
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5) | `99.33% <100%> (+0.04%)` | :arrow_up: |
| [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `59.71% <97.22%> (+2.91%)` | :arrow_up: |
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0%> (-64.16%)` | :arrow_down: |
| [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0%> (-45.66%)` | :arrow_down: |
| [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55% <0%> (-45%)` | :arrow_down: |
| [airflow/providers/apache/hive/hooks/hive.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvaG9va3MvaGl2ZS5weQ==) | `76.02% <0%> (-1.54%)` | :arrow_down: |
| [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `90.9% <0%> (-0.83%)` | :arrow_down: |
| [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.51% <0%> (-0.15%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=footer). Last update [edcad79...9ffcd34](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] brandonwillard commented on issue #7423: [AIRFLOW-3126]
Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
brandonwillard commented on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-609109273
Sorry, I have to move this to my personal fork: #8150.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7423:
[AIRFLOW-3126] Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-586568480
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=h1) Report
> Merging [#7423](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/2ea9278f76bf71aafb5601160602bf7f4194242f?src=pr&el=desc) will **increase** coverage by `0.1%`.
> The diff coverage is `100%`.
[](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7423 +/- ##
=========================================
+ Coverage 86.73% 86.84% +0.1%
=========================================
Files 897 897
Lines 42751 42797 +46
=========================================
+ Hits 37081 37165 +84
+ Misses 5670 5632 -38
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5) | `99.35% <100%> (+0.04%)` | :arrow_up: |
| [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `60.23% <100%> (+3.06%)` | :arrow_up: |
| [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.64% <0%> (+0.14%)` | :arrow_up: |
| [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `92.15% <0%> (+0.28%)` | :arrow_up: |
| [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `91.73% <0%> (+1.65%)` | :arrow_up: |
| [airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=) | `94.36% <0%> (+16.9%)` | :arrow_up: |
| [...roviders/google/cloud/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9wb3N0Z3Jlc190b19nY3MucHk=) | `85.29% <0%> (+32.35%)` | :arrow_up: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `100% <0%> (+50%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=footer). Last update [2ea9278...f026674](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] brandonwillard removed a comment on issue #7423:
[AIRFLOW-3126] Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
brandonwillard removed a comment on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-586559290
How did this get tagged as a GCP-related PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] brandonwillard edited a comment on issue #7423:
[AIRFLOW-3126] Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
brandonwillard edited a comment on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-586559488
Also, why is [`isort` failing](https://travis-ci.org/apache/airflow/jobs/650737702#L1742) on an untouched (and seemingly GCP-related) file?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] brandonwillard edited a comment on issue #7423:
[AIRFLOW-3126] Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
brandonwillard edited a comment on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-586637145
All right, I just checked locally (with `breeze`) and the [uncovered sections of those unrelated MySql files](https://codecov.io/gh/apache/airflow/pull/7423/changes) are definitely being called by their respective tests, which are also passing. As well, the [Travis log messages for those tests](https://travis-ci.org/apache/airflow/jobs/650753318#L1810) say they passed, and the warnings on Travis look the same as the local ones, so I'm not seeing a difference between local and Travis testing.
In other words, I don't know why CodeCov is saying coverage decreased in those untouched and unrelated files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7423:
[AIRFLOW-3126] Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-586568480
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=h1) Report
> Merging [#7423](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/cc6e46f98d88f730a9d7250f143776f01779cb99?src=pr&el=desc) will **decrease** coverage by `53.77%`.
> The diff coverage is `4.34%`.
[](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7423 +/- ##
===========================================
- Coverage 86.66% 32.88% -53.78%
===========================================
Files 878 877 -1
Lines 41163 41196 +33
===========================================
- Hits 35672 13547 -22125
- Misses 5491 27649 +22158
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5) | `17.21% <0%> (-82.08%)` | :arrow_down: |
| [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `16.43% <5.55%> (-40.38%)` | :arrow_down: |
| [...low/contrib/operators/wasb\_delete\_blob\_operator.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93YXNiX2RlbGV0ZV9ibG9iX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [...ing\_platform/example\_dags/example\_display\_video.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kaXNwbGF5X3ZpZGVvLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/contrib/hooks/vertica\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3ZlcnRpY2FfaG9vay5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/contrib/sensors/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvX19pbml0X18ucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/hooks/mssql\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9tc3NxbF9ob29rLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/hooks/webhdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy93ZWJoZGZzX2hvb2sucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/contrib/sensors/emr\_base\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZW1yX2Jhc2Vfc2Vuc29yLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [...irflow/contrib/operators/slack\_webhook\_operator.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zbGFja193ZWJob29rX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| ... and [746 more](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=footer). Last update [cc6e46f...7206b68](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7423:
[AIRFLOW-3126] Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-586568480
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=h1) Report
> Merging [#7423](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1982c3fdca1f04cfc41fc5b5e285d8f01c6b76ab&el=desc) will **decrease** coverage by `22.33%`.
> The diff coverage is `4.34%`.
[](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7423 +/- ##
===========================================
- Coverage 86.97% 64.64% -22.34%
===========================================
Files 927 927
Lines 44963 45009 +46
===========================================
- Hits 39108 29095 -10013
- Misses 5855 15914 +10059
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5) | `16.88% <0.00%> (-82.43%)` | :arrow_down: |
| [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `16.27% <5.55%> (-40.60%)` | :arrow_down: |
| [airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/pig\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9waWdfaG9vay5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/druid\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kcnVpZF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/mssql\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9tc3NxbF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [498 more](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=footer). Last update [1982c3f...2aa2c86](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] brandonwillard commented on issue #7423: [AIRFLOW-3126]
Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
brandonwillard commented on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-586559488
Also, why is [`isort` failing](https://travis-ci.org/apache/airflow/jobs/650737702#L1742) on an untouched (seemingly GCP-related) file?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7423:
[AIRFLOW-3126] Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-586568480
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=h1) Report
> Merging [#7423](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1982c3fdca1f04cfc41fc5b5e285d8f01c6b76ab&el=desc) will **decrease** coverage by `54.37%`.
> The diff coverage is `4.34%`.
[](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7423 +/- ##
===========================================
- Coverage 86.97% 32.60% -54.38%
===========================================
Files 927 927
Lines 44963 45009 +46
===========================================
- Hits 39108 14673 -24435
- Misses 5855 30336 +24481
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5) | `16.88% <0.00%> (-82.43%)` | :arrow_down: |
| [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `16.27% <5.55%> (-40.60%)` | :arrow_down: |
| [airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/pig\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9waWdfaG9vay5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/druid\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kcnVpZF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/mssql\_hook.py](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9tc3NxbF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [795 more](https://codecov.io/gh/apache/airflow/pull/7423/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=footer). Last update [1982c3f...2aa2c86](https://codecov.io/gh/apache/airflow/pull/7423?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7423: [AIRFLOW-3126] Add
option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-586640118
coverage is guideline and it really only works 100% with master builds. we have some optimisations on building times for PR builds (for example no kubernwtes tests are run if no kuberneyes files are touched). so do not worry about coverage decrease in unrelated files
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] brandonwillard closed pull request #7423: [AIRFLOW-3126]
Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
brandonwillard closed pull request #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] brandonwillard commented on issue #7423: [AIRFLOW-3126]
Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
brandonwillard commented on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-586637145
All right, I just checked locally (with `breeze`) and the uncovered sections of those unrelated MySql files are definitely being called by their respective tests, which are also passing. As well, the [Travis log messages for those tests](https://travis-ci.org/apache/airflow/jobs/650753318#L1810) say they passed, and the warnings on Travis look the same as the local ones, so I'm not seeing an obvious divergence between local and Travis testing.
In other words, I'm not seeing a reason for those coverage decreases in those untouched and unrelated files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] brandonwillard edited a comment on issue #7423:
[AIRFLOW-3126] Add option to specify additional K8s volumes
Posted by GitBox <gi...@apache.org>.
brandonwillard edited a comment on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes
URL: https://github.com/apache/airflow/pull/7423#issuecomment-590958733
@davlum, I've tried it and It did not appear to help—specifically, for default worker pod creation under the K8s Executor. It required too much specificity in the template and seemed to deactivate most other configuration options. Can you give an example of how it can be used to add volumes in exactly this way without sacrificing any of the existing functionality/configurability of worker pod creation?
Also, I don't see how it could possibly help with https://github.com/apache/airflow/pull/7405.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services