You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/02/17 04:55:43 UTC
[GitHub] [airflow] dhegberg opened a new pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
dhegberg opened a new pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437
…ng to Cloudwatch
---
Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
<sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dhegberg removed a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
dhegberg removed a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#issuecomment-588388174
@feluelle Let me know if there are any other changes you'd like me to make.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#issuecomment-586842394
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=h1) Report
> Merging [#7437](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/946bdc23c039637b0383e1269f99bdd1b2426565?src=pr&el=desc) will **increase** coverage by `53.24%`.
> The diff coverage is `93.18%`.
[](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7437 +/- ##
==========================================
+ Coverage 32.95% 86.2% +53.24%
==========================================
Files 878 879 +1
Lines 41188 41263 +75
==========================================
+ Hits 13573 35569 +21996
+ Misses 27615 5694 -21921
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/log/cloudwatch\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY2xvdWR3YXRjaF90YXNrX2hhbmRsZXIucHk=) | `100% <100%> (ø)` | |
| [airflow/config\_templates/airflow\_local\_settings.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2FpcmZsb3dfbG9jYWxfc2V0dGluZ3MucHk=) | `62.5% <25%> (+20.19%)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-39.44%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.38% <0%> (-25.52%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [airflow/bin/airflow](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vYWlyZmxvdw==) | `84.61% <0%> (ø)` | :arrow_up: |
| [airflow/plugins\_manager.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `86.95% <0%> (+0.72%)` | :arrow_up: |
| ... and [748 more](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=footer). Last update [946bdc2...0823ca9](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#issuecomment-586842394
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=h1) Report
> Merging [#7437](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/946bdc23c039637b0383e1269f99bdd1b2426565?src=pr&el=desc) will **decrease** coverage by `0.79%`.
> The diff coverage is `29.54%`.
[](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7437 +/- ##
=========================================
- Coverage 32.95% 32.15% -0.8%
=========================================
Files 878 878
Lines 41188 41250 +62
=========================================
- Hits 13573 13265 -308
- Misses 27615 27985 +370
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/config\_templates/airflow\_local\_settings.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2FpcmZsb3dfbG9jYWxfc2V0dGluZ3MucHk=) | `39.28% <0%> (-3.03%)` | :arrow_down: |
| [airflow/utils/log/cloudwatch\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY2xvdWR3YXRjaF90YXNrX2hhbmRsZXIucHk=) | `32.5% <32.5%> (ø)` | |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `24.48% <0%> (-70.41%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `28.16% <0%> (-58.46%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/pod.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZC5weQ==) | `44.73% <0%> (-47.37%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5) | `37.14% <0%> (-40%)` | :arrow_down: |
| [airflow/kubernetes/pod\_generator.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9nZW5lcmF0b3IucHk=) | `20.5% <0%> (-38%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `39.21% <0%> (-35.3%)` | :arrow_down: |
| ... and [9 more](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=footer). Last update [946bdc2...0823ca9](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#discussion_r386245177
##########
File path: airflow/utils/log/cloudwatch_task_handler.py
##########
@@ -0,0 +1,115 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import watchtower
+from cached_property import cached_property
+
+from airflow.configuration import conf
+from airflow.utils.log.file_task_handler import FileTaskHandler
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class CloudwatchTaskHandler(FileTaskHandler, LoggingMixin):
+ """
+ CloudwatchTaskHandler is a python log handler that handles and reads task instance logs.
+
+ It extends airflow FileTaskHandler and uploads to and reads from Cloudwatch.
+
+ :param base_log_folder: base folder to store logs locally
+ :type base_log_folder: str
+ :param log_group_arn: ARN of the Cloudwatch log group for remote log storage
+ with format arn:aws:logs:{region name}:{account id}:log-group:{group name}
Review comment:
```suggestion
with format ``arn:aws:logs:{region name}:{account id}:log-group:{group name}``
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#discussion_r386018939
##########
File path: airflow/utils/log/cloudwatch_task_handler.py
##########
@@ -0,0 +1,114 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import watchtower
+from cached_property import cached_property
+
+from airflow.configuration import conf
+from airflow.utils.log.file_task_handler import FileTaskHandler
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class CloudwatchTaskHandler(FileTaskHandler, LoggingMixin):
+ """
+ CloudwatchTaskHandler is a python log handler that handles and reads task instance logs.
+
+ It extends airflow FileTaskHandler and uploads to and reads from Cloudwatch.
+
+ :param base_log_folder: base folder to store logs locally
+ :type base_log_folder: str
+ :param log_group_arn: ARN of the Cloudwatch log group for remote log storage
+ :type log_group_arn: str
+ :param filename_template: template for file name (local storage) or log stream name (remote)
+ :type filename_template: str
+ """
+ def __init__(self, base_log_folder, log_group_arn, filename_template):
+ super().__init__(base_log_folder, filename_template)
+ split_arn = log_group_arn.split(':') # arn:aws:logs:<region name>:<account id>:log-group:<group name>
Review comment:
I would suggest to add this comment to the docstring above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#issuecomment-586842394
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=h1) Report
> Merging [#7437](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/a9ad0a929851b6912e0bb8551f1ff80b50281944?src=pr&el=desc) will **decrease** coverage by `0.28%`.
> The diff coverage is `95.55%`.
[](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7437 +/- ##
==========================================
- Coverage 86.82% 86.54% -0.29%
==========================================
Files 891 897 +6
Lines 42081 42749 +668
==========================================
+ Hits 36538 36997 +459
- Misses 5543 5752 +209
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/log/cloudwatch\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY2xvdWR3YXRjaF90YXNrX2hhbmRsZXIucHk=) | `100% <100%> (ø)` | |
| [airflow/config\_templates/airflow\_local\_settings.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2FpcmZsb3dfbG9jYWxfc2V0dGluZ3MucHk=) | `64.28% <33.33%> (-1.1%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-45.08%)` | :arrow_down: |
| [airflow/utils/process\_utils.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9wcm9jZXNzX3V0aWxzLnB5) | `69.11% <0%> (-30.89%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0%> (-25.21%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [airflow/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9fX2luaXRfXy5weQ==) | `92% <0%> (-8%)` | :arrow_down: |
| [airflow/providers/google/cloud/operators/dlp.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9kbHAucHk=) | `96.25% <0%> (-3.75%)` | :arrow_down: |
| ... and [67 more](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=footer). Last update [a9ad0a9...a1db32a](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] boring-cyborg[bot] commented on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#issuecomment-593443592
Awesome work, congrats on your first merged pull request!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#discussion_r385128092
##########
File path: docs/howto/write-logs.rst
##########
@@ -115,6 +115,29 @@ To configure it, you must additionally set the endpoint url to point to your loc
You can do this via the Connection Extra ``host`` field.
For example, ``{"host": "http://localstack:4572"}``
+.. _write-logs-amazon-cloudwatch:
+
+Writing Logs to Amazon Cloudwatch
+---------------------------------
+
+
+Enabling remote logging
+'''''''''''''''''''''''
+
+To enable this feature, ``airflow.cfg`` must be configured as follows:
+
+.. code-block:: ini
+
+ [logging]
+ # Airflow can store logs remotely in AWS Cloudwatch. Users must supply a log group
+ # ARN (starting with 'cloudwatch://...') and an Airflow connection
+ # id that provides write and read access to the log location.
+ remote_logging = True
+ remote_base_log_folder = cloudwatch://arn:aws:logs:<region name>:<account id>:log-group:<group name>:*
Review comment:
What is the `*` on the end here for? What other possible values would it have?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dhegberg commented on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
dhegberg commented on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#issuecomment-593313805
Updated to reflect latest comments.
@ashb Thanks for taking the time to review this.
@feluelle Let me know if you have any other suggestions I should address.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#discussion_r382901838
##########
File path: setup.py
##########
@@ -157,6 +157,7 @@ def write_version(filename: str = os.path.join(*["airflow", "git_version"])):
]
aws = [
'boto3~=1.10',
+ 'watchtower>=0.7.3',
Review comment:
`~=0.7.3` please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#issuecomment-586842394
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=h1) Report
> Merging [#7437](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/946bdc23c039637b0383e1269f99bdd1b2426565?src=pr&el=desc) will **decrease** coverage by `0.79%`.
> The diff coverage is `29.54%`.
[](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7437 +/- ##
=========================================
- Coverage 32.95% 32.15% -0.8%
=========================================
Files 878 878
Lines 41188 41250 +62
=========================================
- Hits 13573 13265 -308
- Misses 27615 27985 +370
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/config\_templates/airflow\_local\_settings.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2FpcmZsb3dfbG9jYWxfc2V0dGluZ3MucHk=) | `39.28% <0%> (-3.03%)` | :arrow_down: |
| [airflow/utils/log/cloudwatch\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY2xvdWR3YXRjaF90YXNrX2hhbmRsZXIucHk=) | `32.5% <32.5%> (ø)` | |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `24.48% <0%> (-70.41%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `28.16% <0%> (-58.46%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/pod.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZC5weQ==) | `44.73% <0%> (-47.37%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5) | `37.14% <0%> (-40%)` | :arrow_down: |
| [airflow/kubernetes/pod\_generator.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9nZW5lcmF0b3IucHk=) | `20.5% <0%> (-38%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `39.21% <0%> (-35.3%)` | :arrow_down: |
| ... and [9 more](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=footer). Last update [946bdc2...0823ca9](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dhegberg commented on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
dhegberg commented on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#issuecomment-592448097
@ashb I've updated based on your latest comments. Please take a look when you get the chance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle merged pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
feluelle merged pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#discussion_r382901557
##########
File path: airflow/utils/log/cloudwatch_task_handler.py
##########
@@ -0,0 +1,101 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import watchtower
+from cached_property import cached_property
+
+from airflow.configuration import conf
+from airflow.utils.log.file_task_handler import FileTaskHandler
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class CloudwatchTaskHandler(FileTaskHandler, LoggingMixin):
+ """
+ CloudwatchTaskHandler is a python log handler that handles and reads task instance logs.
+
+ It extends airflow FileTaskHandler and uploads to and reads from Cloudwatch.
+ """
+ def __init__(self, base_log_folder, log_group, region_name, filename_template):
Review comment:
These should go in the class-level doc string, not the constructor's (as only one of these is displayed)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dhegberg commented on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
dhegberg commented on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#issuecomment-588388174
@feluelle Let me know if there are any other changes you'd like me to make.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj edited a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#issuecomment-587000868
Can you also add a note in the documentation?
https://airflow.readthedocs.io/en/latest/howto/write-logs.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#issuecomment-586842394
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=h1) Report
> Merging [#7437](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/a9ad0a929851b6912e0bb8551f1ff80b50281944?src=pr&el=desc) will **decrease** coverage by `0.28%`.
> The diff coverage is `95.55%`.
[](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7437 +/- ##
==========================================
- Coverage 86.82% 86.54% -0.29%
==========================================
Files 891 897 +6
Lines 42081 42749 +668
==========================================
+ Hits 36538 36995 +457
- Misses 5543 5754 +211
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/log/cloudwatch\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY2xvdWR3YXRjaF90YXNrX2hhbmRsZXIucHk=) | `100% <100%> (ø)` | |
| [airflow/config\_templates/airflow\_local\_settings.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2FpcmZsb3dfbG9jYWxfc2V0dGluZ3MucHk=) | `64.28% <33.33%> (-1.1%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-45.08%)` | :arrow_down: |
| [airflow/utils/process\_utils.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9wcm9jZXNzX3V0aWxzLnB5) | `69.11% <0%> (-30.89%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0%> (-25.21%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [airflow/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9fX2luaXRfXy5weQ==) | `92% <0%> (-8%)` | :arrow_down: |
| [airflow/providers/google/cloud/operators/dlp.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9kbHAucHk=) | `96.25% <0%> (-3.75%)` | :arrow_down: |
| ... and [67 more](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=footer). Last update [a9ad0a9...1600c07](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dhegberg commented on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
dhegberg commented on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#issuecomment-587114047
> Can you also add a note in the documentation?
> https://airflow.readthedocs.io/en/latest/howto/write-logs.html
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj edited a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#issuecomment-587000868
Can you also add note in the documentation?
https://airflow.readthedocs.io/en/latest/howto/write-logs.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#discussion_r380590652
##########
File path: airflow/utils/log/cloudwatch_task_handler.py
##########
@@ -0,0 +1,101 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import watchtower
+from cached_property import cached_property
+
+from airflow.configuration import conf
+from airflow.utils.log.file_task_handler import FileTaskHandler
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class CloudwatchTaskHandler(FileTaskHandler, LoggingMixin):
+ """
+ CloudwatchTaskHandler is a python log handler that handles and reads task instance logs.
+
+ It extends airflow FileTaskHandler and uploads to and reads from Cloudwatch.
+ """
+ def __init__(self, base_log_folder, log_group, region_name, filename_template):
Review comment:
Please document the arguments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#issuecomment-586842394
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=h1) Report
> Merging [#7437](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/946bdc23c039637b0383e1269f99bdd1b2426565?src=pr&el=desc) will **increase** coverage by `53.42%`.
> The diff coverage is `93.18%`.
[](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7437 +/- ##
===========================================
+ Coverage 32.95% 86.37% +53.42%
===========================================
Files 878 879 +1
Lines 41188 41232 +44
===========================================
+ Hits 13573 35615 +22042
+ Misses 27615 5617 -21998
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/log/cloudwatch\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY2xvdWR3YXRjaF90YXNrX2hhbmRsZXIucHk=) | `100% <100%> (ø)` | |
| [airflow/config\_templates/airflow\_local\_settings.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2FpcmZsb3dfbG9jYWxfc2V0dGluZ3MucHk=) | `62.5% <25%> (+20.19%)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-39.44%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.38% <0%> (-25.52%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [airflow/bin/airflow](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vYWlyZmxvdw==) | `84.61% <0%> (ø)` | :arrow_up: |
| [airflow/plugins\_manager.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `86.95% <0%> (+0.72%)` | :arrow_up: |
| ... and [751 more](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=footer). Last update [946bdc2...7dc7e1c](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dhegberg commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
dhegberg commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#discussion_r385254375
##########
File path: docs/howto/write-logs.rst
##########
@@ -115,6 +115,29 @@ To configure it, you must additionally set the endpoint url to point to your loc
You can do this via the Connection Extra ``host`` field.
For example, ``{"host": "http://localstack:4572"}``
+.. _write-logs-amazon-cloudwatch:
+
+Writing Logs to Amazon Cloudwatch
+---------------------------------
+
+
+Enabling remote logging
+'''''''''''''''''''''''
+
+To enable this feature, ``airflow.cfg`` must be configured as follows:
+
+.. code-block:: ini
+
+ [logging]
+ # Airflow can store logs remotely in AWS Cloudwatch. Users must supply a log group
+ # ARN (starting with 'cloudwatch://...') and an Airflow connection
+ # id that provides write and read access to the log location.
+ remote_logging = True
+ remote_base_log_folder = cloudwatch://arn:aws:logs:<region name>:<account id>:log-group:<group name>:*
Review comment:
When you display the ARNs in the cloudwatch console or via a describe-log-groups call, they all have a '*' suffix.
The docs however describe the arn without a suffix: https://docs.aws.amazon.com/IAM/latest/UserGuide/list_amazoncloudwatchlogs.html#amazoncloudwatchlogs-resources-for-iam-policies
I'll go the ARN format that is consistent with the docs and if users include the * when copying the ARN it will work anyway.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#discussion_r380592715
##########
File path: airflow/utils/log/cloudwatch_task_handler.py
##########
@@ -0,0 +1,101 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import watchtower
+from cached_property import cached_property
+
+from airflow.configuration import conf
+from airflow.utils.log.file_task_handler import FileTaskHandler
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class CloudwatchTaskHandler(FileTaskHandler, LoggingMixin):
+ """
+ CloudwatchTaskHandler is a python log handler that handles and reads task instance logs.
+
+ It extends airflow FileTaskHandler and uploads to and reads from Cloudwatch.
+ """
+ def __init__(self, base_log_folder, log_group, region_name, filename_template):
+ super().__init__(base_log_folder, filename_template)
+ self.handler = None
+ self.log_group = log_group
+ self.region_name = region_name
+ self.closed = False
+
+ @cached_property
+ def hook(self):
+ """
+ Returns AwsLogsHook.
+ """
+ remote_conn_id = conf.get('logging', 'REMOTE_LOG_CONN_ID')
+ try:
+ from airflow.providers.amazon.aws.hooks.logs import AwsLogsHook
+ return AwsLogsHook(aws_conn_id=remote_conn_id, region_name=self.region_name)
+ except Exception: # pylint: disable=broad-except
+ self.log.error(
+ 'Could not create an AwsLogsHook with connection id "%s". '
+ 'Please make sure that airflow[aws] is installed and '
+ 'the Cloudwatch logs connection exists.', remote_conn_id
+ )
+
+ def _render_filename(self, ti, try_number):
+ # Replace unsupported log group name characters
+ return super()._render_filename(ti, try_number).replace(':', '_')
+
+ def set_context(self, ti):
+ self.handler = watchtower.CloudWatchLogHandler(
+ log_group=self.log_group,
+ stream_name=self._render_filename(ti, ti.try_number),
+ boto3_session=self.hook.get_session(self.region_name)
+ )
+
+ def close(self):
+ """
+ Close the handler responsible for the upload of the local log file to Cloudwatch.
+ """
+ # When application exit, system shuts down all handlers by
+ # calling close method. Here we check if logger is already
+ # closed to prevent uploading the log to remote storage multiple
+ # times when `logging.shutdown` is called.
+ if self.closed:
+ return
+
+ if self.handler is not None:
+ self.handler.close()
+ # Mark closed so we don't double write if close is called twice
+ self.closed = True
+
+ def _read(self, task_instance, try_number, metadata=None):
+ stream_name = self._render_filename(task_instance, try_number)
+ return '*** Reading remote log from Cloudwatch log_group: {} log_stream: {}.\n{}\n'.format(
+ self.log_group, stream_name, self.get_cloudwatch_logs(stream_name=stream_name)
+ ), {'end_of_log': True}
+
+ def get_cloudwatch_logs(self, stream_name):
Review comment:
Same here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dhegberg commented on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
dhegberg commented on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#issuecomment-589738521
@feluelle @mik-laj Can you take another look at this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#discussion_r382901434
##########
File path: airflow/config_templates/airflow_local_settings.py
##########
@@ -172,6 +173,20 @@
}
DEFAULT_LOGGING_CONFIG['handlers'].update(S3_REMOTE_HANDLERS)
+ elif REMOTE_BASE_LOG_FOLDER.startswith('cloudwatch://'):
+ split_arn = urlparse(REMOTE_BASE_LOG_FOLDER).netloc.split(':')
+ CLOUDWATCH_REMOTE_HANDLERS: Dict[str, Dict[str, str]] = {
+ 'task': {
+ 'class': 'airflow.utils.log.cloudwatch_task_handler.CloudwatchTaskHandler',
+ 'formatter': 'airflow',
+ 'base_log_folder': str(os.path.expanduser(BASE_LOG_FOLDER)),
+ 'log_group': split_arn[6],
Review comment:
I think this sort of parsing should be handled in the TaskHandler, not here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#discussion_r382901609
##########
File path: airflow/utils/log/cloudwatch_task_handler.py
##########
@@ -0,0 +1,110 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import watchtower
+from cached_property import cached_property
+
+from airflow.configuration import conf
+from airflow.utils.log.file_task_handler import FileTaskHandler
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class CloudwatchTaskHandler(FileTaskHandler, LoggingMixin):
+ """
+ CloudwatchTaskHandler is a python log handler that handles and reads task instance logs.
+
+ It extends airflow FileTaskHandler and uploads to and reads from Cloudwatch.
+ """
+ def __init__(self, base_log_folder, log_group, region_name, filename_template):
+ """
+ :param base_log_folder: base folder to store logs locally
+ :param log_group: name of the Cloudwatch log group for remote log storage
+ :param region_name: AWS region name
+ :param filename_template: template for file name (local storage) or log stream name (remote)
+ """
+ super().__init__(base_log_folder, filename_template)
+ self.handler = None
+ self.log_group = log_group
+ self.region_name = region_name
+ self.closed = False
+
+ @cached_property
+ def hook(self):
+ """
+ Returns AwsLogsHook.
+ """
+ remote_conn_id = conf.get('logging', 'REMOTE_LOG_CONN_ID')
+ try:
+ from airflow.providers.amazon.aws.hooks.logs import AwsLogsHook
+ return AwsLogsHook(aws_conn_id=remote_conn_id, region_name=self.region_name)
+ except Exception: # pylint: disable=broad-except
+ self.log.error(
+ 'Could not create an AwsLogsHook with connection id "%s". '
+ 'Please make sure that airflow[aws] is installed and '
+ 'the Cloudwatch logs connection exists.', remote_conn_id
+ )
+
+ def _render_filename(self, ti, try_number):
+ # Replace unsupported log group name characters
+ return super()._render_filename(ti, try_number).replace(':', '_')
+
+ def set_context(self, ti):
+ self.handler = watchtower.CloudWatchLogHandler(
Review comment:
Should probably call super here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#discussion_r385123141
##########
File path: airflow/utils/log/cloudwatch_task_handler.py
##########
@@ -0,0 +1,114 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import watchtower
+from cached_property import cached_property
+
+from airflow.configuration import conf
+from airflow.utils.log.file_task_handler import FileTaskHandler
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class CloudwatchTaskHandler(FileTaskHandler, LoggingMixin):
+ """
+ CloudwatchTaskHandler is a python log handler that handles and reads task instance logs.
+
+ It extends airflow FileTaskHandler and uploads to and reads from Cloudwatch.
+
+ :param base_log_folder: base folder to store logs locally
+ :type base_log_folder: str
+ :param log_group_arn: ARN of the Cloudwatch log group for remote log storage
+ :type log_group_arn: str
+ :param filename_template: template for file name (local storage) or log stream name (remote)
+ :type filename_template: str
+ """
+ def __init__(self, base_log_folder, log_group_arn, filename_template):
+ super().__init__(base_log_folder, filename_template)
+ split_arn = log_group_arn.split(':')
Review comment:
Could you add an example ARN here in a comment? Makes it easier to tell/sanity check what the 4th and 7th parts are.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dhegberg commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
dhegberg commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#discussion_r385254375
##########
File path: docs/howto/write-logs.rst
##########
@@ -115,6 +115,29 @@ To configure it, you must additionally set the endpoint url to point to your loc
You can do this via the Connection Extra ``host`` field.
For example, ``{"host": "http://localstack:4572"}``
+.. _write-logs-amazon-cloudwatch:
+
+Writing Logs to Amazon Cloudwatch
+---------------------------------
+
+
+Enabling remote logging
+'''''''''''''''''''''''
+
+To enable this feature, ``airflow.cfg`` must be configured as follows:
+
+.. code-block:: ini
+
+ [logging]
+ # Airflow can store logs remotely in AWS Cloudwatch. Users must supply a log group
+ # ARN (starting with 'cloudwatch://...') and an Airflow connection
+ # id that provides write and read access to the log location.
+ remote_logging = True
+ remote_base_log_folder = cloudwatch://arn:aws:logs:<region name>:<account id>:log-group:<group name>:*
Review comment:
When you display the ARNs in the cloudwatch console or via a describe-log-groups call, they all have a '*' suffix.
The docs however describe the arn without a suffix: https://docs.aws.amazon.com/IAM/latest/UserGuide/list_amazoncloudwatchlogs.html#amazoncloudwatchlogs-resources-for-iam-policies
I'm happy to go with whichever format you think is less confusing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#issuecomment-586842394
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=h1) Report
> Merging [#7437](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/946bdc23c039637b0383e1269f99bdd1b2426565?src=pr&el=desc) will **increase** coverage by `53.42%`.
> The diff coverage is `93.18%`.
[](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7437 +/- ##
===========================================
+ Coverage 32.95% 86.37% +53.42%
===========================================
Files 878 879 +1
Lines 41188 41263 +75
===========================================
+ Hits 13573 35641 +22068
+ Misses 27615 5622 -21993
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/log/cloudwatch\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY2xvdWR3YXRjaF90YXNrX2hhbmRsZXIucHk=) | `100% <100%> (ø)` | |
| [airflow/config\_templates/airflow\_local\_settings.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2FpcmZsb3dfbG9jYWxfc2V0dGluZ3MucHk=) | `62.5% <25%> (+20.19%)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-39.44%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.38% <0%> (-25.52%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [airflow/bin/airflow](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vYWlyZmxvdw==) | `84.61% <0%> (ø)` | :arrow_up: |
| [airflow/plugins\_manager.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `86.95% <0%> (+0.72%)` | :arrow_up: |
| ... and [748 more](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=footer). Last update [946bdc2...0823ca9](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#discussion_r386019415
##########
File path: tests/utils/log/test_cloudwatch_task_handler.py
##########
@@ -0,0 +1,223 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+from unittest import mock
+from unittest.mock import call
+
+from watchtower import CloudWatchLogHandler
+
+from airflow.models import DAG, TaskInstance
+from airflow.operators.dummy_operator import DummyOperator
+from airflow.providers.amazon.aws.hooks.logs import AwsLogsHook
+from airflow.utils.log.cloudwatch_task_handler import CloudwatchTaskHandler
+from airflow.utils.state import State
+from airflow.utils.timezone import datetime
+
+try:
+ import boto3
+ import moto
+ from moto import mock_logs
+except ImportError:
+ mock_logs = None
+
+
+@unittest.skipIf(mock_logs is None,
+ "Skipping test because moto.mock_logs is not available")
+@mock_logs
+class TestCloudwatchTaskHandler(unittest.TestCase):
+
+ def setUp(self):
+ super().setUp()
Review comment:
```suggestion
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dhegberg commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
dhegberg commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#discussion_r385254375
##########
File path: docs/howto/write-logs.rst
##########
@@ -115,6 +115,29 @@ To configure it, you must additionally set the endpoint url to point to your loc
You can do this via the Connection Extra ``host`` field.
For example, ``{"host": "http://localstack:4572"}``
+.. _write-logs-amazon-cloudwatch:
+
+Writing Logs to Amazon Cloudwatch
+---------------------------------
+
+
+Enabling remote logging
+'''''''''''''''''''''''
+
+To enable this feature, ``airflow.cfg`` must be configured as follows:
+
+.. code-block:: ini
+
+ [logging]
+ # Airflow can store logs remotely in AWS Cloudwatch. Users must supply a log group
+ # ARN (starting with 'cloudwatch://...') and an Airflow connection
+ # id that provides write and read access to the log location.
+ remote_logging = True
+ remote_base_log_folder = cloudwatch://arn:aws:logs:<region name>:<account id>:log-group:<group name>:*
Review comment:
When you display the ARNs in the cloudwatch console or via a describe-log-groups call, they all have a '*' suffix.
The docs however describe the arn without a suffix: https://docs.aws.amazon.com/IAM/latest/UserGuide/list_amazoncloudwatchlogs.html#amazoncloudwatchlogs-resources-for-iam-policies
Ill go the ARN format that is consistent with the docs and if users include the * when copying the ARN it will work anyway.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dhegberg commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
dhegberg commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#discussion_r380767089
##########
File path: airflow/utils/log/cloudwatch_task_handler.py
##########
@@ -0,0 +1,101 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import watchtower
+from cached_property import cached_property
+
+from airflow.configuration import conf
+from airflow.utils.log.file_task_handler import FileTaskHandler
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class CloudwatchTaskHandler(FileTaskHandler, LoggingMixin):
+ """
+ CloudwatchTaskHandler is a python log handler that handles and reads task instance logs.
+
+ It extends airflow FileTaskHandler and uploads to and reads from Cloudwatch.
+ """
+ def __init__(self, base_log_folder, log_group, region_name, filename_template):
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#issuecomment-586842394
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=h1) Report
> Merging [#7437](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/946bdc23c039637b0383e1269f99bdd1b2426565?src=pr&el=desc) will **increase** coverage by `53.42%`.
> The diff coverage is `93.18%`.
[](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7437 +/- ##
===========================================
+ Coverage 32.95% 86.38% +53.42%
===========================================
Files 878 878
Lines 41188 41296 +108
===========================================
+ Hits 13573 35673 +22100
+ Misses 27615 5623 -21992
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/log/cloudwatch\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY2xvdWR3YXRjaF90YXNrX2hhbmRsZXIucHk=) | `100% <100%> (ø)` | |
| [airflow/config\_templates/airflow\_local\_settings.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2FpcmZsb3dfbG9jYWxfc2V0dGluZ3MucHk=) | `62.5% <25%> (+20.19%)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-39.44%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.38% <0%> (-25.52%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [airflow/config\_templates/default\_celery.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2RlZmF1bHRfY2VsZXJ5LnB5) | `52.94% <0%> (ø)` | :arrow_up: |
| [airflow/contrib/hooks/aws\_hook.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F3c19ob29rLnB5) | `100% <0%> (ø)` | :arrow_up: |
| [airflow/bin/airflow](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vYWlyZmxvdw==) | `84.61% <0%> (ø)` | :arrow_up: |
| ... and [765 more](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=footer). Last update [946bdc2...afcf5d0](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dhegberg commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
dhegberg commented on a change in pull request #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#discussion_r380766956
##########
File path: airflow/utils/log/cloudwatch_task_handler.py
##########
@@ -0,0 +1,101 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import watchtower
+from cached_property import cached_property
+
+from airflow.configuration import conf
+from airflow.utils.log.file_task_handler import FileTaskHandler
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class CloudwatchTaskHandler(FileTaskHandler, LoggingMixin):
+ """
+ CloudwatchTaskHandler is a python log handler that handles and reads task instance logs.
+
+ It extends airflow FileTaskHandler and uploads to and reads from Cloudwatch.
+ """
+ def __init__(self, base_log_folder, log_group, region_name, filename_template):
+ super().__init__(base_log_folder, filename_template)
+ self.handler = None
+ self.log_group = log_group
+ self.region_name = region_name
+ self.closed = False
+
+ @cached_property
+ def hook(self):
+ """
+ Returns AwsLogsHook.
+ """
+ remote_conn_id = conf.get('logging', 'REMOTE_LOG_CONN_ID')
+ try:
+ from airflow.providers.amazon.aws.hooks.logs import AwsLogsHook
+ return AwsLogsHook(aws_conn_id=remote_conn_id, region_name=self.region_name)
+ except Exception: # pylint: disable=broad-except
+ self.log.error(
+ 'Could not create an AwsLogsHook with connection id "%s". '
+ 'Please make sure that airflow[aws] is installed and '
+ 'the Cloudwatch logs connection exists.', remote_conn_id
+ )
+
+ def _render_filename(self, ti, try_number):
+ # Replace unsupported log group name characters
+ return super()._render_filename(ti, try_number).replace(':', '_')
+
+ def set_context(self, ti):
+ self.handler = watchtower.CloudWatchLogHandler(
+ log_group=self.log_group,
+ stream_name=self._render_filename(ti, ti.try_number),
+ boto3_session=self.hook.get_session(self.region_name)
+ )
+
+ def close(self):
+ """
+ Close the handler responsible for the upload of the local log file to Cloudwatch.
+ """
+ # When application exit, system shuts down all handlers by
+ # calling close method. Here we check if logger is already
+ # closed to prevent uploading the log to remote storage multiple
+ # times when `logging.shutdown` is called.
+ if self.closed:
+ return
+
+ if self.handler is not None:
+ self.handler.close()
+ # Mark closed so we don't double write if close is called twice
+ self.closed = True
+
+ def _read(self, task_instance, try_number, metadata=None):
+ stream_name = self._render_filename(task_instance, try_number)
+ return '*** Reading remote log from Cloudwatch log_group: {} log_stream: {}.\n{}\n'.format(
+ self.log_group, stream_name, self.get_cloudwatch_logs(stream_name=stream_name)
+ ), {'end_of_log': True}
+
+ def get_cloudwatch_logs(self, stream_name):
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#issuecomment-586842394
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=h1) Report
> Merging [#7437](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/a9ad0a929851b6912e0bb8551f1ff80b50281944?src=pr&el=desc) will **decrease** coverage by `0.46%`.
> The diff coverage is `93.18%`.
[](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7437 +/- ##
==========================================
- Coverage 86.82% 86.36% -0.47%
==========================================
Files 891 892 +1
Lines 42081 42125 +44
==========================================
- Hits 36538 36382 -156
- Misses 5543 5743 +200
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/log/cloudwatch\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY2xvdWR3YXRjaF90YXNrX2hhbmRsZXIucHk=) | `100% <100%> (ø)` | |
| [airflow/config\_templates/airflow\_local\_settings.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2FpcmZsb3dfbG9jYWxfc2V0dGluZ3MucHk=) | `62.5% <25%> (-2.89%)` | :arrow_down: |
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0%> (-64.16%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0%> (-45.66%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-45.08%)` | :arrow_down: |
| [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55% <0%> (-45%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.38% <0%> (-25.52%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| ... and [6 more](https://codecov.io/gh/apache/airflow/pull/7437/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=footer). Last update [a9ad0a9...107fc6c](https://codecov.io/gh/apache/airflow/pull/7437?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
feluelle commented on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#issuecomment-593443949
Nice work @dhegberg! 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #7437: [AIRFLOW-2325] Add CloudwatchTaskHandler option for remote task loggi…
URL: https://github.com/apache/airflow/pull/7437#issuecomment-587000868
Can you also add the documentation?
https://airflow.readthedocs.io/en/latest/howto/write-logs.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services