You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/01/17 22:19:14 UTC
[GitHub] [airflow] jaketf opened a new pull request #7197: [AIRFLOW-6586]
Improvements to gcs sensor
jaketf opened a new pull request #7197: [AIRFLOW-6586] Improvements to gcs sensor
URL: https://github.com/apache/airflow/pull/7197
refactors GoogleCloudStorageUploadSessionCompleteSensor to use set instead of number of objects
add poke mode only decorator
assert that poke_mode_only applied to child of BaseSensorOperator
---
Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [ ] Description above provides context of the change
- [ ] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
- [ ] Unit tests coverage for changes (not needed for documentation changes)
- [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [ ] Relevant documentation is updated including usage instructions.
- [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
<sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7197:
[AIRFLOW-6586][WIP] Improvements to gcs sensor
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7197: [AIRFLOW-6586][WIP] Improvements to gcs sensor
URL: https://github.com/apache/airflow/pull/7197#issuecomment-616140990
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=h1) Report
> Merging [#7197](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8517cb189872aebdd70e9d5d584478cd7fb65d0f&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7197 +/- ##
======================================
Coverage 6.23% 6.23%
======================================
Files 946 946
Lines 45665 45665
======================================
Hits 2846 2846
Misses 42819 42819
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=footer). Last update [39a5bd6...39a5bd6](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7197: [AIRFLOW-6586][WIP]
Improvements to gcs sensor
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7197: [AIRFLOW-6586][WIP] Improvements to gcs sensor
URL: https://github.com/apache/airflow/pull/7197#issuecomment-616140990
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=h1) Report
> Merging [#7197](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8517cb189872aebdd70e9d5d584478cd7fb65d0f&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7197 +/- ##
======================================
Coverage 6.23% 6.23%
======================================
Files 946 946
Lines 45665 45665
======================================
Hits 2846 2846
Misses 42819 42819
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=footer). Last update [39a5bd6...39a5bd6](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7197:
[AIRFLOW-6586][WIP] Improvements to gcs sensor
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7197: [AIRFLOW-6586][WIP] Improvements to gcs sensor
URL: https://github.com/apache/airflow/pull/7197#issuecomment-616140990
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=h1) Report
> Merging [#7197](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8517cb189872aebdd70e9d5d584478cd7fb65d0f&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7197 +/- ##
======================================
Coverage 6.23% 6.23%
======================================
Files 946 946
Lines 45665 45665
======================================
Hits 2846 2846
Misses 42819 42819
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=footer). Last update [061b0e1...061b0e1](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7197:
[AIRFLOW-6586][WIP] Improvements to gcs sensor
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7197: [AIRFLOW-6586][WIP] Improvements to gcs sensor
URL: https://github.com/apache/airflow/pull/7197#issuecomment-616140990
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=h1) Report
> Merging [#7197](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8517cb189872aebdd70e9d5d584478cd7fb65d0f&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7197 +/- ##
======================================
Coverage 6.23% 6.23%
======================================
Files 946 946
Lines 45665 45665
======================================
Hits 2846 2846
Misses 42819 42819
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=footer). Last update [39a5bd6...39a5bd6](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on issue #7197: [AIRFLOW-6586][WIP]
Improvements to gcs sensor
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #7197: [AIRFLOW-6586][WIP] Improvements to gcs sensor
URL: https://github.com/apache/airflow/pull/7197#issuecomment-610417544
@jaketf any update?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] stale[bot] commented on issue #7197: [AIRFLOW-6586][WIP]
Improvements to gcs sensor
Posted by GitBox <gi...@apache.org>.
stale[bot] commented on issue #7197: [AIRFLOW-6586][WIP] Improvements to gcs sensor
URL: https://github.com/apache/airflow/pull/7197#issuecomment-610329387
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7197:
[AIRFLOW-6586][WIP] Improvements to gcs sensor
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7197: [AIRFLOW-6586][WIP] Improvements to gcs sensor
URL: https://github.com/apache/airflow/pull/7197#issuecomment-616140990
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=h1) Report
> Merging [#7197](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8517cb189872aebdd70e9d5d584478cd7fb65d0f&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7197 +/- ##
======================================
Coverage 6.23% 6.23%
======================================
Files 946 946
Lines 45665 45665
======================================
Hits 2846 2846
Misses 42819 42819
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=footer). Last update [39a5bd6...39a5bd6](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7197:
[AIRFLOW-6586][WIP] Improvements to gcs sensor
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7197: [AIRFLOW-6586][WIP] Improvements to gcs sensor
URL: https://github.com/apache/airflow/pull/7197#issuecomment-616140990
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=h1) Report
> Merging [#7197](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8517cb189872aebdd70e9d5d584478cd7fb65d0f&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7197 +/- ##
======================================
Coverage 6.23% 6.23%
======================================
Files 946 946
Lines 45665 45665
======================================
Hits 2846 2846
Misses 42819 42819
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=footer). Last update [39a5bd6...39a5bd6](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7197:
[AIRFLOW-6586][WIP] Improvements to gcs sensor
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7197: [AIRFLOW-6586][WIP] Improvements to gcs sensor
URL: https://github.com/apache/airflow/pull/7197#issuecomment-616140990
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=h1) Report
> Merging [#7197](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8517cb189872aebdd70e9d5d584478cd7fb65d0f&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7197 +/- ##
======================================
Coverage 6.23% 6.23%
======================================
Files 946 946
Lines 45665 45665
======================================
Hits 2846 2846
Misses 42819 42819
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=footer). Last update [39a5bd6...39a5bd6](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] jaketf commented on issue #7197: [AIRFLOW-6586][WIP]
Improvements to gcs sensor
Posted by GitBox <gi...@apache.org>.
jaketf commented on issue #7197: [AIRFLOW-6586][WIP] Improvements to gcs sensor
URL: https://github.com/apache/airflow/pull/7197#issuecomment-613476722
Apologies for letting this go stale I will revive this later this week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7197:
[AIRFLOW-6586][WIP] Improvements to gcs sensor
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7197: [AIRFLOW-6586][WIP] Improvements to gcs sensor
URL: https://github.com/apache/airflow/pull/7197#issuecomment-616140990
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=h1) Report
> Merging [#7197](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8517cb189872aebdd70e9d5d584478cd7fb65d0f&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7197 +/- ##
======================================
Coverage 6.23% 6.23%
======================================
Files 946 946
Lines 45665 45665
======================================
Hits 2846 2846
Misses 42819 42819
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=footer). Last update [39a5bd6...39a5bd6](https://codecov.io/gh/apache/airflow/pull/7197?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7197: [AIRFLOW-6586][WIP] Improvements to gcs sensor
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7197: [AIRFLOW-6586][WIP] Improvements to gcs sensor
URL: https://github.com/apache/airflow/pull/7197#discussion_r382006081
##########
File path: airflow/sensors/decorators/poke_mode_only.py
##########
@@ -0,0 +1,75 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+import inspect
+import os
+from functools import wraps
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+
+
+def poke_mode_only(cls):
+ """
+ Class Decorator for child classes of BaseSensorOperator to indicate
+ that instances of this class are only safe to use poke mode.
+
+ Will decorate all methods in the class to assert they did not change
+ the mode from 'poke'.
+
+ :param cls: BaseSensor class to enforce methods only use 'poke' mode.
+ :type cls: type
+ """
+ def decorate(cls):
+ assert issubclass(cls, BaseSensorOperator)
Review comment:
Please use explicit exception. We've agreed to not use asserts in production code:
https://lists.apache.org/thread.html/bcf2d23fcd79e21b3aac9f32914e1bf656e05ffbcb8aa282af497a2d%40%3Cdev.airflow.apache.org%3E
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services