You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/01/31 15:09:33 UTC

[GitHub] [airflow] michalslowikowski00 opened a new pull request #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator

michalslowikowski00 opened a new pull request #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator
URL: https://github.com/apache/airflow/pull/7307
 
 
   ---
   Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ ] Description above provides context of the change
   - [ ] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   <sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator
URL: https://github.com/apache/airflow/pull/7307#issuecomment-580914401
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=h1) Report
   > Merging [#7307](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/f3ad5cf6185b9d406d0fb0a4ecc0b5536f79217a?src=pr&el=desc) will **increase** coverage by `53.04%`.
   > The diff coverage is `95%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7307/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #7307       +/-   ##
   ==========================================
   + Coverage   32.86%   85.9%   +53.04%     
   ==========================================
     Files         867     871        +4     
     Lines       40564   40610       +46     
   ==========================================
   + Hits        13332   34887    +21555     
   + Misses      27232    5723    -21509
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/providers/google/cloud/hooks/gcs.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2djcy5weQ==) | `85.51% <100%> (+70.86%)` | :arrow_up: |
   | [airflow/providers/google/cloud/operators/gcs.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9nY3MucHk=) | `86.51% <100%> (+57.17%)` | :arrow_up: |
   | [...providers/google/cloud/example\_dags/example\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX2djcy5weQ==) | `96.29% <89.47%> (+96.29%)` | :arrow_up: |
   | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-43.8%)` | :arrow_down: |
   | [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `50.67% <0%> (-34.46%)` | :arrow_down: |
   | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: |
   | [airflow/bin/airflow](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vYWlyZmxvdw==) | `84.61% <0%> (ø)` | :arrow_up: |
   | ... and [743 more](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=footer). Last update [f3ad5cf...7739d23](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] michalslowikowski00 commented on a change in pull request #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator

Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator
URL: https://github.com/apache/airflow/pull/7307#discussion_r373946095
 
 

 ##########
 File path: airflow/providers/google/cloud/hooks/gcs.py
 ##########
 @@ -323,6 +324,27 @@ def delete(self, bucket_name, object_name):
 
         self.log.info('Blob %s deleted.', object_name)
 
+    def delete_bucket(self, bucket_name: str, force: bool = False):
+        """
+        Delete a bucket object from the Google Cloud Storage
 
 Review comment:
   Done.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] michalslowikowski00 commented on issue #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator

Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on issue #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator
URL: https://github.com/apache/airflow/pull/7307#issuecomment-581781667
 
 
   Thank you @nuclearpinguin 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator
URL: https://github.com/apache/airflow/pull/7307#issuecomment-580914401
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=h1) Report
   > Merging [#7307](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/a2d6a2f85e07c38be479e91e4a27981f308f4711?src=pr&el=desc) will **decrease** coverage by `0.44%`.
   > The diff coverage is `94.87%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7307/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff            @@
   ##           master   #7307      +/-   ##
   =========================================
   - Coverage   85.34%   84.9%   -0.45%     
   =========================================
     Files         863     863              
     Lines       40484   40523      +39     
   =========================================
   - Hits        34552   34406     -146     
   - Misses       5932    6117     +185
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/providers/google/cloud/hooks/gcs.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2djcy5weQ==) | `85.51% <100%> (+0.53%)` | :arrow_up: |
   | [airflow/providers/google/cloud/operators/gcs.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9nY3MucHk=) | `86.44% <100%> (+0.81%)` | :arrow_up: |
   | [...providers/google/cloud/example\_dags/example\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX2djcy5weQ==) | `96.29% <89.47%> (-3.71%)` | :arrow_down: |
   | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0%> (-72.16%)` | :arrow_down: |
   | [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0%> (-54.93%)` | :arrow_down: |
   | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50% <0%> (-50%)` | :arrow_down: |
   | [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0%> (-46.67%)` | :arrow_down: |
   | [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `49.65% <0%> (-38.78%)` | :arrow_down: |
   | [airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=) | `61.53% <0%> (-38.47%)` | :arrow_down: |
   | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `85% <0%> (-8.34%)` | :arrow_down: |
   | ... and [2 more](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=footer). Last update [a2d6a2f...06a05ef](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] michalslowikowski00 commented on issue #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator

Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on issue #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator
URL: https://github.com/apache/airflow/pull/7307#issuecomment-581268174
 
 
   @potiuk Thank you.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator
URL: https://github.com/apache/airflow/pull/7307#issuecomment-580914401
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=h1) Report
   > Merging [#7307](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/a2d6a2f85e07c38be479e91e4a27981f308f4711?src=pr&el=desc) will **decrease** coverage by `0.44%`.
   > The diff coverage is `94.87%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7307/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff            @@
   ##           master   #7307      +/-   ##
   =========================================
   - Coverage   85.34%   84.9%   -0.45%     
   =========================================
     Files         863     863              
     Lines       40484   40523      +39     
   =========================================
   - Hits        34552   34406     -146     
   - Misses       5932    6117     +185
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/providers/google/cloud/hooks/gcs.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2djcy5weQ==) | `85.51% <100%> (+0.53%)` | :arrow_up: |
   | [airflow/providers/google/cloud/operators/gcs.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9nY3MucHk=) | `86.44% <100%> (+0.81%)` | :arrow_up: |
   | [...providers/google/cloud/example\_dags/example\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX2djcy5weQ==) | `96.29% <89.47%> (-3.71%)` | :arrow_down: |
   | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0%> (-72.16%)` | :arrow_down: |
   | [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0%> (-54.93%)` | :arrow_down: |
   | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50% <0%> (-50%)` | :arrow_down: |
   | [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0%> (-46.67%)` | :arrow_down: |
   | [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `49.65% <0%> (-38.78%)` | :arrow_down: |
   | [airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=) | `61.53% <0%> (-38.47%)` | :arrow_down: |
   | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `85% <0%> (-8.34%)` | :arrow_down: |
   | ... and [2 more](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=footer). Last update [a2d6a2f...06a05ef](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator

Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator
URL: https://github.com/apache/airflow/pull/7307#discussion_r373678646
 
 

 ##########
 File path: airflow/providers/google/cloud/hooks/gcs.py
 ##########
 @@ -323,6 +324,27 @@ def delete(self, bucket_name, object_name):
 
         self.log.info('Blob %s deleted.', object_name)
 
+    def delete_bucket(self, bucket_name: str, force: bool = False):
+        """
+        Delete a bucket object from the Google Cloud Storage
 
 Review comment:
   ```suggestion
           Delete a bucket object from the Google Cloud Storage.
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] michalslowikowski00 commented on a change in pull request #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator

Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator
URL: https://github.com/apache/airflow/pull/7307#discussion_r373946147
 
 

 ##########
 File path: airflow/providers/google/cloud/operators/gcs.py
 ##########
 @@ -625,3 +625,32 @@ def execute(self, context: Dict):
                 object_name=self.destination_object,
                 filename=destination_file.name
             )
+
+
+class GCSDeleteBucketOperator(BaseOperator):
+    """
+    Deletes bucket from a Google Cloud Storage.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:GCSDeleteBucketOperator`
+
+    :param bucket_name: name of the bucket which will be deleted
+    :type bucket_name: str
+    """
+
+    template_fields = ('bucket_name', "gcp_conn_id")
+
+    @apply_defaults
+    def __init__(self,
+                 bucket_name: str,
+                 gcp_conn_id: str = 'google_cloud_default',
+                 *args, **kwargs) -> None:
+        super().__init__(*args, **kwargs)
+
+        self.bucket_name = bucket_name
+        self.gcp_conn_id = gcp_conn_id
+
+    def execute(self, context):
+        hook = GCSHook(gcp_conn_id=self.gcp_conn_id, )
+        hook.delete_bucket(self.bucket_name)
 
 Review comment:
   Thank you. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator

Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator
URL: https://github.com/apache/airflow/pull/7307#discussion_r373682965
 
 

 ##########
 File path: airflow/providers/google/cloud/example_dags/example_gcs.py
 ##########
 @@ -127,7 +128,32 @@
         task_id="delete_files", bucket_name=BUCKET_1, objects=[BUCKET_FILE_LOCATION]
     )
 
+    # [START howto_operator_gcs_delete_bucket]
+    delete_bucket_1 = GCSDeleteBucketOperator(task_id="delete_bucket", bucket_name=BUCKET_1)
+    delete_bucket_2 = GCSDeleteBucketOperator(task_id="delete_bucket", bucket_name=BUCKET_2)
+    # [END howto_operator_gcs_delete_bucket]
+
     [create_bucket1, create_bucket2] >> list_buckets >> list_buckets_result
     [create_bucket1, create_bucket2] >> upload_file
     upload_file >> [download_file, copy_file]
     upload_file >> gcs_bucket_create_acl_entry_task >> gcs_object_create_acl_entry_task >> delete_files
+
+    create_bucket1 >> delete_bucket_1
+    create_bucket2 >> delete_bucket_2
+    create_bucket2 >> copy_file
+    create_bucket1 >> copy_file
+    list_buckets >> delete_bucket_1
+    upload_file >> delete_bucket_1
+    create_bucket1 >> upload_file >> delete_bucket_1
+    transform_file >> delete_bucket_1
+    gcs_bucket_create_acl_entry_task >> delete_bucket_1
+    gcs_object_create_acl_entry_task >> delete_bucket_1
+    download_file >> delete_bucket_1
+    copy_file >> delete_bucket_1
+    copy_file >> delete_bucket_2
+    delete_files >> delete_bucket_1
 
 Review comment:
   The previous version was incorrect in a few cases. This can be simplified if you look at the fact that some of the requirements are included in others, but the current version makes it easier to understand the relationship between tasks. I note that in Airflow we define requirements and Airflow can automatically detect redundancy and optimize it. If there is something the computer can do, the computer should do it.
   
   I'm worried about task `transform_file`, which has no upstream dependencies, so so this task can be executed before the bucket is created.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] michalslowikowski00 commented on a change in pull request #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator

Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator
URL: https://github.com/apache/airflow/pull/7307#discussion_r373946078
 
 

 ##########
 File path: airflow/providers/google/cloud/operators/gcs.py
 ##########
 @@ -625,3 +625,32 @@ def execute(self, context: Dict):
                 object_name=self.destination_object,
                 filename=destination_file.name
             )
+
+
+class GCSDeleteBucketOperator(BaseOperator):
+    """
+    Deletes bucket from a Google Cloud Storage.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:GCSDeleteBucketOperator`
+
+    :param bucket_name: name of the bucket which will be deleted
+    :type bucket_name: str
+    """
+
+    template_fields = ('bucket_name', "gcp_conn_id")
+
+    @apply_defaults
+    def __init__(self,
+                 bucket_name: str,
 
 Review comment:
   I agree, thank you.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator

Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator
URL: https://github.com/apache/airflow/pull/7307#discussion_r373678546
 
 

 ##########
 File path: airflow/providers/google/cloud/example_dags/example_gcs.py
 ##########
 @@ -127,7 +128,32 @@
         task_id="delete_files", bucket_name=BUCKET_1, objects=[BUCKET_FILE_LOCATION]
     )
 
+    # [START howto_operator_gcs_delete_bucket]
+    delete_bucket_1 = GCSDeleteBucketOperator(task_id="delete_bucket", bucket_name=BUCKET_1)
+    delete_bucket_2 = GCSDeleteBucketOperator(task_id="delete_bucket", bucket_name=BUCKET_2)
+    # [END howto_operator_gcs_delete_bucket]
+
     [create_bucket1, create_bucket2] >> list_buckets >> list_buckets_result
     [create_bucket1, create_bucket2] >> upload_file
     upload_file >> [download_file, copy_file]
     upload_file >> gcs_bucket_create_acl_entry_task >> gcs_object_create_acl_entry_task >> delete_files
+
+    create_bucket1 >> delete_bucket_1
+    create_bucket2 >> delete_bucket_2
+    create_bucket2 >> copy_file
+    create_bucket1 >> copy_file
+    list_buckets >> delete_bucket_1
+    upload_file >> delete_bucket_1
+    create_bucket1 >> upload_file >> delete_bucket_1
+    transform_file >> delete_bucket_1
+    gcs_bucket_create_acl_entry_task >> delete_bucket_1
+    gcs_object_create_acl_entry_task >> delete_bucket_1
+    download_file >> delete_bucket_1
+    copy_file >> delete_bucket_1
+    copy_file >> delete_bucket_2
+    delete_files >> delete_bucket_1
 
 Review comment:
   Can it be simplified? I think the previous version was much more clearer.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator

Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator
URL: https://github.com/apache/airflow/pull/7307#discussion_r373679320
 
 

 ##########
 File path: airflow/providers/google/cloud/operators/gcs.py
 ##########
 @@ -625,3 +625,32 @@ def execute(self, context: Dict):
                 object_name=self.destination_object,
                 filename=destination_file.name
             )
+
+
+class GCSDeleteBucketOperator(BaseOperator):
+    """
+    Deletes bucket from a Google Cloud Storage.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:GCSDeleteBucketOperator`
+
+    :param bucket_name: name of the bucket which will be deleted
+    :type bucket_name: str
+    """
+
+    template_fields = ('bucket_name', "gcp_conn_id")
+
+    @apply_defaults
+    def __init__(self,
+                 bucket_name: str,
+                 gcp_conn_id: str = 'google_cloud_default',
+                 *args, **kwargs) -> None:
+        super().__init__(*args, **kwargs)
+
+        self.bucket_name = bucket_name
+        self.gcp_conn_id = gcp_conn_id
+
+    def execute(self, context):
+        hook = GCSHook(gcp_conn_id=self.gcp_conn_id, )
+        hook.delete_bucket(self.bucket_name)
 
 Review comment:
   ```suggestion
           hook.delete_bucket(bucket_name=self.bucket_name, force=self.force)
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin merged pull request #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator

Posted by GitBox <gi...@apache.org>.
nuclearpinguin merged pull request #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator
URL: https://github.com/apache/airflow/pull/7307
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator
URL: https://github.com/apache/airflow/pull/7307#issuecomment-580914401
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=h1) Report
   > Merging [#7307](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/f3ad5cf6185b9d406d0fb0a4ecc0b5536f79217a?src=pr&el=desc) will **increase** coverage by `53.04%`.
   > The diff coverage is `95%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7307/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #7307       +/-   ##
   ==========================================
   + Coverage   32.86%   85.9%   +53.04%     
   ==========================================
     Files         867     871        +4     
     Lines       40564   40610       +46     
   ==========================================
   + Hits        13332   34887    +21555     
   + Misses      27232    5723    -21509
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/providers/google/cloud/hooks/gcs.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2djcy5weQ==) | `85.51% <100%> (+70.86%)` | :arrow_up: |
   | [airflow/providers/google/cloud/operators/gcs.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9nY3MucHk=) | `86.51% <100%> (+57.17%)` | :arrow_up: |
   | [...providers/google/cloud/example\_dags/example\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX2djcy5weQ==) | `96.29% <89.47%> (+96.29%)` | :arrow_up: |
   | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-43.8%)` | :arrow_down: |
   | [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `50.67% <0%> (-34.46%)` | :arrow_down: |
   | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: |
   | [airflow/bin/airflow](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vYWlyZmxvdw==) | `84.61% <0%> (ø)` | :arrow_up: |
   | ... and [743 more](https://codecov.io/gh/apache/airflow/pull/7307/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=footer). Last update [f3ad5cf...7739d23](https://codecov.io/gh/apache/airflow/pull/7307?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator
URL: https://github.com/apache/airflow/pull/7307#issuecomment-581136406
 
 
   Please rebase to latest master @michalslowikowski00   - we had a failing master drama and there is much higher chance the build will succeed. We have still some intermittent errors, but there should be less often.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator

Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7307: [AIRFLOW-6676] added GCSDeleteBucketOperator
URL: https://github.com/apache/airflow/pull/7307#discussion_r373679194
 
 

 ##########
 File path: airflow/providers/google/cloud/operators/gcs.py
 ##########
 @@ -625,3 +625,32 @@ def execute(self, context: Dict):
                 object_name=self.destination_object,
                 filename=destination_file.name
             )
+
+
+class GCSDeleteBucketOperator(BaseOperator):
+    """
+    Deletes bucket from a Google Cloud Storage.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:GCSDeleteBucketOperator`
+
+    :param bucket_name: name of the bucket which will be deleted
+    :type bucket_name: str
+    """
+
+    template_fields = ('bucket_name', "gcp_conn_id")
+
+    @apply_defaults
+    def __init__(self,
+                 bucket_name: str,
 
 Review comment:
   ```suggestion
                    bucket_name: str,
                    force: bool = True,
   ```
   It's good idea to add this option to operator. In my opinion it should default to `True`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services