You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/02/02 14:24:31 UTC
[GitHub] [airflow] rachael-ds opened a new pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
rachael-ds opened a new pull request #14033:
URL: https://github.com/apache/airflow/pull/14033
Updated documentation and example dags for:
CloudDLPDeidentifyContentOperator
GCSObjectExistenceSensor
GCSObjectsWithPrefixExistenceSensor
Some of the other operators for DLP and sensors GCS don't work as I would have expected - I'll raise a separate bug for this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #14033: Documentation and example dag for CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#issuecomment-784398014
Awesome work, congrats on your first merged pull request!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rachael-ds commented on a change in pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
rachael-ds commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r570913695
##########
File path: docs/apache-airflow-providers-google/sensors/cloud/gcs.rst
##########
@@ -0,0 +1,75 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+Google Cloud Storage Sensors
Review comment:
Thanks for the heads up - done!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r568830202
##########
File path: docs/apache-airflow-providers-google/sensors/cloud/gcs.rst
##########
@@ -0,0 +1,75 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+Google Cloud Storage Sensors
Review comment:
We describe the sensors and ordinary operators in one guide, so that there is one place with all the information for the service. Transfer operators are a bit more problematic as they integrate with two services in one operator, so they have separate guides. Can you move this guide to ``docs/apache-airflow-providers-google/operators/cloud/gcs.rst``?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r571323573
##########
File path: tests/providers/google/cloud/operators/test_gcs_system.py
##########
@@ -22,6 +22,7 @@
from tests.test_utils.gcp_system_helpers import CLOUD_DAG_FOLDER, GoogleSystemTest, provide_gcp_context
+@pytest.mark.system("google.cloud")
Review comment:
This test have already system market. See: https://github.com/apache/airflow/blob/d4c4db8a1833d07b1c03e4c062acea49c79bf5d6/tests/test_utils/gcp_system_helpers.py#L121
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rachael-ds commented on a change in pull request #14033: Documentation and example dag for CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
rachael-ds commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r571465672
##########
File path: docs/conf.py
##########
@@ -151,9 +151,7 @@
'README.rst',
]
elif PACKAGE_NAME.startswith('apache-airflow-providers-'):
- exclude_patterns = [
- 'operators/_partials',
- ]
+ exclude_patterns = ['operators/_partials', 'sensors/_partials']
Review comment:
Not anymore
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rachael-ds commented on a change in pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
rachael-ds commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r570919993
##########
File path: airflow/providers/google/cloud/example_dags/example_gcs.py
##########
@@ -151,6 +156,41 @@
copy_file >> delete_bucket_2
delete_files >> delete_bucket_1
+with models.DAG(
+ "example_gcs_sensors",
+ start_date=days_ago(1),
+ schedule_interval=None,
+ tags=['example'],
+) as dag2:
+ create_bucket = GCSCreateBucketOperator(
+ task_id="create_bucket", bucket_name=BUCKET_1, project_id=PROJECT_ID
+ )
+ upload_file = LocalFilesystemToGCSOperator(
+ task_id="upload_file",
+ src=PATH_TO_UPLOAD_FILE,
+ dst=BUCKET_FILE_LOCATION,
+ bucket=BUCKET_1,
+ )
+ # [START howto_sensor_object_exists_task]
+ gcs_object_exists = GCSObjectExistenceSensor(
+ bucket=BUCKET_1,
+ object=PATH_TO_UPLOAD_FILE,
+ mode='poke',
+ task_id="gcs_object_exists_task",
+ )
+ # [END howto_sensor_object_exists_task]
+ # [START howto_sensor_object_with_prefix_exists_task]
+ gcs_object_with_prefix_exists = GCSObjectsWtihPrefixExistenceSensor(
+ bucket=BUCKET_1,
+ prefix=PATH_TO_UPLOAD_FILE_PREFIX,
+ mode='poke',
+ task_id="gcs_object_with_prefix_exists_task",
+ )
+ # [END howto_sensor_object_with_prefix_exists_task]
+ delete_bucket = GCSDeleteBucketOperator(task_id="delete_bucket", bucket_name=BUCKET_1)
+
+ create_bucket >> upload_file >> [gcs_object_exists, gcs_object_with_prefix_exists] >> delete_bucket
Review comment:
The system test for this DAG will confirm that the GCSObjectExistenceSensor and GCSObjectsWtihPrefixExistenceSensor can be called in this context successfully.
It doesn't explicitly test for poke functionality (I see we have separate unit tests for this) - I was hesitant to have one task executing the sensor while try to have another upload a file in the same dag (I'm not sure how the tests handle task concurrency/whether this is even best practice). Open to thoughts/suggestions on this thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r571323095
##########
File path: docs/spelling_wordlist.txt
##########
@@ -101,6 +101,7 @@ Davydov
De
Decrypt
Decrypts
+DeidentifyConfig
Review comment:
```suggestion
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #14033: Documentation and example dag for CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r574242560
##########
File path: airflow/providers/google/cloud/example_dags/example_gcs.py
##########
@@ -151,6 +156,41 @@
copy_file >> delete_bucket_2
delete_files >> delete_bucket_1
+with models.DAG(
+ "example_gcs_sensors",
+ start_date=days_ago(1),
+ schedule_interval=None,
+ tags=['example'],
+) as dag2:
+ create_bucket = GCSCreateBucketOperator(
+ task_id="create_bucket", bucket_name=BUCKET_1, project_id=PROJECT_ID
+ )
+ upload_file = LocalFilesystemToGCSOperator(
+ task_id="upload_file",
+ src=PATH_TO_UPLOAD_FILE,
+ dst=BUCKET_FILE_LOCATION,
+ bucket=BUCKET_1,
+ )
+ # [START howto_sensor_object_exists_task]
+ gcs_object_exists = GCSObjectExistenceSensor(
+ bucket=BUCKET_1,
+ object=PATH_TO_UPLOAD_FILE,
+ mode='poke',
+ task_id="gcs_object_exists_task",
+ )
+ # [END howto_sensor_object_exists_task]
+ # [START howto_sensor_object_with_prefix_exists_task]
+ gcs_object_with_prefix_exists = GCSObjectsWtihPrefixExistenceSensor(
+ bucket=BUCKET_1,
+ prefix=PATH_TO_UPLOAD_FILE_PREFIX,
+ mode='poke',
+ task_id="gcs_object_with_prefix_exists_task",
+ )
+ # [END howto_sensor_object_with_prefix_exists_task]
+ delete_bucket = GCSDeleteBucketOperator(task_id="delete_bucket", bucket_name=BUCKET_1)
+
+ create_bucket >> upload_file >> [gcs_object_exists, gcs_object_with_prefix_exists] >> delete_bucket
Review comment:
It is enough to test the green path as it will allow us to detect 99% of the problems. For other cases, we have unit tests. These tests are mainly aimed at checking whether the operator works at a high level, e.g. API has not been deleted, or the parameters have not changed and this test checks it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r571321860
##########
File path: docs/apache-airflow-providers-google/operators/cloud/data_loss_prevention.rst
##########
@@ -296,10 +296,25 @@ Unlike storage methods (Jobs) content method are synchronous, stateless methods.
De-identify Content
"""""""""""""""""""
+De-identification is the process of removing identifying information from data.
+Configuration information defines how you want the sensitive data de-identified.
-To de-identify potentially sensitive info from a content item, you can use
+This config can either be saved and persisted in de-identification templates or defined in a DeidentifyConfig object:
Review comment:
```suggestion
This config can either be saved and persisted in de-identification templates or defined in a ``DeidentifyConfig`` object:
```
Google style guide:
> The following list includes items that should be in code font, but it's not an exhaustive list.
> ...
> * Class names.
https://developers.google.com/style/code-in-text
or
```suggestion
This config can either be saved and persisted in de-identification templates or defined in a :class:`~google.cloud.dlp_v2.types.DeidentifyConfig` object:
```
The second solultion will make the link clickable if you add the inventory URL. See: https://github.com/apache/airflow/blob/d4c4db8a1833d07b1c03e4c062acea49c79bf5d6/docs/exts/docs_build/third_party_inventories.py#L29-L51
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rachael-ds commented on a change in pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
rachael-ds commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r570914222
##########
File path: airflow/providers/google/cloud/example_dags/example_dlp.py
##########
@@ -177,3 +178,31 @@
)
# [END howto_operator_dlp_delete_job_trigger]
create_trigger >> update_trigger >> delete_trigger
+
+DEIDENTIFY_CONFIG = {
Review comment:
Good shout - I've added some more general info as as well as a snippet of the de-identify config
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #14033: Documentation and example dag for CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#issuecomment-782445013
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r568830202
##########
File path: docs/apache-airflow-providers-google/sensors/cloud/gcs.rst
##########
@@ -0,0 +1,75 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+Google Cloud Storage Sensors
Review comment:
We describe the sensors and ordinary operators in one guide, so that there is one place with all the information for the service. Transfer operators are a bit more problematic as they integrate with two services in one operator, so they have separate guides.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rachael-ds commented on a change in pull request #14033: Documentation and example dag for CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
rachael-ds commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r571466651
##########
File path: docs/apache-airflow-providers-google/operators/cloud/gcs.rst
##########
@@ -111,37 +118,79 @@ More information
See Google Cloud Storage insert documentation to `create a ACL entry for ObjectAccess
<https://cloud.google.com/storage/docs/json_api/v1/objectAccessControls/insert>`_.
-Reference
----------
-
-For further information, look at:
-
-* `Client Library Documentation <https://googleapis.github.io/google-cloud-python/latest/storage/index.html>`__
-* `Product Documentation <https://cloud.google.com/storage/docs/>`__
.. _howto/operator:GCSDeleteBucketOperator:
-Deleting Bucket
-^^^^^^^^^^^^^^^
+GCSDeleteBucketOperator
Review comment:
Noted - I'll just revert my changes around this for now thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#issuecomment-771669663
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, itβs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better π.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rachael-ds commented on a change in pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
rachael-ds commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r570913887
##########
File path: airflow/providers/google/cloud/example_dags/example_dlp.py
##########
@@ -177,3 +178,31 @@
)
# [END howto_operator_dlp_delete_job_trigger]
create_trigger >> update_trigger >> delete_trigger
+
+DEIDENTIFY_CONFIG = {
+ "info_type_transformations": {
+ "transformations": [
+ {
+ "primitive_transformation": {
+ "replace_config": {"new_value": {"string_value": "[deidentified_number]"}}
+ }
+ }
+ ]
+ }
+}
+
+with models.DAG(
Review comment:
Sure - done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] leahecole commented on a change in pull request #14033: Documentation and example dag for CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
leahecole commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r579534183
##########
File path: docs/apache-airflow-providers-google/sensors/cloud/gcs.rst
##########
@@ -0,0 +1,75 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+Google Cloud Storage Sensors
+============================
+
+Cloud Storage allows world-wide storage and retrieval of any amount of data at any time.
+You can use Cloud Storage for a range of scenarios including serving website content,
+storing data for archival and disaster recovery, or distributing large data objects to users via direct download.
+
+.. contents::
+ :depth: 1
+ :local:
+
+Prerequisite Tasks
+^^^^^^^^^^^^^^^^^^
+
+.. include::/sensors/_partials/prerequisite_tasks.rst
+
+.. _howto/sensor:GCSObjectExistenceSensor:
+
+GCSObjectExistenceSensor
+------------------------
+
+Use the :class:`~airflow.providers.google.cloud.sensors.gcs.GCSObjectExistenceSensor` to wait (poll) for the existence of a file in Google Cloud Storage.
+
+.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_gcs_sensors.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_sensor_object_exists_task]
+ :end-before: [END howto_sensor_object_exists_task]
+
+.. _howto/sensor:GCSObjectsWtihPrefixExistenceSensor:
+
+GCSObjectsWtihPrefixExistenceSensor
Review comment:
Looks like it's fixed yay community for being so speedy!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rachael-ds commented on pull request #14033: Documentation and example dag for CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
rachael-ds commented on pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#issuecomment-784139920
Thanks Leah & Kamil for the review. The final tests all look good - are we ok to merge @mik-laj ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rachael-ds commented on a change in pull request #14033: Documentation and example dag for CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
rachael-ds commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r571466730
##########
File path: tests/providers/google/cloud/operators/test_gcs_system.py
##########
@@ -22,6 +22,7 @@
from tests.test_utils.gcp_system_helpers import CLOUD_DAG_FOLDER, GoogleSystemTest, provide_gcp_context
+@pytest.mark.system("google.cloud")
Review comment:
Missed that - thanks, reverted
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r571322768
##########
File path: docs/conf.py
##########
@@ -151,9 +151,7 @@
'README.rst',
]
elif PACKAGE_NAME.startswith('apache-airflow-providers-'):
- exclude_patterns = [
- 'operators/_partials',
- ]
+ exclude_patterns = ['operators/_partials', 'sensors/_partials']
Review comment:
Is it needed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r568831836
##########
File path: airflow/providers/google/cloud/example_dags/example_dlp.py
##########
@@ -177,3 +178,31 @@
)
# [END howto_operator_dlp_delete_job_trigger]
create_trigger >> update_trigger >> delete_trigger
+
+DEIDENTIFY_CONFIG = {
+ "info_type_transformations": {
+ "transformations": [
+ {
+ "primitive_transformation": {
+ "replace_config": {"new_value": {"string_value": "[deidentified_number]"}}
+ }
+ }
+ ]
+ }
+}
+
+with models.DAG(
Review comment:
Can you add system tests? We have a rule that each DAG for GCP services should have system tests to verify its correctness.
https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-system-tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rachael-ds commented on a change in pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
rachael-ds commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r570915343
##########
File path: docs/apache-airflow-providers-google/sensors/cloud/gcs.rst
##########
@@ -0,0 +1,75 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+Google Cloud Storage Sensors
+============================
+
+Cloud Storage allows world-wide storage and retrieval of any amount of data at any time.
+You can use Cloud Storage for a range of scenarios including serving website content,
+storing data for archival and disaster recovery, or distributing large data objects to users via direct download.
+
+.. contents::
+ :depth: 1
+ :local:
+
+Prerequisite Tasks
+^^^^^^^^^^^^^^^^^^
+
+.. include::/sensors/_partials/prerequisite_tasks.rst
+
+.. _howto/sensor:GCSObjectExistenceSensor:
+
+GCSObjectExistenceSensor
+------------------------
+
+Use the :class:`~airflow.providers.google.cloud.sensors.gcs.GCSObjectExistenceSensor` to wait (poll) for the existence of a file in Google Cloud Storage.
+
+.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_gcs_sensors.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_sensor_object_exists_task]
+ :end-before: [END howto_sensor_object_exists_task]
+
+.. _howto/sensor:GCSObjectsWtihPrefixExistenceSensor:
+
+GCSObjectsWtihPrefixExistenceSensor
Review comment:
Yeah I noticed this - I'll raise a separate issue for this. Changing the operator name could impact those already using the operators so I'll reach out to Airflow community to see what's best practice for this :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] leahecole commented on a change in pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
leahecole commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r569945414
##########
File path: airflow/providers/google/cloud/example_dags/example_dlp.py
##########
@@ -177,3 +178,31 @@
)
# [END howto_operator_dlp_delete_job_trigger]
create_trigger >> update_trigger >> delete_trigger
+
+DEIDENTIFY_CONFIG = {
Review comment:
Would it be worth also including the config in the documentation right before the example? Given the fact that this config isn't just a string, I think it might be worth showing
##########
File path: docs/apache-airflow-providers-google/sensors/cloud/gcs.rst
##########
@@ -0,0 +1,75 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+Google Cloud Storage Sensors
+============================
+
+Cloud Storage allows world-wide storage and retrieval of any amount of data at any time.
+You can use Cloud Storage for a range of scenarios including serving website content,
+storing data for archival and disaster recovery, or distributing large data objects to users via direct download.
+
+.. contents::
+ :depth: 1
+ :local:
+
+Prerequisite Tasks
+^^^^^^^^^^^^^^^^^^
+
+.. include::/sensors/_partials/prerequisite_tasks.rst
+
+.. _howto/sensor:GCSObjectExistenceSensor:
+
+GCSObjectExistenceSensor
+------------------------
+
+Use the :class:`~airflow.providers.google.cloud.sensors.gcs.GCSObjectExistenceSensor` to wait (poll) for the existence of a file in Google Cloud Storage.
+
+.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_gcs_sensors.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_sensor_object_exists_task]
+ :end-before: [END howto_sensor_object_exists_task]
+
+.. _howto/sensor:GCSObjectsWtihPrefixExistenceSensor:
+
+GCSObjectsWtihPrefixExistenceSensor
Review comment:
I just looked - is this a typo in the sensor? I did look at the source code and thsi at least matches that code, but if this is a typo we should definitely fix it π¬ Or does Wtih stand for something?
##########
File path: docs/apache-airflow-providers-google/sensors/cloud/gcs.rst
##########
@@ -0,0 +1,75 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+Google Cloud Storage Sensors
+============================
+
+Cloud Storage allows world-wide storage and retrieval of any amount of data at any time.
+You can use Cloud Storage for a range of scenarios including serving website content,
+storing data for archival and disaster recovery, or distributing large data objects to users via direct download.
+
+.. contents::
+ :depth: 1
+ :local:
+
+Prerequisite Tasks
Review comment:
Oops I hadn't seen the other prerequisites file - I think it's not rendering properly
##########
File path: docs/apache-airflow-providers-google/sensors/cloud/gcs.rst
##########
@@ -0,0 +1,75 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+Google Cloud Storage Sensors
+============================
+
+Cloud Storage allows world-wide storage and retrieval of any amount of data at any time.
+You can use Cloud Storage for a range of scenarios including serving website content,
+storing data for archival and disaster recovery, or distributing large data objects to users via direct download.
+
+.. contents::
+ :depth: 1
+ :local:
+
+Prerequisite Tasks
Review comment:
The two headings you add with the two example sensors show up under Prerequisite tasks - is that where you want them? Or is it that prerequisite tasks is empty and it's rendering incorrectly?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#issuecomment-771669663
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, itβs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better π.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] leahecole commented on pull request #14033: Documentation and example dag for CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
leahecole commented on pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#issuecomment-777166021
I'm happy with the DAGs and docs, but it looks like some of the tests aren't passing, and from what I can tell (disclaimer, I have such limited experience messing around with the system tests) it seems like it might be related to your PR. @rachael-ds are you seeing the same output when you run the tests locally?
I also see you have a question about system tests and DAG concurrency. I'm not 100% sure the answer - might be worth sending a quick note to the devlist to ask.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r571322501
##########
File path: docs/apache-airflow-providers-google/operators/cloud/gcs.rst
##########
@@ -111,37 +118,79 @@ More information
See Google Cloud Storage insert documentation to `create a ACL entry for ObjectAccess
<https://cloud.google.com/storage/docs/json_api/v1/objectAccessControls/insert>`_.
-Reference
----------
-
-For further information, look at:
-
-* `Client Library Documentation <https://googleapis.github.io/google-cloud-python/latest/storage/index.html>`__
-* `Product Documentation <https://cloud.google.com/storage/docs/>`__
.. _howto/operator:GCSDeleteBucketOperator:
-Deleting Bucket
-^^^^^^^^^^^^^^^
+GCSDeleteBucketOperator
Review comment:
> Avoid using code items in headings when possible. If you must mention a code item in a heading, add a descriptive noun to the item in code font.
See: https://developers.google.com/style/headings
There is also a second cause. Guides are read by people who don't know class names, and a good table of contents allows us to find the right section faster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rachael-ds closed pull request #14033: Documentation and example dag for CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
rachael-ds closed pull request #14033:
URL: https://github.com/apache/airflow/pull/14033
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r568830202
##########
File path: docs/apache-airflow-providers-google/sensors/cloud/gcs.rst
##########
@@ -0,0 +1,75 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+Google Cloud Storage Sensors
Review comment:
We describe the sensors and ordinary operators in one guide, so that there is one place with all the information for the service. Transfer operators are a bit more problematic as they integrate with two services in one operator, so they have separate guides.
##########
File path: docs/apache-airflow-providers-google/sensors/cloud/gcs.rst
##########
@@ -0,0 +1,75 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+Google Cloud Storage Sensors
Review comment:
We describe the sensors and ordinary operators in one guide, so that there is one place with all the information for the service. Transfer operators are a bit more problematic as they integrate with two services in one operator, so they have separate guides. Can you move this guide to ``docs/apache-airflow-providers-google/operators/cloud/gcs.rst``?
##########
File path: airflow/providers/google/cloud/example_dags/example_dlp.py
##########
@@ -177,3 +178,31 @@
)
# [END howto_operator_dlp_delete_job_trigger]
create_trigger >> update_trigger >> delete_trigger
+
+DEIDENTIFY_CONFIG = {
+ "info_type_transformations": {
+ "transformations": [
+ {
+ "primitive_transformation": {
+ "replace_config": {"new_value": {"string_value": "[deidentified_number]"}}
+ }
+ }
+ ]
+ }
+}
+
+with models.DAG(
Review comment:
Can you add system tests? We have a rule that each DAG for GCP services should have system tests to verify its correctness.
https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-system-tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#issuecomment-772110136
[The Workflow run](https://github.com/apache/airflow/actions/runs/531793994) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rachael-ds commented on a change in pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
rachael-ds commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r570927095
##########
File path: docs/apache-airflow-providers-google/sensors/cloud/gcs.rst
##########
@@ -0,0 +1,75 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+Google Cloud Storage Sensors
+============================
+
+Cloud Storage allows world-wide storage and retrieval of any amount of data at any time.
+You can use Cloud Storage for a range of scenarios including serving website content,
+storing data for archival and disaster recovery, or distributing large data objects to users via direct download.
+
+.. contents::
+ :depth: 1
+ :local:
+
+Prerequisite Tasks
+^^^^^^^^^^^^^^^^^^
+
+.. include::/sensors/_partials/prerequisite_tasks.rst
+
+.. _howto/sensor:GCSObjectExistenceSensor:
+
+GCSObjectExistenceSensor
+------------------------
+
+Use the :class:`~airflow.providers.google.cloud.sensors.gcs.GCSObjectExistenceSensor` to wait (poll) for the existence of a file in Google Cloud Storage.
+
+.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_gcs_sensors.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_sensor_object_exists_task]
+ :end-before: [END howto_sensor_object_exists_task]
+
+.. _howto/sensor:GCSObjectsWtihPrefixExistenceSensor:
+
+GCSObjectsWtihPrefixExistenceSensor
Review comment:
Raised here: https://github.com/apache/airflow/issues/14097
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rachael-ds commented on a change in pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
rachael-ds commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r570915494
##########
File path: docs/apache-airflow-providers-google/sensors/cloud/gcs.rst
##########
@@ -0,0 +1,75 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+Google Cloud Storage Sensors
+============================
+
+Cloud Storage allows world-wide storage and retrieval of any amount of data at any time.
+You can use Cloud Storage for a range of scenarios including serving website content,
+storing data for archival and disaster recovery, or distributing large data objects to users via direct download.
+
+.. contents::
+ :depth: 1
+ :local:
+
+Prerequisite Tasks
Review comment:
Cheers - I think it should be fine now
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r571322690
##########
File path: docs/apache-airflow-providers-google/operators/cloud/gcs.rst
##########
@@ -111,37 +118,79 @@ More information
See Google Cloud Storage insert documentation to `create a ACL entry for ObjectAccess
<https://cloud.google.com/storage/docs/json_api/v1/objectAccessControls/insert>`_.
-Reference
----------
-
-For further information, look at:
-
-* `Client Library Documentation <https://googleapis.github.io/google-cloud-python/latest/storage/index.html>`__
-* `Product Documentation <https://cloud.google.com/storage/docs/>`__
.. _howto/operator:GCSDeleteBucketOperator:
-Deleting Bucket
-^^^^^^^^^^^^^^^
+GCSDeleteBucketOperator
+-----------------------
Deleting Bucket allows you to remove bucket object from the Google Cloud Storage.
It is performed through the
:class:`~airflow.providers.google.cloud.operators.gcs.GCSDeleteBucketOperator` operator.
+Using the operator
+""""""""""""""""""
+
Review comment:
```suggestion
```
This section is not too long to be broken into smaller parts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rachael-ds commented on a change in pull request #14033: Documentation and example dag for CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
rachael-ds commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r571465672
##########
File path: docs/conf.py
##########
@@ -151,9 +151,7 @@
'README.rst',
]
elif PACKAGE_NAME.startswith('apache-airflow-providers-'):
- exclude_patterns = [
- 'operators/_partials',
- ]
+ exclude_patterns = ['operators/_partials', 'sensors/_partials']
Review comment:
Not anymore - reverted
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] leahecole commented on pull request #14033: Documentation and example dag for CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
leahecole commented on pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#issuecomment-782445189
LGTM from the docs standpoint and it seems like things are passing, @mik-laj lmk if you're also happy!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #14033: Doc + example dag for: CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#issuecomment-772110136
[The Workflow run](https://github.com/apache/airflow/actions/runs/531793994) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] leahecole merged pull request #14033: Documentation and example dag for CloudDLPDeidentifyContentOperator, GCSObjectExistenceSensor, GCSObjectsWithPrefixExistenceSensor
Posted by GitBox <gi...@apache.org>.
leahecole merged pull request #14033:
URL: https://github.com/apache/airflow/pull/14033
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org