[GitHub] [airflow] OmerJog commented on a change in pull request #5166: [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor

[GitBox <gi...@apache.org>]

OmerJog commented on a change in pull request #5166: [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor
URL: https://github.com/apache/airflow/pull/5166#discussion_r277987601
 
 

 ##########
 File path: airflow/contrib/sensors/gcs_sensor.py
 ##########
 @@ -160,3 +162,105 @@ def poke(self, context):
             google_cloud_storage_conn_id=self.google_cloud_conn_id,
             delegate_to=self.delegate_to)
         return bool(hook.list(self.bucket, prefix=self.prefix))
+
+
+class GoogleCloudStorageUploadSessionCompleteSensor(BaseSensorOperator):
+    """
+    Checks for changes in the number of files at prefix in Google Cloud Storage
+    bucket and returns True if the inactivity period has passed with no
+    increase in the number of files. Note, it is recommended to use reschedule
+    mode if you expect this sensor to run for hours.
+
+    :param bucket: The Google cloud storage bucket where the objects are.
+        expected.
+    :type bucket: str
+    :param prefix: The name of the prefix to check in the Google cloud
+        storage bucket.
+    :param inactivity_period: The total seconds of inactivity to designate
+        an upload session is over. Note, this mechanism is not real time and
+        this operator may not return until a poke_interval after this period
+        has passed with no additional files sensed.
+    :type inactivity_period: int
+    :param min_files: The minimum number of files needed for upload session
+        to be considered valid.
+    :type min_files: int
+    :param previous_num_files: The previous number of files before the next
+        iteration.
 
 Review comment:
   This note is a bit hard to understand.  previous + before + next in one sentence really makes it unclear.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services