You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/07 13:50:44 UTC

[GitHub] [airflow] anaynayak opened a new pull request #18807: Update S3PrefixSensor to support checking multiple prefixes within a bucket

anaynayak opened a new pull request #18807:
URL: https://github.com/apache/airflow/pull/18807


   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   Used the alternate approach as mentioned in https://github.com/apache/airflow/issues/15001#issuecomment-934527384
   
   Summary of changes:
   1. Support prefix as a str or list. Haven't changed the name to avoid changes to existing consumers.
   2. Removed `self.full_url` since we can't build one for multiple prefixes 
   3. Updated existing test to pytest
   4. Added tests for multiple prefixes
   5. Added context type as Dict[str, Any]. There are other variations across the codebase. 
   
   closes: https://github.com/apache/airflow/issues/15001
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] anaynayak commented on a change in pull request #18807: Update S3PrefixSensor to support checking multiple prefixes within a bucket

Posted by GitBox <gi...@apache.org>.
anaynayak commented on a change in pull request #18807:
URL: https://github.com/apache/airflow/pull/18807#discussion_r724958748



##########
File path: airflow/providers/amazon/aws/sensors/s3_prefix.py
##########
@@ -33,7 +33,7 @@ class S3PrefixSensor(BaseSensorOperator):
     :param bucket_name: Name of the S3 bucket
     :type bucket_name: str
     :param prefix: The prefix being waited on. Relative path from bucket root level.
-    :type prefix: str
+    :type prefix: str or list

Review comment:
       Makes sense. Will do this change.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] github-actions[bot] commented on pull request #18807: Update S3PrefixSensor to support checking multiple prefixes within a bucket

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #18807:
URL: https://github.com/apache/airflow/pull/18807#issuecomment-940431557


   The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] anaynayak commented on a change in pull request #18807: Update S3PrefixSensor to support checking multiple prefixes within a bucket

Posted by GitBox <gi...@apache.org>.
anaynayak commented on a change in pull request #18807:
URL: https://github.com/apache/airflow/pull/18807#discussion_r726771459



##########
File path: airflow/providers/amazon/aws/sensors/s3_prefix.py
##########
@@ -67,18 +67,15 @@ def __init__(
         super().__init__(**kwargs)
         # Parse
         self.bucket_name = bucket_name
-        self.prefix = prefix
+        self.prefix = [prefix] if isinstance(prefix, str) else prefix
         self.delimiter = delimiter
-        self.full_url = "s3://" + bucket_name + '/' + prefix
         self.aws_conn_id = aws_conn_id
         self.verify = verify
         self.hook: Optional[S3Hook] = None
 
-    def poke(self, context):
+    def poke(self, context: Dict[str, Any]):
         self.log.info('Poking for prefix : %s in bucket s3://%s', self.prefix, self.bucket_name)
-        return self.get_hook().check_for_prefix(
-            prefix=self.prefix, delimiter=self.delimiter, bucket_name=self.bucket_name
-        )
+        return all(self._check_for_prefix(prefix) for prefix in self.prefix)

Review comment:
       I can raise another PR to change the name to `prefixes` . Had the same thought going on in my mind 😄 . Had called out the same as point 1 on the [description](https://github.com/apache/airflow/pull/18807#issue-1020070170).I'm curious how we handle such backward incompatible changes. Do we update UPDATING.md ? 
   
   To support finer control over `all` or `any`, was also suggesting passing a callable which lets the user decide on a per key basis. Default implementation could continue to be `all` based. Can raise another PR to do both these changes if it makes sense.
   
   The change will lead to an extra `__init__` parameter:
   ```
       def __init__(
           self,
           ...,
           callback: Callable[[Dict[str, bool]], bool] = lambda prefix_available: all(prefix_available.values()),
           **kwargs,
       ):
   
       def poke(self, context: Dict[str, Any]):
           self.log.info('Poking for prefix : %s in bucket s3://%s', self.prefix, self.bucket_name)
           # callback can choose to return true even if any/subset/all of the keys are present
           return self.callback({prefix: self._check_for_prefix(prefix) for prefix in self.prefix})
   
   ```
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] anaynayak commented on a change in pull request #18807: Update S3PrefixSensor to support checking multiple prefixes within a bucket

Posted by GitBox <gi...@apache.org>.
anaynayak commented on a change in pull request #18807:
URL: https://github.com/apache/airflow/pull/18807#discussion_r725432280



##########
File path: airflow/providers/amazon/aws/sensors/s3_prefix.py
##########
@@ -58,7 +58,7 @@ def __init__(
         self,
         *,
         bucket_name: str,
-        prefix: str,
+        prefix: Union[str, Sequence[str]],

Review comment:
       yes makes sense. Have fixed it in https://github.com/apache/airflow/pull/18807/commits/a0fd5ed9b80b9d2c3085d9f50cc3db37cf342ca6




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #18807: Update S3PrefixSensor to support checking multiple prefixes within a bucket

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #18807:
URL: https://github.com/apache/airflow/pull/18807#discussion_r726573093



##########
File path: airflow/providers/amazon/aws/sensors/s3_prefix.py
##########
@@ -67,18 +67,15 @@ def __init__(
         super().__init__(**kwargs)
         # Parse
         self.bucket_name = bucket_name
-        self.prefix = prefix
+        self.prefix = [prefix] if isinstance(prefix, str) else prefix
         self.delimiter = delimiter
-        self.full_url = "s3://" + bucket_name + '/' + prefix
         self.aws_conn_id = aws_conn_id
         self.verify = verify
         self.hook: Optional[S3Hook] = None
 
-    def poke(self, context):
+    def poke(self, context: Dict[str, Any]):
         self.log.info('Poking for prefix : %s in bucket s3://%s', self.prefix, self.bucket_name)
-        return self.get_hook().check_for_prefix(
-            prefix=self.prefix, delimiter=self.delimiter, bucket_name=self.bucket_name
-        )
+        return all(self._check_for_prefix(prefix) for prefix in self.prefix)

Review comment:
       I just realised this would cause some `_check_for_prefix` to not be called if any of the prefixes fail the check, because `all` returns eagerly. Not sure if that would be an issue in practice.
   
   Also we should've changed the attribute from `prefix` to `prefixes` 🙂 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #18807: Update S3PrefixSensor to support checking multiple prefixes within a bucket

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #18807:
URL: https://github.com/apache/airflow/pull/18807#discussion_r724684240



##########
File path: airflow/providers/amazon/aws/sensors/s3_prefix.py
##########
@@ -33,7 +33,7 @@ class S3PrefixSensor(BaseSensorOperator):
     :param bucket_name: Name of the S3 bucket
     :type bucket_name: str
     :param prefix: The prefix being waited on. Relative path from bucket root level.
-    :type prefix: str
+    :type prefix: str or list

Review comment:
       ```suggestion
       :type prefix: str or list of str
   ```

##########
File path: airflow/providers/amazon/aws/sensors/s3_prefix.py
##########
@@ -58,7 +58,7 @@ def __init__(
         self,
         *,
         bucket_name: str,
-        prefix: str,
+        prefix: Union[str, Sequence[str]],

Review comment:
       This union probably isn't that useful since `str` also satisfies `Sequence[str]` 😆




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] anaynayak commented on a change in pull request #18807: Update S3PrefixSensor to support checking multiple prefixes within a bucket

Posted by GitBox <gi...@apache.org>.
anaynayak commented on a change in pull request #18807:
URL: https://github.com/apache/airflow/pull/18807#discussion_r724966829



##########
File path: airflow/providers/amazon/aws/sensors/s3_prefix.py
##########
@@ -58,7 +58,7 @@ def __init__(
         self,
         *,
         bucket_name: str,
-        prefix: str,
+        prefix: Union[str, Sequence[str]],

Review comment:
       I tried to read up a bit around this since I was confused how a Sequence[str] could represent both. I understand that a str is also a Sequence[str] and hence it works with just Sequence[str] without the Union. 
   
   Noticed a few interesting threads in the process around [mypy](https://github.com/python/mypy/issues/11001) and [typing](https://github.com/python/typing/issues/256). 
   
   On a related note, I see one more occurrence of `Union[str, Sequence[str]]` in the project for the impersonation_chain variables. Personally I prefer to explicitly specify as `Union[str, Sequence[str]]` but can easily change this for the current PR.
   
   @uranusjr  Let me know what you think




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] anaynayak commented on pull request #18807: Update S3PrefixSensor to support checking multiple prefixes within a bucket

Posted by GitBox <gi...@apache.org>.
anaynayak commented on pull request #18807:
URL: https://github.com/apache/airflow/pull/18807#issuecomment-939226870


   @uranusjr thank you  for reviewing the PR.
   
   Few points from the summary which I wanted to clarify:
   1. Removal of `self.full_url` from the class hopefully doesn't require a change on [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md)
   2. Usage of `@cached_property` for the hook. Noticed this in other places. Can add it here for consistency.
   3. Haven't done a similar change across other S3 sensors.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk merged pull request #18807: Update S3PrefixSensor to support checking multiple prefixes within a bucket

Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #18807:
URL: https://github.com/apache/airflow/pull/18807


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #18807: Update S3PrefixSensor to support checking multiple prefixes within a bucket

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #18807:
URL: https://github.com/apache/airflow/pull/18807#discussion_r726882552



##########
File path: airflow/providers/amazon/aws/sensors/s3_prefix.py
##########
@@ -67,18 +67,15 @@ def __init__(
         super().__init__(**kwargs)
         # Parse
         self.bucket_name = bucket_name
-        self.prefix = prefix
+        self.prefix = [prefix] if isinstance(prefix, str) else prefix
         self.delimiter = delimiter
-        self.full_url = "s3://" + bucket_name + '/' + prefix
         self.aws_conn_id = aws_conn_id
         self.verify = verify
         self.hook: Optional[S3Hook] = None
 
-    def poke(self, context):
+    def poke(self, context: Dict[str, Any]):
         self.log.info('Poking for prefix : %s in bucket s3://%s', self.prefix, self.bucket_name)
-        return self.get_hook().check_for_prefix(
-            prefix=self.prefix, delimiter=self.delimiter, bucket_name=self.bucket_name
-        )
+        return all(self._check_for_prefix(prefix) for prefix in self.prefix)

Review comment:
       > I'm curious how we handle such backward incompatible changes.
   
   Do you mean changing `prefix` to `prefixes`? As the change stands right now, it's already backwards incoimpatible, so changing the name is not really a concern. It's extremely unlikely (and not idiomatic) to access attributes in a hook, so I shouldn't really matter much. We could even change the attribute to `_prefixes` to discourage access even further.
   
   As for the `all` issue, you can actually fix it quite easily with `all([...])` (explicitly cast the result of all calls to a list).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] anaynayak commented on a change in pull request #18807: Update S3PrefixSensor to support checking multiple prefixes within a bucket

Posted by GitBox <gi...@apache.org>.
anaynayak commented on a change in pull request #18807:
URL: https://github.com/apache/airflow/pull/18807#discussion_r737505894



##########
File path: airflow/providers/amazon/aws/sensors/s3_prefix.py
##########
@@ -67,18 +67,15 @@ def __init__(
         super().__init__(**kwargs)
         # Parse
         self.bucket_name = bucket_name
-        self.prefix = prefix
+        self.prefix = [prefix] if isinstance(prefix, str) else prefix
         self.delimiter = delimiter
-        self.full_url = "s3://" + bucket_name + '/' + prefix
         self.aws_conn_id = aws_conn_id
         self.verify = verify
         self.hook: Optional[S3Hook] = None
 
-    def poke(self, context):
+    def poke(self, context: Dict[str, Any]):
         self.log.info('Poking for prefix : %s in bucket s3://%s', self.prefix, self.bucket_name)
-        return self.get_hook().check_for_prefix(
-            prefix=self.prefix, delimiter=self.delimiter, bucket_name=self.bucket_name
-        )
+        return all(self._check_for_prefix(prefix) for prefix in self.prefix)

Review comment:
       Agree that with the removal of `full_url` the backward compatibility has gone away. For the name change, the worry was more for people passing keyword arguments. Wasn't worried about the access to attributes.  I can create a small PR to rename to _prefixes if that isn't an issue. Let me know if that makes sense.
   
   I now understand what you were referring to with the eager return of `all` . Personally I think it helps avoid a few extra calls to s3 and so should be fine. 
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] anaynayak commented on a change in pull request #18807: Update S3PrefixSensor to support checking multiple prefixes within a bucket

Posted by GitBox <gi...@apache.org>.
anaynayak commented on a change in pull request #18807:
URL: https://github.com/apache/airflow/pull/18807#discussion_r725432222



##########
File path: airflow/providers/amazon/aws/sensors/s3_prefix.py
##########
@@ -33,7 +33,7 @@ class S3PrefixSensor(BaseSensorOperator):
     :param bucket_name: Name of the S3 bucket
     :type bucket_name: str
     :param prefix: The prefix being waited on. Relative path from bucket root level.
-    :type prefix: str
+    :type prefix: str or list

Review comment:
       Fixed with https://github.com/apache/airflow/pull/18807/commits/a0fd5ed9b80b9d2c3085d9f50cc3db37cf342ca6 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #18807: Update S3PrefixSensor to support checking multiple prefixes within a bucket

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #18807:
URL: https://github.com/apache/airflow/pull/18807#discussion_r725410179



##########
File path: airflow/providers/amazon/aws/sensors/s3_prefix.py
##########
@@ -58,7 +58,7 @@ def __init__(
         self,
         *,
         bucket_name: str,
-        prefix: str,
+        prefix: Union[str, Sequence[str]],

Review comment:
       The problem is not with Union but Sequence. `Union[str, List[str]]` is probably a better annotation.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org