You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/08 11:45:00 UTC

[GitHub] [airflow] wegmatho opened a new issue, #24321: S3KeySensor wildcard_match only matching key prefixes instead of full patterns

wegmatho opened a new issue, #24321:
URL: https://github.com/apache/airflow/issues/24321

   ### Apache Airflow Provider(s)
   
   amazon
   
   ### Versions of Apache Airflow Providers
   
   3.4.0
   
   ### Apache Airflow version
   
   2.3.2 (latest released)
   
   ### Operating System
   
   Debian GNU/Linux 10
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   _No response_
   
   ### What happened
   
   For patterns like "*.zip" the S3KeySensor succeeds for all files, does not take full pattern into account i.e. the ".zip" part). 
   Bug introduced in https://github.com/apache/airflow/pull/22737
   
   ### What you think should happen instead
   
   Full pattern match as in version 3.3.0 (in S3KeySensor poke()):
   
   ```
   ...
   if self.wildcard_match:
               return self.get_hook().check_for_wildcard_key(self.bucket_key, self.bucket_name)
   ...
   ```
   
   alternatively the files obtained by `files = self.get_hook().get_file_metadata(prefix, bucket_name)` which only match the prefix should be further filtered.
   
   ### How to reproduce
   
   create a DAG with a key sensor task containing wildcard and suffix, e.g. the following task should succeed only if any ZIP-files are available in "my-bucket", but succeeds for all instead:
   
   `S3KeySensor(task_id="wait_for_file", bucket_name="my-bucket", bucket_key="*.zip", wildcard_match=True)`
   
   ### Anything else
   
   Not directly part of this issue but at the same time I would suggest to include additional file attributes in method _check_key, e.g. the actual key of the files. This way more filters, e.g. exclude specific keys, could be implemented by using the check_fn.
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on issue #24321: S3KeySensor wildcard_match only matching key prefixes instead of full patterns

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #24321:
URL: https://github.com/apache/airflow/issues/24321#issuecomment-1149908156

   cc @vincbeck 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] boring-cyborg[bot] commented on issue #24321: S3KeySensor wildcard_match only matching key prefixes instead of full patterns

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #24321:
URL: https://github.com/apache/airflow/issues/24321#issuecomment-1149809867

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #24321: S3KeySensor wildcard_match only matching key prefixes instead of full patterns

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24321:
URL: https://github.com/apache/airflow/issues/24321#issuecomment-1149876082

   Would it be possible you submit a fix for it? I think that's the fastest way to fix it. And I am about to release provider's release so quick fix to that might be released in a couple of days if you manage to fix it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk closed issue #24321: S3KeySensor wildcard_match only matching key prefixes instead of full patterns

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #24321: S3KeySensor wildcard_match only matching key prefixes instead of full patterns
URL: https://github.com/apache/airflow/issues/24321


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] vincbeck commented on issue #24321: S3KeySensor wildcard_match only matching key prefixes instead of full patterns

Posted by GitBox <gi...@apache.org>.
vincbeck commented on issue #24321:
URL: https://github.com/apache/airflow/issues/24321#issuecomment-1150290410

   I'll look into it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org