[GitHub] [airflow] eladkal commented on issue #8464: Add operator to list subfolders from s3 bucket

[GitBox <gi...@apache.org>]

eladkal commented on issue #8464:
URL: https://github.com/apache/airflow/pull/8464#issuecomment-616324375


   LGTM but I have two questions that I want to raise:
   
   1. If we have this operator do we need the S3ListOperator ?
   This raises a question that i asked a while ago but didn't have the item to check it.
   Why the S3Hook has `list_prefixes` & `list_keys` functions? Is there a use case for `list_keys` that isn't covered by `list_prefixes`?
   
   The way I see it `list_prefixes` include subfolders while list_keys exclude them. So can't `list_prefixes` just have additional parameter to set it?
   
   The fact that we have both `list_prefixes` & `list_keys` resulted in having two sensors S3KeySensor & S3PrefixSensor. Now it will also result in having two operators.
   
   2. Maybe worth assembling all the S3 operators (S3ListOperator , S3DeleteObjectsOperator , etc.. ) into a single file similar to [gcs.py](https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/gcs.py)?
   
   WDYT?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org