[GitHub] [incubator-druid] dclim opened a new issue #7573: Using 'prefixes' with S3 firehose fails to retrieve a single object with only s3:GetObject permissions

[GitBox <gi...@apache.org>]

dclim opened a new issue #7573: Using 'prefixes' with S3 firehose fails to retrieve a single object with only s3:GetObject permissions
URL: https://github.com/apache/incubator-druid/issues/7573
 
 
   ### Affected Version
   
   0.14.0
   
   ### Description
   
   The S3 firehose supports the user providing a list of `uris` or `prefixes` to fetch. When using `prefixes`, the implementation issues a 'list objects' call to get the list of objects that match the given prefix. There is a fallback in the implementation where if the request fails due to authorization issues (user does not have `s3:ListBucket` permissions), it will try to retrieve the object directly, using the `s3:GetObject` grant. This makes sense since a typical auth strategy is to grant read permission to specific objects but not allow a user to list out path prefixes. However, the fallback still issues a 'list objects' request which fails if a user has `s3:GetObject` but not `s3:ListBucket` which defeats the purpose of the fallback.
   
   See:
   https://github.com/apache/incubator-druid/blob/master/extensions-core/s3-extensions/src/main/java/org/apache/druid/firehose/s3/StaticS3FirehoseFactory.java#L150
   
   https://github.com/apache/incubator-druid/blob/master/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3Utils.java#L247
   
   The workaround is to use `uris` if you only have `s3:GetObject` permissions, but in the case where you are building an application where you don't know beforehand what permissions the user has, it would be nice to be able to use `prefixes` only and have it work for all cases.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org