You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2019/06/12 23:29:08 UTC
[GitHub] [incubator-druid] dclim opened a new issue #7875: Make required S3
permissions consistent for pushing segments and task logs
dclim opened a new issue #7875: Make required S3 permissions consistent for pushing segments and task logs
URL: https://github.com/apache/incubator-druid/issues/7875
### Description
The code for pushing task logs and segments to S3 deep storage is mostly shared, but the segment code path has some additional logic to possibly get the bucket owner (by fetching the bucket ACL) and set the ACL of the segment object to give the bucket owner full access. See:
https://github.com/apache/incubator-druid/blob/master/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3DataSegmentPusher.java#L147
and compare with:
https://github.com/apache/incubator-druid/blob/master/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3TaskLogs.java#L138
The additional ACL permissions required for pushing segments can lead to scenarios where the task logs succeed in being written to S3 but the segment PUT request fails with an Access Denied. There is no reason that the ACL on the segment should be set but not on the task log, and making this consistent will avoid future debugging headache.
Additionally - consider having a fallback where if the putObject request fails after setting the ACL, it retries without setting the ACL, even if `druid.storage.disableAcl=false`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org