[jira] [Commented] (HADOOP-15576) S3A Multipart Uploader API implementation to work with S3Guard and encryption

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16554542#comment-16554542 ] 

Steve Loughran commented on HADOOP-15576:
-----------------------------------------

Ewan, still expecting this

* S3AMultipartUploader to always go through s3a FS and {{WriteOperationsHelper}}, not at the s3 client; retry logic to go in there, along with s3guard introduction. that's for completing & aborting ubploads ({{abortMultipartUpload()}},  {{newUploadPartRequest()}}{{finalizeMultipartUpload()}}. That shoud be enough to wire up S3Guard. If something is missing: add it.
* S3AMultipartUploader to declare retry policy on its ops
* test to include inconsistent s3 client with failures (always) and, if s3guard is enabled inconsistent, turned on. Your store doesn't do inconsistency so that's moot, but it may fail -so we need those retries

You must not be creating any AWS request classes yourself, or calling the s3a client. If there isn't a method in WriteOperationsHelper to create the instance, add it, make sure it works with S3Guard, sets encryption stuff etc. The requirement to declare the retry policy forces you to look at the retry logic of the layers underneath, make sure either they all retry *or the new code adds it*. It will also help future maintenance



> S3A  Multipart Uploader API implementation to work with S3Guard and encryption
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-15576
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15576
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.2
>            Reporter: Steve Loughran
>            Assignee: Ewan Higgs
>            Priority: Blocker
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper; adding any new methods are needed there.
> # Tests. the tests of HDFS-13713. 
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and even if there was, it might not show that S3Guard was bypassed, because there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader, encryption settings are't picked up. Files being uploaded this way *are not being encrypted*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org