[jira] [Commented] (FLINK-38350) Update Hadoop Library to 3.4.2

["Michael Johnson (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-38350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095631#comment-18095631 ] 

Michael Johnson commented on FLINK-38350:
-----------------------------------------

Hi,

I think this is becoming important due to changes in how SSE-C works for S3!

MultiPartUploads to S3 are in 3 parts:
 * CreateMultipartUpload:  [https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html]
 * UploadPart: [https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html] 
 * CompleteMultipartUpload:  [https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html] 

When using SSE-C, I believe this Bug was corrected for only the first 2 parts...but not the 3rd!

Meaning, when calling the CompleteMultipartUpload function, it does not pass the required Encryption Headers!

If you look at the S3 Docs for CompleteMultipartUpload:  [https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html#API_CompleteMultipartUpload_RequestParameters] 

These Headers are still required when SSE-C is used:
 * *[x-amz-server-side-encryption-customer-algorithm|https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html#API_CompleteMultipartUpload_RequestSyntax]*
 * {*}[x-amz-server-side-encryption-customer-key|https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html#API_CompleteMultipartUpload_RequestSyntax]{*}{*}{{*}}
 * {*}[x-amz-server-side-encryption-customer-key-MD5|https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html#API_CompleteMultipartUpload_RequestSyntax]{*}{*}{{*}}

Hadoop library is used internally by Apache Flink for Snapshotting when SSE-C is required...as such, it will cause Snapshotting to fail on the 3rd stage!

Upon deeper investigation:
 * Flink 2.2.0 and Flink 2.3.0 are using Hadoop 3.3.4:  [https://github.com/apache/flink/blob/1baacaaade29ecc9dbc13b66c390b646a45d9544/flink-filesystems/pom.xml#L37] 
 * Hadoop 3.3.4 uses AWS SDK Java 1.12.262:  [https://github.com/apache/hadoop/blob/a585a73c3e02ac62350c136643a5e7f6095a3dbb/hadoop-project/pom.xml#L187] 
 * AWS SDK Java 1.12.262 does NOT implement SSE-C for CompleteMultipartUpload:  [https://github.com/aws/aws-sdk-java/blob/1.12.262/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/model/CompleteMultipartUploadRequest.java#L41] 

However:
 * Next highest version of Hadoop is 3.4.0, which is using AWS SDK Java 1.12.599:  [https://github.com/apache/hadoop/blob/branch-3.4.0/hadoop-project/pom.xml#L189] 
 * AWS SDK Java 1.12.599 DOES implement SSE-C for CompleteMultipartUpload:  [https://github.com/aws/aws-sdk-java/blob/1.12.599/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/model/CompleteMultipartUploadRequest.java#L41] 

So I think the necessary thing is that Flink will need to upgrade Hadoop to at least 3.4.0!

> Update Hadoop Library to 3.4.2
> ------------------------------
>
>                 Key: FLINK-38350
>                 URL: https://issues.apache.org/jira/browse/FLINK-38350
>             Project: Flink
>          Issue Type: Improvement
>          Components: FileSystems
>    Affects Versions: 2.0-preview, 1.20.2
>            Reporter: Paul Ashley
>            Priority: Major
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Hadoop 3.4.2 which includes support for AWS conditional writes in https://issues.apache.org/jira/browse/HADOOP-19256
>  
> It would be valuable for Flink to include this, as it would allow S3 based table sinks to be able to guarantee write integrity without relying on other concurrency controls (or accepting data loss).  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)