You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Sergey Nuyanzin (Jira)" <ji...@apache.org> on 2022/04/20 20:37:00 UTC

[jira] [Commented] (FLINK-27333) Upgrade flink-filesystems hadoop version to version 3.3.2

    [ https://issues.apache.org/jira/browse/FLINK-27333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17525270#comment-17525270 ] 

Sergey Nuyanzin commented on FLINK-27333:
-----------------------------------------

Isn't this one a duplicate for https://issues.apache.org/jira/browse/FLINK-27308 ?

> Upgrade flink-filesystems hadoop version to version 3.3.2
> ---------------------------------------------------------
>
>                 Key: FLINK-27333
>                 URL: https://issues.apache.org/jira/browse/FLINK-27333
>             Project: Flink
>          Issue Type: Improvement
>          Components: FileSystems
>    Affects Versions: 1.14.3
>            Reporter: Chinmay Sumant
>            Priority: Major
>
> We have a security requirement to client side encrypt flink state for certain flink applications that process sensitive data.
> Currently, there is no feature that supports this out of the box on AWS S3 backend. 
> We found that one way to do it is to use flink-s3-fs-hadoop compiled against hadoop 3.3.2 for checkpoints as hadoop 3.3.2 provides out of the box AWS client side encryption using AWS KMS keys before writing the data to S3.  (https://issues.apache.org/jira/browse/HADOOP-13887) 
> We were able to change the flink-filesystems shaded hadoop version from existing 3.2.2 version to version 3.3.2 and compile with minimal code changes. The resultant flink-s3-fs-hadoop jar was used in the checkpoint plugin path for our flink jobs and worked well for checkpoints/savepoints upto 250 GB each with client side encryption using AWS KMS.
> Filing this Jira to request to take these changes upstream and also to check if there are concerns with changing the hadoop version that may affect any other components since our observations have been limited to plugin jar and checkpoints using flink-s3-fs-hadoop filesystem. 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)