You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/04/20 21:30:35 UTC
[GitHub] [flink] chinmayms opened a new pull request, #19540: [FLINK-27333] Flink filesystems hadoop version upgrade to 3.3.2
chinmayms opened a new pull request, #19540:
URL: https://github.com/apache/flink/pull/19540
## What is the purpose of the change
This PR upgrades flink-filesystems hadoop version to 3.3.2 and makes the necessary code changes as part of changes to constructors in hadoop 3.3.2 code.
Context: We have a security requirement to client side encrypt flink state for certain flink applications that process sensitive data.
One way to do it is to use flink-s3-fs-hadoop compiled against hadoop 3.3.2 for checkpoints as hadoop 3.3.2 provides out of the box AWS client side encryption using AWS KMS keys before writing the data to S3. (https://issues.apache.org/jira/browse/HADOOP-13887)
## Brief change log
Change shaded hadoop version to 3.3.2 in flink-filesystems pom.xml
Update arguments for methods connected to `WriteOperationHelper` since hadoop 3.3.2 introduces 3 new arguments to its constructor.
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Tested on flink jobs with jar flink-s3-fs-hadoop in checkpoint plugin path at Apple Flink Infrastructure for Flink version 1.14.3
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (yes)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] chinmayms commented on a diff in pull request #19540: [FLINK-27333] Flink filesystems hadoop version upgrade to 3.3.2
Posted by GitBox <gi...@apache.org>.
chinmayms commented on code in PR #19540:
URL: https://github.com/apache/flink/pull/19540#discussion_r854653554
##########
flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3FileSystemFactory.java:
##########
@@ -94,6 +94,11 @@ protected URI getInitURI(URI fsUri, org.apache.hadoop.conf.Configuration hadoopC
@Override
protected S3AccessHelper getS3AccessHelper(FileSystem fs) {
final S3AFileSystem s3Afs = (S3AFileSystem) fs;
- return new HadoopS3AccessHelper(s3Afs, s3Afs.getConf());
+ return new HadoopS3AccessHelper(
+ s3Afs,
+ s3Afs.getConf(),
+ s3Afs.createStoreContext().getInstrumentation(),
Review Comment:
Makes sense. Changed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on pull request #19540: [FLINK-27333] Flink filesystems hadoop version upgrade to 3.3.2
Posted by GitBox <gi...@apache.org>.
zentol commented on PR #19540:
URL: https://github.com/apache/flink/pull/19540#issuecomment-1104798346
Duplicate of #19540.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] stevenzwu commented on a diff in pull request #19540: [FLINK-27333] Flink filesystems hadoop version upgrade to 3.3.2
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on code in PR #19540:
URL: https://github.com/apache/flink/pull/19540#discussion_r854610694
##########
flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3FileSystemFactory.java:
##########
@@ -94,6 +94,11 @@ protected URI getInitURI(URI fsUri, org.apache.hadoop.conf.Configuration hadoopC
@Override
protected S3AccessHelper getS3AccessHelper(FileSystem fs) {
final S3AFileSystem s3Afs = (S3AFileSystem) fs;
- return new HadoopS3AccessHelper(s3Afs, s3Afs.getConf());
+ return new HadoopS3AccessHelper(
+ s3Afs,
+ s3Afs.getConf(),
+ s3Afs.createStoreContext().getInstrumentation(),
Review Comment:
we can probably do the extraction inside the HadoopS3AccessHelper constructor without changing here and the constructor signature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol closed pull request #19540: [FLINK-27333] Flink filesystems hadoop version upgrade to 3.3.2
Posted by GitBox <gi...@apache.org>.
zentol closed pull request #19540: [FLINK-27333] Flink filesystems hadoop version upgrade to 3.3.2
URL: https://github.com/apache/flink/pull/19540
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19540: [FLINK-27333] Flink filesystems hadoop version upgrade to 3.3.2
Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #19540:
URL: https://github.com/apache/flink/pull/19540#issuecomment-1104475376
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c36003b4103651f35d414a4eac88e7a3dd3f34e3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c36003b4103651f35d414a4eac88e7a3dd3f34e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c36003b4103651f35d414a4eac88e7a3dd3f34e3 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on pull request #19540: [FLINK-27333] Flink filesystems hadoop version upgrade to 3.3.2
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on PR #19540:
URL: https://github.com/apache/flink/pull/19540#issuecomment-1104767411
To fix failing it is also required updating versions in NOTICE files in a way similar to how it was done at https://github.com/apache/flink/pull/16786
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org