You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "xx789633 (via GitHub)" <gi...@apache.org> on 2026/02/13 02:52:28 UTC
[PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
xx789633 opened a new pull request, #2674:
URL: https://github.com/apache/fluss/pull/2674
<!--
*Thank you very much for contributing to Fluss - we are happy that you want to help us improve Fluss. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [GitHub issue](https://github.com/apache/fluss/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no issue.
- Name the pull request in the format "[component] Title of the pull request", where *[component]* should be replaced by the name of the component being changed. Typically, this corresponds to the component label assigned to the issue (e.g., [kv], [log], [client], [flink]). Skip *[component]* if you are unsure about which is the best component.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes.
- Each pull request should address only one issue, not mix up code from multiple issues.
**(The sections below can be removed for hotfixes or typos)**
-->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #xxx
<!-- What is the purpose of the change -->
### Brief change log
<!-- Please describe the changes made in this pull request and explain how they address the issue -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
Posted by "affo (via GitHub)" <gi...@apache.org>.
affo commented on PR #2674:
URL: https://github.com/apache/fluss/pull/2674#issuecomment-5043127688
@litiliu +1 here for the initiative
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on code in PR #2674:
URL: https://github.com/apache/fluss/pull/2674#discussion_r2802891680
##########
fluss-filesystems/fluss-fs-hadoop-shaded/src/main/resources/META-INF/NOTICE:
##########
@@ -9,28 +9,42 @@ This project bundles the following dependencies under the Apache Software Licens
- com.fasterxml.jackson.core:jackson-annotations:2.15.3
- com.fasterxml.jackson.core:jackson-core:2.15.3
- com.fasterxml.jackson.core:jackson-databind:2.15.3
-- com.fasterxml.woodstox:woodstox-core:5.3.0
+- com.fasterxml.woodstox:woodstox-core:5.4.0
- com.google.guava:failureaccess:1.0
- com.google.guava:guava:27.0-jre
- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
- com.google.j2objc:j2objc-annotations:1.1
- commons-beanutils:commons-beanutils:1.9.4
- commons-collections:commons-collections:3.2.2
-- commons-io:commons-io:2.8.0
-- commons-logging:commons-logging:1.1.3
-- org.apache.commons:commons-compress:1.21
-- org.apache.commons:commons-configuration2:2.1.1
+- commons-io:commons-io:2.16.1
+- commons-logging:commons-logging:1.2
+- io.dropwizard.metrics:metrics-core:3.2.4
+- io.netty:netty-buffer:4.1.100.Final
+- io.netty:netty-codec:4.1.100.Final
+- io.netty:netty-common:4.1.100.Final
+- io.netty:netty-handler:4.1.100.Final
+- io.netty:netty-resolver:4.1.100.Final
+- io.netty:netty-transport:4.1.100.Final
+- io.netty:netty-transport-classes-epoll:4.1.100.Final
+- io.netty:netty-transport-native-epoll:4.1.100.Final
+- io.netty:netty-transport-native-unix-common:4.1.100.Final
+- org.apache.commons:commons-compress:1.26.1
+- org.apache.commons:commons-configuration2:2.10.1
- org.apache.commons:commons-lang3:3.18.0
-- org.apache.commons:commons-text:1.4
-- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1
-- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1
-- org.apache.hadoop:hadoop-annotations:3.3.4
-- org.apache.hadoop:hadoop-auth:3.3.4
-- org.apache.hadoop:hadoop-common:3.3.4
-- org.apache.kerby:kerb-core:1.0.1
-- org.apache.kerby:kerby-asn1:1.0.1
-- org.apache.kerby:kerby-pkix:1.0.1
-- org.apache.kerby:kerby-util:1.0.1
+- org.apache.commons:commons-text:1.10.0
+- org.apache.hadoop:hadoop-annotations:3.4.1
+- org.apache.hadoop:hadoop-auth:3.4.1
+- org.apache.hadoop:hadoop-common:3.4.1
+- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.3.0
+- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25:1.3.0
+- org.apache.kerby:kerb-core:2.0.3
+- org.apache.kerby:kerb-crypto:2.0.3
+- org.apache.kerby:kerb-util:2.0.3
+- org.apache.kerby:kerby-asn1:2.0.3
+- org.apache.kerby:kerby-config:2.0.3
+- org.apache.kerby:kerby-pkix:2.0.3
+- org.apache.kerby:kerby-util:2.0.3
+- org.codehaus.jettison:jettison:1.5.4
Review Comment:
Are the new dependencies really needed? Especially the netty and kerby dependecies. Netty is a big dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
Posted by "xx789633 (via GitHub)" <gi...@apache.org>.
xx789633 commented on code in PR #2674:
URL: https://github.com/apache/fluss/pull/2674#discussion_r2803114355
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java:
##########
@@ -70,31 +70,34 @@ public S3DelegationTokenProvider(String scheme, Configuration conf) {
public ObtainedSecurityToken obtainSecurityToken() {
LOG.info("Obtaining session credentials token with access key: {}", accessKey);
- AWSSecurityTokenService stsClient =
- AWSSecurityTokenServiceClientBuilder.standard()
- .withRegion(region)
- .withCredentials(
- new AWSStaticCredentialsProvider(
- new BasicAWSCredentials(accessKey, secretKey)))
+ StsClient stsClient =
Review Comment:
I don't think so. According to the rustfs discussion at https://github.com/orgs/rustfs/discussions/1746, rustfs currently only support STS AssumeRole action and does not yet support GetSessionToken, which is precisely what our code attempts to invoke (https://github.com/apache/fluss/blob/main/fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java#L80)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #2674:
URL: https://github.com/apache/fluss/pull/2674#discussion_r2802937454
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java:
##########
@@ -70,31 +70,34 @@ public S3DelegationTokenProvider(String scheme, Configuration conf) {
public ObtainedSecurityToken obtainSecurityToken() {
LOG.info("Obtaining session credentials token with access key: {}", accessKey);
- AWSSecurityTokenService stsClient =
- AWSSecurityTokenServiceClientBuilder.standard()
- .withRegion(region)
- .withCredentials(
- new AWSStaticCredentialsProvider(
- new BasicAWSCredentials(accessKey, secretKey)))
+ StsClient stsClient =
Review Comment:
Seems this will also fix #2661 ? Right?
Could you please help verify it?
Also check minio can be supported or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #2674:
URL: https://github.com/apache/fluss/pull/2674#discussion_r2870239144
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java:
##########
@@ -70,31 +70,34 @@ public S3DelegationTokenProvider(String scheme, Configuration conf) {
public ObtainedSecurityToken obtainSecurityToken() {
LOG.info("Obtaining session credentials token with access key: {}", accessKey);
- AWSSecurityTokenService stsClient =
- AWSSecurityTokenServiceClientBuilder.standard()
- .withRegion(region)
- .withCredentials(
- new AWSStaticCredentialsProvider(
- new BasicAWSCredentials(accessKey, secretKey)))
+ StsClient stsClient =
Review Comment:
Actaully, it does't matter if it don't support GetSessionToken. We can still just use STS tokens just like we did for oss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
Posted by "xx789633 (via GitHub)" <gi...@apache.org>.
xx789633 commented on code in PR #2674:
URL: https://github.com/apache/fluss/pull/2674#discussion_r2802937215
##########
fluss-filesystems/fluss-fs-hadoop-shaded/src/main/resources/META-INF/NOTICE:
##########
@@ -9,28 +9,42 @@ This project bundles the following dependencies under the Apache Software Licens
- com.fasterxml.jackson.core:jackson-annotations:2.15.3
- com.fasterxml.jackson.core:jackson-core:2.15.3
- com.fasterxml.jackson.core:jackson-databind:2.15.3
-- com.fasterxml.woodstox:woodstox-core:5.3.0
+- com.fasterxml.woodstox:woodstox-core:5.4.0
- com.google.guava:failureaccess:1.0
- com.google.guava:guava:27.0-jre
- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
- com.google.j2objc:j2objc-annotations:1.1
- commons-beanutils:commons-beanutils:1.9.4
- commons-collections:commons-collections:3.2.2
-- commons-io:commons-io:2.8.0
-- commons-logging:commons-logging:1.1.3
-- org.apache.commons:commons-compress:1.21
-- org.apache.commons:commons-configuration2:2.1.1
+- commons-io:commons-io:2.16.1
+- commons-logging:commons-logging:1.2
+- io.dropwizard.metrics:metrics-core:3.2.4
+- io.netty:netty-buffer:4.1.100.Final
+- io.netty:netty-codec:4.1.100.Final
+- io.netty:netty-common:4.1.100.Final
+- io.netty:netty-handler:4.1.100.Final
+- io.netty:netty-resolver:4.1.100.Final
+- io.netty:netty-transport:4.1.100.Final
+- io.netty:netty-transport-classes-epoll:4.1.100.Final
+- io.netty:netty-transport-native-epoll:4.1.100.Final
+- io.netty:netty-transport-native-unix-common:4.1.100.Final
+- org.apache.commons:commons-compress:1.26.1
+- org.apache.commons:commons-configuration2:2.10.1
- org.apache.commons:commons-lang3:3.18.0
-- org.apache.commons:commons-text:1.4
-- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1
-- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1
-- org.apache.hadoop:hadoop-annotations:3.3.4
-- org.apache.hadoop:hadoop-auth:3.3.4
-- org.apache.hadoop:hadoop-common:3.3.4
-- org.apache.kerby:kerb-core:1.0.1
-- org.apache.kerby:kerby-asn1:1.0.1
-- org.apache.kerby:kerby-pkix:1.0.1
-- org.apache.kerby:kerby-util:1.0.1
+- org.apache.commons:commons-text:1.10.0
+- org.apache.hadoop:hadoop-annotations:3.4.1
+- org.apache.hadoop:hadoop-auth:3.4.1
+- org.apache.hadoop:hadoop-common:3.4.1
+- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.3.0
+- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25:1.3.0
+- org.apache.kerby:kerb-core:2.0.3
+- org.apache.kerby:kerb-crypto:2.0.3
+- org.apache.kerby:kerb-util:2.0.3
+- org.apache.kerby:kerby-asn1:2.0.3
+- org.apache.kerby:kerby-config:2.0.3
+- org.apache.kerby:kerby-pkix:2.0.3
+- org.apache.kerby:kerby-util:2.0.3
+- org.codehaus.jettison:jettison:1.5.4
Review Comment:
it seems those dependencies are always included somewhere (like in https://github.com/apache/fluss/blob/main/fluss-filesystems/fluss-fs-azure/src/main/resources/META-INF/NOTICE)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
Posted by "affo (via GitHub)" <gi...@apache.org>.
affo commented on code in PR #2674:
URL: https://github.com/apache/fluss/pull/2674#discussion_r2811241841
##########
fluss-filesystems/fluss-fs-hadoop-shaded/src/main/resources/META-INF/NOTICE:
##########
@@ -9,28 +9,42 @@ This project bundles the following dependencies under the Apache Software Licens
- com.fasterxml.jackson.core:jackson-annotations:2.15.3
- com.fasterxml.jackson.core:jackson-core:2.15.3
- com.fasterxml.jackson.core:jackson-databind:2.15.3
-- com.fasterxml.woodstox:woodstox-core:5.3.0
+- com.fasterxml.woodstox:woodstox-core:5.4.0
- com.google.guava:failureaccess:1.0
- com.google.guava:guava:27.0-jre
- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
- com.google.j2objc:j2objc-annotations:1.1
- commons-beanutils:commons-beanutils:1.9.4
- commons-collections:commons-collections:3.2.2
-- commons-io:commons-io:2.8.0
-- commons-logging:commons-logging:1.1.3
-- org.apache.commons:commons-compress:1.21
-- org.apache.commons:commons-configuration2:2.1.1
+- commons-io:commons-io:2.16.1
+- commons-logging:commons-logging:1.2
+- io.dropwizard.metrics:metrics-core:3.2.4
+- io.netty:netty-buffer:4.1.100.Final
+- io.netty:netty-codec:4.1.100.Final
+- io.netty:netty-common:4.1.100.Final
+- io.netty:netty-handler:4.1.100.Final
+- io.netty:netty-resolver:4.1.100.Final
+- io.netty:netty-transport:4.1.100.Final
+- io.netty:netty-transport-classes-epoll:4.1.100.Final
+- io.netty:netty-transport-native-epoll:4.1.100.Final
+- io.netty:netty-transport-native-unix-common:4.1.100.Final
+- org.apache.commons:commons-compress:1.26.1
+- org.apache.commons:commons-configuration2:2.10.1
- org.apache.commons:commons-lang3:3.18.0
-- org.apache.commons:commons-text:1.4
-- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1
-- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1
-- org.apache.hadoop:hadoop-annotations:3.3.4
-- org.apache.hadoop:hadoop-auth:3.3.4
-- org.apache.hadoop:hadoop-common:3.3.4
-- org.apache.kerby:kerb-core:1.0.1
-- org.apache.kerby:kerby-asn1:1.0.1
-- org.apache.kerby:kerby-pkix:1.0.1
-- org.apache.kerby:kerby-util:1.0.1
+- org.apache.commons:commons-text:1.10.0
+- org.apache.hadoop:hadoop-annotations:3.4.1
+- org.apache.hadoop:hadoop-auth:3.4.1
+- org.apache.hadoop:hadoop-common:3.4.1
+- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.3.0
+- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25:1.3.0
+- org.apache.kerby:kerb-core:2.0.3
+- org.apache.kerby:kerb-crypto:2.0.3
+- org.apache.kerby:kerb-util:2.0.3
+- org.apache.kerby:kerby-asn1:2.0.3
+- org.apache.kerby:kerby-config:2.0.3
+- org.apache.kerby:kerby-pkix:2.0.3
+- org.apache.kerby:kerby-util:2.0.3
+- org.codehaus.jettison:jettison:1.5.4
Review Comment:
@xx789633 well, that is an investigation, thank you 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
Posted by "affo (via GitHub)" <gi...@apache.org>.
affo commented on code in PR #2674:
URL: https://github.com/apache/fluss/pull/2674#discussion_r2804027113
##########
fluss-filesystems/fluss-fs-hadoop-shaded/src/main/resources/META-INF/NOTICE:
##########
@@ -9,28 +9,42 @@ This project bundles the following dependencies under the Apache Software Licens
- com.fasterxml.jackson.core:jackson-annotations:2.15.3
- com.fasterxml.jackson.core:jackson-core:2.15.3
- com.fasterxml.jackson.core:jackson-databind:2.15.3
-- com.fasterxml.woodstox:woodstox-core:5.3.0
+- com.fasterxml.woodstox:woodstox-core:5.4.0
- com.google.guava:failureaccess:1.0
- com.google.guava:guava:27.0-jre
- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
- com.google.j2objc:j2objc-annotations:1.1
- commons-beanutils:commons-beanutils:1.9.4
- commons-collections:commons-collections:3.2.2
-- commons-io:commons-io:2.8.0
-- commons-logging:commons-logging:1.1.3
-- org.apache.commons:commons-compress:1.21
-- org.apache.commons:commons-configuration2:2.1.1
+- commons-io:commons-io:2.16.1
+- commons-logging:commons-logging:1.2
+- io.dropwizard.metrics:metrics-core:3.2.4
+- io.netty:netty-buffer:4.1.100.Final
+- io.netty:netty-codec:4.1.100.Final
+- io.netty:netty-common:4.1.100.Final
+- io.netty:netty-handler:4.1.100.Final
+- io.netty:netty-resolver:4.1.100.Final
+- io.netty:netty-transport:4.1.100.Final
+- io.netty:netty-transport-classes-epoll:4.1.100.Final
+- io.netty:netty-transport-native-epoll:4.1.100.Final
+- io.netty:netty-transport-native-unix-common:4.1.100.Final
+- org.apache.commons:commons-compress:1.26.1
+- org.apache.commons:commons-configuration2:2.10.1
- org.apache.commons:commons-lang3:3.18.0
-- org.apache.commons:commons-text:1.4
-- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1
-- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1
-- org.apache.hadoop:hadoop-annotations:3.3.4
-- org.apache.hadoop:hadoop-auth:3.3.4
-- org.apache.hadoop:hadoop-common:3.3.4
-- org.apache.kerby:kerb-core:1.0.1
-- org.apache.kerby:kerby-asn1:1.0.1
-- org.apache.kerby:kerby-pkix:1.0.1
-- org.apache.kerby:kerby-util:1.0.1
+- org.apache.commons:commons-text:1.10.0
+- org.apache.hadoop:hadoop-annotations:3.4.1
+- org.apache.hadoop:hadoop-auth:3.4.1
+- org.apache.hadoop:hadoop-common:3.4.1
+- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.3.0
+- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25:1.3.0
+- org.apache.kerby:kerb-core:2.0.3
+- org.apache.kerby:kerb-crypto:2.0.3
+- org.apache.kerby:kerb-util:2.0.3
+- org.apache.kerby:kerby-asn1:2.0.3
+- org.apache.kerby:kerby-config:2.0.3
+- org.apache.kerby:kerby-pkix:2.0.3
+- org.apache.kerby:kerby-util:2.0.3
+- org.codehaus.jettison:jettison:1.5.4
Review Comment:
@xx789633 these new ones are introduced by Hadoop 3.4 right?
If we set them as exclusions, would everything fall apart?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
Posted by "xx789633 (via GitHub)" <gi...@apache.org>.
xx789633 commented on PR #2674:
URL: https://github.com/apache/fluss/pull/2674#issuecomment-3949981803
I see Flink is also working on a similar issue: https://github.com/apache/flink/pull/27026. Let's wait and see how it progresses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
Posted by "affo (via GitHub)" <gi...@apache.org>.
affo commented on code in PR #2674:
URL: https://github.com/apache/fluss/pull/2674#discussion_r2871191374
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java:
##########
@@ -70,31 +70,34 @@ public S3DelegationTokenProvider(String scheme, Configuration conf) {
public ObtainedSecurityToken obtainSecurityToken() {
LOG.info("Obtaining session credentials token with access key: {}", accessKey);
- AWSSecurityTokenService stsClient =
- AWSSecurityTokenServiceClientBuilder.standard()
- .withRegion(region)
- .withCredentials(
- new AWSStaticCredentialsProvider(
- new BasicAWSCredentials(accessKey, secretKey)))
+ StsClient stsClient =
Review Comment:
@luoyuxia I plan to restructure this part of the code by addressing https://github.com/apache/fluss/issues/2662 anyhow 🤝
The idea is to allow more flexible `AWSCredentialsProviders` to delegate tokens to clients.
Would probably fix under some peculiar configuration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on code in PR #2674:
URL: https://github.com/apache/fluss/pull/2674#discussion_r2810475733
##########
fluss-filesystems/fluss-fs-hadoop-shaded/src/main/resources/META-INF/NOTICE:
##########
@@ -9,28 +9,33 @@ This project bundles the following dependencies under the Apache Software Licens
- com.fasterxml.jackson.core:jackson-annotations:2.15.3
- com.fasterxml.jackson.core:jackson-core:2.15.3
- com.fasterxml.jackson.core:jackson-databind:2.15.3
-- com.fasterxml.woodstox:woodstox-core:5.3.0
+- com.fasterxml.woodstox:woodstox-core:5.4.0
- com.google.guava:failureaccess:1.0
- com.google.guava:guava:27.0-jre
- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
- com.google.j2objc:j2objc-annotations:1.1
- commons-beanutils:commons-beanutils:1.9.4
- commons-collections:commons-collections:3.2.2
-- commons-io:commons-io:2.8.0
-- commons-logging:commons-logging:1.1.3
-- org.apache.commons:commons-compress:1.21
-- org.apache.commons:commons-configuration2:2.1.1
+- commons-io:commons-io:2.16.1
+- commons-logging:commons-logging:1.2
+- io.dropwizard.metrics:metrics-core:3.2.4
Review Comment:
`io.dropwizard.metrics:metrics-core` is not used by `hadoop-common` (https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/dependency-analysis.html), can you remove it from all the NOTICE files in filesystem modules?
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java:
##########
@@ -70,31 +70,34 @@ public S3DelegationTokenProvider(String scheme, Configuration conf) {
public ObtainedSecurityToken obtainSecurityToken() {
LOG.info("Obtaining session credentials token with access key: {}", accessKey);
- AWSSecurityTokenService stsClient =
- AWSSecurityTokenServiceClientBuilder.standard()
- .withRegion(region)
- .withCredentials(
- new AWSStaticCredentialsProvider(
- new BasicAWSCredentials(accessKey, secretKey)))
+ StsClient stsClient =
Review Comment:
@luoyuxia , do you have other solutions to resolve the rustfs issue?
##########
fluss-filesystems/fluss-fs-gs/src/main/resources/META-INF/NOTICE:
##########
@@ -75,21 +85,25 @@ This project bundles the following dependencies under the Apache Software Licens
- io.opencensus:opencensus-impl-core:0.31.0
- io.opencensus:opencensus-proto:0.2.0
- io.perfmark:perfmark-api:0.26.0
-- org.apache.commons:commons-compress:1.21
-- org.apache.commons:commons-configuration2:2.1.1
+- org.apache.commons:commons-compress:1.26.1
+- org.apache.commons:commons-configuration2:2.10.1
- org.apache.commons:commons-lang3:3.18.0
-- org.apache.commons:commons-text:1.4
-- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1
-- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1
-- org.apache.hadoop:hadoop-annotations:3.3.4
-- org.apache.hadoop:hadoop-auth:3.3.4
-- org.apache.hadoop:hadoop-common:3.3.4
+- org.apache.commons:commons-text:1.10.0
+- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.3.0
+- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25:1.3.0
+- org.apache.hadoop:hadoop-annotations:3.4.1
+- org.apache.hadoop:hadoop-auth:3.4.1
+- org.apache.hadoop:hadoop-common:3.4.1
- org.apache.httpcomponents:httpclient:4.5.13
- org.apache.httpcomponents:httpcore:4.4.13
-- org.apache.kerby:kerb-core:1.0.1
-- org.apache.kerby:kerby-asn1:1.0.1
-- org.apache.kerby:kerby-pkix:1.0.1
-- org.apache.kerby:kerby-util:1.0.1
+- org.apache.kerby:kerb-core:2.0.3
+- org.apache.kerby:kerb-crypto:2.0.3
+- org.apache.kerby:kerb-util:2.0.3
+- org.apache.kerby:kerby-asn1:2.0.3
+- org.apache.kerby:kerby-config:2.0.3
+- org.apache.kerby:kerby-pkix:2.0.3
+- org.apache.kerby:kerby-util:2.0.3
+- org.codehaus.jettison:jettison:1.5.4
Review Comment:
`org.codehaus.jettison:jettison` is not used by `hadoop-common` (https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/dependency-analysis.html), can you remove it from all the NOTICE files in filesystem modules?
##########
fluss-filesystems/fluss-fs-s3/src/main/resources/META-INF/NOTICE:
##########
@@ -3,57 +3,96 @@ Copyright 2025-2026 The Apache Software Foundation
This project bundles the following dependencies under the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)
-- com.amazonaws:aws-java-sdk-core:1.12.319
-- com.amazonaws:aws-java-sdk-dynamodb:1.12.319
-- com.amazonaws:aws-java-sdk-kms:1.12.319
-- com.amazonaws:aws-java-sdk-s3:1.12.319
-- com.amazonaws:aws-java-sdk-sts:1.12.319
-- com.amazonaws:jmespath-java:1.12.319
- com.fasterxml.jackson.core:jackson-annotations:2.15.3
- com.fasterxml.jackson.core:jackson-core:2.15.3
- com.fasterxml.jackson.core:jackson-databind:2.15.3
-- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.15.3
-- com.fasterxml.woodstox:woodstox-core:5.3.0
+- com.fasterxml.woodstox:woodstox-core:5.4.0
- com.google.guava:failureaccess:1.0
- com.google.guava:guava:27.0-jre
- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
- com.google.j2objc:j2objc-annotations:1.1
- commons-beanutils:commons-beanutils:1.9.4
- commons-codec:commons-codec:1.15
- commons-collections:commons-collections:3.2.2
-- commons-io:commons-io:2.8.0
-- commons-logging:commons-logging:1.1.3
-- joda-time:joda-time:2.8.1
-- org.apache.commons:commons-compress:1.21
-- org.apache.commons:commons-configuration2:2.1.1
+- commons-io:commons-io:2.16.1
+- commons-logging:commons-logging:1.2
+- io.dropwizard.metrics:metrics-core:3.2.4
+- io.netty:netty-buffer:4.1.100.Final
+- io.netty:netty-codec:4.1.100.Final
+- io.netty:netty-codec-http:4.1.100.Final
+- io.netty:netty-codec-http2:4.1.100.Final
+- io.netty:netty-common:4.1.100.Final
+- io.netty:netty-handler:4.1.100.Final
+- io.netty:netty-resolver:4.1.100.Final
+- io.netty:netty-transport:4.1.100.Final
+- io.netty:netty-transport-classes-epoll:4.1.100.Final
+- io.netty:netty-transport-native-epoll:4.1.100.Final
+- io.netty:netty-transport-native-unix-common:4.1.100.Final
Review Comment:
Please remove all the netty dependency in the NOTICE files (including NOTICE files in other filesystem modules).
See the warning log
https://github.com/apache/fluss/actions/runs/22016381283/job/63618691913?pr=2674
##########
fluss-filesystems/fluss-fs-s3/src/main/resources/META-INF/NOTICE:
##########
@@ -3,57 +3,96 @@ Copyright 2025-2026 The Apache Software Foundation
This project bundles the following dependencies under the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)
-- com.amazonaws:aws-java-sdk-core:1.12.319
-- com.amazonaws:aws-java-sdk-dynamodb:1.12.319
-- com.amazonaws:aws-java-sdk-kms:1.12.319
-- com.amazonaws:aws-java-sdk-s3:1.12.319
-- com.amazonaws:aws-java-sdk-sts:1.12.319
-- com.amazonaws:jmespath-java:1.12.319
- com.fasterxml.jackson.core:jackson-annotations:2.15.3
- com.fasterxml.jackson.core:jackson-core:2.15.3
- com.fasterxml.jackson.core:jackson-databind:2.15.3
-- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.15.3
-- com.fasterxml.woodstox:woodstox-core:5.3.0
+- com.fasterxml.woodstox:woodstox-core:5.4.0
- com.google.guava:failureaccess:1.0
- com.google.guava:guava:27.0-jre
- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
- com.google.j2objc:j2objc-annotations:1.1
- commons-beanutils:commons-beanutils:1.9.4
- commons-codec:commons-codec:1.15
- commons-collections:commons-collections:3.2.2
-- commons-io:commons-io:2.8.0
-- commons-logging:commons-logging:1.1.3
-- joda-time:joda-time:2.8.1
-- org.apache.commons:commons-compress:1.21
-- org.apache.commons:commons-configuration2:2.1.1
+- commons-io:commons-io:2.16.1
+- commons-logging:commons-logging:1.2
+- io.dropwizard.metrics:metrics-core:3.2.4
+- io.netty:netty-buffer:4.1.100.Final
+- io.netty:netty-codec:4.1.100.Final
+- io.netty:netty-codec-http:4.1.100.Final
+- io.netty:netty-codec-http2:4.1.100.Final
+- io.netty:netty-common:4.1.100.Final
+- io.netty:netty-handler:4.1.100.Final
+- io.netty:netty-resolver:4.1.100.Final
+- io.netty:netty-transport:4.1.100.Final
+- io.netty:netty-transport-classes-epoll:4.1.100.Final
+- io.netty:netty-transport-native-epoll:4.1.100.Final
+- io.netty:netty-transport-native-unix-common:4.1.100.Final
+- org.apache.commons:commons-compress:1.26.1
+- org.apache.commons:commons-configuration2:2.10.1
- org.apache.commons:commons-lang3:3.18.0
-- org.apache.commons:commons-text:1.4
-- org.apache.hadoop:hadoop-annotations:3.3.4
-- org.apache.hadoop:hadoop-auth:3.3.4
-- org.apache.hadoop:hadoop-aws:3.3.4
-- org.apache.hadoop:hadoop-common:3.3.4
-- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1
-- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1
+- org.apache.commons:commons-text:1.10.0
+- org.apache.hadoop:hadoop-annotations:3.4.1
+- org.apache.hadoop:hadoop-auth:3.4.1
+- org.apache.hadoop:hadoop-aws:3.4.1
+- org.apache.hadoop:hadoop-common:3.4.1
+- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.3.0
+- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25:1.3.0
- org.apache.httpcomponents:httpclient:4.5.13
- org.apache.httpcomponents:httpcore:4.4.13
-- org.apache.kerby:kerb-core:1.0.1
-- org.apache.kerby:kerby-asn1:1.0.1
-- org.apache.kerby:kerby-pkix:1.0.1
-- org.apache.kerby:kerby-util:1.0.1
-- org.wildfly.openssl:wildfly-openssl:1.0.7.Final
+- org.apache.kerby:kerb-core:2.0.3
+- org.apache.kerby:kerb-crypto:2.0.3
+- org.apache.kerby:kerb-util:2.0.3
+- org.apache.kerby:kerby-asn1:2.0.3
+- org.apache.kerby:kerby-config:2.0.3
+- org.apache.kerby:kerby-pkix:2.0.3
+- org.apache.kerby:kerby-util:2.0.3
+- org.codehaus.jettison:jettison:1.5.4
+- org.wildfly.openssl:wildfly-openssl:1.1.3.Final
- org.xerial.snappy:snappy-java:1.1.10.4
-- software.amazon.ion:ion-java:1.0.2
-
-This project bundles the following dependencies under BSD-2 License (https://opensource.org/licenses/BSD-2-Clause).
-See bundled license files for details.
-
-- dnsjava:dnsjava:2.1.7
+- software.amazon.awssdk:annotations:2.24.6
+- software.amazon.awssdk:apache-client:2.24.6
+- software.amazon.awssdk:arns:2.24.6
+- software.amazon.awssdk:auth:2.24.6
+- software.amazon.awssdk:aws-core:2.24.6
+- software.amazon.awssdk:aws-json-protocol:2.24.6
+- software.amazon.awssdk:aws-query-protocol:2.24.6
+- software.amazon.awssdk:aws-xml-protocol:2.24.6
+- software.amazon.awssdk:checksums:2.24.6
+- software.amazon.awssdk:checksums-spi:2.24.6
+- software.amazon.awssdk:crt-core:2.24.6
+- software.amazon.awssdk:dynamodb:2.24.6
+- software.amazon.awssdk:endpoints-spi:2.24.6
+- software.amazon.awssdk:http-auth:2.24.6
+- software.amazon.awssdk:http-auth-aws:2.24.6
+- software.amazon.awssdk:http-auth-spi:2.24.6
+- software.amazon.awssdk:http-client-spi:2.24.6
+- software.amazon.awssdk:identity-spi:2.24.6
+- software.amazon.awssdk:json-utils:2.24.6
+- software.amazon.awssdk:kms:2.24.6
+- software.amazon.awssdk:metrics-spi:2.24.6
+- software.amazon.awssdk:netty-nio-client:2.24.6
+- software.amazon.awssdk:profiles:2.24.6
+- software.amazon.awssdk:protocol-core:2.24.6
+- software.amazon.awssdk:regions:2.24.6
+- software.amazon.awssdk:s3:2.24.6
+- software.amazon.awssdk:sdk-core:2.24.6
+- software.amazon.awssdk:sts:2.24.6
+- software.amazon.awssdk:third-party-jackson-core:2.24.6
+- software.amazon.awssdk:utils:2.24.6
+- software.amazon.eventstream:eventstream:1.0.1
This project bundles the following dependencies under the MIT (https://opensource.org/licenses/MIT)
See bundled license files for details.
- org.checkerframework:checker-qual:2.5.2
- org.codehaus.mojo:animal-sniffer-annotations:1.17
+- org.reactivestreams:reactive-streams:1.0.4
Review Comment:
Which dependency is bringing in this transitive dependency? Can we avoid including it? If it must be included, we are required by ASF policy to add a reference to its license in `META-INF/licenses`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
Posted by "xx789633 (via GitHub)" <gi...@apache.org>.
xx789633 commented on code in PR #2674:
URL: https://github.com/apache/fluss/pull/2674#discussion_r2836010058
##########
fluss-filesystems/fluss-fs-hadoop-shaded/src/main/resources/META-INF/NOTICE:
##########
@@ -9,28 +9,33 @@ This project bundles the following dependencies under the Apache Software Licens
- com.fasterxml.jackson.core:jackson-annotations:2.15.3
- com.fasterxml.jackson.core:jackson-core:2.15.3
- com.fasterxml.jackson.core:jackson-databind:2.15.3
-- com.fasterxml.woodstox:woodstox-core:5.3.0
+- com.fasterxml.woodstox:woodstox-core:5.4.0
- com.google.guava:failureaccess:1.0
- com.google.guava:guava:27.0-jre
- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
- com.google.j2objc:j2objc-annotations:1.1
- commons-beanutils:commons-beanutils:1.9.4
- commons-collections:commons-collections:3.2.2
-- commons-io:commons-io:2.8.0
-- commons-logging:commons-logging:1.1.3
-- org.apache.commons:commons-compress:1.21
-- org.apache.commons:commons-configuration2:2.1.1
+- commons-io:commons-io:2.16.1
+- commons-logging:commons-logging:1.2
+- io.dropwizard.metrics:metrics-core:3.2.4
Review Comment:
maybe we can use localstack to replace rustfs?: https://docs.localstack.cloud/aws/services/sts/#api-coverage it supports GetSessionToken API which is precisely what our code attempts to invoke @luoyuxia @wuchong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
Posted by "xx789633 (via GitHub)" <gi...@apache.org>.
xx789633 commented on code in PR #2674:
URL: https://github.com/apache/fluss/pull/2674#discussion_r2836006840
##########
fluss-filesystems/fluss-fs-gs/src/main/resources/META-INF/NOTICE:
##########
@@ -75,21 +85,25 @@ This project bundles the following dependencies under the Apache Software Licens
- io.opencensus:opencensus-impl-core:0.31.0
- io.opencensus:opencensus-proto:0.2.0
- io.perfmark:perfmark-api:0.26.0
-- org.apache.commons:commons-compress:1.21
-- org.apache.commons:commons-configuration2:2.1.1
+- org.apache.commons:commons-compress:1.26.1
+- org.apache.commons:commons-configuration2:2.10.1
- org.apache.commons:commons-lang3:3.18.0
-- org.apache.commons:commons-text:1.4
-- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1
-- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1
-- org.apache.hadoop:hadoop-annotations:3.3.4
-- org.apache.hadoop:hadoop-auth:3.3.4
-- org.apache.hadoop:hadoop-common:3.3.4
+- org.apache.commons:commons-text:1.10.0
+- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.3.0
+- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25:1.3.0
+- org.apache.hadoop:hadoop-annotations:3.4.1
+- org.apache.hadoop:hadoop-auth:3.4.1
+- org.apache.hadoop:hadoop-common:3.4.1
- org.apache.httpcomponents:httpclient:4.5.13
- org.apache.httpcomponents:httpcore:4.4.13
-- org.apache.kerby:kerb-core:1.0.1
-- org.apache.kerby:kerby-asn1:1.0.1
-- org.apache.kerby:kerby-pkix:1.0.1
-- org.apache.kerby:kerby-util:1.0.1
+- org.apache.kerby:kerb-core:2.0.3
+- org.apache.kerby:kerb-crypto:2.0.3
+- org.apache.kerby:kerb-util:2.0.3
+- org.apache.kerby:kerby-asn1:2.0.3
+- org.apache.kerby:kerby-config:2.0.3
+- org.apache.kerby:kerby-pkix:2.0.3
+- org.apache.kerby:kerby-util:2.0.3
+- org.codehaus.jettison:jettison:1.5.4
Review Comment:
After investigating the transitive dependency tree, here's where each of the new dependencies comes from:
- netty: new in hadoop 3.4. Hadoop's netty usage is for internal RPC/transport, not needed for filesystem operations.
- io.dropwizard.metrics: new in hadoop 3.4. Hadoop metrics framework, not needed for filesystem operations.
- org.codehaus.jettison: new in hadoop 3.4. Not needed.
- org.reactivestream: from AWS SDK v2 http-client-spi. It is core part of AWS SDK v2's async model, cannot be exclude.
By the way, the size of the fluss-fs-s3 jar is only ~40MB. Not too much difference. @wuchong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
Posted by "xx789633 (via GitHub)" <gi...@apache.org>.
xx789633 commented on code in PR #2674:
URL: https://github.com/apache/fluss/pull/2674#discussion_r2807381847
##########
fluss-filesystems/fluss-fs-hadoop-shaded/src/main/resources/META-INF/NOTICE:
##########
@@ -9,28 +9,42 @@ This project bundles the following dependencies under the Apache Software Licens
- com.fasterxml.jackson.core:jackson-annotations:2.15.3
- com.fasterxml.jackson.core:jackson-core:2.15.3
- com.fasterxml.jackson.core:jackson-databind:2.15.3
-- com.fasterxml.woodstox:woodstox-core:5.3.0
+- com.fasterxml.woodstox:woodstox-core:5.4.0
- com.google.guava:failureaccess:1.0
- com.google.guava:guava:27.0-jre
- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
- com.google.j2objc:j2objc-annotations:1.1
- commons-beanutils:commons-beanutils:1.9.4
- commons-collections:commons-collections:3.2.2
-- commons-io:commons-io:2.8.0
-- commons-logging:commons-logging:1.1.3
-- org.apache.commons:commons-compress:1.21
-- org.apache.commons:commons-configuration2:2.1.1
+- commons-io:commons-io:2.16.1
+- commons-logging:commons-logging:1.2
+- io.dropwizard.metrics:metrics-core:3.2.4
+- io.netty:netty-buffer:4.1.100.Final
+- io.netty:netty-codec:4.1.100.Final
+- io.netty:netty-common:4.1.100.Final
+- io.netty:netty-handler:4.1.100.Final
+- io.netty:netty-resolver:4.1.100.Final
+- io.netty:netty-transport:4.1.100.Final
+- io.netty:netty-transport-classes-epoll:4.1.100.Final
+- io.netty:netty-transport-native-epoll:4.1.100.Final
+- io.netty:netty-transport-native-unix-common:4.1.100.Final
+- org.apache.commons:commons-compress:1.26.1
+- org.apache.commons:commons-configuration2:2.10.1
- org.apache.commons:commons-lang3:3.18.0
-- org.apache.commons:commons-text:1.4
-- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1
-- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1
-- org.apache.hadoop:hadoop-annotations:3.3.4
-- org.apache.hadoop:hadoop-auth:3.3.4
-- org.apache.hadoop:hadoop-common:3.3.4
-- org.apache.kerby:kerb-core:1.0.1
-- org.apache.kerby:kerby-asn1:1.0.1
-- org.apache.kerby:kerby-pkix:1.0.1
-- org.apache.kerby:kerby-util:1.0.1
+- org.apache.commons:commons-text:1.10.0
+- org.apache.hadoop:hadoop-annotations:3.4.1
+- org.apache.hadoop:hadoop-auth:3.4.1
+- org.apache.hadoop:hadoop-common:3.4.1
+- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.3.0
+- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25:1.3.0
+- org.apache.kerby:kerb-core:2.0.3
+- org.apache.kerby:kerb-crypto:2.0.3
+- org.apache.kerby:kerb-util:2.0.3
+- org.apache.kerby:kerby-asn1:2.0.3
+- org.apache.kerby:kerby-config:2.0.3
+- org.apache.kerby:kerby-pkix:2.0.3
+- org.apache.kerby:kerby-util:2.0.3
+- org.codehaus.jettison:jettison:1.5.4
Review Comment:
Hi @wuchong @affo, I have excluded netty dependency.
netty dependency is introduced by Hadoop-common 3.4. I have checked the source code of hadoop-common 3.4 and found it does not use netty at all. The Hadoop community likely declared these dependencies in hadoop-common to unify Netty version management across all sub-modules (such as hadoop-hdfs, hadoop-yarn, and other server-side components), but hadoop-common itself does not actually use them. By the way, netty in Hadoop is mainly used for IPC/RPC, while Fluss only calls standard filesystem calls.
For kerby, those dependencies were there before this pull request. We just upgrade their versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
Posted by "litiliu (via GitHub)" <gi...@apache.org>.
litiliu commented on PR #2674:
URL: https://github.com/apache/fluss/pull/2674#issuecomment-5001845363
@xx789633 ，hi，
https://github.com/apache/flink/pull/27026 has become active again and was recently updated
to Hadoop 3.4.3 with AWS SDK v2.
This PR has been inactive for several months and now conflicts with
the latest main. Are you still planning to continue updating it?
If not, I would like to open a successor PR based on the latest Fluss
main, while crediting and reusing the relevant work from this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
Posted by "polyzos (via GitHub)" <gi...@apache.org>.
polyzos closed pull request #2674: [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2
URL: https://github.com/apache/fluss/pull/2674
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 [fluss]
Posted by "polyzos (via GitHub)" <gi...@apache.org>.
polyzos commented on PR #2674:
URL: https://github.com/apache/fluss/pull/2674#issuecomment-6251730849
Superseded by merged #3699; main has Hadoop 3.4.3 and AWS SDK v2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org