You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "sd4324530 (via GitHub)" <gi...@apache.org> on 2026/01/20 14:05:28 UTC
[PR] [filesystem] fix the size of fluss-fs-s3 is too big [fluss]
sd4324530 opened a new pull request, #2419:
URL: https://github.com/apache/fluss/pull/2419
### Purpose
Linked issue: close https://github.com/apache/fluss/issues/2412
### Brief change log
Fixed the issue of the `fluss-fs-s3` jar becoming too big due to Hadoop version upgrades.
1. Exclude dependency from `hadoop-aws` version 3.4.0:
```
<groupId>software.amazon.awssdk</groupId>
<artifactId>bundle</artifactId>
```
2. Added `InvalidCredentialsException` to replace `NoAwsCredentialsException`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [filesystem] fix the size of fluss-fs-s3 is too big [fluss]
Posted by "sd4324530 (via GitHub)" <gi...@apache.org>.
sd4324530 commented on code in PR #2419:
URL: https://github.com/apache/fluss/pull/2419#discussion_r2710846398
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/exception/InvalidCredentialsException.java:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.fs.s3.exception;
+
+/** Exception thrown when no credentials were found. */
+public class InvalidCredentialsException extends RuntimeException {
+
+ public static final String E_NO_AWS_CREDENTIALS = "No AWS Credentials";
+
+ public InvalidCredentialsException(String credentialProvider) {
Review Comment:
Thank you so much for reminding me. I checked it again carefully.
Fluss v0.8 depends on hadoop version 3.3.4.
I found that the inheritance chain of `NoAwsCredentialsException` has changed when upgrading from 3.3.4 to 3.4.0:
hadoop 3.3.4:
```mermaid
classDiagram
direction RL
class AmazonClientException
class CredentialInitializationException
class NoAuthWithAWSException
class NoAwsCredentialsException
AmazonClientException --|> CredentialInitializationException
CredentialInitializationException --|> NoAuthWithAWSException
NoAuthWithAWSException --|> NoAwsCredentialsException
```
hadoop 3.4.0:
```mermaid
classDiagram
direction RL
class SdkClientException
class CredentialInitializationException
class NoAuthWithAWSException
class NoAwsCredentialsException
SdkClientException--|> CredentialInitializationException
CredentialInitializationException --|> NoAuthWithAWSException
NoAuthWithAWSException --|> NoAwsCredentialsException
```
In Hadoop 3.3.4, `AmazonClientException` originates from `aws-java-sdk-core-1.12.319.jar`, which is only 1MB in size.
In Hadoop 3.4.0, `SdkClientException` originates from `bundle-2.23.19.jar`, which is 500+MB in size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [filesystem] fix the size of fluss-fs-s3 is too big [fluss]
Posted by "sd4324530 (via GitHub)" <gi...@apache.org>.
sd4324530 commented on code in PR #2419:
URL: https://github.com/apache/fluss/pull/2419#discussion_r2710846398
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/exception/InvalidCredentialsException.java:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.fs.s3.exception;
+
+/** Exception thrown when no credentials were found. */
+public class InvalidCredentialsException extends RuntimeException {
+
+ public static final String E_NO_AWS_CREDENTIALS = "No AWS Credentials";
+
+ public InvalidCredentialsException(String credentialProvider) {
Review Comment:
Thank you so much for reminding me. I checked it again carefully.
Fluss v0.8 depends on hadoop version 3.3.4.
I found that the extends chain of `NoAwsCredentialsException` has changed when upgrading from 3.3.4 to 3.4.0:
hadoop 3.3.4:
```mermaid
classDiagram
direction RL
class AmazonClientException
class CredentialInitializationException
class NoAuthWithAWSException
class NoAwsCredentialsException
AmazonClientException --|> CredentialInitializationException
CredentialInitializationException --|> NoAuthWithAWSException
NoAuthWithAWSException --|> NoAwsCredentialsException
```
hadoop 3.4.0:
```mermaid
classDiagram
direction RL
class SdkClientException
class CredentialInitializationException
class NoAuthWithAWSException
class NoAwsCredentialsException
SdkClientException--|> CredentialInitializationException
CredentialInitializationException --|> NoAuthWithAWSException
NoAuthWithAWSException --|> NoAwsCredentialsException
```
In Hadoop 3.3.4, `AmazonClientException` originates from `aws-java-sdk-core-1.12.319.jar`, which is only 1MB in size.
see: https://github.com/apache/fluss/blob/release-0.8/fluss-filesystems/fluss-fs-s3/pom.xml#L188-L192
In Hadoop 3.4.0, `SdkClientException` originates from `bundle-2.23.19.jar`, which is 500+MB in size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [filesystem] fix the size of fluss-fs-s3 is too big [fluss]
Posted by "sd4324530 (via GitHub)" <gi...@apache.org>.
sd4324530 commented on PR #2419:
URL: https://github.com/apache/fluss/pull/2419#issuecomment-3791030094
@luoyuxia
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [filesystem] fix the size of fluss-fs-s3 is too big [fluss]
Posted by "sd4324530 (via GitHub)" <gi...@apache.org>.
sd4324530 commented on code in PR #2419:
URL: https://github.com/apache/fluss/pull/2419#discussion_r2710846398
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/exception/InvalidCredentialsException.java:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.fs.s3.exception;
+
+/** Exception thrown when no credentials were found. */
+public class InvalidCredentialsException extends RuntimeException {
+
+ public static final String E_NO_AWS_CREDENTIALS = "No AWS Credentials";
+
+ public InvalidCredentialsException(String credentialProvider) {
Review Comment:
Thank you so much for reminding me. I checked it again carefully.
Fluss v0.8 depends on hadoop version 3.3.4.
I found that the inheritance chain of `NoAwsCredentialsException` has changed when upgrading from 3.3.4 to 3.4.0:
hadoop 3.3.4:
```mermaid
classDiagram
direction RL
class AmazonClientException
class CredentialInitializationException
class NoAuthWithAWSException
class NoAwsCredentialsException
AmazonClientException --|> CredentialInitializationException
CredentialInitializationException --|> NoAuthWithAWSException
NoAuthWithAWSException --|> NoAwsCredentialsException
```
hadoop 3.4.0:
```mermaid
classDiagram
direction RL
class SdkClientException
class CredentialInitializationException
class NoAuthWithAWSException
class NoAwsCredentialsException
SdkClientException--|> CredentialInitializationException
CredentialInitializationException --|> NoAuthWithAWSException
NoAuthWithAWSException --|> NoAwsCredentialsException
```
In Hadoop 3.3.4, `AmazonClientException` originates from `aws-java-sdk-core-1.12.319.jar`, which is only 1MB in size.
see: https://github.com/apache/fluss/blob/release-0.8/fluss-filesystems/fluss-fs-s3/pom.xml#L188-L192
In Hadoop 3.4.0, `SdkClientException` originates from `bundle-2.23.19.jar`, which is 500+MB in size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [filesystem] fix the size of fluss-fs-s3 is too big [fluss]
Posted by "sd4324530 (via GitHub)" <gi...@apache.org>.
sd4324530 commented on PR #2419:
URL: https://github.com/apache/fluss/pull/2419#issuecomment-3794944531
@luoyuxia
I adapted the code in the `fluss-fs-s3` module to AWS Java SDK v2 and removed the v1 version dependency.
Please take another review if you have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [filesystem] fix the size of fluss-fs-s3 is too big [fluss]
Posted by "sd4324530 (via GitHub)" <gi...@apache.org>.
sd4324530 commented on code in PR #2419:
URL: https://github.com/apache/fluss/pull/2419#discussion_r2711108407
##########
fluss-filesystems/fluss-fs-s3/pom.xml:
##########
@@ -233,6 +233,10 @@
<groupId>org.slf4j</groupId>
<artifactId>slf4j-reload4j</artifactId>
</exclusion>
+ <exclusion>
Review Comment:
@luoyuxia Hadoop 3.4.1 provides a "tar" file instead of the "jar" file we usually use. Simply adjusting the version number in the pom.xml file will not reduce the final package size.
I think we still need to manually exclude the fat JAR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [filesystem] fix the size of fluss-fs-s3 is too big [fluss]
Posted by "litiliu (via GitHub)" <gi...@apache.org>.
litiliu commented on PR #2419:
URL: https://github.com/apache/fluss/pull/2419#issuecomment-5001658075
@luoyuxia @sd4324530 Why do we chose to degrade the Hadoop version, instead of excluding the whole aws-bundle and include the necessary aws sub-serice?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [filesystem] fix the size of fluss-fs-s3 is too big [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #2419:
URL: https://github.com/apache/fluss/pull/2419#discussion_r2711340036
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/exception/InvalidCredentialsException.java:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.fs.s3.exception;
+
+/** Exception thrown when no credentials were found. */
+public class InvalidCredentialsException extends RuntimeException {
+
+ public static final String E_NO_AWS_CREDENTIALS = "No AWS Credentials";
+
+ public InvalidCredentialsException(String credentialProvider) {
Review Comment:
Thank you for the explantion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [filesystem] fix the size of fluss-fs-s3 is too big [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia merged PR #2419:
URL: https://github.com/apache/fluss/pull/2419
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [filesystem] fix the size of fluss-fs-s3 is too big [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #2419:
URL: https://github.com/apache/fluss/pull/2419#discussion_r2710747950
##########
fluss-filesystems/fluss-fs-s3/pom.xml:
##########
@@ -233,6 +233,10 @@
<groupId>org.slf4j</groupId>
<artifactId>slf4j-reload4j</artifactId>
</exclusion>
+ <exclusion>
Review Comment:
https://hadoop.apache.org/release/3.4.1.html
"We have also introduced a lean tar which is a small tar file that does not contain the AWS SDK because the size of AWS SDK is itself 500 MB. This can ease usage for non AWS users. Even AWS users can add this jar explicitly if desired."
If we use 3.4.1, the fat jar won't be included.
##########
fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/exception/InvalidCredentialsException.java:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.fs.s3.exception;
+
+/** Exception thrown when no credentials were found. */
+public class InvalidCredentialsException extends RuntimeException {
+
+ public static final String E_NO_AWS_CREDENTIALS = "No AWS Credentials";
+
+ public InvalidCredentialsException(String credentialProvider) {
Review Comment:
I'm curious about the class not found issue. I also check the v0.8 jar for s3
```jar tf fluss-fs-s3-0.8.0-incubating.jar | grep SdkClientException
com/amazonaws/SdkClientException.class
```
The same with the jar build with this pr
```
com/amazonaws/SdkClientException.class
```
So, it looks to me that the same problem will also happen in v0.8, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [filesystem] fix the size of fluss-fs-s3 is too big [fluss]
Posted by "sd4324530 (via GitHub)" <gi...@apache.org>.
sd4324530 commented on code in PR #2419:
URL: https://github.com/apache/fluss/pull/2419#discussion_r2710857012
##########
fluss-filesystems/fluss-fs-s3/pom.xml:
##########
@@ -233,6 +233,10 @@
<groupId>org.slf4j</groupId>
<artifactId>slf4j-reload4j</artifactId>
</exclusion>
+ <exclusion>
Review Comment:
Thanks, this should be a better solution. I'll test it and give my conclusion later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [filesystem] fix the size of fluss-fs-s3 is too big [fluss]
Posted by "sd4324530 (via GitHub)" <gi...@apache.org>.
sd4324530 commented on PR #2419:
URL: https://github.com/apache/fluss/pull/2419#issuecomment-3791056862
@luoyuxia
Replace the full `aws-sdk-v2` dependency in `hadoop-aws` with the minimum required module.
I successfully tested data writing and reading using `minio`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org