You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "zuston (via GitHub)" <gi...@apache.org> on 2026/01/29 08:35:43 UTC

[I] Exclude hadoop dep in fluss-fs-hdfs module [fluss]

zuston opened a new issue, #2511:
URL: https://github.com/apache/fluss/issues/2511

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
   
   
   ### Motivation
   
   According to the documentation at
   https://fluss.apache.org/docs/streaming-lakehouse/integrate-data-lakes/iceberg/#2-remote-storage-support,
   when placing fluss-fs-hdfs into FLINK_HOME/lib, if the Hadoop version provided by the environment is not consistent with the one bundled in fluss-fs-hdfs, the Flink job may fail at runtime.
   
   Currently, fluss-fs-hdfs packages the entire Hadoop dependency inside the jar, which makes it tightly coupled to a specific Hadoop version. To avoid classpath conflicts and improve compatibility with the runtime environment, we should remove the embedded Hadoop dependencies from this jar.
   
   Furthermore, to better align with the Hadoop version used in existing clusters, it would be preferable to introduce a Maven profile mechanism that allows users to explicitly specify the Hadoop version to compile against.
   
   ### Solution
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Willingness to contribute
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Exclude hadoop dep in fluss-fs-hdfs module [fluss]

Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on issue #2511:
URL: https://github.com/apache/fluss/issues/2511#issuecomment-3816327503

   @zuston good arguemnet. What's your opinion @luoyuxia ?
   
   We actually provided `fluss-fs-hadoop` which is a `fluss-fs-hdfs` without bundling hadoop dep. So I think, what we need is a better documentation to guide users, when to use which one. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Exclude hadoop dep in fluss-fs-hdfs module [fluss]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on issue #2511:
URL: https://github.com/apache/fluss/issues/2511#issuecomment-3816255535

   cc @luoyuxia @wuchong 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Exclude hadoop dep in fluss-fs-hdfs module [fluss]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on issue #2511:
URL: https://github.com/apache/fluss/issues/2511#issuecomment-3816344709

   In our cases, we have a existing hadoop cluster with 3.2.2 version. So I have to downgrade fluss-fs to 3.2.2 and then exclude all hadoop deps. Now it works! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org