[jira] [Commented] (FLINK-22575) Offer a more recent flink-shaded-hadoop-2-uber

["Chesnay Schepler (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-22575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339689#comment-17339689 ] 

Chesnay Schepler commented on FLINK-22575:
------------------------------------------

While I appreciate that you only propose maintaining a single version it does feel like an overly optimistic take on the matter. It is likely that hadoop 3 would need a similar treatment in the future. Additionally, this is very much a long-term commitment that I'd ideally avoid.

Generally speaking I'd prefer if we could look at what is required form Hadoop specifically for each connector; a general-purpose Hadoop distribution will just result in users using it yet again for other use cases like Yarn etc., where we then end up with the same issues we had in the past; a lack of a clear feature set and the in-feasibility of verifying that everything still works as it should.

Have other approaches been considered, like moving more Flink dependencies to flink-shaded? What dependencies are commonly clashing?

> Offer a more recent flink-shaded-hadoop-2-uber
> ----------------------------------------------
>
>                 Key: FLINK-22575
>                 URL: https://issues.apache.org/jira/browse/FLINK-22575
>             Project: Flink
>          Issue Type: Improvement
>          Components: BuildSystem / Shaded, Table SQL / Ecosystem
>            Reporter: Timo Walther
>            Assignee: Timo Walther
>            Priority: Major
>
> FLINK-17685 stopped offering a {{flink-shaded-hadoop-2-uber}}. However, it seems that other downstream projects would benefit from supporting a pre-bundled Hadoop for a long period of time. Esp. when offering uber JARs for SQL connectors/formats, a default Hadoop version is required that does not interfere with Flink dependencies. 
> We suggest to revert FLINK-17685 and offer a Hadoop 2.10.1. However, this is just an intermediate solution and will only be supported for Hadoop 2. User's are advised to use HADOOP_CLASSPATH.
> In the long-term our connectors should make use of the plugin architecture.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)