[jira] [Commented] (FLINK-11026) Rework creation of sql-client connector/format jars

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711361#comment-16711361 ] 

ASF GitHub Bot commented on FLINK-11026:
----------------------------------------

zentol opened a new pull request #7251: [FLINK-11026][ES6] Rework creation of fat sql-client jars 
URL: https://github.com/apache/flink/pull/7251
 
 
   Based on #7247.
   
   ## What is the purpose of the change
   
   This PR is a PoC for reworking the packaging of jars specific to the sql-client (which basically are just fat-jars). Only the `flink-connector-elasticsearch6` module is covered here; if accepted the same principle should be applied to the kafka connectors (0.10, 0.11, 2) and all formats.
   
   Instead of defining separate shade-plugin execution with a custom artifactSuffix this PR adds a dedicated `flink-sql-connector-elasticsearch6` module which only contains the packaging logic. This is a similar approach that we've already been using for `flink-shaded-hadoop2-uber`.
   
   The main motivation for this is licensing; for accurate notice files it is necessary to be able to supply each artifact with distinct NOTICE files.
   
   This cannot be done within a single module in a reasonable way. We would have to un-package each created jar, add the appropriate license files, and re-pack them again. We'd end up with tightly-coupled plugin definitions (since the names have to match!) and an overall more complicated (and slower!) build.
   
   ## Brief change log
   
   * add new `flink-sql-connector-elasticsearch6` module containing the sql-client-specific shade-plugin configuration and apply the following modifications
     *  set `executionId` to `shade-flink`
     * disable `shadedArtifactAttached` so only a single jar is deployed
     * remove sql-jar suffix as it is no longer necessary
   * remove sqlJars profile from `flink-connector-elasticsearch6`
   * add `sqlJars` profile to `flink-connectors` to support skipping the creation of sql jars
   
   ## Verifying this change
   
   Covered by sql-client E2E test.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes)
   
   ## Documentation
   
   I have not checked the documentation yet for references that would have to be changed.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Rework creation of sql-client connector/format jars
> ---------------------------------------------------
>
>                 Key: FLINK-11026
>                 URL: https://issues.apache.org/jira/browse/FLINK-11026
>             Project: Flink
>          Issue Type: Improvement
>          Components: Build System, SQL Client
>    Affects Versions: 1.5.5, 1.6.2, 1.7.0
>            Reporter: Chesnay Schepler
>            Assignee: Chesnay Schepler
>            Priority: Major
>              Labels: pull-request-available
>
> For the SQL client we currently have a separate {{sql-jars}} profile in various connectors/formats that create an additional fat jar with a separate classifier.
> One of the core maven mantras is "One artifact per module.", and we see the importance of this mantra as our current packaging strategy makes it impossible to provide different NOTICE files for the created jars (regular and sql-jar).
> Currently we would be forced to provide the same file for both jars, which will cause problems for any downstream users that wants to handle NOTICE files properly. We would cause the same issue we had with netty, which categorically claims to be bundling dependencies although it doesn't, forcing us to manually cut out the valid parts.
> My proposal is to move custom packaging strategies into their own module that depend on the original module.
>  I will use {{flink-connector-elasticsearch6}} as an example, which packages both a regular jar without any included dependencies, and a sql jar bundling everything.
>  * create a separate {{flink-sql-connector-elasticsearch6/}}{{flink-connector-elasticsearch6-uber}}{{}} module
>  * this module depends on {{flink-connector-elasticsearch6}}, and bundles all dependencies
>  * move the current shading logic for the sql jar out of the {{sql-jars}} profile{{}}
>  * add a {{sql-jars}} profile to {{flink-connectors}} for skipping the creation of these jars



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)