[Feature][Connector] Split connector jar from release core jar

[范佳 <fa...@qq.com.INVALID>]

Hi all guys,
Now all the connector jar in the binary distribution package of Seatunnel are packaged into one jar file: core .
This makes it impossible for us to implement multi-version support for the same component.
So, I designed a new package and submit method with other wonderful people.

Check it out:
https://github.com/apache/incubator-seatunnel/issues/1669 <https://github.com/apache/incubator-seatunnel/issues/1669>

Hope get your great advise and idea.

Jia  Fan

Re: [Feature][Connector] Split connector jar from release core jar

[wenjun <we...@apache.org>]

Hi Fanjia,

Thanks for opening this discussion, and thanks to other guys providing
the idea, it seems we have found a way to split the connector jars
from our fat job jar.

There are three things that need to do:
1. For Spark engine, we can use `--jars` to submit the connector jars
to the Spark cluster, this is already supported.
2. For Flink engine, we can add our connector jars into Fliink
environment by `pipeline.jars`, this needs to write a client like
`CliFrontend` since Flink doesn't support using `pipeline.jars` by
shell.
3. Change the current distribution way, packaging the connectors into
our plugin directory.

If I have misunderstood, please let me know.

Thanks,
Wenjun Ruan


On Wed, Apr 20, 2022 at 2:24 PM 范佳 <fa...@qq.com.invalid> wrote:
>
> Hi all guys,
> Now all the connector jar in the binary distribution package of Seatunnel are packaged into one jar file: core .
> This makes it impossible for us to implement multi-version support for the same component.
> So, I designed a new package and submit method with other wonderful people.
>
> Check it out:
> https://github.com/apache/incubator-seatunnel/issues/1669 <https://github.com/apache/incubator-seatunnel/issues/1669>
>
> Hope get your great advise and idea.
>
> Jia  Fan
>