You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/04/07 03:34:02 UTC
[GitHub] [incubator-seatunnel] BenJFan opened a new issue, #1669: [Feature][Connector] Split connector jar from release core jar
BenJFan opened a new issue, #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669
### Search before asking
- [X] I had searched in the [feature](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
### Description
Now all the connector jar in the binary distribution package of Seatunnel are packaged into one jar file: core . This makes it impossible for us to implement multi-version support for the same component.
### Now
```
.
apache-seatunnel
| - - lib
| - - seatunnel-core-spark.jar
| - - seatunnel-core-flink.jar
| - - plugins
| - - config
| - - bin
```
### After (Example for mulit-version Elasticsearch 6.x and 7.x)
```
.
apache-seatunnel
| - - lib
| - - seatunnel-core-spark.jar
| - - seatunnel-core-flink.jar
| - - connectors
| - - flink
| - - seatunnel-connector-flink-elasticsearch7.jar
| - - seatunnel-connector-flink-kafka0.10.jar
| - - other-all-lasted-version-connecotr-single.jar
| - - spark
| - - opt
| - - flink
| - - seatunnel-connector-flink-elasticsearch6.jar
| - - seatunnel-connector-flink-kafka0.08.jar
| - - other-all-older-version-connecotr-single.jar
| - - spark
| - - plugins
| - - config
| - - bin
```
After this finish, user can use differcult version connecotor on one release version, just need move jar from opt folder to connectors folder.
### Usage Scenario
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] BenJFan commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
BenJFan commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1092640550
> > Can flink load additional jars through the -C parameter?
>
> Maybe we can, I will test this, thanks.
-C,--classpath <url> Adds a URL to each user code
classloader on all nodes in the
cluster. The paths must specify a
protocol (e.g. file://) and be
accessible on all nodes (e.g. by means
of a NFS share). You can use this
option multiple times for specifying
more than one URL. The protocol must
be supported by the {@link
java.net.URLClassLoader}.
This parameter should be jar can be accessible on all nodes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] BenJFan commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
BenJFan commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1094474225
> > currently the deployment directory `plugins` was designed to put connectors's related dependencies, we'd better keep this rule for compatibility.
>
> the previous structure is like this:
>
> ```
> | - - plugins
> | - - elasticsearch7/lib
> | - - seatunnel-connector-flink-elasticsearch7.jar
> | - - kafka0.10/lib
> | - - seatunnel-connector-flink-kafka0.10.jar
> ```
>
> it doesn't distinguish engines,maybe different with your design.
Thanks, seem like plugins directory used for store some plugin configuration files. https://github.com/apache/incubator-seatunnel/pull/1651
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1092644491
>
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] bigdataf commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
bigdataf commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1092689617
> A jar shared storage is required, but it seems that hdfs is not supported,There is also the -yt parameter, but yarn mode is supported,and not universal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] bigdataf commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
bigdataf commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1092628584
Can flink load additional jars through the -C parameter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1091827217
> There have one problem, seem like flink cli submit job accpted only one jar once time. Have any solution?
It seems flink just accept a shadow jar. Can we merge the multiple jars into a shadow jar by code when we need to submit a seatunnel job?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1092638396
> Can flink load additional jars through the -C parameter?
Maybe we can, I will test this, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1092548007
I investigate a way that don't need to merge the jar with Flink engine, we can rewrite `CliFrontend` to submit our dependencies. But this way is hack, and we may need to maintain the different version of `CliFronted`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] kalencaya commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
kalencaya commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1099738232
> > surprised and glad to people have similar ideas with me.
> > I have been working for integrate flink and seatunnel with web admin and I'm willing to share my works for you:
> >
> > * submit seatunnel job. flink cluster support Standalone、YARN、Native Kubernetes resource providers and Session、Application、Per-Job deployment mode, so it is a hard work for supporting all of them. Although flink provides cli、rest、python and scala REPL for job submission and monitor, there are still absence of Java client for submiting jobs and access flink cluster and jobs. I have spent much time providing such a Java client and published it to maven central repository.
> > * thin seatunnel jar. Seatunnel jar contains all connector plugins and would distribute to all flink cluster nodes with job. As we finded, flink support application deployment mode for reducing job jar dependencies network overload. For this problem, I have concluded two solutions, just same as mentioned before:
> >
> > * one is `--classpath` parameters. Actually it is great for solving seatunnel distribution between nodes, people just configures seatunnel environment in advance on each nodes like jdk or something else, which is easy and receive huge payback.
> > * another is `PipelineOptions#JARS` option. `Clifrontend` would add job jar and libs as `PipelineOptions#JARS` option, then job jar and their dependencies would be upload to JobManager. Through our Java client, we have demonstrated the practicability that a thin seatunnel jar containing none connector plugin and specified connector plugin jars job submission.
>
> Thanks guys advise. @kalencaya @tmljob . I had readed flinkx's code, it also use `PipelineOptions#JARS` option. I will learn how make this way work on SeaTunnel. Thanks again. By the way, @kalencaya is the jar will also upload to TaskManager?
Of course! Flink handles them well between JobManager and TaskManager.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yx91490 commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
yx91490 commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1092377687
currently the deployment directory `plugins` was designed to put connectors's related dependencies, we'd better keep this rule for compatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] BenJFan commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
BenJFan commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1092574345
> I investigate a way that don't need to merge the jar with Flink engine, we can rewrite `CliFrontend` to submit our dependencies. But this way is hack, and we may need to maintain the different version of `CliFronted`.
This method can work on cluster mode？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] BenJFan commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
BenJFan commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1099077819
> surprised and glad to people have similar ideas with me.
>
> I have been working for integrate flink and seatunnel with web admin and I'm willing to share my works for you:
>
> * submit seatunnel job. flink cluster support Standalone、YARN、Native Kubernetes resource providers and Session、Application、Per-Job deployment mode, so it is a hard work for supporting all of them. Although flink provides cli、rest、python and scala REPL for job submission and monitor, there are still absence of Java client for submiting jobs and access flink cluster and jobs. I have spent much time providing such a Java client and published it to maven central repository.
> * thin seatunnel jar. Seatunnel jar contains all connector plugins and would distribute to all flink cluster nodes with job. As we finded, flink support application deployment mode for reducing job jar dependencies network overload. For this problem, I have concluded two solutions, just same as mentioned before:
>
> * one is `--classpath` parameters. Actually it is great for solving seatunnel distribution between nodes, people just configures seatunnel environment in advance on each nodes like jdk or something else, which is easy and receive huge payback.
> * another is `PipelineOptions#JARS` option. `Clifrontend` would add job jar and libs as `PipelineOptions#JARS` option, then job jar and their dependencies would be upload to JobManager. Through our Java client, we have demonstrated the practicability that a thin seatunnel jar containing none connector plugin and specified connector plugin jars job submission.
Thanks guys advise. @kalencaya @tmljob . I had readed flinkx's code, it also use `PipelineOptions#JARS` option. I will learn how make this way work on SeaTunnel. Thanks again. By the way, @kalencaya is the jar will also upload to TaskManager?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1092598640
> > I investigate a way that don't need to merge the jar with Flink engine, we can rewrite `CliFrontend` to submit our dependencies. But this way is hack, and we may need to maintain the different version of `CliFronted`.
>
> This method can work on cluster mode？
I don't test, but it should work, `CliFrontend` will submit the jar to `JobManager`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1092644759
> > > Can flink load additional jars through the -C parameter?
> >
> >
> > Maybe we can, I will test this, thanks.
>
> ```
> -C,--classpath <url> Adds a URL to each user code
> classloader on all nodes in the
> cluster. The paths must specify a
> protocol (e.g. file://) and be
> accessible on all nodes (e.g. by means
> of a NFS share). You can use this
> option multiple times for specifying
> more than one URL. The protocol must
> be supported by the {@link
> java.net.URLClassLoader}.
> ```
>
> This parameter should be jar can be accessible on all nodes
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] BenJFan commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
BenJFan commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1091044283
Also we can use program to check user's config need which connectors, then only load the connectors user must use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] BenJFan commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
BenJFan commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1092379147
> currently the deployment directory `plugins` was designed to put connectors's related dependencies, we'd better keep this rule for compatibility.
Ok, change connectors directory to plugins directory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] BenJFan commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
BenJFan commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1092375996
> > There have one problem, seem like flink cli submit job accpted only one jar once time. Have any solution?
>
> It seems flink just accept a shadow jar. Can we merge the multiple jars into a shadow jar by code when we need to submit a seatunnel job?
This is a solution, but maybe cost more time, we can consider about this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] tmljob commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
tmljob commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1096671410
For the problem of splitting plug-in packages under flink, please refer to the implementation of chunjun (flinkx).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun closed issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
ruanwenjun closed issue #1669: [Feature][Connector] Split connector jar from release core jar
URL: https://github.com/apache/incubator-seatunnel/issues/1669
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yx91490 commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
yx91490 commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1094225261
> currently the deployment directory `plugins` was designed to put connectors's related dependencies, we'd better keep this rule for compatibility.
the previous structure is like this:
```
| - - plugins
| - - elasticsearch7/lib
| - - seatunnel-connector-flink-elasticsearch7.jar
| - - kafka0.10/lib
| - - seatunnel-connector-flink-kafka0.10.jar
```
it doesn't distinguish engines,maybe different with your design.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] BenJFan commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
BenJFan commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1091104779
There have one problem, seem like flink cli submit job accpted only one jar once time. Have any solution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] kalencaya commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
kalencaya commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1096744296
surprised and glad to people have similar ideas with me.
I have been working for integrate flink and seatunnel with web admin and I'm willing to share my works for you:
* submit seatunnel job. flink cluster support Standalone、YARN、Native Kubernetes resource providers and Session、Application、Per-Job deployment mode, so it is a hard work for supporting all of them. Although flink provides cli、rest、python and scala REPL for job submission and monitor, there are still absence of Java client for submiting jobs and access flink cluster and jobs. I have spent much time providing such a Java client and published it to maven central repository.
* thin seatunnel jar. Seatunnel jar contains all connector plugins and would distribute to all flink cluster nodes with job. As we finded, flink support application deployment mode for reducing job jar dependencies network overload. For this problem, I have concluded two solutions, just same as mentioned before:
* one is `--classpath` parameters. Actually it is great for solving seatunnel distribution between nodes, people just configures seatunnel environment in advance on each nodes like jdk or something else, which is easy and receive huge payback.
* another is `PipelineOptions#JARS` option. `Clifrontend` would add job jar and libs as `PipelineOptions#JARS` option, then job jar and their dependencies would be upload to JobManager. Through our Java client, we have demonstrated the practicability that a thin seatunnel jar containing none connector plugin and specified connector plugin jars job submission.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] kalencaya commented on issue #1669: [Feature][Connector] Split connector jar from release core jar
Posted by GitBox <gi...@apache.org>.
kalencaya commented on issue #1669:
URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1096745337
Here is my flink Java client repository url: https://github.com/flowerfine/flinkful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org