You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@linkis.apache.org by Chen Xia <ca...@gmail.com> on 2022/01/19 02:12:41 UTC
[DISCUSS] Optimize the third-party dependency jar packages
At present, the compiled binary package of linkis is relatively large,
nearly 500MB, and there are many jar packages that third party depend on.
currently existing problems
- The package body is too large and exceeds the size of general
conventional projects, which increases the difficulty of downloading the
material package.
- The same dependency uses different versions directly or indirectly in
different modules, which can easily lead to package conflicts.
- Whether the linkis-mg-gateway/linkis-mg-eureka service can not import
all the packages, like other services, in the startup script, add the
general package path of linkis-commons to the classpath
- When the engine plugin starts, the default is to load the general
package lib/linkis-commons/public-module/*. Currently, In the compiled
binary of engine plugin, there are many jar that are duplicated in
lib/linkis-commons , so the engine dependencies are also can be simplified.
Optimization method:
- Step 1: You can first sort out the list of third-party dependencies.
Click the link below to download the data based on version 1.0.3 : Download
the third-party dependency list data:download third-party dependencies
list txt
<https://github.com/apache/incubator-linkis/files/7888883/third-party-dependencies-with-plugin.txt>
- Step 2: According to the principle of single introduction, try to
simplify, other modules can exclude existing jar packages in
lib/linkis-commons/public-module/, and common dependencies can be placed in
lib/linkis-commons/public-module/ middle
The corresponding issue[1] has been created. Is there anything that needs
special attention in the distributed deployment scenario?
[1] https://github.com/apache/incubator-linkis/issues/1313
---------------------------
目前linkis的编译后的部署安装包比较大，已经近500MB，第三方依赖的jar包比较多。目前存在的问题
- 包体过大，超出一般常规项目的大小，增加了物料包下载的难度。
- 同一依赖项在不同模块，直接或间接的使用了不同版本，容易导致包冲突问题。
-
linkis-mg-gateway/linkis-mg-eureka服务是否可以不用引入全部包，和其他服务一样，启动脚本中，将linkis-commons的通用包路径加入到classpath中
- 引擎插件启动时，默认是加载通用包 lib/linkis-commons/public-module/*，目前引擎插件编译的包中，存在不少与
lib/linkis-commons中重复的包，所以引擎的依赖包是不是也可以进行精简。
优化方式：
- 步骤一：可以先梳理出第三方依赖详情列表，点击下方链接可以下载基于1.0.3版本梳理的数据：下载第三方依赖列表数据，
-
步骤二：根据单一引入，尽量精简的原则，其他模块可以排除lib/linkis-commons/public-module/中已存在的jar包，通用的依赖可以尽量放在lib/linkis-commons/public-module/中
已经创建了对应的issue[1]，对于分布式部署场景这种处理方式是否有什么需要特别注意的地方？
[1] https://github.com/apache/incubator-linkis/issues/1313
Re: [DISCUSS] Optimize the third-party dependency jar packages
Posted by Shuai Di <sh...@apache.org>.
+1.
We should reduce the package size to ease the distribution and deployment.
There's similar opinion from the Incubator community too.
Shuai Di
Best Regards.
Chen Xia <ca...@gmail.com> 于2022年1月19日周三 10:13写道：
> At present, the compiled binary package of linkis is relatively large,
> nearly 500MB, and there are many jar packages that third party depend on.
> currently existing problems
>
> - The package body is too large and exceeds the size of general
> conventional projects, which increases the difficulty of downloading the
> material package.
> - The same dependency uses different versions directly or indirectly in
> different modules, which can easily lead to package conflicts.
> - Whether the linkis-mg-gateway/linkis-mg-eureka service can not import
> all the packages, like other services, in the startup script, add the
> general package path of linkis-commons to the classpath
> - When the engine plugin starts, the default is to load the general
> package lib/linkis-commons/public-module/*. Currently, In the compiled
> binary of engine plugin, there are many jar that are duplicated in
> lib/linkis-commons , so the engine dependencies are also can be
> simplified.
>
> Optimization method:
>
> - Step 1: You can first sort out the list of third-party dependencies.
> Click the link below to download the data based on version 1.0.3 :
> Download
> the third-party dependency list data:download third-party dependencies
> list txt
> <
> https://github.com/apache/incubator-linkis/files/7888883/third-party-dependencies-with-plugin.txt
> >
> - Step 2: According to the principle of single introduction, try to
> simplify, other modules can exclude existing jar packages in
> lib/linkis-commons/public-module/, and common dependencies can be
> placed in
> lib/linkis-commons/public-module/ middle
>
>
> The corresponding issue[1] has been created. Is there anything that needs
> special attention in the distributed deployment scenario?
>
> [1] https://github.com/apache/incubator-linkis/issues/1313
>
> ---------------------------
>
> 目前linkis的编译后的部署安装包比较大，已经近500MB，第三方依赖的jar包比较多。目前存在的问题
>
> - 包体过大，超出一般常规项目的大小，增加了物料包下载的难度。
> - 同一依赖项在不同模块，直接或间接的使用了不同版本，容易导致包冲突问题。
> -
>
> linkis-mg-gateway/linkis-mg-eureka服务是否可以不用引入全部包，和其他服务一样，启动脚本中，将linkis-commons的通用包路径加入到classpath中
> - 引擎插件启动时，默认是加载通用包 lib/linkis-commons/public-module/*，目前引擎插件编译的包中，存在不少与
> lib/linkis-commons中重复的包，所以引擎的依赖包是不是也可以进行精简。
>
> 优化方式：
>
> - 步骤一：可以先梳理出第三方依赖详情列表，点击下方链接可以下载基于1.0.3版本梳理的数据：下载第三方依赖列表数据，
> -
>
> 步骤二：根据单一引入，尽量精简的原则，其他模块可以排除lib/linkis-commons/public-module/中已存在的jar包，通用的依赖可以尽量放在lib/linkis-commons/public-module/中
>
> 已经创建了对应的issue[1]，对于分布式部署场景这种处理方式是否有什么需要特别注意的地方？
>
> [1] https://github.com/apache/incubator-linkis/issues/1313
>