You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@linkis.apache.org by "casionone (via GitHub)" <gi...@apache.org> on 2023/03/20 15:40:53 UTC
[GitHub] [linkis] casionone opened a new issue, #4397: [Feature] Support scala script deal with custom jar package
casionone opened a new issue, #4397:
URL: https://github.com/apache/linkis/issues/4397
## Description
You can create a new scala code script to execute the jar package command, and the jar package is stored in the shared directory (also supports hdfs mode)
Support spark configuration on the management console Support additional jar path parameters spark Support additional jar packages Multiple separated by commas Does not support directory-level settings (the set directory will be automatically filtered), need to be specific to the jar package
The default value is ""
Example :
```
spark.external.default.jars=file:///mnt/bdap/hadoop/qualitis/0236_uat.jar,file:///mnt/bdap/hadoop/qualitis/0236_uat2.jar,file:///mnt/bdap /hadoop/qualities
file:///mnt/bdap/hadoop/qualitis will be ignored automatically
```
other:
The current hive library table export involves the scala execution function. The org.apache.linkis.engineplugin.spark.imexport.ExportData.exportData function is a resource that needs to be specified to the engine to load by specifying the jar package path
```
val destination =
"""{"path":"/mnt/bdap/hadoop/casionxia/student0308gbk1.csv","pathType":"share","hasHeader":true,"isCsv":true,"isOverwrite":false," fieldDelimiter":",","encoding":"GBK","nullValue":"NULL"}"""
val dataInfo =
"""{"database":"stacyyan_ind","tableName":"student","isPartition":false,"partition":"","columns":"*"}"""
org.apache.linkis.engineplugin.spark.imexport.ExportData.exportData(spark, dataInfo, destination)
```
## 描述
可以新建scala代码脚本执行jar包命令,jar包存放于 放在共享目录（也支持 hdfs方式）
支持在管理台配置spark 支持额外jar的路径参数 spark 支持额外的jar包 多个以逗号分隔 不支持目录级别的设置（设置的目录会被自动过滤），需要具体到jar包
默认值为 ""
示例 :
```
spark.external.default.jars=file:///mnt/bdap/hadoop/qualitis/0236_uat.jar,file:///mnt/bdap/hadoop/qualitis/0236_uat2.jar,file:///mnt/bdap/hadoop/qualitis
file:///mnt/bdap/hadoop/qualitis 会被自动忽略
```
其他：
当前hive 库表导出就涉及到scala 执行函数，org.apache.linkis.engineplugin.spark.imexport.ExportData.exportData 函数是需要通过指定jar包路径来告知引擎加载的资源
```
val destination =
"""{"path":"/mnt/bdap/hadoop/casionxia/student0308gbk1.csv","pathType":"share","hasHeader":true,"isCsv":true,"isOverwrite":false,"fieldDelimiter":",","encoding":"GBK","nullValue":"NULL"}"""
val dataInfo =
"""{"database":"stacyyan_ind","tableName":"student","isPartition":false,"partition":"","columns":"*"}"""
org.apache.linkis.engineplugin.spark.imexport.ExportData.exportData(spark,dataInfo,destination)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] github-actions[bot] commented on issue #4397: [Feature] Support scala script deal with custom jar package
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4397:
URL: https://github.com/apache/linkis/issues/4397#issuecomment-1476471339
## :blush: Welcome to the Apache Linkis community!!
We are glad that you are contributing by opening this issue.
Please make sure to include all the relevant context.
We will be here shortly.
If you are interested in contributing to our website project, please let us know!
You can check out our contributing guide on
:point_right: [How to Participate in Project Contribution](https://linkis.apache.org/community/how-to-contribute).
### Community
|WeChat Assistant|WeChat Public Account|
|-|-|
|<img src="https://linkis.apache.org/Images/wedatasphere_contact_01.png" width="128"/>|<img src="https://linkis.apache.org/Images/gzh_01.png" width="128"/>|
### Mailing Lists
|Name|Description|Subscribe|Unsubscribe|Archive|
|:-----|:--------|:------|:-------|:-----|
| [dev@linkis.apache.org](mailto:dev@linkis.apache.org) | community activity information | [subscribe](mailto:dev-subscribe@linkis.apache.org) | [unsubscribe](mailto:dev-unsubscribe@linkis.apache.org) | [archive](http://mail-archives.apache.org/mod_mbox/linkis-dev) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] jacktao007 closed issue #4397: [Feature] Support scala script deal with custom jar package
Posted by "jacktao007 (via GitHub)" <gi...@apache.org>.
jacktao007 closed issue #4397: [Feature] Support scala script deal with custom jar package
URL: https://github.com/apache/linkis/issues/4397
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org