You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@linkis.apache.org by GitBox <gi...@apache.org> on 2022/03/30 03:46:21 UTC
[GitHub] [incubator-linkis] CCweixiao opened a new pull request #1829: Script custom variable run type and suffix constraint configuration.
CCweixiao opened a new pull request #1829:
URL: https://github.com/apache/incubator-linkis/pull/1829
### What is the purpose of the change
引擎类型、脚本类型自定义后，脚本中的变量解析异常，需要在源码中修改几个脚本，基于此，需要把此类配置抽取出来作为配置文件中的一个配置项。https://github.com/apache/incubator-linkis/issues/1751
After the engine type and script type are customized, the variable parsing in the script is abnormal, and several scripts need to be modified in the source code. Based on this, such configuration needs to be extracted as a configuration item in the configuration file. https://github.com/apache/incubator-linkis/issues/1751
### Brief change log
增加配置项，以及对应的配置解析工具方法, Add configuration items, and the corresponding configuration parsing tool method
val CODE_TYPE_AND_RUN_TYPE_RELATION = CommonVars("wds.linkis.storage.codeType.runType.relation", "sql=>hql|sql|jdbc|hive|psql|sh|shell,python=>python|py,java=>java,scala=>scala")
```java
def getCodeTypeAndRunTypeRelationMap: Map[String, List[String]] = {
if (CODE_TYPE_AND_RUN_TYPE_RELATION.getValue == null || "".equals(CODE_TYPE_AND_RUN_TYPE_RELATION.getValue)) {
return Map()
}
CODE_TYPE_AND_RUN_TYPE_RELATION.getValue.split(",")
.filter(x => x != null && !"".equals(x))
.map(x => {
val confArr = x.split("=>")
if (confArr.length == 2) (confArr(0), for (x <- confArr(1).split("\\|").toList) yield x.trim) else null
}).filter(x => x != null).toMap
}
```
只是还有几个问题需要确认：
1. 对应配置项CODE_TYPE_AND_RUN_TYPE_RELATION 的定义是否合理，放在StorageConfiguration中是否合适？
2. CustomVariableUtils文件中，runType匹配关系如下
```java
runType match {
case "hql" | "sql" | "jdbc" | "hive"| "psql" => codeType = SQL_TYPE
case "python" | "py" => codeType = PY_TYPE
case "java" => codeType = JAVA_TYPE
case "scala" => codeType = SCALA_TYPE
case "sh" | "shell" => codeType = SQL_TYPE
case _ => return (false, code)
}
```
其中sh shell类型对应codeType是否合理？
3. orchestrator模块中有一个文件也叫CustomVariableUtils，其中用到了上述配置，但是缺少linkis-storage模块依赖，
```xml
<dependency>
<groupId>org.apache.linkis</groupId>
<artifactId>linkis-storage</artifactId>
<version>${linkis.version}</version>
<scope>provided</scope>
</dependency>
```
是否可以直接加呢？会不会引起依赖冲突
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For additional commands, e-mail: commits-help@linkis.apache.org
[GitHub] [incubator-linkis] CCweixiao commented on pull request #1829: Script custom variable run type and suffix constraint configuration.
Posted by GitBox <gi...@apache.org>.
CCweixiao commented on pull request #1829:
URL: https://github.com/apache/incubator-linkis/pull/1829#issuecomment-1082704636
> `org.apache.linkis.governance.common.paser.CodeType` has also mapping functions similar to codetype
>
> ```
> object CodeType extends Enumeration {
> type CodeType = Value
> val Python, SQL, Scala, Shell, Other, Remain, JSON = Value
>
> def getType(codeType: String): CodeType = codeType.toLowerCase() match {
> case "python" | "pyspark" | "py" => Python
> case "sql" | "hql" | "psql" => SQL
> case "scala" => Scala
> case "shell" | "sh" => Shell
> case _ => Other
> }
> }
> ```
>
> `"shell" | "sh" => Shell` is inconsistent with `case "sh" | "shell" => codeType = SQL_TYPE` ,this also confuses me I understand that this mapping relationship should be global. Is it more reasonable to put it in the public module of linkis.
目前只碰到脚本中参数解析的时候，遇到需要在代码中维护这些codeType和runType的变量，我待会看下别的地方是否有相同引用，以做统一
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For additional commands, e-mail: commits-help@linkis.apache.org
[GitHub] [incubator-linkis] casionone commented on pull request #1829: Script custom variable run type and suffix constraint configuration.
Posted by GitBox <gi...@apache.org>.
casionone commented on pull request #1829:
URL: https://github.com/apache/incubator-linkis/pull/1829#issuecomment-1082697588
In addition, I think it is necessary to add unit tests for
`StorageConfiguration.getCodeTypeAndRunTypeRelationMap`
`StorageConfiguration.getSuffixBelongToRunTypeOrNot`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For additional commands, e-mail: commits-help@linkis.apache.org
[GitHub] [incubator-linkis] casionone commented on pull request #1829: Script custom variable run type and suffix constraint configuration.
Posted by GitBox <gi...@apache.org>.
casionone commented on pull request #1829:
URL: https://github.com/apache/incubator-linkis/pull/1829#issuecomment-1082696320
`org.apache.linkis.governance.common.paser.CodeType` has also mapping functions similar to codetype
```
object CodeType extends Enumeration {
type CodeType = Value
val Python, SQL, Scala, Shell, Other, Remain, JSON = Value
def getType(codeType: String): CodeType = codeType.toLowerCase() match {
case "python" | "pyspark" | "py" => Python
case "sql" | "hql" | "psql" => SQL
case "scala" => Scala
case "shell" | "sh" => Shell
case _ => Other
}
}
```
`"shell" | "sh" => Shell` is inconsistent with `case "sh" | "shell" => codeType = SQL_TYPE` ,this also confuses me
I understand that this mapping relationship should be global. Is it more reasonable to put it in the public module of linkis.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For additional commands, e-mail: commits-help@linkis.apache.org
[GitHub] [incubator-linkis] CCweixiao commented on pull request #1829: Script custom variable run type and suffix constraint configuration.
Posted by GitBox <gi...@apache.org>.
CCweixiao commented on pull request #1829:
URL: https://github.com/apache/incubator-linkis/pull/1829#issuecomment-1082703500
>
> In addition, I think it is necessary to add unit tests code for `StorageConfiguration.getCodeTypeAndRunTypeRelationMap` `StorageConfiguration.getSuffixBelongToRunTypeOrNot`
待会的提交，一并带上测试用例
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For additional commands, e-mail: commits-help@linkis.apache.org
[GitHub] [incubator-linkis] casionone edited a comment on pull request #1829: Script custom variable run type and suffix constraint configuration.
Posted by GitBox <gi...@apache.org>.
casionone edited a comment on pull request #1829:
URL: https://github.com/apache/incubator-linkis/pull/1829#issuecomment-1082697588
In addition, I think it is necessary to add unit tests code for
`StorageConfiguration.getCodeTypeAndRunTypeRelationMap`
`StorageConfiguration.getSuffixBelongToRunTypeOrNot`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For additional commands, e-mail: commits-help@linkis.apache.org
[GitHub] [incubator-linkis] peacewong closed pull request #1829: Script custom variable run type and suffix constraint configuration.
Posted by GitBox <gi...@apache.org>.
peacewong closed pull request #1829:
URL: https://github.com/apache/incubator-linkis/pull/1829
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For additional commands, e-mail: commits-help@linkis.apache.org
[GitHub] [incubator-linkis] peacewong commented on pull request #1829: Script custom variable run type and suffix constraint configuration.
Posted by GitBox <gi...@apache.org>.
peacewong commented on pull request #1829:
URL: https://github.com/apache/incubator-linkis/pull/1829#issuecomment-1082698283
Thanks.
Question 1: This is reasonable, but I think the CodeTypeAndRunTypeRelationMap should be made a singleton instead of regenerating every time.
问题1：这个是合理的，但是我认为应该将CodeTypeAndRunTypeRelationMap作为单例，而不是每次都重新生成
Question 2 seems to be a bug, it should be classified as a python class or defined as a shell class, because their comments all start with "#", not the "--" of sql
问题2这个看是Bug，应该归类为python类或者定义为shell类，因为他们的注释都是“#”开头，而不是sql的“--”开头
Question 3 CustomVariableUtils in the orchestrator module will be removed, the code is being merged
问题3 orchestrator模块中的CustomVariableUtils将会被移除掉，正在合并代码
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For additional commands, e-mail: commits-help@linkis.apache.org