You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/09/15 06:37:42 UTC
[GitHub] [incubator-seatunnel] CRUDKanade opened a new issue, #2736: [Bug] --variable无法替换配置文件里的${}变量
CRUDKanade opened a new issue, #2736:
URL: https://github.com/apache/incubator-seatunnel/issues/2736
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
按官网https://seatunnel.apache.org/zh-CN/docs/2.1.3/command/usage , 描述使用 --variable 命令无法替换${}变量
### SeaTunnel Version
2.1.2
### SeaTunnel Config
```conf
env {
spark.executor.memory = "1g"
spark.executor.cores = "1"
spark.executor.instances = "2"
spark.app.name = "test"
}
source {
jdbc {
driver = "com.mysql.cj.jdbc.Driver"
url = "jdbc:mysql://xxxx:xxxx/xxx?"
user = ****
password = ****
table = "user"
result_table_name = "tb_6875311198656"
}
}
transform {
sql {
sql = """select ${city}"""
result_table_name = "tb_6875316324672"
}
}
sink {
Console {}
}
```
### Running Command
```shell
./bin/start-seatunnel-spark.sh --master local[4] --deploy-mode client --config /root/seatunnel/seatunnel-2.1.2/config/test.conf -i city=1
```
### Error Exception
```log
2022-09-07 08:55:54 ERROR Seatunnel:60 -
===============================================================================
2022-09-07 08:55:54 ERROR Seatunnel:63 - Fatal Error,
2022-09-07 08:55:54 ERROR Seatunnel:65 - Please submit bug report in https://github.com/apache/incubator-seatunnel/issues
2022-09-07 08:55:54 ERROR Seatunnel:67 - Reason:Execute Spark task error
2022-09-07 08:55:54 ERROR Seatunnel:68 - Exception StackTrace:java.lang.RuntimeException: Execute Spark task error
at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:79)
at org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:39)
at org.apache.seatunnel.core.spark.SeatunnelSpark.main(SeatunnelSpark.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.spark.sql.catalyst.parser.ParseException:
no viable alternative at input 'select '(line 1, pos 7)
== SQL ==
select
-------^^^
at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:239)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:115)
at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:69)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
at org.apache.seatunnel.spark.transform.Sql.process(Sql.scala:28)
at org.apache.seatunnel.spark.SparkEnvironment.transformProcess(SparkEnvironment.java:159)
at org.apache.seatunnel.spark.batch.SparkBatchExecution.start(SparkBatchExecution.java:50)
at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:76)
... 12 more
2022-09-07 08:55:54 ERROR Seatunnel:69 -
===============================================================================
Exception in thread "main" java.lang.RuntimeException: Execute Spark task error
at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:79)
at org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:39)
at org.apache.seatunnel.core.spark.SeatunnelSpark.main(SeatunnelSpark.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.spark.sql.catalyst.parser.ParseException:
no viable alternative at input 'select '(line 1, pos 7)
== SQL ==
select
-------^^^
at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:239)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:115)
at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:69)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
at org.apache.seatunnel.spark.transform.Sql.process(Sql.scala:28)
at org.apache.seatunnel.spark.SparkEnvironment.transformProcess(SparkEnvironment.java:159)
at org.apache.seatunnel.spark.batch.SparkBatchExecution.start(SparkBatchExecution.java:50)
at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:76)
... 12 more
```
### Flink or Spark Version
spark 2.3.2
### Java or Scala Version
jdk1.8
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] aruizhong commented on issue #2736: [Bug] --variable无法替换配置文件里的${}变量
Posted by "aruizhong (via GitHub)" <gi...@apache.org>.
aruizhong commented on issue #2736:
URL: https://github.com/apache/seatunnel/issues/2736#issuecomment-1680732585
> > I use 2.3.0 and the developing 2.3.1, found the same bug. it can replace with environment variable which use 'export param = param_val' from shell, but it fails when parameter is passed with '--variable'
>
> I ran it with seatunnel engine.
have you solved the problem that the parameter can not take effect with seatunnel Zeta Engine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] aruizhong commented on issue #2736: [Bug] --variable无法替换配置文件里的${}变量
Posted by "aruizhong (via GitHub)" <gi...@apache.org>.
aruizhong commented on issue #2736:
URL: https://github.com/apache/seatunnel/issues/2736#issuecomment-1680731185
>
have you solved the problem that the parameter can not take effect with seatunnel Zeta Engine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CriysHot commented on issue #2736: [Bug] --variable无法替换配置文件里的${}变量
Posted by GitBox <gi...@apache.org>.
CriysHot commented on issue #2736:
URL: https://github.com/apache/incubator-seatunnel/issues/2736#issuecomment-1282140346
for example:
sql = '"""select ${city}"""'
add a single quotation mark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CRUDKanade closed issue #2736: [Bug] --variable无法替换配置文件里的${}变量
Posted by GitBox <gi...@apache.org>.
CRUDKanade closed issue #2736: [Bug] --variable无法替换配置文件里的${}变量
URL: https://github.com/apache/incubator-seatunnel/issues/2736
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] general7078 commented on issue #2736: [Bug] --variable无法替换配置文件里的${}变量
Posted by "general7078 (via GitHub)" <gi...@apache.org>.
general7078 commented on issue #2736:
URL: https://github.com/apache/incubator-seatunnel/issues/2736#issuecomment-1437981672
I use 2.3.0 and the developing 2.3.1, found the same bug. it can replace with environment variable which use 'export param = param_val' from shell, but it fails when parameter is passed with '--variable'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] aruizhong commented on issue #2736: [Bug] --variable无法替换配置文件里的${}变量
Posted by "aruizhong (via GitHub)" <gi...@apache.org>.
aruizhong commented on issue #2736:
URL: https://github.com/apache/seatunnel/issues/2736#issuecomment-1680728302
> > I use 2.3.0 and the developing 2.3.1, found the same bug. it can replace with environment variable which use 'export param = param_val' from shell, but it fails when parameter is passed with '--variable'
>
> I ran it with seatunnel engine.
solved the problem that the parameter can not take effect with seatunnel Zeta Engine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] liuyongfei1 commented on issue #2736: [Bug] --variable无法替换配置文件里的${}变量
Posted by GitBox <gi...@apache.org>.
liuyongfei1 commented on issue #2736:
URL: https://github.com/apache/incubator-seatunnel/issues/2736#issuecomment-1247946920
sql {
sql = "'"select ${city}"'"
result_table_name = "tb_6875316324672"
}
it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] general7078 commented on issue #2736: [Bug] --variable无法替换配置文件里的${}变量
Posted by "general7078 (via GitHub)" <gi...@apache.org>.
general7078 commented on issue #2736:
URL: https://github.com/apache/incubator-seatunnel/issues/2736#issuecomment-1437983478
> I use 2.3.0 and the developing 2.3.1, found the same bug. it can replace with environment variable which use 'export param = param_val' from shell, but it fails when parameter is passed with '--variable'
I ran it with seatunnel engine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org