You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/09/28 10:08:57 UTC
[GitHub] [incubator-seatunnel] chenhu opened a new issue, #2935: ST Parse Json configuration file has some issue
chenhu opened a new issue, #2935:
URL: https://github.com/apache/incubator-seatunnel/issues/2935
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
ST Parse Json configuration file has some issue,
I has a json configuration file,and the parse result is :
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 22/09/28 14:19:09 INFO ConfigBuilder: Loading config file: /tmp/datax-executor/jsonjobTmp-abf3e9ce6c6046bf8377598857eba6ef.conf
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 22/09/28 14:19:09 INFO ConfigBuilder: parsed config file: {
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "transform" : [
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] [
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "orders_t",
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "orders",
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "Sql",
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "select toYYYYMMDD(createdate) as day, * from orders"
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] ]
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] ],
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "sink" : [
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] [
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "system",
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "com.mysql.jdbc.driver",
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "orders",
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "orders",
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "Jdbc",
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "root",
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "jdbc:mysql://192.168.110.228:3306/parking",
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "true"
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] ]
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] ],
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "source" : [
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] [
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "system",
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "1",
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "com.mysql.jdbc.Driver",
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "ID",
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "20",
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "10",
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "orders",
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "Jdbc",
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "root",
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "orders",
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "jdbc:mysql://192.168.110.228:3306/parking?useSSL=true&Unicode=true&characterEncoding=UTF-8"
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] ]
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] ],
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "env" : {
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "spark.default.parallelism" : "10"
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] }
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] }
### SeaTunnel Version
2.1.3
### SeaTunnel Config
```conf
{
"env": {
"spark.default.parallelism": "10"
},
"source": [
{
"password": "system",
"jdbc.lowBound": "1",
"driver": "com.mysql.jdbc.Driver",
"jdbc.partitionColumn": "ID",
"jdbc.upperBound": "20",
"jdbc.numberPartitions": "10",
"result_table_name": "orders",
"plugin_name": "Jdbc",
"user": "root",
"table": "orders",
"url": "jdbc:mysql://192.168.110.228:3306/parking?useSSL=true&Unicode=true&characterEncoding=UTF-8"
}
],
"transform": [
{
"result_table_name": "orders_t",
"source_table_name": "orders",
"plugin_name": "Sql",
"sql": "select toYYYYMMDD(createdate) as day, * from orders"
}
],
"sink": [
{
"password": "system",
"driver": "com.mysql.jdbc.driver",
"dbTable": "orders",
"source_table_name": "orders",
"plugin_name": "Jdbc",
"user": "root",
"url": "jdbc:mysql://192.168.110.228:3306/parking",
"useSsl": "true"
}]
}
```
### Running Command
```shell
./bin/start-seatunnel-spark.sh --master local -c ~/Downloads/test.conf --deploy-mode client
```
### Error Exception
```log
ERROR Seatunnel: Exception StackTrace:org.apache.seatunnel.shade.com.typesafe.config.ConfigException$WrongType: /tmp/datax-executor/jsonjobTmp-abf3e9ce6c6046bf8377598857eba6ef.conf: 1: source has type list of LIST rather than list of OBJECT
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getHomogeneousWrappedList(SimpleConfig.java:452)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getObjectList(SimpleConfig.java:460)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfigList(SimpleConfig.java:465)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at org.apache.seatunnel.config.EnvironmentFactory.checkIsContainHive(EnvironmentFactory.java:62)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at org.apache.seatunnel.config.EnvironmentFactory.getEnvironment(EnvironmentFactory.java:47)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at org.apache.seatunnel.config.ExecutionContext.<init>(ExecutionContext.java:49)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:44)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:36)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at org.apache.seatunnel.Seatunnel.run(Seatunnel.java:48)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at org.apache.seatunnel.SeatunnelSpark.main(SeatunnelSpark.java:27)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at java.lang.reflect.Method.invoke(Method.java:498)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53]
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 22/09/28 14:19:10 ERROR Seatunnel:
```
### Flink or Spark Version
spark3.3 on hadoop3
### Java or Scala Version
java8
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] closed issue #2935: ST Parse Json configuration file has some issue
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #2935: ST Parse Json configuration file has some issue
URL: https://github.com/apache/incubator-seatunnel/issues/2935
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #2935: ST Parse Json configuration file has some issue
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #2935:
URL: https://github.com/apache/incubator-seatunnel/issues/2935#issuecomment-1296400074
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #2935: ST Parse Json configuration file has some issue
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #2935:
URL: https://github.com/apache/incubator-seatunnel/issues/2935#issuecomment-1306415856
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ashulin commented on issue #2935: ST Parse Json configuration file has some issue
Posted by GitBox <gi...@apache.org>.
ashulin commented on issue #2935:
URL: https://github.com/apache/incubator-seatunnel/issues/2935#issuecomment-1263441215
linked #2933
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org