You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "EthanWang95 (via GitHub)" <gi...@apache.org> on 2023/04/27 06:57:21 UTC
[GitHub] [incubator-seatunnel] EthanWang95 opened a new issue, #4686: Parameter transfer error -i
EthanWang95 opened a new issue, #4686:
URL: https://github.com/apache/incubator-seatunnel/issues/4686
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
When using ST to synchronize data from PG to ES, the ${tablename} variable was set in PG's query, but in the command - i table_ name=cbi_ open_ api_ Desc has not taken effect
### SeaTunnel Version
2.3.0
### SeaTunnel Config
```conf
env {
# seatunnel defined streaming batch duration in seconds
## [spark.app.name](http://spark.app.name) = "SeaTunnel"
#spark.sql.catalogImplementation = "hive"
spark.executor.instances = 1
spark.executor.cores = 2
spark.executor.memory = "4g"
}
source {
jdbc {
driver = "com.mysql.jdbc.Driver"
url = "jdbc:mysql://******:3306/****?useUnicode=true&characterEncoding=utf8&useSSL=false"
user = "***"
password = "****"
query = "select * from ${table_name}"
}
}
transform {
# split data by specific delimiter
# you can also use other filter plugins, such as sql
# sql {
# sql = "select * from accesslog where request_time > 1000"
#}
}
sink {
# choose stdout output plugin to output data to console
#Console {}
elasticsearch {
hosts = ["10.122.33.95:9201"]
index = "seatunneltest_tst_20230426_01"
index_type = "seatunneltest"
}
}
```
### Running Command
```shell
./bin/start-seatunnel-spark-connector-v2.sh --master yarn --deploy-mode client --config config/toes/spark.batch.pg.to.es.conf -i^Cable_name=cbi_open_api_desc
```
### Error Exception
```log
2023-04-27 14:12:38 INFO AbstractPluginDiscovery:170 - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='jdbc'} from classpath
2023-04-27 14:12:39 INFO YarnSchedulerBackend$YarnDriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.122.33.241:45332) with ID 1
2023-04-27 14:12:39 WARN JdbcSource:139 - get row type info exception
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '{table_name}' at line 1
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.Util.getInstance(Util.java:408)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:944)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3976)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3912)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2486)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1858)
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1197)
at com.mysql.jdbc.PreparedStatement.getMetaData(PreparedStatement.java:2524)
at org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.JdbcDialect.getResultSetMetaData(JdbcDialect.java:183)
at org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSource.initTableField(JdbcSource.java:133)
at org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSource.prepare(JdbcSource.java:82)
at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.initializePlugins(SourceExecuteProcessor.java:88)
at org.apache.seatunnel.core.starter.spark.execution.AbstractPluginExecuteProcessor.<init>(AbstractPluginExecuteProcessor.java:49)
at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.<init>(SourceExecuteProcessor.java:51)
at org.apache.seatunnel.core.starter.spark.execution.SparkExecution.<init>(SparkExecution.java:50)
at org.apache.seatunnel.core.starter.spark.command.SparkApiTaskExecuteCommand.execute(SparkApiTaskExecuteCommand.java:54)
at org.apache.seatunnel.core.starter.Seatunnel.run(Seatunnel.java:39)
at org.apache.seatunnel.core.starter.spark.SeatunnelSpark.main(SeatunnelSpark.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```
### Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] closed issue #4686: Parameter transfer error -i
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #4686: Parameter transfer error -i
URL: https://github.com/apache/seatunnel/issues/4686
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EthanWang95 commented on issue #4686: Parameter transfer error -i
Posted by "EthanWang95 (via GitHub)" <gi...@apache.org>.
EthanWang95 commented on issue #4686:
URL: https://github.com/apache/incubator-seatunnel/issues/4686#issuecomment-1537083684
> Is that still not work yet?
Yes, according to the logs, it didn't work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parameter transfer error -i [seatunnel]
Posted by "fatCatGm (via GitHub)" <gi...@apache.org>.
fatCatGm commented on issue #4686:
URL: https://github.com/apache/seatunnel/issues/4686#issuecomment-1883295597
You can give it a try；
jdbc {
driver = "com.mysql.jdbc.Driver"
url = "jdbc:mysql://******:3306/****?useUnicode=true&characterEncoding=utf8&useSSL=false"
user = "***"
password = "****"
query = "select * from "${table_name}""
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] commented on issue #4686: Parameter transfer error -i
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4686:
URL: https://github.com/apache/seatunnel/issues/4686#issuecomment-1590245967
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] 666fishsun commented on issue #4686: Parameter transfer error -i
Posted by "666fishsun (via GitHub)" <gi...@apache.org>.
666fishsun commented on issue #4686:
URL: https://github.com/apache/incubator-seatunnel/issues/4686#issuecomment-1536486722
@EthanWang95 Is that still not work yet?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] commented on issue #4686: Parameter transfer error -i
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4686:
URL: https://github.com/apache/seatunnel/issues/4686#issuecomment-1577724724
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] davidzollo commented on issue #4686: Parameter transfer error -i
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on issue #4686:
URL: https://github.com/apache/seatunnel/issues/4686#issuecomment-1643040137
not support replace the variables in the conf file for now, the command -i is used to directly pass the param value to Spark or Flink engine, it seems like {FLINK_HOME}/bin/flink -Dtable_ name=cbi_ open_ api_ Desc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org